alt Hacker NewsI don't understand why you'd use a RLHF-aligned chatbot model for that purpose: this thing has been heavily tuned to satisfy the human interacting with it, of course it's going to fail following higher level instruction at some point and start blindly following the human desire.
Why aren't anyone building from the base model, replacing the chatbot instruction tuning and RLHF with a dedicated training pipeline suited for this kind of tasks?
Because the pretrained chatbot is the flagship product of an AI company in 2025. They want to sell this product to customers who can't spell RLHF, never mind have the (substantial) resources to do their own training.
If Anthropic were getting into the vending machine business, or even selling a custom product to the vending machine industry, they'd start somewhere else. But because they need to sell a story of "we used Claude to replace XYZ business function", they started with Claude.