alt Hacker Newshow feasible is it build a SOTA specialized model for some use case e.g. deal sourcing by using this as pre-trained model or a LORA or similar pattern on top? Gonna shoot my shot at a billion dollar business
Replies
dsrtslnd23 • today at 7:14 PM
yes would also be interested in that - using knowledge distillation and other special knowledge sources to post-train on top of an open model like GLM-5.3. I was wondering that when Elon Musk tweeted "Specialist AI’s (single language, single area of knowledge) are another 100X" (src: https://x.com/elonmusk/status/2089968914596045178) - maybe he knows something regarding specialist model training the general public does not know?
Not feasible. We’ve seen again and again that generalized models—somewhat surprisingly—dunk on specialized ones in almost all cases.
The first case of this I remember was Bloomberg thinking that their extensive private data about markets would give a home-trained model better performance in finance tasks. The next version of ChatGPT beat them.
With how often new versions of frontier models are released, you likely won’t finish your work before it’s obsolete. The labs have whole teams dedicated to specific getting training data for specific industries (finance is one), and more powerful generalized models make better decisions even without specialized knowledge.
Your best bet is to get really good at training for something and then sell your company to one of the frontier labs for their post-training efforts.