alt Hacker NewsIt's pretty clear, because OpenAI has no clue what they are doing. If I was the CEO of OpenAI, I would have invested significantly in catastrophic forgetting mitigations and built a model capable of continual learning.
If you have a model that can learn as you go, then the concept of accuracy on a static benchmark would become meaningless, since a perfect continual learning model would memorize all the answers within a few passes and always achieve a 100% score on every question. The only relevant metrics would be sample efficiency and time to convergence. i.e. how quickly does the system learn?
Replies
It's not obvious that there are such mitigations.
I personally would have gone for search/reasoning as has been done. It's the reason path.
It's actually great if the end result is that the incumbent with infinite money that has unrealistic aspirations of capturing a huge section of the sector lights all the money on fire. It's what happened with Magic Leap - and I think everyone can agree that the house of Saud tossing their money into a brilliant blaze like that is probably better than anything else they would have wanted to do with that money. And if we get some modest movements forward in that technical space because of that, all the better. Sometimes capitalism can be great, because it funnels all the greed into some hubris project like this and all the people that are purely motivated by greed can go spin their wheels off in the corner and minimize the damage they do. And then some little startup like Deepseek can come along and do 90% of the job for 1% of the money
> I would have invested significantly in catastrophic forgetting mitigations and built a model capable of continual learning.
You say it as if it's an easy thing to do. These things take time man.