alt Hacker NewsThere’s still 50-500x cost reduction in “this is only an engineering problem” low hanging fruit from specialized chips to run the models + improved distillation.
Entirely feasible that by 2031, Fable 5 (or greater) intelligence level models will run cool on smart phones, if not sooner.
Replies
andai • today at 2:54 PM
I saw a 1B model yesterday that was fine tuned on Fable output. I found that hilarious, but it did actually make all the scores go up.
(Actually talking to it, it was about as coherent as you'd expect, i.e. 3/10)
The floor for "actually usable model" keeps dropping though. (Seems to be about 27B right now?)
You're betting on getting getting ridiculously powerful chips to run on batteries in a tiny housing without cooling, while we can't even get enough RAM? It would be a terrible waste of resources. Now we already have TFLOPs wasting in our pockets and backpacks, then we'll have PFLOPs idling, because there's so much time between prompts. Much more efficient to batch it on a server.