alt Hacker NewsSort of. Phi models were just trained on GPT outputs though.
Replies
not_a_bot_4sho • yesterday at 8:11 PM
By design. The whole point of Phi is the "textbooks is all you need" theory on curated training data, as opposed to kitchen sinks.
Sort of. Phi models were just trained on GPT outputs though.
By design. The whole point of Phi is the "textbooks is all you need" theory on curated training data, as opposed to kitchen sinks.
For those that don't know about this. Phi was announced with a paper called "Textbooks are all you need". What they did was use GPT 3.5 and created synthetic textbook chapters and exercises.
They also did some more interesting work like showing very small models can be coherent as long as you have very simple children's book style training data (TinyStories is pretty famous).
Lots of these ideas are still used. Learning facts at scale with active reading is an ICLR 2026 paper from Meta AI that does a lot of similar work.