alt Hacker News>Distillation “attacks” are not attacks. The frontier labs “distilled” all existing human written knowledge into their models
So why didnt we have these LLMs in 2005?
Replies
Is this some form of rage bait? 2005 we hadn't the GPUs, we have today. There are other factors, but I think this is the big one. The mathematics of building an LLM are really old, we just hadn't the hardware to do the needed calculations.
Because the transformer architecture that enabled modern LLMs wasn't invented until 2017[1]?
1: That's the "T" in GPT fyi, even though Google is the author of the research paper that changed everything
Moore's Law or something. Were you alive in 2005? The Nintendo DS getting the Opera browser was a big deal. THAT 2005 with today's LLMs? Hilarious.
We didn't have the compute required (GPUs powerful enough to parallelize forward and backward pass). This compute is what allows us to train from human knowledge or distillation.
because you had neither the chips or the information in 2005. You have probably on the order of 5000x to 10000x more GPU compute today than you had in 2005 and three to four magnitudes more openly available data.
The first "L" in LLM does the work. In 2005 you had no Github, Stackoverflow, Youtube, common crawl and no archive of digital ebooks.
Answer the question "how much does 5 cents of LLM computation in July 2026 cost in July 2005" and you'll have the answer to your question.
Don't forget to account for all the costs. It's not just that CPUs are X times slower. Memory is X times smaller, too, and networks are X time slower. And all this hardware is many times more expensive.
If I'm getting my mental estimation right, training a 2026-frontier-class LLM in 2005 would be somewhere on the order of all the computation power in the world at the time. It's not that many more factors of magnitude before you end up at "all the computation power in the world up to that point".