alt Hacker NewsSurely the only problem isn’t the knowledge of how to train and the confidence it would work. Wouldn’t it also have required a 1 or 2 order of magnitude increase in computing power over 2005-era supercomputers to be able to train a modern LLM in a reasonable time (“reasonable” meaning “finish training before GPT-3 comes out in 15 years”)?
Replies
bananaflag • today at 12:52 PM
And also "the whole Internet" (including scanned books etc.) on which to train was much smaller.
On the other hand, Google was openly collecting data with a view towards building AI.
I think it’s plausible that attention could have been invented earlier and GPT demonstrated at smaller scale on earlier hardware. Then hardware timelines might have accelerated (or alternative approaches explored) and we’d have gotten to where we are earlier.