alt Hacker NewsI think it’s plausible that attention could have been invented earlier and GPT demonstrated at smaller scale on earlier hardware. Then hardware timelines might have accelerated (or alternative approaches explored) and we’d have gotten to where we are earlier.
NVIDIA moved as fast as possible back then, there were huge performance gains with each new GPU generation, hard to imagine they might have gone even faster.