alt Hacker News>It's extremely difficult to increase compute by 100 times, but with sufficient investment in talent, achieving a 10x increase in compute is more feasible.
The article explains how in reality the opposite is true. Especially when you look at it long term. Compute power grows exponentially, humans do not.
Replies
Humans don't grow exponentially indefinitely. But there's only something in the order of 100k AI researchers employed in the big labs right now. Meanwhile, there's around 20mn software engineers globally, and around 200k math graduates per year.
The number of humans who could feasibly work on this problem is pretty high, and the labs could grow an order of magnitude, and still only be tapping into the top 1-2% of engineers & mathematicians. They could grow two orders of magnitude before they've absorbed all of the above-average engineers & mathematicians in the world.
Human do write code that scalable with compute.
The performance is always raw performance * software efficiency. You can use shitty software and waste all these FLOPs.
Algorithmic improvements in new fields are often bigger than hardware improvements.
If the bitter lesson were true we'd be getting sota results out of two layer neural networks using tanh as activation functions.
It's a lazy blog post that should be thrown out after a minute of thought by anyone in the field.