logoalt Hacker News

nightski • yesterday at 3:41 PM • 2 replies • view on HN

Not only that we are one model innovation away from them being worthless. You need $1T in compute because transformers are so ungodly inefficient. That compute will still be useful even if we find a more efficient architecture, but the margin on it will be so much less.

Deepseek was a sign of this on a smaller scale.


Replies

xmcp123 • yesterday at 3:58 PM

Can you expand on that deepseek part?

➕ show 1 reply
logicchains • yesterday at 4:30 PM

>That compute will still be useful even if we find a more efficient architecture

Mathematically there's no way to have lossless attention without O(n^2) cost; some magical massively more efficient architecture is not going to pop up the same way nobody's going to find a way to sort an arbitrary list in log(n) time. There may still be more opportunities for incremental gains (like using the bare-minimum number of full-quadratic-attention layers necessary), but they're not going to be orders of magnitude more efficient without sacrificing intelligence.

➕ show 1 reply