Not only that we are one model innovation away from them being worthless. You need $1T in compute because transformers are so ungodly inefficient. That compute will still be useful even if we find a more efficient architecture, but the margin on it will be so much less.
Deepseek was a sign of this on a smaller scale.
>That compute will still be useful even if we find a more efficient architecture
Mathematically there's no way to have lossless attention without O(n^2) cost; some magical massively more efficient architecture is not going to pop up the same way nobody's going to find a way to sort an arbitrary list in log(n) time. There may still be more opportunities for incremental gains (like using the bare-minimum number of full-quadratic-attention layers necessary), but they're not going to be orders of magnitude more efficient without sacrificing intelligence.
Can you expand on that deepseek part?