>That compute will still be useful even if we find a more efficient architecture
Mathematically there's no way to have lossless attention without O(n^2) cost; some magical massively more efficient architecture is not going to pop up the same way nobody's going to find a way to sort an arbitrary list in log(n) time. There may still be more opportunities for incremental gains (like using the bare-minimum number of full-quadratic-attention layers necessary), but they're not going to be orders of magnitude more efficient without sacrificing intelligence.
Counter proof by existence - humans use significantly less energy and don’t do intelligence through an n^2 process. The human brains seem to operate closer to SSMs like Mamba with a recurrent architecture and attention is implemented using sparse distributed weights. So yes, it’s possible to have very high intelligence without O(n^2). Is it as good as transformers? Hard to tell right now.