he said specifically lossless attention. As in that algorithm. what you just said is irrelevant to the conversation because all of the current AI hype is built on the back of attention
> That compute will still be useful even if we find a more efficient architecture
Fixing N^2 attention would require a different architecture, but the person I was responding to was treating attention as if it’s the only architecture we have for intelligence when we know that not only do we have less competitive intelligence for cheaper, human intelligence doesn’t use n^2 attention. You have also fallen into the trap of misunderstanding the statement same as the person I replied to.
Here’s what was literally quoted in the response:
> That compute will still be useful even if we find a more efficient architecture
Fixing N^2 attention would require a different architecture, but the person I was responding to was treating attention as if it’s the only architecture we have for intelligence when we know that not only do we have less competitive intelligence for cheaper, human intelligence doesn’t use n^2 attention. You have also fallen into the trap of misunderstanding the statement same as the person I replied to.