Lookahead Sparse Attention should be playing a big role as well, as it dramatically slashes memory consumption.