It's not just the pointless chaff, the SSH protocol is inherently very chatty, and SFTP even more so. The solution, for a high-performance game, is don't use SSH. Either run it over Wireguard or grab some standard crypto library and encrypt the packets yourself. You'll probably make a few minor mistakes but unless the other player is the NSA it'll be good enough.

For that matter, why does it need to be encrypted at all? What's the threat model?

If there really is a genuine need to encrypt and low latency is critical, consider using a stream cipher mode like AES-CTR to pregenerate keystream at times when the CPU is lightly loaded. Then when you need to encrypt (say) 128 bytes you peel off that many bytes of keystream and encrypt at close to zero cost. Just remember to also MAC the encrypted data, since AES-CTR provides zero integrity protection.