alt Hacker NewsI wonder how bad the performance would be if they plain ignored the whole rotary encoding dance and just back-filled precisely the parts of the cache that changed directly. Would it break the model? Confuse the model? Or would the model internally correct for it?
The rotary issue is a simple rotation, cheap, no reason to skip it.
But this is the kind of thing you could ask your agent to test locally.