alt Hacker NewsCausal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.
Replies
cma • yesterday at 10:27 AM
The paper says:
"Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA."
And adding to that, there is also the recurrent state in the Kimi Delta Attention. I wouldn't call it position information but more sth like "position sensitivity"