I wonder if Kimis KDA architecture is a way closer to online models(models that learn during interference).As KDA updates its state matrix: St =St−1 +βt (vt −St−1 kt )ktT

The expression (vt −St−1 kt )ktT is literally the gradient step of a linear regression model updating its weights (S) on a single sample using Mean Squared Error loss.

Interesting what other labs comes up with from this.