alt Hacker NewsI'm surprised more attention isn't paid to this research direction, that nobody has tried to generalize it for example by combining the recurrence concept with next token prediction. That said despite the considerable gains this seems to just be some hyperparameter tweaking rather than a foundational improvement.
Replies
whiplash451 • yesterday at 11:50 PM
Not just hyper parameter tweaking. Not foundational research either. But rather engineering improvements that compound with each other (conswiglu layers, muon optimizer)
> nobody has tried to generalize it for example by combining the recurrence concept with next token prediction
Here you go: https://arxiv.org/abs/2502.05171