All modern models are MoE already, no?
That's not the case. Some are dense and some are hybrid.
MOE is not the holy grail, as there are drawbacks eg. less consistency, expert under/over-use
alt Hacker News
That's not the case. Some are dense and some are hybrid.
MOE is not the holy grail, as there are drawbacks eg. less consistency, expert under/over-use