All modern models are MoE already, no?
That's not the case. Some are dense and some are hybrid.
MOE is not the holy grail, as there are drawbacks eg. less consistency, expert under/over-use
That's not the case. Some are dense and some are hybrid.
MOE is not the holy grail, as there are drawbacks eg. less consistency, expert under/over-use