alt Hacker NewsDoes MoE help with multimodality? Can it in general enable reasoning in imagery (technical drawings, diagrams, schematics) rather than text-based?
Does MoE help with multimodality? Can it in general enable reasoning in imagery (technical drawings, diagrams, schematics) rather than text-based?
MoE has nothing to do with multimodality.
MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing.
Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization.
As for reasoning in non-text modalities, you might find this paper interesting :) https://huggingface.co/papers/2502.05171