MoE's in the abstract often get presented as if theres explicit layers of experts for any given domain of knowledge, like your coding tasks are being routed to coding experts, but it's really not that at all.

THe original MoE paper from Noam Shazeer et al. is worth a read on this bit, though the paper is admittedly pretty dense. But TL;DR is that each expert layer is learning highly abstract, localized structural and syntactic patterns in the data to minimize the loss function, and its doing this token-by-token (which in some cases may have some domain clustering, but that's just incidental).

When you start batching your queries, even if they all seem like theyre in a single domain, if you visualized the activations you'd notice that most if not all of the network is lighting up on the batched forward pass.