You could think of it as a standard decoder only LLM (almost all modern ones we use everyday), with some layers (experts) having parallel networks and conditionally based on the input token (per token) - the token is routed through some of these layers. In the case of a non MoE (dense) - each token goes through all layers, so the inference engine has to read all the layers and do a matrix (layer) times vector (token) computation, while in the case of MoE the number of layers per token that has to do the compute is substantially lesser, so one can expect much higher tps than a dense model at the same number of parameters (size - 7B, 27B etc)
alt Hacker News
You could think of it as a standard decoder only LLM (almost all modern ones we use everyday), with some layers (experts) having parallel networks and conditionally based on the input token (per token) - the token is routed through some of these layers. In the case of a non MoE (dense) - each token goes through all layers, so the inference engine has to read all the layers and do a matrix (layer) times vector (token) computation, while in the case of MoE the number of layers per token that has to do the compute is substantially lesser, so one can expect much higher tps than a dense model at the same number of parameters (size - 7B, 27B etc)