alt Hacker News> We also further increased the sparsity of the Mixture of Experts (MoE): with the Stable LatentMoE framework, the model efficiently activates 16 out of 896 experts. Together with improvements in training methodology and data recipes, these structural advances give K3 roughly 2.5x the overall scaling efficiency of K2, converting compute into capability more effectively.
Assuming experts are uniformly distributed (I’m really not that familiar with the deep details there), that’s 2800/896*16 = 50 billion active parameters just for the active/expert part. Wild stuff, and I’m glad there’s at least some companies still publishing (and pushing, for open-weight models) total parameter count.
And: It sounds very believable that this would result in efficiency gains wrt. to compute necessary for “good”-quality inference. Does anyone know whether there currently even are any SOTA or near-SOTA models that are dense still?
Replies
2.5x the scaling efficiency, so 4 times the price? What is happening here? Did the subsidies dry up with the discrepancy between chinese and US models?
No, you can't divide the entire size by the expert count. A lot of weights are constant for all tokens, so total active count is ((2800-(shared)/896)*16 + (shared))