That's an interesting approach, but... what's the performance impact? It seems the check was done only against really trivial tasks. Why not run some known coding benchmark against the original and pruned one?

Also Gemma is Moe not modem (https://arxiv.org/html/2410.07490v1) so will try to balance export choice for each token - I'm suspicious about expectations for this model.