I pretty much got that idea when reading the post, on how it didnt have improvements on a Gemma 270M model which didn't have any looping issue. However, it still made me look at some bottlenecks in models I had trained (like MLP network for stylus input automation and a 17M reranker with NLI stance and noul heads) and allowed me to finetune without training, but just by adjusting some weights. But what's pretty cool is the zero shot sandbox development method they showed for a Go model. I tried to do that for a 17M re-ranker from zero weights and Astra built it using its reasoning without me having to use a GPU and find training datasets that are not CC by NC, and add some hours of tweaking and checkpoints. I don't know if its the best way to build something, but we might be building small models (maybe not LLMs) like with just programming the weights instead of finetuning.

In the end I got to replace a 68M model with a 17M model, and improve stlyus tool recognition accuracy from 88 to 96%. Was a pretty fun experiment too.