alt Hacker NewsSpeaking for myself. I’ve been aware of a lot of concepts around training, but stayed away from it until very recently.
A few weeks ago, I formed a hypothesis that I could get a smaller model good at the specific standards I have for my git commit messages, which are hard to get out of most LLMs today.
I spent the last four to five weeks on traditional training. It improved in a lot of areas, but not enough to be usable for me.
Yesterday, the article the author of this post is critical of gave me enough of an aha that maybe the problems I was seeing could be solved by having a larger LLM change the weights directly, without going through training.
In the last twelve-ish hours I’ve made more headway than I have in 2+ weeks. I’m using one of my last well performing checkpoints (full fine-tune + changes to the decision head/LoRA, etc.) and gave the model free rein to change any of the layers.
Its last two actions: (a) removed 16 participant-order directions from a head matrix, which improved one type of reasoning and degraded another, and (b) restored one of the 16, which reversed the degradation while keeping the core improvement.
All that to say, this might be known to people who’ve been around training for years, even decades, but it’s new to a lot of us. And we’re excited about it.
Replies
> it’s new to a lot of us. And we’re excited about it.
Yes definitely.
Seems to me as if the weighs fit in the big model's context, it (the big model) reads all weights and doesn't need python.
But if the weights don't fit in context, then it uses python.
Isn't that pretty obvious in a way? That if too many weights, can't look at all. I don't think there's a need for OP to be dismissive when the AI uses python to look at a subset of the weights. It just has to?
To me, Astra understanding a 200k weights small model (the stuck car) is amazing enough already.
Wow. Are you saying you trained a smaller model from scratch (from what training data?), then used a larger model (which model?) to tweak its weights to fix this problem? Or did you post-train/fine-tune an existing model? I'm very curious what models, hardware, and training data you used!
I assume you run a benchmark on all the tasks on every change the model recommends? Do you have it set up like nanogpt speed run thingy that karpathy for a thumbs up or down for every change when you did this with astra?
From the Fine Article you're calling critical:
>> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights
Out of interest, did you use an assistant like Claude Code or Codex to help with the training experiments?
Would you be willing to share what the specific standards for your git commit messages are, and with what models you used that those standards couldn't be met?
Generally speaking you should be able to do something like that in AGENTS, I am really surprised that you have had an issue with most LLMs today getting them to follow a standard for SDLC. Personally, I like conventional commits a lot.