In my latest experiments with editing the weights directly, it has edited the weights of my decision head and LoRa checkpoint.

Essentially, my training introduced knowledge and structure which wasn't translating into results. I tried training in a few different ways, different ranks, different things, and I wasn't making any progress.

With weight editing, a larger model was able to eliminate some undesirable pathways which the model was often using. Without going into the nitty-gritty, it basically took my best LoRa checkpoint and gave it the skills that I was unable to get with further training.