alt Hacker NewsAnyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again?
I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.
Replies
If dirt-simple type operations like copy-paste yield useful improvements with even a small probability that would seem to open things up for adaptive reconfiguration and whole other classes of optimizations like genetic algorithms.
Found it: https://news.ycombinator.com/item?id=47500709
Part 3 might be the best introduction: https://dnhkng.github.io/posts/sapir-whorf/
tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed back to the target language. And you can repeat the middle to get a stronger model. Which neatly fits Anthropic's findings here that something similar to CoT is happening in those middle layers
Three months ago. I wonder if Anthropic's J-Space research was actually inspired by those blog posts
I always thought that area of research had the coolest name, too: “mechanistic interpretability”
Yeah! I still think about that sometimes. Mind-blowing that worked at all, let alone improved performance.
> I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.
Too bad the frontier models are closed weights.
Maybe the research community and whole rest of the world will build on open and all the advances will happen in open ecosystems instead.
Source for those interested
https://dnhkng.github.io/posts/rys/