And here we go...

As others have said, while still absolutely insane, I don't find it completely out-of-the-blue surprising. One of the recurring abilities of (current-generation) LLMs seem to be to rapidly grasp very complex systems, where humans would need months or years to reach the same level of expertise.

You can see this in the way a coding agent can "understand" and explain sprawling legacy codebases in seconds.

You can also see it in the projects that are using them to reverse-engineer proprietary IoT protocols or computer games.

But those were all systems where we had detailed knowledge about what structures are there to be reverse-enginered.

If the models can transfer this ability to some other systems where we don't know the underlying structure then I think we will be in for a lot more craziness in the future. (Protein interaction networks? DNA sequences? Animal languages?)

As for weights, if the models really can analyze the semantics behind specific weights or directions, I have my hopes that they could also give a textual explanation or help designing tools to make "handcrafting" weights more practical in general.