16,000 words into the essay: "What I mean by clean"
...and it still doesn't answer it. What is this actually about?
It means that modern neural networks are way more interpretable than older ones. We can understand how they work way more easily than earlier ones.
I think "clean" means something like each neuron encodes exactly one concept and it's possible to discover what the concept is.
Clean means no alchemy.
It means that NNs don't (as previously thought) smoosh unrelated concepts together into the same neurons, which would have made interpretability very difficult.
For instance, if you had a single neuron that fires for "puppies" and "murder", it would be much more difficult to tell if the model is thinking something nice or something not so nice.