>But to be clear, nobody understands why the evolved weights of a NN make the decisions that they do.

We might not understand particular "emergent" capabilities, but the low level mechanism is not just understood, but a deterministic algorithm with a handful of basic componets, that are well understood themselves.