There is a philosophical problem hidden in the core of interpretability: Can a mind understand a mind much greater than itself?
Seems more of an empirical problem to me.
Now or in the near future we will be studying minds much greater than our own, and we will find out how well we can understand them.
If they don't kill us all before then, of course.
The differences b/w knowing, understanding and interpreting are suddenly very real and highly relevant.
Even if it can, we can only identify a very wide set of reasons that can interpret the outputs. But what is root cause, that is a very deep rabbit hole.
Let’s say a guy got electrocuted while showering. I can say that the cause is the guy going for a shower. For others the cause is the electric wiring system failure.