> Never before have we looked at image classifications and decided that's how the eye works

Actually we have, several times. But the way we arrived at those conclusions is worth observing:

1. ML people figure out how the ML mechanism works.

2. Neuroscientists independently figure out how brains do it.

3. Observe any analogies that may or may not exist between the two underlying mechanisms.

I can't help but notice how that's a scientific way of doing it. By contrast, the way people arrive at similar conclusions when talking about LLMs tends to consist of observing that two things are cosmetically similar, so they must be the same. That's not just pseudoscientific; it's the mode of reasoning that leads people to believe in sympathetic magic.