No there is a logical errror in there. You are implicitly asserting that the trained thing is an imitation, whereas it is only the output that is being imitated.

A flip way of saying it is that we are evolving a process that exhibits the signs of what we call thinking. Why should we not say it is actually thinking?

How certain are you that in your brain there isn’t a process very similar?