We don't have a reward function for "human understanding". We reward the appearance of understanding. We define goals that we cannot conceive of reaching without something like understanding happening. There is something happening, but it is alien and counter-intuitive - it makes bizarre mistakes that betray it - and we don't know what it is. I'm pretty sure it is not human understanding.