The big take away for is the fact that the ONLY reason why chatgpt was able to get to this counterexample was because of the knowledge of the person driving the conversation.
I don't think chatgpt could have come to this on its own without the amount of steering he did, which just validates the idea that AI is not a replacement for human expertise but an amplifier.
> I don't think chatgpt could have come to this on its own without the amount of steering he did, which just validates the idea that AI is not a replacement for human expertise but an amplifier.
The problem is that, what happens to human expertise as people start to use AI earlier and earlier in their careers, so that in 50 years? The problem is that Terry Tao spent decades as a mathematician before ever encoutering AI. Of course he and people his age will be able to drive AI somewhat sanely and use it to their advantage.
But as more people grow up with AI, they will likely not reach levels like Terry Tao because their exposure to AI and the temptation to use it will certainly dull raw human intellect over time.
Maybe true, I'm not sure, but this isn't the conversation where the counterexample was found.