We had a company hackathon in the fall of 2023. One of the teams did a project where the pulled a bunch of expense data out of the DB, shoved it into a prompt, and asked ChatGPT to summarize the expenses and give recommendations. They then treated the output as if it were factual, without validating any of the results, and talked about turning it into a customer product.

That was my oh shit moment. As in "oh shit, they think this random text generator can reason and think."

That was pretty much the writing on the wall for me.