Same here. LLMs are great at spitting out well-known solutions to problems instead of the best one. The "long tail" of solutions is usually lost due to how tokens are sampled from the LLM's probability distribution.

What I found to help a lot is to ask for e.g. 10 different solutions to a problem and then choosing one of them. Sometimes, this even leads to borderline creative solutions if there aren't 10 different ones.

One thing that SO had was you could see multiple solutions and implementations for something. Sometimes the "best" solution isn't very readable code, sometimes you are able to understand the problem better when you see a bunch of people solving it in different ways and arguing about it like angry monkeys.

It really could be bad though.

Much of what Claude insists you do probably came from SO.

I'm not sure we're better off without SO in the long run.

You're right but that site has been sputtering culturally for some time. I put a lot of effort into editing questions and answers on ServerFault (part of SO) but I feel that time was wasted. I think they knew for a while they just wanted to sell it and just stopped caring. A number of editors were allowed to be jerks for too long and it went to their heads. I wish I could take back all that effort.

well, SO is probably the highest quality data source for a language model and the rest of the internet is just diluting the final latent space limited by Jon Skeet.

What you are noticing in a long term is the "community" knowledge and communication which the chatGPT is now kind of destroying. In some sense, it is no different from the difference between studying along and studying with your peers at a university.

You can definitely study alone and achieve perfect grades, but studying with your peers is how you build relationships for future life and take your community forward as a whole.

was just gonna post the same thing

needed to implement a language feature that was a bit complicated and im not familiar with it so just planned with claude to do it, and after each write/fix cycle it just wouldn't work right.... gave up, went back to SO copy pasted the (not perfect but enough to start from) answer and worked up from there...

at the same time my knowledge grew and im more confident to do this same capability myself whereas reiterating with claude it was just a slog and i didn't learn much...

i think i may be starting to sour on these "do it all for me" usage scenarios for ai... especially for unfamiliar areas...

Agreed. Which is also odd, if you think about it. Surely with the amount of compute Anthropic and others have available, they could test each of the solutions in the SO data they surely have and rank them based on efficiency/elegance/other criteria and remove poor solutions from their training data.

I am thinking to make canned encyclopaedia of stackoverflow answers.

Claude/Grok/Gemini/Chatgpt answers are often so… how to say it… misleading? I have to stop the conversation as it leads nowhere (and it is not a skill issue :)

agreed; sometimes the most useful info on SO wasn't in the replies, but the follow-up comments to the replies

We may need to create a community driven version of SO. Hard for it to be a successful business these days.

What are you looking for and finding on stack overflow that isn't begginer to intermediate level?

And, the AI trained on Stack Overflow. So if no one is posting new questions, and new answers. What will AI train on next, for the next thing.

Definitely not better off. SO was fairly mean spirited, but nowhere else has such a vast trove of high quality answers to common software problems been collected. SO likely trained many of these models with its answers, and I don't know what software development will look like when it dies.