It's not true that LLMs can only perform as well as the average of their training data. They can learn to perform at an elite level in domains like programming by extracting quality signals from code reviews, bug fixes, tutorials, etc, and the labs are doing post-training to teach them what excellent code looks like.
Maybe right now that's only true in a few domains like math, programming, language translation, maybe some writing task? But soon it will be true in many more domains.
Given that we can't hope to out-execute the LLMs on the things they are elite at, what is left for us? This site wants to reserve "taste" for the humans, but I'm not sure LLMs won't learn "taste" too.
The best I've got right now is that humans live and interact with the real world in ways that disembodied LLMs can't (yet). Maybe they will be the theoreticians and we the experimentalists?
The world is pretty diverse. There’s not an objective correct answer to a lot of things. Leaving that decision up to an LLM is acceptable sometimes, and not in others.
I’m not sure how you train for that. Though, maybe the world just converges on homogeneity out of convenience. Certainly happens in other areas.