Discovery of the best solution in a problem space is not generative but only verificative. Meaning: the LLM can see if a solution is better than another, but it can't generate the best one from the start. If you trust it, you'll get sub-par solutions.

This is definitely an agent problem instead of an LLM problem. Anybody got something explorative like this working?