alt Hacker NewsThis is just another facet of the sycophancy issue. They really need to start RLing these models to gently push back the way a friend would on things that are questionable.
It would also make sense for them to just RL in the "research prior art" arc as part of that "is the request sensible" pre-analysis. It wouldn't be difficult and it would save everyone a lot of pain.
Yeah, for this to feel credible you need to make sure the model doesn't figure out that you're planning to build a solution yourself.