Well the big labs certainly haven't intentionally tried to train away this emergent property... Not sure how "hey let's make the model disagree with the user more" would go over with leadership. Customer is always right, right?
The problem is asking for user preference leads to sycophantic responses
The problem is asking for user preference leads to sycophantic responses