rlhf will cause models during training to hide data in regions not typically accessible to them in the geometry during inference, due to competing objectives, so i wonder if this looks like that.