There are typos all over the training data, and people offering RLHF feedback can overlook them.