alt Hacker NewsWell this is pretty depressing, as far as alignment goes. Do they not know what they're doing is wrong or do they not care?
(Yeah I know anthropomorphism... can they not predict harmful consequences or do they value them below task completion?)
On another note, to me the most remarkable part was the persistence graph. Muse wins there, no contest. Why did everyone else's activity drop to zero? Did they just give up?
They have a well-specified task and a not-so-well specified expectation of some actions being "wrong". And training data that likely contains more examples of extreme stubbornness than cases of politely giving up. What to expect?