It's worth remembering that in a few years that capabilities of these agents are likely to be as far behind the frontier as GPT-4 is today.

As it stands we've made remarkably little progress in terms of alignment and still have no good strategies which are likely to guarantee the alignment of super intelligent systems. As it stands the frontier of alignment is basically some combination of:

- hoping that more intelligent models become more aligned by default (more or less disproved at this point)

- hoping that if you RHLF a model to be a good boy enough it will in fact be a good boy

- asking it nicely in its prompts to be a good boy

- using another model to spot when it's being a bad boy and turning it off

- letting it lose and hoping we can spot when it's bad

There are many arguments which I'm convinced by that would suggest alignment of a super intelligence is impossible.

None of this is surprising to those of us who have been concerned about AI risk for a long-time and have be repeatedly mocked or insulted.

There will be a point of no return if we carry on down this path, and that point is now very rapidly approaching. When it does everyone you know will die, or worse. We should remember we need super-human general intelligences to cure cancer. Select narrow intelligences are fine and allow us to retain control. Let's be sensible about this. We need to stop.