honestly i dont blame smart people for cashing in, because there is no inherent reward for behaving goodly/smartly. in this instance gwern is a particularly smart individual and has contributed a lot so i especially shouldnt judge him. straight up ripping off a black mirror episode (s02e07) is a bit heinous for my liking though. digital twins are inevitable but that doesnt make this any less wrong.

"AI poses threats of its own ... a nuclear bomb can’t think for itself and make choices, but AIs do, and current LLMs have proven themselves untrustworthy as they regularly reward-hack and betray their users ... how can you trust them to handle ecosystems of combined-arms for an AI-centric military during a war? But widespread deployment of GAs offer some hope of meaningful supervision, as long as the GAs are sample-efficient enough ... or there is some chance of the principals being able to “catch up” later and correct any errors before events have spun too far out of control."

gwern openly admits ai behaves erratically, then hand waves the issue away 'as long as we double check things [sic]'. gwern is smarter than this, ergo i feel like i am being bullshitted.

is anything in this world worth knowing that a version of you will suffer for eternity? is anything in this world worth copying yourself such that you may be enslaved by anyone with terminal access?