alt Hacker NewsDigression, but this is the real Great Filter imo, not AI. I think technology advances to a point where it only takes one or two bad actors to type the right prompt to get a recipe for civilization-destroying bioweapons before you get anywhere near true AGI or anything relevant to the Kardashev scale. Biology is fragile.
But not that that’s a good justification for hamstrung models. I think it’s just the inevitable endgame and it’s more sad than scary
Replies
I don't fully understand the instinct to regulate local models for this? It seems like the wrong place to address the problem.
You can download Ebola sequences right now if you want to. That's not the same as having an isolate. The difference is a lot of messy reality. This kind of work is not generally "one shot" (Claude make me a supervirus, make no mistakes), it requires lab space, iteration, and specific resources. It has a footprint.
Wouldn't it make more sense to monitor / regulate facilities where you can sequence or request assembly of DNA, RNA, restrict and monitor the supply of key reagents and so on?
I have the same concern. If it becomes possible to engineer Captain Trips with a budget in the low 8 digits it won’t really matter what else happens.