alt Hacker NewsWas this a "build better sandboxing" and "don't tell people to eat glue" safety leader, or a Roko's Basilisk believing safety leader?
A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.
Replies
Given that he’s citing the need to learn from safety in other fields, I’d say the former.
We don't actually need anybody worrying about silly hypothetical scenarios — at least not as paid employees. There are already a surplus of sci-fi authors doing that.
>we clearly need a much stronger focus on the problems we are seeing now
I think it's a little more complicated than that. As Dean Ball put it:
>Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it worse by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.
https://x.com/deanwball/status/2104622726140883355
The important distinction, in my view, is between solutions which at least attempt to address the root problem, and solutions which sorta just patch things up (like better sandboxing). Addressing the root problem is both more robust in the short term, and also more likely to generalize in the long term. Resist the urge to focus on band-aid solutions, even if they are easier.
Today’s current problems were all hypothetical several years ago. At that time people claimed that the “real pressing problems” were misinformation and DEI issues. If we pretend that hypothetical problems can be safely ignored because there’s “no evidence” that they are real, we will keep getting surprised.
It puzzles me how doomers try to predict past the singularity. Isn't that _by definition_ unpredictable?
I'm reading If Anyone Builds It Everyone Dies, and there's so much sheer stupidity that has to happen for their 10+ pages of extinction scenario to occur.
I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all. My number 1 question: why do they think an RSI capable model would be first developed OUTSIDE a frontier lab? The labs have more compute, more data, more human brains working on the problem. Also thousands of variations of that same model that escaped. The escaping model somehow acquires the millions (billions???) of dollars it takes to run training to somehow RSI itself into infinity then decides to kill us all, all before the frontier labs manage to achieve RSI?
They entirely discount human alpha/economics. In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't. If we can't build a "software factory", how can an AI automate a bioweapons lab? Let's say AI steals crypto to fund itself. Do you think hackers aren't _already_ using AI to steal crypto? Don't discount human alpha!
Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.