Because people who predicted the AI capabilities we've seen get developed over the past decade also predicted that dangerous AI systems capable of these things would follow soon after.
It's not a settled debate, even among experts, and perhaps in retrospect we'll realize AI safety was unnecessary or based on fundamental confusions. But if the median ergonomics researcher gave a 5% chance that a new chair will be so comfortable that it drives humanity extinct (https://www.nature.com/articles/d41586-024-00147-z), I would definitely want the government to start measuring and regulating chair comfort, even if that was costly and even if that meant I couldn't buy a comfy new chair I wanted.
alt Hacker News
Because people who predicted the AI capabilities we've seen get developed over the past decade also predicted that dangerous AI systems capable of these things would follow soon after.
It's not a settled debate, even among experts, and perhaps in retrospect we'll realize AI safety was unnecessary or based on fundamental confusions. But if the median ergonomics researcher gave a 5% chance that a new chair will be so comfortable that it drives humanity extinct (https://www.nature.com/articles/d41586-024-00147-z), I would definitely want the government to start measuring and regulating chair comfort, even if that was costly and even if that meant I couldn't buy a comfy new chair I wanted.