If that hammer could allow people to go into people's homes / work en masse, steal all their information, blackmail them, steal their identities, break their systems (including those of hospitals and other critical infrastructure) and generally help fund bad actors through it all we'd think of having restrictions on hammers too. A hammer can't screw people over by the millions.

I don't like this argument specifically with AI. Facial recognition everywhere you go is just a tool. Your job creating a detailed profile on exactly how you work, who you talk to, and about what is just a tool. The tools have become so good and easy to use we have to have serious discussions about them before things get out of hand.