alt Hacker NewsI'd like to question that premise. Open models are great for research, privacy, cost, customisation and a host of other things... But they're also going to be the engine that breaks the world. I'm already surprised that we haven't seen Llama / Qwen + Whisper automated spear phishing at scale. It's literally just a matter of time. Given the incredible progress in image and video generation, it's been clear for a couple of years now that real time identity theft voice and video calls are going to be an enormous problem. I don't think anyone realises how much of a problem. I haven't seen a single effective solution proposed to proving digital identity given these new attack vectors. It's akin to nuclear waste in that way - the benefits are obvious, but the problems so vexatious (not to mention expensive) that little space is given to acknowledging them, let alone solving them.
If anyone has come across a robust solution to digital identity verification - not to mention verification of news media etc - which is robust enough to withstand the cyber attacks and social engineering the frontier models are capable of... I haven't seen it.
A world where business and communication is conducted primarily online cannot coexist with low cost, widely distributed, undetectable identity theft.
Replies
> A world where business and communication is conducted primarily online cannot coexist with low cost, widely distributed, undetectable identity theft.
what if we dont use centralized identity at all?
web of trust failed in the 90s because exchanging keys is hard when all you have is desktop computers, wired connections and old school hackers who dont care about ux. today you could set up a system where the whole process is tap two phones together with nfc and confirm, everything gets auto downloaded and shared with your whole network.
now this doesnt work directly for a random online business where none of your contact personally know anyone, but governments can act as an authority that cross signs their citizens keys. that means if you want to stop bots from creating accounts all you have to do is make new users show a certificate from one of the authorities you trust.
its cheap to verify, decentralized and not hard to integrate with existing PKI. the biggest technical problem is key management as always, but that can be solved with something like ethereum style social recovery setups or a did:plc type multi key scheme.
that gives you a robust identity scheme. browsers and social media can integrate it to show a trust rating for content based on who signed it. messaging apps can show a warning or refuse calls from unknown users. fraud is only possible against a person who somehow has no irl contacts, never used a government service in their life and trusts random strangers online. at that point they deserve it.
the one massive, unpredictable question is if anybody is willing to go ahead and adopt it on a scale big enough to create network effects. it would take a state level force and carefully designed OS integrations so its more convenient than email/password signup.
but even if all of this fails i still think a world of scams is better than a world controlled by a couple big countries (America, China, maybe EU) or for-profit corporations. open source ai might lead to anarchy but closed ai will definitely get us to tyranny. i dont know about you but i would take the first option.
> the problems so vexatious (not to mention expensive) that little space is given to acknowledging them, let alone solving them.
These problems will exist regardless of whether or not we get open model access. I'd personally rather that OpenAI and Anthropic aren't profiting off these scammers, creating a perverse incentive that open model providers don't have.
Sounds good. We can stop conducting business online and get back to being humans.
Spear phishing isn't a real problem. Competent organizations have already implemented sufficient defenses and controls, regardless of whether the attacker is a human or LLM. Idiots will continue getting scammed but that's nothing new. I expect that many small businesses and local governments will be essentially forced to outsource their IT infrastructure to large vendors that can maintain hardened systems appropriate to the escalating threat level.
(Nuclear waste isn't an actual problem either. For civilian powerplants the highly radioactive waste can generally be stored indefinitely at the reactor site.)