>but don't lose sight of what LLMs can do off the leash.
there is no such thing as an LLM "off the leash", it's not a dog, and even if it was a dog the person responsible is the owner. What is this bizarre attitude to a piece of software that makes people think existing laws don't apply?
If your LLM agent hacks a bank, you have hacked a bank, you will go to prison and that's entirely sufficient. People have been hacking banks for decades now, it didn't require the government to regulate C compilers and Emacs.
There is a baseline level of competence and motivation needed to commit crimes.
Decades ago few people would walk into a record store and steal CDs. Napster came along smashing all barriers entry, and it became weird not to steal music.
Its not really the legality that matters, it's the barrier on one hand and the cognitive ability on the other. Drop both and you get huge spikes in crime.
This is overly reductive.
If your web browser hacks a bank, but you didn't know and didn't expect it to, have you hacked a bank? Why is an LLM different? What happened to mens rea?