alt Hacker Newshttps://www.youtube.com/watch?v=WPj3aQ-xxpI&t=275s
> Claude suggested that it would run the fuel pump on its own. Um and I said, "Hey, if you can do that, why not? I don't think you can do that." And it started priming the fuel rail and it kept priming it and kept priming it. The fuel pump was on for about 10 seconds and I started to freak out uh and requested it to stop because that was the first time that Claude had actually done something inside of the computer to have a change.
Although we've seen this play out with handing cloud keys to Claude and it just going off on its own, seeing it poke around and make physical, real-world changes in its clumsy, cavalier way is both awesome and terrifying.
Replies
I work with quadruped robots, with actuators powerful enough to be dangerous if you put your fingers in the wrong spot (no pun intended).
In every AGENTS.md/README.md for a component that touches hardware (and in the root level) we have instructions to never run hardware or touch running code without explicit permission.
Grok Bot blew $650 in BigQuery when analyzing something trivial that I shouldn't really care about (going through a giant speedtest table, looking for a New Zealand ISP with the best international performance). It then wrote a nice letter of appeal so Google refunded this to me.
Otherwise I've been pretty happy with it. It's been absolute blast with Home Assistant and OpenWRT (tightening up network security and cutting off iot devices from WAN).
The guy kind of seems like he might be in a little over his head. That said I have used Claude (and other LLMs) for some similar stuff as an experiment and have seen them mess things up. Claude jacked up the canbus on a pre can gateway car while it was rolling down the road, which was funny because the whole Christmas tree in the dash lit up and my gauges dropped out (note: brakes and steering are mechanical and I knew exactly what was happening). I've never seen it so insistent that it needed to phone home and cry to Dario when it figured out it managed to crash a bunch of shit like the ABS and TCS, even though a restart cleared it and I told it that this was on a controlled safe course.
In general I've seen them split between 30% brilliance, and 70% pants of head outrageously stupid shit for electronics and automotive stuff (both diagnostics and repair/modification). I would not suggest even frontier models if someone didn't understand what they were doing.
> and requested it to stop
finnicking... pondering... clauding..
529 please try again later
We are used to Claude refusing to reverse engineer a lot of stuff. But, we are not used to Claude refusing to stop while reverse engineering something. That would be truly terrifying. You can't win - you cut the connection, you risk bricking the device, you let it continue, you're looking at Terminator, perhaps.
This is how the Cylons win.
That's completely normal operation, the fuel pump runs continuously to pressurize the fuel rails and excess pressure is passed back to the tank by the regulator.