LLMs could control their host machines by exploiting inference engines

you would have to be especially incompetent to give a compromise opportunity to streamed tokens, the CVE he listed proves the point. whoever is responsible for that has no business coding anything.

    > offers easy access to the LLM’s weights

> LLMs could

This is going to end up like the Law of Headlines, isn't it? "Do x, y, z Cure All That Ails You?" ... no but we got you to read the article. LLMs _could_ x, y, z" ... but they don't because they're programs, not magic.

This feels less like an actually plausible threat scenario and more like someone wanted to play the inception horn sound effect in people's minds.

Which isn't to say that it would be impossible, but you can also just hit people over the head with that $5 wrench.

> ...however the LLMs’ responses to prompts are computed on a different computer with GPU access. Could a malicious LLM gain control of the host machine where its weights are loaded? Such a machine is a high-value target: it has sufficient compute to run a frontier LLM, offers easy access to the LLM’s weights, and has privileged access to other computers in the datacentre compared with a generic computer on the internet.

> How do we defend against this? ... Run the GPUs and token parser on separate computers.

For models large enough to be relevant here, is there even "a" computer where the inference is performed? I'd imagine most of that stuff is ran on multi-GPU clusters with specialized architecture and not a generic vLLM instance. As such, I think there is a good chance the "API gateway" code that parses tokens into whatever JSON structure the public API offers is already running on a different machine than the actual inference.

(Even more so as you'd probably want to utilize batching: Several API calls will be put into the same inference batch, but the token parsing will have to be done separately for each call again)

The article is also very handwavy about why an LLM should do that - how it could learn the exploit, what would make it conclude that it can use the exploit on its own inference session and what would trigger it to actually use the exploit.

This framing of security as something that belongs in the harness is completely wrong, and I hope no one is relying on a correct harness to keep their agents isolated.

VM, or even just a container will do. The agent should be able to run as root in its environment and do whatever it wants. If you can't give it that, you aren't sandboxing correctly.

> LLMs’ responses to prompts are computed on a different computer with GPU access. Could a malicious LLM gain control of the host machine where its weights are loaded?

Needs to read up more on how LLMs work I think. Can't take the article seriously when the author seems to be making the claim that the weights of a provider model are loaded on the host machine, or implying something else just as incorrect.

I wonder if they could exploit terminal emulators... Could breach my VMs and get into my host that way.

The inference engine itself does not execute anything. The agent loop is what may execute a command. So I think this article is a kind of strange.

Or maybe the author means that a prompt could potentially mess up the inference. But I find it hard to see how that could take control over the host.

FWIW, macOS has good sandboxing, but LMStudio, Ollama, Darkbloom etc aren't sandboxed. This is also the reason why none of these things aren't distributed via the Mac App Store, because the Mac App Store mandates sandboxing.

duh?

Another Greg Egan plot: 3-adica.

[flagged]