AirLLM 70B inference with single 4GB GPU

For anyone wondering “how slow is this?”

IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token

https://github.com/lyogavin/airllm/releases/tag/v3.1.0

Seeing a lot of these “run 1TB models with 1GB RAM” projects recently. Most seem vibe coded and probably won’t be maintained.

Hoping a winner emerges with some real momentum behind it.

What device do I need and how much will it cost to install one at home so that it works as quickly as the Claude Code answer (and it answers quite slowly)?

I’m still slightly confused on what this adds.

Let’s say I wanted to run a full size open weight model. I have a 128GB m3 max laptop.

Does this basically load layers in and out on demand? So I still have to download the full model to disk, but the RAM requirements go way down? The readme calls out that one still needs to connect HuggingFace, which leads me to believe that maybe you don’t even need to download the full model?

I guess the use case is something like: you have a slightly obsolete Mac or PC or a whole bunch of them, and just need to compose one or more convincing spam emails, but it's fine if it takes a full week to do it?

We will see if this project has legs. This is the kind of efficiency we desperately need. Now if we can address efficiency with llm training.

Running 70B on a 4GB GPU is wild. Really impressive engineering feat for resource-constrained environments.

Same problem, different day