Some context for why this hit us harder than most: we run a ~20-agent dev harness — coding agents that work in parallel, open PRs, and push commits more or less around the clock. Great for shipping. Rough on your CI meter. A team of humans pushes a few dozen times a day; 20 tireless agents push a lot more, and every push fires the full test matrix. Our GitHub Actions bill climbed accordingly — nobody made a bad decision, the machines just never stop typing.

So we did the obvious thing: moved CI to a single self-hosted box. Fixed monthly cost instead of a per-minute meter, and the builds actually got faster because the box keeps a warm Docker cache and doesn't cold-start every job.

It sounds like a five-minute job — spin up a VM, run config.sh, done. It mostly is. But three things bit us that nobody warns you about, and they're the reason we bothered to package this up:

1. OOM. A memory-constrained box running a couple of heavy jobs at once just gets its builds OOM-killed, silently, and you spend an afternoon blaming flaky tests. Fix: give the box swap — specifically one swapfile per runner — and set vm.swappiness=10 so it only leans on swap under real pressure. Swap isn't the hot path; it's the airbag.

2. "No space left on device" — but the disk looks half empty. The culprit was /tmp silt: dead job-workspace directories that never get cleaned up, slowly filling the disk until a big build tips over. Fix: give each runner its own /scratch (so they don't fight over one /tmp), and add a systemd-tmpfiles rule that ages /tmp + scratch every 6h. Set-and-forget.

3. The test database. Our suite needs Postgres. One shared instance every job writes into means parallel runs clobber each other's data and burn through connections; the other extreme, a fresh Postgres container per job, is heavier and slower than it needs to be. What worked: one small native Postgres on the box (128MB shared_buffers — deliberately tiny), shared by all runners, with each run isolated in its own schema — create it on the way in, point search_path at it via DATABASE_URL, drop it in an always-run cleanup step. Small-and-shared beats N heavy instances on a constrained box, and per-run schemas make concurrent runs safe without paying for a container spin-up on every job.

The operating rule we landed on for an 8 GB box: provision 3 runners, but cap concurrency at 2. The third is a warm spare and gives you headroom; running all three heavy jobs at once is how you meet the OOM killer again. Bigger box, bump both numbers — it's just policy.

We wrote it up as a few idempotent scripts + a README so it's reproducible instead of tribal knowledge:

https://github.com/senoff/self-hosted-ci-runner

- provision-box.sh — swap / per-runner scratch / reaper - install-runners.sh — download, register, systemd-install N runners - provision-postgres.sh — one small shared Postgres, with a per-run-schema example workflow - check-runners.sh — how many are actually online (a dropped runner silently slow-walks every PR)

Everything's parameterized through a config.env, and the README has the exact reference box config (a small Hetzner VM, Ubuntu 24.04, Docker) with copy-paste setup steps.

Fair warning / the tradeoffs: self-hosted runners are great for a private repo you control. Do not point them at a public repo that runs untrusted-PR workflows — that's remote code execution on your box by design. One box is also a single point of failure, and you own patching it now. For our case (private repo, trusted team + agents, predictable cost) it's been a clear win.