Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware

I feel like these types of benchmarks mostly test what the mainstream tells you is possible and or works. You rarely get to hear the different implementations of inference engines, different types of setups (e.g. hybrid ones or older hardware) and their pros and cons. Also it seems these were tested with very minimal token usage.

Currently on my 9 year old laptop with a GTX 1070 Max-Q 8GB vram, 16 gb ddr4 ram and i7 8750h. I am running a finetuned Qwen 3.5-35B-A3B (Tiel-Coder-35B-A3B-GGUF-MTP) with 4-bit quant on a llamacpp fork (Turboquant) with 131k context at around 16-19 tk/s and 350-500 tk/s prefil using Pi harness. It is amazing for writing scripts and doing medium coding tasks and summarizing text for me. After around 100k context it goes down to 9 tk/s but that is to be expected.

If you ever ask any frontier LLM or these types of benchmarks, they would probably say that the best model I can run on my system is a crappy 9B model or something, which is just completely false. So I really wonder how much more you can pull out from more powerful systems and much better open weight models.

around this time, i was thinking about running a local LLM for the company i work for. i'm trying to figure out what kind of hardware i'd need and what level of performance i could expect. i tried a few things on your website but how much do the results reflect real world usage?

in my use case, the model might need to analyze a given task first then generate and execute a database query and finally send the results back to the client. to what extent could your setup handle this kind of workflow and how accurate are the results?

good job.

Why can't tools show a progress bar for the prefilling step?

I would like to see in such simulators:

- multi GPU configuration (hetero and homo)

- PCIe gen chooser (3.0/4.0/5.0)

- is nvlink enabled (2 cards, 2 x 2 cards, 3 x 2 cards)

- is PCIe bifurcated or CPU native (threadripper pro for example)

It would be great if numbers weren't off, would include optimised versions (ie. speculative decoding), context size, include allowed parallel inference (single agents use parallel inference a lot) etc.

I don't think in general people care about long list of models, they just care about the best one.

Also dual gpu is common setup, there should be ie. 2x 4090.

It would be nice if it would give a feel for the quality too ...

Shame they are not including old crap like my RX580 which is doing some impressive stuff with Qwen 9B Models.

just go rent one for a few minutes and try it out ... it'll cost you well under $0.50 if you're just trying to get a feel for if it's right for you. There's a lot of things like context window that this doesn't capture.

Depending on who you sign up for, that's within the introductory credits (usually like 5-10$) so it's actually $0.

Cool idea but I'd like to also be able to set a custom value of tok/s, ignoring all the other options

Would be nice if we could choose, say, multiple 4090s or otherwise customize the configuration (or maybe I just missed the button for that?)

Cool examples, this is interesting to test. I've noticed that OpenAI and Meta play a lot with animations and fades to mask the delays.

Model list is a bit limited, and whole harness affects the speed, so the libs used in harness and their versions might be useful to show for each individual model setup.

Doesn't seem to work? It just spits out a fixed answer about RAM. qwen3.8 27b

I don't know what basis these numbers are estimated on, but my M2 Ultra 128GB runs almost twice as fast as what's reported for the M3 Ultra 512GB.

I wished there was such a tool before picking my hardware, it's useful. But the estimates needs to be somewhat correct.

Doesn't provide for hybrid setups RTX5070 + RTX5080 for example.

This is a great idea but I'm far less concerned about output token speed and way more about the ability to parallelize prefill - for agentic coding that's pretty much way more important than output speed.

[dead]

[dead]