alt Hacker NewsI run a crowd sourced website to collect data on the best and cheapest hardware setup for local LLM here: https://inferbench.com/
Source code: https://github.com/BinSquare/inferbench
Replies
kilpikaarna • yesterday at 11:45 PM
Nice! Though for older hardware it would be nice if the price reflected the current second hand market (harder to get data for, I know). Eg. Nvidia RTX 3070 ranks as second best GPU in tok/s/$ even at the MSRP of $499. But you can get one for half that now.
➕ show 1 reply
jsight • today at 2:02 AM
It seems like verification might need to be improved a bit? I looked at Mistral-Large-123B. Someone is claiming 12 tokens/sec on a single RTX 3090 at FP16.
Perhaps some filter could cut out submissions that don't really make sense?
Cool site, I noticed the 3090 is on there twice.
https://inferbench.com/gpu/NVIDIA%20GeForce%20RTX%203090
https://inferbench.com/gpu/NVIDIA%20RTX%203090