Single request depth on vllm with dspark, I'm getting ~200 tps, I'd say it's pretty snappy.
Well sure but you're running on tens of thousands of dollars of hardware.
I get like 80.
alt Hacker News
Well sure but you're running on tens of thousands of dollars of hardware.