alt Hacker NewsThese numbers are a lot lower than I expected from such pricey hardware.
I'll stick with my old AMD datacenter cards thanks. An Instinct MI50 32GB would cost around $550 or so. It has 1TB/s vram bandwidth thanks to its HBM2.
Replies
jdright • today at 4:40 PM
any numbers to share? also, what inference engine do you use and with which api?
I am also running some old AMD datacenter cards, 2x MI25 in my case. Getting around 30 tokens/second with short context.
Tensor parallel in llama.cpp using RCCL (disabled by default in llama.cpp for some reason). Surprisingly, for these cards HIP is actually faster than Vulkan, unlike the 9070 XT where Vulkan still wins.
ROCm nightlies do actually support these old cards, just not the ROCm stable releases.