logoalt Hacker News

bozdemir • today at 7:50 AM • 1 reply • view on HN

Exagerating? I setup llm endpoints on corporate level in a company on H200 gpus, that serves hundreds of software engineers, I have observed differences, even on different versions of nvidia drivers. I shared my day to day experience. It is more and more important if you have very limited VRAM like 24gb on a 4090 RTX desktop or RTX 5090 Laptop gpu. I set up and use llms on my personal devices also, when I can't use cloud models on classified company materials. Here is an example problem what I daily report and deal with.

https://github.com/ggml-org/llama.cpp/issues/27330


Replies

jimnotgym • today at 7:54 AM

I can believe different drivers make a difference, but that wasn't the point under debate

➕ show 1 reply