alt Hacker NewsUnified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't
Comments
I'm writing my own inference engine for Strix Halo and the same model. I already have 30%+ performance plus a more graceful decay over long contexts; that said, their point stands: memory bandwidth is what you really want.
For some reason, this reminds me of my last shared memory system. It was an Athlon XP 1800+ with VIA ProSavage back around 2002. It was just barely able to run CS 1.6.
Think future generations of AMD could get quite interesting. They’re no doubt seeing people whining about mem throughput specifically
Uhh the 5090 alone is double the cost of their quoted PC prices.
If compute is not the bottleneck, memory is easy-ish to produce (the hard part is mostly on the fab side); what stops a Chinese NVIDIA (huawei) from being 10x cheaper?
Do unified memory CPUs suffer from the same memory shortages as normal memory?
I guess they're just welding the memory to the CPU chip, but still curious.
"Can't" is not really correct.
Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.
The current “big GPU” has 96gb of memory, but that’s not a “consumer GPU” apparently, while a $5000 Spark is a “consumer PC” I guess. In any case you’re probably better off running a large open weights model on the cloud.
Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).
> Put two machines on a desk, each about $2,000. One is a tower with an NVIDIA RTX 5090: 32GB of the fastest consumer memory ever shipped, 1,792 GB/s. The other is a mini PC the size of a paperback, an AMD Ryzen AI Max+ 395 "Strix Halo" box with 128GB of soldered memory at roughly 256 GB/s.
Doesn't change the conclusions of the article, but each of those machines is more like $4k+
https://www.microcenter.com/product/711961/amd-ryzen-ai-halo...