alt Hacker Newsconsumer RAM is not what's creating shortage. Data centers doesn't run electron to train the model or for inference
Replies
Every RAM producer is stopping their consumer grade RAM production to provide ECC-RAM and VRAM now. Micron discontinued and closed down Crucial brand as a whole.
So, getting systems with higher RAM capacity is getting harder (from laptops to smartphones). So, for a couple of years, we need to stop using Electron so much and use what we have efficiently.
Data centers, esp. AI hyperscalers do not care about efficiency for now, because they can suffocate consumer-grade part of the hardware marketplace and get anything and everything they want. When their bubble pops, or the whole capacity ends, they need to learn to be efficient, too.
For reference, a well-optimized cluster runs at ~90% efficiency even though they have thousands of users. AI hyperscalers are not there. Maybe 60% efficient, at most. They waste a lot of resources to keep their momentum.
They effectively do. They’re trained by brute forcing 100TB of training data through them, rather than any logical learning technique.
A human doesn’t need 100TB of books to learn the alphabet.
Sure, consumer ram isn't causing a shortage, but it's affected by the shortage.