alt Hacker NewsElectricity cost seems to be about $30/month for a 32B model on a GPU. It's probably better on Apple hardware.
https://github.com/QuantiusBenignus/Zshelf/discussions/2
Not accounting for hardware, of course :)
Replies
NorwegianDude • today at 12:19 PM
The price, processed tokens, and output can be anything, it just depends on what GPU it is.
Nvidia GPUs are much more efficient than Apple hardware for inference(and training).
My Mac Studio uses about 60–80 watts whenever I’m running a model (as measured by the system metrics), so it’s less than 2 kWh/day at full blast. Electricity is like 0.125 €/kWh, so that 24-hour period would be <0.25 €.
Not accounting hardware in my costs, since I didn’t buy my hardware for running models. Running models is just something it can do in addition to what I got it for.