alt Hacker NewsI find these local infra costs crazy heavy duty. I always assume these folks are getting their values worth or that infra is serving multiple people's requests at and justifying the cost. Like I once thought of buying a 48-64 GB RAM laptop (not even clsoe to what these models need) and I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access. So I assume the usage is also heavy duty, or if nothing then for privacy and safety and then I guess the price, if one can pay, will totally be worth it.
Replies
> I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access
The advantage is no more 5-hours or weekly session limit anymore. Also no chance your proprietary code ends up in someone repo.
You just need to imagine someone making big bucks in tech + being a little bit money ignorant, having a big house and a nice basement and cheap energy prices.
Than it just makes sense that they already do this.
But to be fair, if i move, enough space for battery, solar and a basement which can take proper ML hardware + heat exchanger, is in my future vision.
If the only variable is money, local models will never make any sense at these prices of infra/subscriptions.
For a lot of people, there are other variables though.
Looking ahead, do we think token costs will remain where they are today? Go down? Up?
Your investment in hardware is of course fixed, the token cost amortized over time.
I suppose the other question is, do we think down the road that the models capable of being run locally will improve?