alt Hacker NewsThat's only half the reason it's expensive.
The other reason is that it would likely take years to spend $4000 (plus the real cost of electricity) worth of tokens on a 3rd-party provider that's running a similar limited, DS Flash type model. By that time, the hardware will be obsolete, assuming it's still operational.
Replies
KronisLV • today at 6:27 PM
But if you can use cloud models, why wouldn’t you use SOTA? For 2400 USD or less per year you can get pretty huge amounts of benefit out of that (though at the whim of whoever you are giving the money to).
> it would likely take years to spend $4000 (plus the real cost of electricity)
Since that cluster only yields 20-30 tok/s on that size of model, at least a decade before the hardware breaks-even with current token costs, and that's not counting electricity. Assuming continued downward pressure on token prices, and the cost of electricity, it never pays for itself.