From the other discussion: "running at full capacity, 24/7. Why would you do that?".
Also, I'd say if you're using Mac anyway that changes the economics, depends whether you've had to upgrade specs to run the size models you want. Going up to 64, 96 or 128GB adds 1000s to the price.
However when I'm doing non local inference, I rarely use a model as small as 32B.
alt Hacker News
From the other discussion: "running at full capacity, 24/7. Why would you do that?".
Also, I'd say if you're using Mac anyway that changes the economics, depends whether you've had to upgrade specs to run the size models you want. Going up to 64, 96 or 128GB adds 1000s to the price.
However when I'm doing non local inference, I rarely use a model as small as 32B.