Running very large models on Mac is unusable at 10 tok/sec. You get more average inference over the day using free Google Gemini.
And for the price of a Mac that can run a large model, you can get 2 3090s humming along running a small model so fast that it can simulate a lot of the behavior in large models just through sheer number of context it generates. For example, editing code means that by the time your large model on your Mac is finished writing a file, the smaller models have generated the code, written the code to file, ran it, and debugged any issues.
So given that, which one of these is true about you?
1. You are paid by Apple to push marketing on HN
2. You are a hardcore Apple fanboy and just think that owning a Mac studio is a flex
alt Hacker News
Running very large models on Mac is unusable at 10 tok/sec. You get more average inference over the day using free Google Gemini.
And for the price of a Mac that can run a large model, you can get 2 3090s humming along running a small model so fast that it can simulate a lot of the behavior in large models just through sheer number of context it generates. For example, editing code means that by the time your large model on your Mac is finished writing a file, the smaller models have generated the code, written the code to file, ran it, and debugged any issues.
So given that, which one of these is true about you?
1. You are paid by Apple to push marketing on HN
2. You are a hardcore Apple fanboy and just think that owning a Mac studio is a flex