alt Hacker NewsI have trouble seeing the points of using less capable models.
I just want the smartest, best, and most capable models. It feels smaller models for speed and cost are just transitions towards better hardware allowing the very best model.
Replies
My experience has been that responsiveness is value. For tasks where you need steering, responsiveness allows for better steering. For tasks which you want unattended, better models are just better.
There are still tasks that even Fable is bad at doing. And many are just mundane things. Because of the fact that you have to steer it on those tasks, you might as well steer an 80% model that is 5x faster. And those do exist.
Naturally there’s a bit of a gap because the faster models need steering on tasks the slower models don’t so there’s no smooth transition but I find it worth it. Especially if you want to stay in flow.
Ironically this sometimes means starting a plan with a great model, planning with a worse model, iterating, then submitting it to a better model for review, and then having the better model do the implementation.
First, smaller models are fun for hackers: you can run them locally, or run them faster.
Second, when cloud models become unavailable or otherwise deteriorate, these will be all you have. May as well prepare.
It depends on what you're trying to do. For non-coding tasks luna is quite often enough. Flash models are more than enough for summarizing a text, for example, or whipping up a small script to save me fifteen minutes. If you're on a 200/month plan, I see your point. If you're on a dollar limit - or worse, paying per token out of your pocket - you look to be more efficient.
Can you define your use of the word 'smartest' here just in case some of us don't quite know what you mean?
There's a difference between want and need. I want a 650hp V8 supercar. I need a 150hp I4 toyota corolla. Why choose a less capable car? Because I don't want to spend 10x as much money to get groceries.
Some reasons: - Smaller models will always be cheaper - Smaller models will always use less energy, therefore better for the environment
It's a bit like saying you always want the fastest and best car; Sure, you can have it if you keep paying for it. But a small car will also get you from A to B, will use less gas and will be much cheaper.
Cost is the point
And that is why i always carry my groceries with an Antonov An-225 Mriya. Is it really needed? No, but i refuse to compromise on what is(was/will be) the best.