Compare the performance of a 980 and a 5050 and I am sure that will answer your question.
Also models baked into the silicon are able to achieve efficiency that is simply impossible to achieve with programmable circuits, there is a general slowdown in the raw capabilities that transformers can achieve and agentic tool use is simply an amplifier that will reach a wall eventually. It wouldn't surprise me if we saw within 5 to 10 years accelerator cards that you're able to purchase and plug into via usb-c that are able to achieve thousands of tok/s as well as api costs going down to what we already see with subscriptions today.
There has been quite a lot of off-ramping going on where people feel satisfied with the performance they're getting out of the models and simply staying there instead of using SOTA.
alt Hacker News
Compare the performance of a 980 and a 5050 and I am sure that will answer your question.
Also models baked into the silicon are able to achieve efficiency that is simply impossible to achieve with programmable circuits, there is a general slowdown in the raw capabilities that transformers can achieve and agentic tool use is simply an amplifier that will reach a wall eventually. It wouldn't surprise me if we saw within 5 to 10 years accelerator cards that you're able to purchase and plug into via usb-c that are able to achieve thousands of tok/s as well as api costs going down to what we already see with subscriptions today.
There has been quite a lot of off-ramping going on where people feel satisfied with the performance they're getting out of the models and simply staying there instead of using SOTA.