alt Hacker NewsWhich raises the question, which are the fastest frontier models? are the enterprise hosted Anthropic models faster than what Anthropic serves?
Somehow no one talks about LLM speed.
Replies
LoganDark • yesterday at 11:54 PM
> Somehow no one talks about LLM speed.
When I've raised speeds about local inference I've been told 60-75 t/s is perfectly usable. It makes sense that people aren't talking about speed yet since you either already have a response fast enough to wait for, or you go do something else and check back in a few minutes.
I would love to wait for the latter type of tasks though, because those are typically the ones that require the most work from me to verify and I don't want my attention divided with multitasking.
OAI has announced an upcoming 750tok/s 5.6 served through their cerebras acquisition