logoalt Hacker News

notnullorvoid • today at 5:00 PM • 5 replies • view on HN

It's already happened hasn't it. Google's TPUs are apparently much more efficient at inference workloads. However the hardware isn't available to the public like Nvidia chips are.

I'm unsure about the energy efficiency of Tenstorrent cards, but they are at the moment much more cost effective than Nvidia. But software support is far behind, I suspect if it wasn't the cost would be more equal.


Replies

epistasis • today at 5:48 PM

Inference has two steps, prefill and decode, with very different requirements on the compute to memory-bandwidth ratio.

Already these are run on different hardware, at least as I understand, and the computational needs could further drive differentiation.

woodrowbarlow • today at 5:59 PM

there's also stuff like Talaas, where the ASIC is the actual model weights

➕ show 1 reply
dominotw • today at 5:08 PM

Doesnt google themselves use nvidia chips. why do they need nvdia chips then?

➕ show 2 replies
trollbridge • today at 5:12 PM

TPUs aren’t as effective at training.

Inference has already been commoditised.

➕ show 1 reply
hrncn • today at 5:24 PM

You have basically managed to sum up why NVDA keeps dominating, no on can "just buy a few TPUs" and that has been the case since the first TPUs were announced years ago (yes yes GPU shortages and so on, but no one questions general availability of GPUs), and the current Tenstorrent model support page looks like an invitation to beta test HW that might (or might not) be able to run current and future models at some point.

But the next gen of Tenstorrent HW looks great on paper - problem is, I could have written that before the current gen was released.

All of the allegedly competing HW looks like that and has looked like that for years, either a lack of availability, capability, or both.