alt Hacker News>> As you can understand, during V3 training, NVIDIA GPUs were still used, but the NVIDIA ecosystem was no longer employed.
Ironic that these large LLMs are eroding Nividia's moat. In the next paragraph he talks about Nvidia digging its own grave. I wonder if Nividia is aware of this and the frequent release cycle is a response to this development ?
> Ironic that these large LLMs are eroding Nividia's moat.
That's not what the quoted part meant. During v3 development they only had access to hardware limited variants of H series GPUs. Those had less interconnect bandwidth IIRC. So, at the time, the low-level wizards that ds employed bypassed the official APIs (i.e. the nvda ecosystem) and hand wrote alternatives to say nccl, to better use those limited GPUs. I remember them publishing some of it as well. It had to do with allocating memory, moving stuff around, etc. Basically bypassing some limitations by going lower than the official APIs support.
There is no eroding of their moat, as long as they sell GPUs. ANd they're selling GPUs like crazy. The moat speaks for itself, if I may :)