alt Hacker NewsThis is a really funny sounding post. They sound like they just found out that increasing your automation gives you increased capabilities at faster speeds. They also sound like they just realized AI makes hard things easier.
But what really kills me is the idea that these companies are using Python for production inference. I mean really? Have you seen how bloated and slow Python is? Do global locks really sound like a strategy for fast dynamic computation?
Replies
Most of the fastest inference and training code in production today is written in Python. There are no global locks on the GPU except the ones you put there
Python acts as an orchestrator of accelerator libraries and does none of the inference math directly
Someone tell this man about vLLM!
> Have you seen how bloated and slow Python is?
Yes, but it's calling C code.
It's not that they "just found out" - what they are saying is that while they were previously dogfooding because it's good practice, now that their models are so much stronger they are using them because it helps accelerate.
If you look at how many years the whole NVIDIA and CUDA ecosystem has been evolving, it's certainly impressive how they've just stood up and optimized this CUDA-free 100,000 node cluster in just a few months.