alt Hacker News> The total number of concurrent requests represents the number of simultaneous requests moving through the pipeline at any point in time. Unlike a GPU batch, those requests don’t wait for one another to finish, they stream through the pipeline continuously.
What do they mean by “unlike a GPU batch”? It’s not a hardware limitation of GPUs right? It’s just the scaffolding to do this in software isn’t available right
Replies
edg5000 • today at 4:39 AM
From reading xpct's comment in this thread it seems like they use effectively a very deep and wide pipeline by tiling the cores without central clocking or dispatch. In theory if a GPU is centrally organized it could simulate this behaviour, but I imagine that creates transfer overhead. Just speculating though.
It's describing a different type of parallelism - pipeline parallelism. The network is laid out spatially on the wafer in CB's case. So weights are stationary and activations flow. A GPU is going to process the network one layer at a time. GPUs also use other types of parallelism (tensor and expert mainly in inference). Pipeline parallelism doesn't make much sense on a GPU for inference.