alt Hacker NewsGrep (or ripgrep at least) is i/o bottlenecked at this point. It's impossible to process data at faster than i/o speeds, since you have to get the data to the processor somehow. That doesn't change whether that processing is grep on a CPU, or LLM on an ASIC.
Replies
serbuvlad • today at 5:13 PM
ripgrep may be.
I did a toy project once implementing a limited version of grep on an FPGA and was able to get some speedup over GNU grep at the time, though marginal.
In any case, LLMs aren't IO bound :))
➕ show 3 replies
sanderjd • today at 5:13 PM
i/o could still be sped up though? And I dunno, maybe this actually is an argument for how the llm version could end up being faster, because there is lots of investment in crazy fast i/o hardware and protocols to get the data into the chips.
GPUs and inference ASICS also have large amounts of high bandwidth memory, plus lots of high speed storage cache, and dedicated very high bandwidth scale-out and scale-up networks. Because they are also often bound by I/O bandwidth.
If your problem is grepping crazy amounts of data, the infrastructure for LLMs isn't a bad place to look for an example.