Show HN: A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)

Hi everyone, I’m a friend of Mike’s; he’s having issues replying to the post at the moment, but hopes to post a thorough reply to the comments as soon as possible

But the problem is not that your model is fast.

Sure, you can go ASIC and go even faster, but the thing around GPU is that they scale well for both training and inference, and the technical floor is low.

The level to get into FPGA design is insanely high, you've got to read timing diagrams, you need to know combinatorial and sequential logics and good sense of boolean algebra, you need to have an asynchronous signal based mindset which is vastly different from CPU/GPU, you need to know netlist and you need to endure the time it takes for the EDA to finish generating it. Yosys is still years behind Xilinx

There is a reason GPUs are called accelerators; it sacrifices and does not try to really specialize on one particular thing, except high parallel dataflow and branch-free calculation. Otherwise we will all be using DSPs

I didn't expect the 2,000 connection sweep to stay flat, since all of them are sharing one stream. What does per user latency look like at that end of the sweep?

Ignore the naysayers!

Any article, even the really good ones on HN, while they get positive comments, for whatever reason, always get a lot of negative ones, too...

That is, the negative comments are absolutely unavoidable, even for people accomplishing great things!

I personally think that what you've done is brilliant, absolutely brilliant!

I can't wait to see more in this space...

Brilliant, absolutely brilliant!

[dead]

[dead]