alt Hacker NewsIf you are currently trying to build a web-based clone of something approximating ChatGPT, pretty much everything in this article is absolutely mandatory.
The streaming response UI/UX is a total nightmare to make work smoothly. You have to come up with clever heuristics around where to chop up the stream of changes and how to batch the work relative to frame updates to make it not look like confetti during a streaming response.
I really don’t understand why the LLM web chat clients have all converged on this UI where one character appears at a time. It is kind of fun the first time (like you are watching some Harry Potter special effects where an enchanted book writes itself). But really they could send sentence or paragraph long chunks of text, it would be fine.
Seems like an artifact of the time when tokens-per-second were low enough that people needed to be informed that the machine was actually doing something.