alt Hacker NewsBut that AI engineer who is implementing speculative decoding is still just doing basic plumbing that has little to do with the actual reasoning. Yes, he/she might make the process faster, but they will know just as little about why/how the reasoning works as when they implemented a naive, slow version of the inference.
What "actual reasoning" are you referring to? I believe you're making my point for me.
Speculative decoding requires the implementer to understand:
- How the initial prompt is processed by the LLM
- How to retrieve all the probabilities of previously observed tokens in the prompt (this also help people understand things like the probability of the entire prompt itself, the entropy of the prompt etc).
- Details of how the logits generate the distribution of next tokens
- Precise details of the sampling process + the rejection sampling logic for comparing the two models
- How each step of the LLM is run under-the-hood as the response is processed.
Hardly just plumbing, especially since, to my knowledge, there are not a lot of hand-holding tutorials on this topic. You need to really internalize what's going on and how this is going to lead to a 2-5x speed up in inference.
Building all of this yourself gives you a lot of visibility into how the model behaves and how "reasoning" emerges from the sampling process.
edit: Anyone who can perform speculative decoding work also has the ability to inspect the reasoning steps of an LLM and do experiments such as rewinding the thought process of the LLM and substituting a reasoning step to see how it impacts the results. If you're just prompt hacking you're not going to be able to perform these types of experiments to understand exactly how the model is reasoning and what's important to it.