It’s cheaper sure, but it’s very slow. It’s not a drop in replacement
I think we don't have a good draft model for better speculative decoding yet (e.g. DFlash 2). Once we do, it will be faster.
alt Hacker News
I think we don't have a good draft model for better speculative decoding yet (e.g. DFlash 2). Once we do, it will be faster.