The prefill is the only blocking part, and you can prefill the whole context up to the point where they diverge, then prefill each question and decode the one token in parallel for each question.
If you batch vLLM calls with the same prompt prefix to the same process, it'll deduplicate the prompt prefix across batched requests (+/- the block size) and decode in parallel for each.
That's with a vanilla LLM. If you modify the LLM you can pull that in-graph, but it isn't really necessary.
alt Hacker News
Sure it is.
The prefill is the only blocking part, and you can prefill the whole context up to the point where they diverge, then prefill each question and decode the one token in parallel for each question.
If you batch vLLM calls with the same prompt prefix to the same process, it'll deduplicate the prompt prefix across batched requests (+/- the block size) and decode in parallel for each.
That's with a vanilla LLM. If you modify the LLM you can pull that in-graph, but it isn't really necessary.