It's good people moved this quickly on this stuff.

The thing is that the openjev stuff is a ... bit ... of a hack (a good one though):

It does this:

1. Send a throwaway request containing the shared state.

2. Hope SGLang keeps that text in its prefix cache.

3. Send a separate request for every question.

4. Each request repeats the shared beginning (but SGLang hopefully reuses the cached work in.)

5. Compute the complete vocabulary ; hundreds of thousands of possible tokens.

6. Keep only the few special answer tokens.

7. Convert those scores into probabilities.

Obviously this can all be done way more elegantly if you just own the inference engine -- fork / modify SGLang or vllm or llama.cpp, or do what I did in my bespoke inference engine (https://github.com/rdaum/eider/ commit https://github.com/rdaum/eider/commit/b2f981b7ebe0e338f60188...)

that ends up being, instead:

1. Convert the state into one shared prompt.

2. Run that shared prompt through the model once.

3. Fork the model’s internal state once per question.

4. Add a different question to each fork.

5. Ask each fork for its next-token scores.

6. Calculate only 64 possible label scores—not the whole vocabulary.

7. Convert the relevant scores into probabilities and return structured JSON.

I expect we'll see patches for llama.cpp and the others over the next few days/weeks and I also expect most model hosting providers will just end up providing this same service. I don't think Jev themselves have much of a moat. Though maybe it's more about their specific model and the training it gets.