alt Hacker NewsBy design it can't be significantly slower than Jev: the prompt processing (AKA PP) is exactly the same on both and will take most of the time. Then you can process every single "question" in parallel, just predicting one or two tokens (if an answer is ambiguous with a single token) per each question, again in a single batch.
So, fast in the LLM space and comparable with Jev.
That's right. There's only so much optimization that you can make to a transformer-based model and any tricks that Jev is employing, any open-source LLM can also employ.