alt Hacker NewsI'm confused... This has no relation with the Jev team, isn't it?
It's trying to "emulate" Jev behavior using a regular small LLM model (Qwen3 0.6B or MiniCPM5 2B). And with the smallest model it takes like between half to two seconds to run in my M2 Max, so it's not super fast.
I mean, it's faster than asking to a regular LLM, but I think that's not proper to have Jev on the name (also legally...)
Edit: no shade, and I'll give it a try for some ideas. I'd also like to have an open weights Jev but I think the naming is misguiding. I also have to try Jev that, BTW, got access pretty quickly, less than a day I think...
Replies
Performance for this kind of thing should be best on any hardware that has high prefill speeds. As basically this is "do prefill only, measure scores, skip decode entirely".
I don't know how the Mac stuff compares on that front.
I have the same thing replicated in my own bespoke inference engine (for DGX Spark, in Rust & CUDA) and get answers pretty much as fast as the Jev openrouter endpoint.
https://github.com/rdaum/eider/
It's running over Qwen3.6. Getting it working with Qwen3.8 Flash Next now and getting a battery of tests and examples before I go more public with it.
OP's point here is that the overall approach of restricting output token space and using parallel prompts to produce concurrent results and taking the most relevant ones isn't something novel to Jev (not saying there's nothing novel, but a facsimile can be created at the application layer using any small, fast model)