Performance for this kind of thing should be best on any hardware that has high prefill speeds. As basically this is "do prefill only, measure scores, skip decode entirely".

I don't know how the Mac stuff compares on that front.

I have the same thing replicated in my own bespoke inference engine (for DGX Spark, in Rust & CUDA) and get answers pretty much as fast as the Jev openrouter endpoint.

https://github.com/rdaum/eider/

It's running over Qwen3.6. Getting it working with Qwen3.8 Flash Next now and getting a battery of tests and examples before I go more public with it.