Jev Can't Be Calibrated

I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%. I also tested with a fair coin 200 times and got 0.92 probability.

I did several tests and I think Jev is good at problems with a correct answer but weak at problems about actual probabilities whose answers can't be known at all.

Write-up: "Jev Does Not Play Dice" https://kantahayashiai.github.io/posts/jev-does-not-play-dic...

This is why I don't understand why everyone's freaking out about it. By far the biggest problem with LLM classifiers is that they treat every individual business as the blurry average of all businesses in their training data. Being lighter is fine if you control for everything else, but at least at my company we would actually have room for a significantly more expensive / slower classifier if it were demonstrably better at following instructions.

I like this post. I haven't had time to dig into Jev (they aren't accepting new signups), but calibrated probabilities is one of their pitches that caught my attention. And I was wondering how does one offer them on user data. Standard calibration essentially ensures that if a score of 0.8 accompanies a positive prediction (assuming the simple case of binary classification), then if you gathered together all predictions with a score of 0.8, around 80% will be correct.

If you have just one example you're sending to a model, how would they guarantee 80% over your data?

FYI, for an overview, scikit's page on calibration is great [1], and my answer on Quora from a long time ago covers a specific type [2].

[1] https://scikit-learn.org/stable/modules/calibration.html

[2] https://www.quora.com/How-is-isotonic-regression-used-in-pra...

Article says that it _can_ be calibrated, it just isn't out of the box. So the title seems contradicted by the body.

> If you want calibrated probabilities you’ll still need to recalibrate Jev’s probabilities on your own data. The good news is that is cheap. A few hundred labeled examples from your actual data can be enough to fit a Platt scaling on top of Jev’s scores.

I have been running a series of experiments on Jev since its release targeted at understanding it, seeing how it handles real use cases I have, and maybe figure out what it is inside.

Some of my tests do point towards what this post says. I was not successful in getting it's score to align with an existing rubric I had. It 'worked' but it was off and compressed from where I wanted it to be. Not a bad starting point, but I couldn't get it to move to where I intended the rubric to be. It wasn't the most robust test and I didn't spend a lot of time trying, but it wasn't just instantly magical.

However, it does seem genuinely useful just by being fast and cheap and good enough, so I am still a bit hyped.

This hype is caused by the price and the speed since most people don't know about small fast models and use big models for everything.

I'm really looking for a multi-modal image capable version of Jev.

If we could get machine learning type results on images without training, that would be fantastic.

While this is true, it is still possible to take the probabilities and make determinations of recall and false positives using conformal sets, https://crimede-coder.com/blogposts/2026/ConfClassification

(Or just use a model to re-calibrate the probabilities, I like the conformal approach though as those rates are what I often care about.)

You just need some labelled data to generate the "corrected" probabilities (or thresholds to meet the specified error rates).

People on X will use JEV for self driving cars too. That is the height of stupidity. I’ve seen this in so many tweets.

In future, we will see intiatives similar to OpenStreetMap for Textual data or Web similar to high quality non-contaminated steel.

I’m using Jev to classify a blob of text I see in browser with an extension. Calibrated? No. But it’s handy enough. If a large blob of text is likely AI generated, I’m very likely to skip it

Fine tuning LLMs has turned out to be mostly not worth the effort, but I wonder if fine tuning Jev-style models will turn out to be a whole lot more useful.

I spent a couple of days with it and it is fast and cheap but not especially good at classifying.