alt Hacker NewsI’ll go out on a limb and suggest that I don’t think a Jev-like model is particularly useful unless you can fine tune it. The Jev API has zero ability to pass in a prior [0], and, if you can neither pass in a prior nor fine tune for your system, you will get an output that may be almost meaningless.
I’d love to see someone build a model of this sort that can actually accept priors and do something intelligent with them.
[0] You can feed Jev a prior as text. I’ve tried it. It works poorly.
Replies
It seems like jev's major advantage over existing classifiers is that I dont have train it.
If I have to gather and tag data to fine-tune Jev, I can probably just train an "old school" classifier model and make it even cheaper, faster, and just as accurate.
Also one of the more interesting features of Jev is the confidence rating that hardly any Jev-cc talks about.
I suppose a sort of prior-proxy can be encapsulated by a carefully written system prompt.
I think better than priors would be a closed loop where you tell it what the right answer was (or some signal) and they monitor and fine-tune for you
We were able to completely automate 20,100 token prompts with At0m[https://at0m.pienomial.com/].
We believe entire compliance workflows (even multilingual) could be automated.
Would you like to get a demo ?
You can't fine-tune Jev itself but you can train an ML model that uses outputs from Jev as inputs. Which does enable you to 'fine-tune' your overall model.
You can also improve your Jev classifications based on your ongoing data if you're labeling it continuously, especially if you're explaining the reasoning in the feedback labels. You can identify new elements of the rubric and add them to the list of classifications that Jev produces, and then those become new features for your ML model.
Two levers of control for using data to make a Jev-based classifier model continuously better-aligned.