Jev's launch was the best AI era marketing campaign I've seen so far. It was executed and timed so perfectly that Jev became synonymous with decision models overnight.

I think it reveals how many people were only familiar with LLMs as the generic AI tool for everything. It only took one ambitious marketing campaign for someone to come along and present a simpler technology as a new breakthrough to audiences that hadn't considered anything else.

The largest LLMs are really amazing for how much they can accomplish on unseen tasks. That is undeniably remarkable. The mistake that gets made everywhere is thinking that they're the best tool for every task, because LLMs are the only AI tool in most developer's AI toolbelt right now.

I think there are a lot of practical applications where something that works few-shot and doesn't require training is quite appealing if the cost of inference is low. That's the sweet spot for Jev and similar models. If you have time and an eval set to train it, clearly a "real" special-purpose classifier should have much better performance (and for most applications cheaper cost/faster inference), but Jev and friends are way better than using a full-fat transformer LLM for a classification task.

As an example, I've been using winnow running locally as a decision model in openchamber to select between llms for doing various tasks. I don't have an eval set of different coding and other tasks that I could use to train a proper classifier. This I just download off huggingface and it works out of the box and has cut my token spend to a fraction of what it was. So it's good enough to be beneficial.

There are a lot of people working in software that have limited knowledge of information theory or machine learning.

Unfortunately LLMs are magic to them.

When you lack a first principles understanding of how something really works, you will bolt on hacks and duct tape on the wrong approach.

> Just use the LLM to make a thing that's actually good at what you need it to do.

This is the end state of most imagined use cases for “agentic” software IMO. Use the AI to build software, with a classifier or LLM as the expansion joint in awkward spots.

I do agree with you, but I do think you're also making the classic "curse of knowledge" mistake.

You have to know the right words to ask the LLM to make and train a classifier. You and I already know the words. But they are not actually that trivial to learn for an average joe, and people are lazy.

It's because you can "program" it in casual English. No code, no training runs.

Prompted LLM classifiers are good for when training data is expensive and/or you don’t expect a high inference volume. It’s basically choosing lower model development costs in exchange for higher inference costs, which can be good for TCO, time to market, or both. Heck, I’ve worked on some applications that weren’t even feasible before LLM-prompted classification because of the cost to acquire enough data to train a supervised model.

Cost/benefit tips in favor of supervised machine learning when training data is cheap and/or plentiful, or when you expect to be able to amortize model development costs across a whole lot of inference.

The LLM is like the swiss army knife in these situations. It can KINDA do it all, but not very well in a lot of cases. Better to have specialized tools in most cases. And you're right, the LLM is pretty good for figuring out or building out the right tool for the job.

This is like saying we don’t need Dropbox because we have FTP servers.

People vastly underrate the patience of people to learn and setup a classifier. Jev is one API call. Training a good classifier is hard.

People like me who are excited about Jev-style models aren't going to use them to diagnose cancer. We are going to use them to make fairly straightforward decisions very quickly (300 ms or less) and very cheaply ($0.042 per M input tokens - output tokens aren't billed). Even the lower end frontier models will take much longer and be much more expensive, so these decision models are perfect for certain things.

What we've been trying Jev for is interpreting things like short product descriptions provided by customers, and determine which of the natural language categories fits best.

The list of categories are provided by the government and can change several times a year, and the product descriptions are almost random (think the various titles of AliExpress listings of essentially the same product).

What kind of dedicated classifier would be better? Genuinely curious.

Training a model requires labels training data. Getting up and running with an LLM just requires a prompt.

I agree that a good model will outperform an LLM, but the LLM is so much faster/easier to build. That’s the usual tradeoff.

A classifier requires a ton of labeled data to be processed for it to be good. Jev and likes are zero shot classifiers where none of the data is required. Imo both are useful for different tasks. I am not going to use jev based models for compliance or any tasks where being wrong can bankrupt my business. For something trivial like sentiment analysis it’s a god sent.

> For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture

Agreed, and that’s the premise behind Jeffy, open source, bundles pretrained task-specific models that run (and train) locally on CPU: https://jeffyclassify.com/

At the same time, I’ve been working with some people implementing it, and there’s definitely value in having a more general model like Jev, or an LLM, for initial exploration and data labeling, to then use that data and train the task-specific model. It’s a whole workflow

Also, the market/pain insight of Typesafe is super interesting. They realized that a big chunk of the market was using LLMs as classifiers, and they built a solution around that (very much to your point, a task-specific model, rather than a more general LLM), and it worked! (Also they have the ability to generate a lot of attention, given the founder being ex-OpenAI and having raised quite a bit of money too), so it’s not just a technical/engineering thing, it’s a business/market/need/discoverability issue as well

There's a great bit of shortform video content floating around that shows a stammering younger guy in a welding mask:

  > Do you have two pieces of metal that need to become one? 
  > But if it became  two pieces again it wouldn't be a big safety concern?
  > If that's you call Bob's OK welding!

Classifiers can require a lot of examples to train correctly. You can take a decision model's outputs and apply platt scaling to arrive at a good decision boundary a lot faster.

Eventually, yes, it makes sense to move to a pure classifier model as it's probably a lot cheaper. But then you need to learn how to serve your own classifier model correctly. Compute costs are being driven up by demand to serve models and unless you're careful it's easy to use your GPUs less effectively than a company specializing in serving a model.

I think there is a parallel between engineering systems (specialized) and commodity cloud hardware (general purpose).

While no doubt, you can get better performance using a specialized hardware … there becomes a breaking point when the commodity / general purpose is good enough while allowing you lots more flexibility as well as typically being cheaper that is become hard to justify the specialized solution.

There's a significant engineering cost to having a custom classifier.

An off-the-shelf API has way less friction, if the results are good enough why bother?

Use LLMs to implement, process data, test and iterate statistical and NN models.

Once you have a hammer everything is a nail.

>LLMs drive the cost of making those models down to near zero.

The last place I worked and my current job have a tremendously bureaucratic process for standing up new internally trained models. You could easily spend ten times as much mental energy and effort doing the paperwork and revisiting it annually as you do in the care and feeding of the model itself.

If a canned product meets the requirement, there's a lot to be said for using it. Even if it costs a bit more per decision, the all-in costs are WAY lower.

> Just use the LLM to make a thing that's actually good at what you need it to do.

I don’t have a deep understanding of this topic, but wouldn’t you also need a good amount of relevant data to train on?

> Just use the LLM to make a thing that's actually good at what you need it to do.

Yes, this.

> For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture

There’s lots of applications where it’s not being deployed at scale. Also lots of cases where someone who knows they need something classified does not know what a classifier is.

> why people are excited about Jev-style decision models.

I think its a combination of being new to classification models and not using in anger.

LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.

Yes, of course, with time, attention, higher maintenance costs, and slower time to respond to requirements changes you can build a specialized system that performs better than an off the shelf general purpose solution. But you cost money. And you can quit and go work somewhere else, stranding the project. And the data might be garbage you can’t train on without substantial effort. And training takes time, which has opportunity costs that are often substantial. And the requirements may change tomorrow with no notice, invalidating all of your work.

I don’t want to pay someone to maintain a first rate classifier when I can buy a pretty good classifier as a service. I don’t want to pay someone to maintain and build server infrastructure when I can buy that as a service.

I feel like I’m being gaslit on HN when a solution seems obviously valuable to me because it eliminates or modifies constraints, and people dismiss that as if managing constraints wasn’t most of high level engineering work.

I tuned out of HN for a few years, came back in the middle of the AI renaissance, and I am constantly raising my eyebrows and hearing this Upton Sinclair quote in my head:

“It is difficult to get a man to understand something when his salary depends on his not understanding it.”

I have been incredibly impressed by LLMs for coding and data analysis and was expecting the LLMs to perform better so I was surprised by the results especially since this is an old model and nowhere near state of the art for skin cancer diagnosis.

Don’t those decision models mainly route to other models? My understanding is that they wouldn’t handle a request but forward it along with some kind of semantic capability.

For some types of applications, Jev is also useful as a stepping stone, you use it for the cases it's most confident so you already get gains from using it, and then collect all the usage to train your specialized model.

They make very good classifiers and are easy to fine tune for specific use cases.

I think the contention is CNN vs Transformer

it's precisely because you can tune them to do that off a strong base

and because they start off generalized, you can train them against multiple of your workloads (they'll start with some generalized performance/world knowledge about them)

Which allows you to embed proprietary processes/data (world knowledge only you may have) across multiple domains into a model cheaply

> For any application at scale it'd be way more accurate to have a dedicated classifier

"at scale" or the above use case of "specialized domain"

Jev being cheaper than an a full LLM call creates places where other tools might work better later - when the volume or the available data warrants it.

System optimal, code optimal and cost optimal aren't always aligned. Pick the right tool for the job - and dont get married to it. On occasion worse is better, and moreso when your testing or it is a one off/disposable.

I mean, it's the pareto principle, right? 80% of the applications require 20% of the intelligence. If I can drop in something without needing to create high quality training data that's good enough, that's great! Skin cancer probably isn't one of those applications.

The only real usecase I can see for jev style like models are processing long lists of items. i.e server logs or financial transactions or sensor data in robotics.

Internet discovers If statements

subnanosecond response times

model fits on CPU

[dead]