alt Hacker NewsIt's a non-instruction-tuned classifier model trained on a confidence-aware RL variety that generates its own schema and follows it, with a confidence score output. Think BERT on crack, smart enough to be used as a decision maker (conceptually). They call it "not an LLM" because it's non-generative but of course it's a language model in the same way all non-instruction-tuned classifiers are.
Replies
Isn't that the same transformer at the end of the day? It must be faster only because it generates a single token output, just one evaluation of the model. It takes the same input context and has the same O(n^2) attention blocks. It probably takes options as appended to the input and returns a probability over them instead of the whole dictionary. It's post-trained to do that specific job. If so what's the big deal?
I was a bit skeptical when read the initial pr on it, yesterday ran a test involving ~250M tokens, something we measure went from ~60% to >80% success (with almost no tuning) and at less than 50% cost the low-end LLM was running at, looking at it more seriously now ... the servers are US-only currently I understand and ZDR is by request