alt Hacker NewsI took the point as: don't make the LLM the classifier. Use it to turn messy input into useful features, then let a normal model make the actual decision. That gives you thresholds/calibration you can inspect.
What I'm not sure about is how stable those features are when you switch the underlying LLM or model version.
Much like how you shouldn't ask the LLM to solve a (repeated, logical) problem, but you should instead prompt it to generate code that you can inspect/test/fix/reuse.