Here is a short example that came up for me last week.
I had a set of documents I wanted to classify according a taxonomy that is well known (so it is exists in the training data of all the major llm models I tested)
If I have prompt like, `You are an expert classification system. Using the Classification Approach Foo, consider the following and output the category in JSON format, such as {"class":"bar"} `
This works ok, but it works much better if I tell it to output {"class":"bar", "reason": "baz"} and improved with some other approaches like adding "related_class" or "parent_category" which would otherwise be redundant.
Also including some few-shot examples helped, but the biggest benefit came from the "reason" field. Trying justification or other synonyms seems to produce the same output.
I suspect this is something similar to CoT.
This has worked for me for sentiment classification, i had a score and a summary field and the results were much more accurate.
Great, thank you (and hedgehog), that makes sense
Have you tested moving the "reason" field before the "class" field? That may encourage better CoT instead of having the model justify the class after it already picked it. Anecdotally, I saw a 5% boost in performance from a NER system from having the model output the entity's class at the end rather than the beginning.