In my own words - you can't calibrate confidence of a model without a calibration set. Confidence is relative to some labeled dataset, it is established on a set of buckets or intervals, and is population based, something you can't check for a single prediction. Assign per case, verified per population.