Machine Learning
What is model calibration?
Calibration is the property that a model's predicted probabilities mean what they say - among all cases scored at 0.7, roughly 70% should actually occur.
Why ranking is not enough
A model can rank customers perfectly and still output probabilities that are badly wrong in absolute terms. If you only ever take the top N from a list, ranking is all you need. The moment you attach money to the number - expected value, budget allocation, a threshold rule - you need calibration, because an uncalibrated 0.8 might be a real 0.3.
Why gradient boosting needs it
Boosted trees optimised for log loss are often reasonably calibrated, but boosted trees tuned for ranking, or trained on heavily imbalanced data with resampling, usually are not. Resampling in particular shifts the base rate and inflates every predicted probability.
How it is done
Fit a small correction model on a held-out set that maps raw scores to observed frequencies - Platt scaling for a logistic correction, isotonic regression when you have enough data and a non-monotonic distortion. It is a cheap post-processing step, and it must be refit whenever the main model is retrained.
How to check it
Bin the predictions, plot predicted probability against observed frequency per bin, and look for the diagonal. Systematic deviation above or below the line tells you the direction of the bias. This one chart is worth more than another decimal place of AUC in any conversation with a business stakeholder.
We build automation systems for small teams. Free audit call to map your specific workflows - no pitch, just a plan.