Machine Learning
What is model drift?
Model drift is the gradual decay of a deployed model's accuracy as the real world moves away from the data it was trained on.
Two kinds worth telling apart
Data drift is when the inputs change - a new acquisition channel brings in players who behave differently from anyone in the training set. Concept drift is when the relationship itself changes - the same behaviour now means something different because you changed the bonus structure or the product. Data drift is detectable from inputs alone. Concept drift needs outcomes, so you find it later.
How to detect it before it costs you
Monitor the distribution of each input feature against the training distribution, and monitor the distribution of the model's own output scores. A sudden shift in score distribution with no corresponding business event is the earliest cheap warning. Outcome-based monitoring is the ground truth, but it arrives only after the prediction window closes.
Retraining cadence
Scheduled retraining on a fixed cadence is simpler to operate and usually good enough - monthly or quarterly for most retention models. Trigger-based retraining on a drift threshold sounds better and creates more operational surface area. Pick scheduled unless you have a specific reason not to.
The handover version of this
If you buy a model from an agency, the retraining runbook is the deliverable that determines whether the system is still working a year later. Feature pipeline code, training script, evaluation procedure, calibration step, and the cadence - documented well enough that your own team can run it without the agency. Ask for it before signing, not at handover.
We build automation systems for small teams. Free audit call to map your specific workflows - no pitch, just a plan.