The April model went live because it was newer. By May nobody could say whether it was better than March, and March was gone. That is the usual way retraining goes wrong, and it has little to do with whether you retrain monthly or on an alert. The trigger matters less than the rule that decides which version serves traffic.
Two ways to decide a retrain is due
Picture a demand forecast for a distributor in Indore that has been live for six months. The monitoring is up. Someone from operations asks when the model gets retrained. One engineer wants a monthly job, because a monthly cron is easy to schedule and easy to explain. Another wants to wait for the drift alert.
A schedule retrains at a fixed interval, whatever the monitoring says. A signal retrains when a number you already measure crosses a line you set earlier. Neither is right in general. It depends on how the data changes, and on how much monitoring you actually have running, as opposed to planned.
- Predictable cost and timing
- Fits data that changes at a known pace
- Can retrain for no reason, or miss the week that mattered
- Runs as a plain pipeline job
- Retrains when the monitoring says the model moved
- Needs drift and accuracy checks already in place
- Can go months without a retrain, which is fine
- Can fire repeatedly during an upstream incident
My rule is plain. If the data moves at a known rhythm and the outcomes arrive on the same rhythm, use a schedule. Weekly secondary sales for the distributor are like that: the week closes, dispatch figures come in from the depots, and the model can learn from a complete week. If change is irregular, a new competitor in the territory, a price revision, a new district added to the network, use a signal, provided you have the monitoring to detect it.
Most setups end up with both. A long schedule, quarterly perhaps, is the floor. Signals sit on top to catch the months the schedule would have slept through.
Signals that justify a training run
Not every alert should start a retrain. The ones that justify it all mean the same thing: the training data no longer describes the traffic. If you have not set up that monitoring yet, start with the minimum monitoring for a model in production and come back.
- Input drift on a feature the model leans on heavily, sustained for more than a day. One afternoon of odd data is usually a failed upstream job.
- Weekly labelled accuracy below the approval level for two checks in a row. One bad week on a small sample proves little.
- A known upstream change. If the distributor starts carrying a range of packaged snacks it never sold before, the model has never seen them, and nothing in the monitoring needs to confirm it.
- Fresh labels for the period that drifted. This one gets its own section.
Check the labels before you train
Data freshness causes the most common wasted retrain.
Take a model that predicts whether a retailer will pay the distributor's invoice within ninety days. The outcomes for August invoices do not exist until November. A retrain triggered in September, because August looked different, learns nothing about August. It trains on the same labelled rows as last time, with a new timestamp on the artefact.
Before any run, confirm that the training window covers the period that drifted and that its labels have arrived. If they have not, wait. Have the trigger check the label date before it checks anything else.
The candidate has to beat the current model
A new version is a candidate until it has beaten the current one on the same test, and the test is set up before the run, not after.
- 01Triggerschedule or signal
- 02Fresh datawindow and labels checked
- 03Candidate trained
- 04Validatedagainst current, same holdout
- 05Promotedprevious version kept
Roll back to previous if live results disagree
A candidate that fails validation stays a candidate; nothing changes in production.
Three checks cover most cases. Score the candidate and the current model on the same holdout set, refreshed with recent labelled data, and require the candidate to win by a margin agreed in advance. Then look at the scores by segment. A candidate that is better on average and worse for small retailers, or for accounts under two lakh, is not ready, whatever the overall number says. Finally, rerun the regression set: the specific cases behind past incidents, each of which must come out the same or better.
Write the promotion rule down as numbers before the first retrain. A rule written after seeing the candidate's results tends to become whatever the candidate passes. Keep it next to the pipeline config, where the job can read it:
promotion:
holdout: holdout/rolling-last-12-weeks
metric: weighted_mape
must_beat_current_by: 0.01
segments: [region, retailer_size]
max_segment_regression: 0.0
regression_set: tests/known-incidents.csv
sign_off: ops-planning-lead
When the candidate passes offline, it still has to prove itself on live traffic. A shadow run followed by a canary is how we would do that without risking the whole forecast.
Keep the previous version reachable
Promotion replaces the old model on the serving path, not in storage. Keep the previous artefact, a snapshot or pointer for its training data, its feature definitions and its evaluation report. Switching back should be a configuration change, not a rebuild.
Keep it for at least as long as outcomes take to arrive. Validation only uses the past. The first real proof that the new version is better comes when the outcomes for its first period are in, which for the invoice model above is three months. Until then the previous version is the fallback, and a fallback that has to be retrained from an older window at ten at night is not a fallback.
What a retrain costs beyond compute
Compute is the visible cost and usually the smallest. The others never appear on the cloud bill, so they get left out of the decision.
Somebody has to label, where outcomes are not attached automatically. Somebody has to read the validation report and sign off, and if that person is the finance head, their Friday afternoon is not free. Downstream rules need checking too. A cutoff tuned to the old model's scores means something different after a retrain, so the rule that says "send a reminder to every retailer above 0.7" has to be looked at again. And every promotion is a chance to make things worse, with the damage showing up in outcomes weeks later.
That is the case against very short schedules. A weekly retrain with no signal behind it pays all of these costs every week, for a model that may not have moved.
Where retraining pipelines fail
The usual problems are not technical. A schedule retrains and promotes automatically with no comparison against the current model. The holdout set was built at launch and never refreshed, so every candidate looks good on old data. No previous version was kept, so a rollback means a hurried retrain while the business waits.
Two more are worth watching for. A drift signal fires during an upstream outage, the pipeline retrains on broken data, and the broken model is promoted because it scored well against an equally broken holdout. And thresholds in downstream systems are never revisited after a promotion, so a model that genuinely improved produces a worse reminder list on Monday.
Before the first retrain, write the promotion rule: what the candidate must beat, on which data, by how much, and who signs off. Then check that the previous version can be switched back in one change. If you are setting up this kind of MLOps pipeline for the first time, it sits within the AI solutions work we take on.
