A holdout set tells you a candidate model was better on last quarter's data. It cannot tell you what happens when this morning's requests arrive late, half-filled and retried twice. The only place to learn that is production, which is also where a wrong prediction costs money. Shadow runs and canaries let a new model meet live traffic in stages, so you find out without betting the whole system on it.
Where offline evaluation runs out
Take an insurance broker with a model that ranks policies by how likely they are to lapse, so the renewals team calls the riskiest ones first. Version two beats version one on the holdout. The operations head wants to know how it behaves on this month's renewals before it decides who gets the first call.
The holdout is a sample of the past, cleaned with care. Live traffic is neither. Feature values arrive late or missing because the nightly export from the policy system ran late, or a branch uploaded its file at 11 p.m. The calling application retries, times out and sends edge cases nobody collected. A candidate that wins offline can lose live for reasons that have nothing to do with the model and everything to do with the path the data takes to reach it.
A shadow run sees everything, decides nothing
In a shadow run each request goes to both models. The current model's answer goes back to the caller as before. The candidate's answer is written to a log against the same request id and goes nowhere else. Nothing changes for the renewals team or the customer.
You learn how often the two models agree, where they disagree by segment, and how the candidate's latency and error rate hold up under real load and the real feature pipeline. Most feature pipeline mismatches show up here, as a cluster of disagreements that trace back to one field. It is usually something dull, like a date column arriving as DD-MM-YYYY from one branch system and YYYY-MM-DD from another.
You cannot learn whether the candidate's decisions lead to better outcomes, because they were never acted on. A policy the candidate wanted called first was not called first.
Run the shadow until you have enough disagreements to review by hand. A few hundred is usually plenty. Do not run it for a round number of days. It costs a second inference on every request, which is why a shadow run needs a stopping rule written down before it starts.
A canary decides for a small share
A canary lets the candidate answer for a small share of traffic while the current model answers the rest. Start with a share small enough that a bad model cannot do much harm, and size it by how many decisions you need to judge, not by a number that looks tidy on a slide.
Assign by a stable key, such as a hash of the policy number, not per request. The same policyholder then gets one model consistently, and each outcome can be traced to the model that produced it.
- Candidate sees all traffic, decides nothing
- Learns agreement, latency, pipeline mismatches
- No outcome data, because nothing was acted on
- Doubles inference cost while it runs
- Candidate decides for a small, stable share
- Learns real outcomes for that share
- Needs a rollback switch ready first
- Only fair if compared by segment and period
Compare against the current model on the same period, the same traffic mix and the same metric you used in validation. A canary that happens to get mostly motor policies, or mostly customers from one state, will look better or worse for reasons unrelated to the model. Compare by segment, and only step the share up once a full outcome window has passed for the current share. For the renewal model that window is the renewal due date. For a fraud model it might be the chargeback period, which runs to months.
Rollback is a switch, not a deploy
Before the canary starts, the route back to the current model has to be a configuration change that takes effect at once, with the current model still loaded and warm. A rollback that needs a build, a deploy and someone awake at 2 a.m. is an hour of a bad model making decisions.
The router reads its split from config. Rolling back means setting the candidate's share to zero:
model_router:
key: policy_number_hash
current: renewal-rank-v1
candidate: renewal-rank-v2
candidate_share: 0.05
shadow: false
auto_rollback:
error_rate_above: 0.02
p99_latency_ms_above: 800
- 01Request
- 02Routersplit by stable key
- 03Current or candidateone answers, the other may shadow
- 04Decision logboth answers, which was served
- 05Compareby segment, against current
Rollback: set the split back to zero
The router reads its split from configuration, so the rollback is the same mechanism in reverse.
Automate the rollback for service signals. Error rate and slow-tail latency go bad within minutes and need no judgement. Leave quality signals to a person, because a change in outcomes needs a look before anyone acts, and that person should know what a normal week looks like. The monitoring behind those numbers is the same set described in the minimum monitoring for a model in production.
The decision log does the real work
Shadow runs and canaries pay off in the log, long after they end. For every request, record the request id, the model version, the feature values used, the answer from each model that ran, which answer was served, the assignment key and the time.
When outcomes arrive, join them to this log and the comparison finishes itself. When outcomes do not arrive on their own, sample the disagreements and have a person label them. The disagreements carry the information; cases where both models agree tell you little.
Set a retention period. The log holds the same customer data as the requests themselves, and whatever retention rule applies to the source data applies here too.
Mistakes that turn a canary into a guess
- The shadow run logs the candidate's answers without the request id, so nothing can be compared.
- Canary assignment per request, not per customer, so one policyholder flips between models and the outcome belongs to nobody.
- Promotion after the service metrics look fine, before a single outcome has arrived.
- The candidate in shadow reads a slightly different feature pipeline from production, so the disagreements measure the pipeline, not the model.
- No stopping rule, so the shadow run drifts on for weeks.
- A rollback that was never practised and turns out, on the night, to need a deploy.
The order to do it in
- Build the decision log with request id and model version. Confirm it joins to outcomes.
- Run a shadow until a set number of disagreements has been reviewed. Fix the pipeline mismatches.
- Set up the routing switch and automatic rollback on service signals. Test the rollback once.
- Start the canary on a small share, by stable key.
- Wait one outcome window, compare by segment, then step the share up.
- Promote, and keep the previous version loaded until the next outcome window has passed.
Build the decision log first, every time. Every later step reads from it, and a canary without one produces opinions where there should have been a comparison. If the candidate came out of a scheduled or triggered retrain, the promotion rule from retraining on a schedule versus on a signal decides what the canary has to show. Both are part of the MLOps work listed under AI solutions.
