Before anyone trains a churn model, two teams need to agree on what churn is. Sales usually counts an account as lost when the account manager stops getting replies, while finance counts it when the last invoice is ninety days old. A model trained on one definition will look wrong to the other team, and a score that one team distrusts becomes a list nobody works.
Write down what churn means
Take a SaaS company in Pune that sells billing software to pharmacies and small distributors, with account managers for the bigger accounts and self-serve plans for the rest. Ask three teams when an account has churned and you get three answers.
| Team | Counts an account as churned when | What it misses |
|---|---|---|
| Sales | The account manager marks it lost | Self-serve accounts nobody manages |
| Finance | No paid invoice for ninety days | Accounts on annual plans, paid up but unused |
| Product | No login for thirty days | Owners who only log in at month end for GST |
Each is reasonable for its own purpose. For a model, pick one, write the definition down, and put it in the column name, so that churned_no_paid_invoice_90d cannot be mistaken for something else in six months. Then define the prediction window just as exactly. "Will meet that definition within the next sixty days" is a target. "At risk" is not.
The definition also decides which data you need. The finance version needs invoice and payment history, reconciled against the payment gateway, because an invoice marked unpaid in the app and paid in the gateway is the column everyone assumes is clean.
Setting the threshold from team capacity
The score orders accounts by risk. The useful question is how far down the list the team can act, rather than whether an account sits above some natural cut-off. If the two account managers between them can make thirty calls a week, the threshold is whatever score gives roughly thirty managed accounts. Set the line from capacity and move it as capacity changes, including the week one of them is on leave for Diwali.
Two lines work better than one. A contact line, above which an action is created, and a watch line, above which the account is marked but nobody is interrupted. Every flagged account also needs the two or three signals that pushed its score up, shown next to it. Examples are invoices generated in the app down by half over the past month, a UPI autopay that failed twice, or a support ticket open for two weeks. An alert without a reason becomes a cold call, and the person receiving it stops trusting the list within a fortnight. How to present those reasons is its own design problem, covered in explaining a risk score to the person who has to act on it.
Routing each reason to an owner
Route the alert to the person who can act on the reason, in the tool they already work in, with one owner per alert and an expiry. Each reason maps to a specific action, so the person receiving it knows what to do in the first minute.
| Reason | Owner | Where it appears | Action |
|---|---|---|---|
| Usage drop, managed account | Account manager | CRM task | Check-in call that opens with what changed |
| Failed payment or overdue invoice | Billing team | Billing queue | Fix the mandate or card, confirm it went through |
| Ticket past its target | Support lead | Helpdesk, ticket linked | Close the ticket, then call |
| Low usage, self-serve plan | Nobody | Automated WhatsApp message | Short guide to the unused feature |
A failed payment goes to billing rather than sales, because a salesperson calling about money is awkward for everyone. For a contract ending with low usage, the renewal conversation is built around the features the customer never adopted, which for a pharmacy is often the GST report they still prepare in Excel.
We would resist making a discount the default action. A discount given to an account that would have stayed anyway is money lost, and churn scores are never precise enough to tell those accounts apart from the ones that needed it. Keep discounts as a late option that needs a reason of its own and a second approval.
An alert nobody acted on within two weeks is closed as "not actioned" and recorded that way. That record is itself a useful fact about capacity.
- 01Scoreevery account, with reasons
- 02Thresholdset from team capacity
- 03Routeby reason, to the owner's tool
- 04Act or hold backa random share gets no action
- 05Record outcomeaction, date, did it stay
Outcomes retrain the model
The last step is the one that makes the first step improvable.
Measuring whether the calls worked
From the first run, hold back a random share of accounts above the contact line and take no action on them. After sixty or ninety days, compare retention between the accounts that were acted on and the ones held back.
The holdout answers two separate questions. Did the model rank correctly, meaning held-back accounts with high scores churned more often than accounts below the line. Did the action work, meaning acted-on accounts stayed more often than held-back ones with similar scores. Without the holdout, the retention team takes credit for the model's ranking and the model takes credit for the team's calls, and neither claim can be checked in the quarterly review.
- Retention after the action
- Which actions worked for which reasons
- Cost per account kept
- Whether the ranking was right
- The baseline the action is measured against
- Clean labels for retraining
- Accounts the model missed
- Where the threshold should move
Record everything in one table with one row per flagged account per run. The columns are the account, the score, the reasons, whether it was held back, the action taken, who took it, when, and whether the account met the churn definition at the end of the window. That table is the evidence for the programme and the training data for the next model, and it is far easier to set up on day one than to reconstruct from CRM notes later.
Retraining without erasing the signal
The action creates a trap here. If the calls work, the training data now shows high-risk accounts that stayed, and a model retrained on it learns that those signals were never risky. The signals get downweighted, the next version misses the same accounts, and the programme looks like it stopped working when it was working well.
There are two ways round it. Train on the held-back accounts, whose outcomes no action influenced. Or include the action taken as an input, so the model learns the effect of the call separately from the underlying risk. Either way, retrain on a schedule from the recorded outcomes, and keep a plain rule such as "no login in thirty days" running alongside as the baseline the model has to beat. If the rule does almost as well, the case for keeping rules first and models second applies.
How churn programmes stall
The threshold is left at the model's default cut-off, which produces a list three times longer than anyone can work, so the account managers take the top of it and ignore the rest. Scores appear on a dashboard instead of as tasks in the CRM, so nobody sees them during the working day. Every flagged account is offered a discount, because that is the easiest action to script.
Then the data failures. Sales and finance each build a report on their own definition of churn and argue about whose number is right. There is no holdout, so a year in nobody can say whether the programme does anything. Outcomes are recorded in people's heads, and the model is retrained on stale labels. And a score arrives with no reasons, so the call starts with "our system flagged you" and the customer asks what system.
Getting started
- Write down the churn definition and the prediction window, and get sales and finance to sign off on the same sentence.
- Agree the team's weekly capacity and set the contact line from it.
- Write the reason-to-action table with an owner and a tool for each row.
- Deliver alerts as tasks in the owner's tool, with reasons and an expiry.
- Hold back a random share of flagged accounts from the first run.
- Record every action and outcome in one table and review it quarterly.
If you are weighing where a churn model fits among other predictive analytics and ML work, start with step one; it costs a meeting. Set the holdout before the first alert goes out, though. It is the one part of the programme that cannot be added later, because by then the accounts have already been called.
