A credit analyst at an NBFC branch in Nagpur has a loan file open on one side of her desk and the screen on the other. The screen shows a risk score of 82, from a model built by a team in Pune she has never met. If the screen stops at the number, she either ignores it or signs off on it without thinking, and both defeat the point of having a trained person look at the file. The score needs reasons, in the words she already uses, and a way to disagree that leaves a record.
A bare number gets ignored or obeyed
A score with no reasons fails in two ways, and both often show up in the same branch. The analyst already has a view of the case from the bank statement and the field visit report, and the number does not change it, so the model may as well not exist. Or she approves whatever the model says, because questioning it takes effort and leaves her exposed if the account goes bad.
Reasons give her something to check against the file in front of her. A line such as "two missed EMIs in the last six months" can be verified in a minute against the repayment history. The number 82 cannot. That check is what makes the decision hers rather than the model's.
From model output to sentences she would say
Most scoring models can report how much each input, or feature, pushed one particular prediction up or down. Tree-based models have built-in methods for this, and general methods such as SHAP values cover most other models. The raw output is a list of feature names with signed numbers, and nobody acts on a line that reads days_since_last_payment: +0.14.
The design work is a mapping from features to sentences. For each feature, or group of related features, someone who actually does the job writes a template in the branch's own language, with its direction. "Last EMI was {n} days ago" raises the score. "Customer for {n} years" lowers it. Five address fields that move together become one reason, "Address changed recently".
- Screenscore, band, three or four reasons
- Reason templatesfeature group to sentence
- Feature contributionsper case, signed
- Modelscores the case
Only the top layer reaches the screen; the templates do the translating.
The screen then shows the top three or four reasons, ordered by how much they moved the score, each marked as raising or lowering it. If a reason that lowers the score is among the top contributions, show it. She needs to see that the model weighed both sides; otherwise it reads like a prosecutor's summary.
Reasons that behave the same every time
Staff learn to read explanations only if the explanations behave. The same feature should always produce the same wording. The same case scored twice should show the same reasons, which means a deterministic explanation method with fixed settings. Two cases that look alike should show similar reasons. If they do not, people stop reading the panel within a week, and the eye goes straight to the number again.
The reasons also have to agree with the score. When the top reasons all lower risk and the score is still high, she is right to be confused. Usually the cause is a long tail of small contributions, and the honest display is a reason that says so, such as "Several smaller factors", rather than a top-three list that contradicts the number. Version the templates with the model, because a retrained model can shift what a feature means, and last quarter's sentence may no longer be true.
What stays off the screen
Raw feature names and contribution values stay off; they are for the engineer. So does the full list of every feature, because it buries the three that matter. A probability to two decimal places stays off, since it suggests a precision the model does not have. Any reason the analyst cannot verify or act on stays off as well.
Protected characteristics, and features that stand in for them, are a different matter. Suppose the model has learned that a particular pin code or a surname pattern predicts default. If those carry weight in the model, that is a model problem to fix, and leaving them off the screen does not make the decision any fairer. Remove them from the model and retrain.
The override control, and the record it leaves
When she disagrees with the score, she needs a control to record a different decision, a reason chosen from a short fixed list with room for free text, and no penalty for using it. Put it on the same screen as the score, within reach, and keep it to one tap and one choice. An override that needs the branch manager's sign-off is an override nobody uses, which makes the model unchallengeable in practice. The same thinking applies to any system that hands a decision to a person, as in our post on designing the handoff from an agent to a person.
- 01Score and reasons
- 02Analyst decides
- 03Overridereason required
- 04Decision recordmodel version, reasons, who
Weekly review of overrides by reason
The record holds what she saw as well as what she chose.
Record every decision, not only the overrides: case ID, model version, the score, the reasons as they were displayed, the decision, the override reason if any, who decided and when. That record does three jobs. It is the audit trail an internal auditor will ask for sooner or later. It shows where the explanation failed, because a cluster of overrides on one reason means either the model or the template is wrong. And it is the best labelled data you will collect, since disagreements between the model and an experienced analyst are the cases most worth learning from.
Where explanation screens go wrong
The explainer library's output gets pasted straight onto the screen, feature names and all. Reason wording changes between releases, so staff stop trusting it. There is no override control, so the branch works around the system over email and WhatsApp and nothing is recorded. The explanation gets presented as the business reason for the decision, when it is only the model's reason; the business reason is still the analyst's to give. And nobody checks whether decisions got better once staff could see the reasons. A score that never changes what anyone does is the same problem we describe in moving churn prediction from a score to an action.
Check it with five analysts first
Before a score goes on a screen, confirm each of these:
- Each reason template was written with, and reviewed by, someone who does the job.
- The screen shows the top three or four reasons, marked as raising or lowering the score.
- The same case shows the same reasons every time.
- Protected characteristics and their proxies are absent from the model, rather than hidden on the screen.
- An override control exists, asks for a reason, and is recorded with the model version.
- Someone reads the overrides, grouped by reason, every week.
Then sit with five analysts, give each one ten real files with the new screen, and ask them to think aloud. Watch which reasons they check against the file and which they skip. We start that work by writing the reason templates with two people who do the job, before any screen is drawn, because everything else depends on those sentences being ones they would say themselves. Our AI solutions page covers the predictive analytics work this screen sits on top of.
