Rules First, Models Second: When a Trained Model Earns Its Place

Say an NBFC wants to rank the loan enquiries coming in from its branch network, and the first idea on the table is to train a model. Most tasks like this start better as a short list of rules written by the people who already make the call by hand. A trained model earns its place later, when the rules stop keeping up, and you can only tell that has happened if you measured the rules first.

What a rule set does well

A rule is a condition someone can read. Enquiries from firms with more than fifty staff rank above the rest. A cash-on-delivery order above two thousand rupees from a new pin code gets a confirmation call before it ships. A ticket that mentions "GST invoice" goes to the accounts queue. Rules are cheap to write, they explain themselves, and when one is wrong you can point at the line and fix it the same afternoon.

Rules also cover the cases that must never vary. A blocked country, a customer on a sanctions list, a product that cannot ship to a particular state. Those stay rules whatever else you build, because a model gives a probability and a compliance check needs a yes or a no.

The estimator BizHues built for Nebula, a digital accessibility specialist, is a rules engine of this kind. Advisors answer scoping questions about property type, page and template counts, frameworks and the WCAG level required. Rules turn those answers into effort by role, apply pricing, buffers and rush factors, and produce three priced options. The same answers always give the same quote, and the operations team edits the rates and multipliers themselves. Nothing is learned from data, and everything is written down.

Start with the data you already have

Before choosing between rules and a model, look at the records. We ask three questions in the first conversation about any ranking or flagging task.

First, what inputs exist for each case at the moment the decision is made? For a loan enquiry, that means the form answers, the source channel and whatever the bureau pull returned, not anything the branch learned in the meeting afterwards.

Second, what outcome was recorded? Did the enquiry become a disbursed loan, was the order refused at the door, which queue did the agent finally close the ticket in? That column is what a model learns from and what rules get scored against.

Third, how many cases are there, over what period? A few hundred records with outcomes are enough to score a rule set. A model that generalises usually needs more, and needs them to cover the variety you will see in production, including the Diwali month that looks nothing like March.

If no outcome is recorded, start recording it now. Without it, neither rules nor a model can be evaluated, and the argument about which is better stays a matter of opinion.

Score the rules before anything else

Write the rules in code with tests, run them over past records the rules were not written from, and compare the result with the recorded outcomes. For ranking, check how many of the enquiries the rules put at the top actually converted. For flags, count the orders flagged that were fine and the bad ones that slipped through. Keep this scoring script. It becomes the baseline any model has to beat, and it gets reused unchanged when you try one.

1. 01Records with outcomes
2. 02Rules in codewritten with the experts
3. 03Score on held-out records
4. 04Sort the missesby cause
5. 05Add a rule, or train

Repeat on each change

The scoring step is reused unchanged when a model is tried later.

Then read the misses and group them by what went wrong. Take a furniture retailer that sells online and wants to flag cash-on-delivery orders likely to be refused. If most of the misses share a cause a person can describe, such as refused orders all addressed to the same few hostels, add a rule and rerun. If the misses have no pattern anyone can put into words, that is the first sign a model may help.

Signs the rules are no longer enough

The usual signs show up in the maintenance, not on the accuracy chart. The rule list keeps growing and nobody can predict what adding one will do to the others. Rules have exceptions to exceptions, so the order they run in changes the result. The one person who understands the whole set has become the bottleneck for every change, and that person is on leave for Navratri.

Two more are about the data itself. The useful signal has moved into free text, images or long sequences of behaviour, which thresholds cannot capture. Or the business is changing faster than the rules are maintained, with new products, new cities, or a festive season that reshapes order sizes for six weeks. Accuracy against the recorded outcomes stops improving even though rules are still being edited every week.

What you see in the misses	Keep adding rules	Train a model
Cause of the misses	A person can state it	Spread across many weak signals
Kind of input	A few structured fields decide it	Text, images or long histories
Explanation needed	Line by line, for audit	A reason per score is enough
Recorded outcomes	Few, or only recently collected	Enough to train and hold back a test set

Before taking the right-hand column, count what it costs. A model needs labelled outcomes, a held-out test set, a retraining routine for when the data drifts, monitoring to notice that it has drifted, and a way to explain a score to the branch manager who acts on it. It can be wrong in ways nobody wrote down, which is the price of finding patterns nobody wrote down. Put those costs into the comparison before training, not after.

Where rules-versus-model decisions go wrong

The one that hurts most is training a model with no baseline, so nobody can say whether it beats the rules it replaced, and the argument runs for months. Next is scoring rules or a model on the same records that were used to write or train them, which flatters both; hold some records back. Then leakage, meaning inputs only known after the outcome, such as a disbursal date when ranking open enquiries. The score looks excellent in testing and the model is useless the day it goes live.

Two more are worth checking in any review. Replacing every rule at once, including the compliance checks that should have stayed as rules in front of the model. And setting a threshold on the model's score once at launch and never looking at it again as volumes change. A model nobody watches drifts, and the minimum monitoring for a model in production is far less work than finding out from the branch managers.

A decision rule you can apply

1. Find or create the outcome column.
2. Write the rules with the people who make the decision today, in code, with tests.
3. Score the rules on records held back from the writing.
4. Sort the misses by cause.
5. If a cause can be written as a rule, add it and return to step 3.
6. If the misses have no stated cause, train a model on the same records and score it on the same held-out set with the same script.
7. Ship the model only if it clearly beats the rules on records it has not seen, and keep the hard rules running in front of it.

The first step is the outcome column, because everything after it depends on knowing what actually happened. It is usually sitting in Tally or the CRM already, waiting for someone to join it to the inputs. If you are weighing a model for a task like this, bring that joined sheet and the current rules, even if they live only in someone's head, to the first conversation about AI solutions. It answers most of the early questions on its own.