The order history at a kirana supply app records what each shop bought, and the product head reasonably assumes that is enough to build a "you may also need" row on the order screen. It is about half of what a recommendation model needs. The other half is a record of what each shopkeeper was shown and passed over, and most apps have never logged it, which only becomes obvious after the first model is trained.
Define the events before you log them
A recommendation model learns from the pattern of actions across many users and items. Explicit signals such as star ratings are rare and skewed towards the people who feel strongly. Implicit signals are plentiful: views, taps, add to cart, order, skip, return. Before logging any of them, write down what each event means, because the definitions decide what the model can learn.
By "impression" we mean the item was rendered in the visible part of the row on the shopkeeper's screen, not merely returned by the API. By "order" we mean the order was confirmed, not placed and then cancelled the same day. Each event carries a user or session id, an item id, the action, a timestamp, and the context it happened in: which screen, which slot, which position in the list, and what put the item there. A few sample rows show why each field matters.
| Session | Shop | Item | Event | Slot | Position | Source |
|---|---|---|---|---|---|---|
| s-481 | (not logged in) | Atta 10 kg | impression | also_need | 1 | rules |
| s-481 | (not logged in) | Atta 10 kg | tap | also_need | 1 | rules |
| s-481 | shop-2207 | Atta 20 kg | impression | also_need | 6 | rules |
| s-481 | shop-2207 | Mustard oil 1 L | order | search | 2 | search |
Position is the field most often missing. An item tapped because it sat first in the row and an item tapped because the shopkeeper wanted it look identical without it. Impressions are the other gap. If only taps are logged, the model never learns what people passed over, and a 20 kg atta pack that nobody taps in position six is a different fact from one that was never shown.
- Impressionswhat was shown, where, to whom, when
- Interactionstap, cart, order, skip, with position
- Item attributescategory, text, price, availability
- User attributesoptional; often less useful than history
Impressions sit at the top because they are the layer most products never logged.
An impression log does not need to be elaborate. Something like this, written to the warehouse in batches, is enough to start.
CREATE TABLE impressions (
shown_at TIMESTAMP NOT NULL,
session_id TEXT NOT NULL,
shop_id TEXT, -- null until login, stitched later
item_id TEXT NOT NULL,
screen TEXT NOT NULL,
slot TEXT NOT NULL,
position SMALLINT NOT NULL,
source TEXT NOT NULL -- 'model', 'rules', 'random'
);
Identifiers must be stable. A catalogue re-import that assigns new item ids erases the history of every item, so keep one source of truth for the item id and map supplier codes onto it. A shopkeeper who browses before the OTP login and then logs in is two users unless the sessions are stitched together, as in the sample rows above, and on a kirana app that describes most first sessions.
Shops and items with no history
A model that learns from interactions has nothing to say about a shop that just signed up or an item added this morning. Neither is an edge case. On most days new visitors are a large share of the app's traffic, and the newest items, such as the Diwali gift packs listed in September, are the ones the business most wants to move.
For new shops, fall back to something that needs no history: the most ordered items in the category they landed on, items ordered together with the one they are looking at, or a short onboarding question about what kind of shop they run. For new items, use their attributes, such as category, description text and price band, to place them near items with history, and reserve a slot that shows new items on purpose so they collect the interactions they need.
A sparse catalogue, where most items have a handful of interactions and a few have thousands, pushes the model towards recommending the few. That is popularity bias. It has to be measured, because it looks like good performance right up until the long tail stops selling.
Rules come first and stay in service
Rules are the baseline, the fallback and the safety net. "Most ordered in this category this month", "ordered together with this item", and "same brand, in stock for this pin code" can be computed with counts, explained to anyone in the company and shipped in a week. The model has to beat them in evaluation before it earns its place, which is the argument we make at more length in rules first, models second.
The rules stay in service afterwards. They cover new shops, items the model has no confidence about, and the day the model service is down and the row must not be empty.
A live model trains on its own choices
Once a model is live, its own recommendations are the items most people see, so they collect the most taps, so the next training run learns they were the right choice. Left alone, the model narrows towards whatever it recommended first.
- 01Logsimpressions and interactions
- 02Trainon the logs
- 03Recommendthe model's picks fill the row
- 04Log againmark which items the model placed, and where
Each round trains on the previous round's choices
The loop cannot be avoided; logging the source of each item is what makes it correctable.
Three habits keep the loop in check. Record, for every impression, whether the item was placed by the model, the rules or a random ordering, which is what the source column is for. Keep a small share of traffic on the rules or on a random ordering, so there is always data the model did not shape. And when training, weight interactions by how much exposure the item had, so an item tapped once in a thousand impressions does not look like an item tapped once in ten.
Testing on a time split before launch
An offline evaluation replays history. Train on interactions up to a date, then ask whether the model would have ranked the items each shop went on to order after that date. The split must be by time. A random split lets the model see the future and makes every result look better than it will be live. The festive season makes it worse, because a random split trains on Diwali week and tests on Diwali week.
Compare the model with the rules on the same holdout and look at three measures together.
| Measure | What it tells you |
|---|---|
| Held-out orders in the top few | Whether the ranking finds what shops actually bought |
| Share of catalogue ever recommended | Whether the long tail gets any exposure |
| Lean towards popular items | How much the model repeats the bestseller list |
Our bar is plain. The model goes to a live test only if it beats the rules on the time-split holdout without recommending a smaller share of the catalogue. The live test then runs with the rules as the control and a guardrail on the measure the business actually cares about, which is rarely taps. Shadow runs and canaries are a sensible way to run that first live exposure.
Where recommender data goes wrong
Taps are logged and impressions are not, so there is no record of what was ignored. Sessions have no link to a shop, so history vanishes at login. Item ids change on every catalogue import. Cancelled orders are counted as orders because nobody wrote down the definition. A random train-test split leaks the future and flatters the model. The team optimises for taps and gets the recommendation equivalent of clickbait. And there is no fallback, so a new shopkeeper sees an empty row or an error on the first order.
A checklist before training anything
- Impressions and interactions logged with timestamps, positions and the source of each item in the slot.
- Written definitions for each event, agreed with the product team.
- Stable item ids and stitched shop or session ids.
- Item attributes complete enough to place a new item.
- Several months of history, or a plan to collect it while the rules run.
- A rules baseline live, with its performance measured.
- A time-based holdout defined and agreed as the bar.
If we are asked where to begin, it is impression logging. Everything else on the list can be added over time, but interactions that happened before impressions were logged can never be read properly, and the sooner the logging starts, the sooner there is a history worth training on.
