A D2C brand selling on its own site and on two marketplaces collects a few thousand pieces of customer text a month, from product reviews, app store comments, WhatsApp support chats and the free-text box at the end of a survey. The first report anyone builds from that pile is a sentiment line, up or down each week. It confirms what the team already felt from the support queue, and it does not tell anybody what to change.
What a sentiment score averages away
"Negative" covers a late delivery to a pin code the courier barely serves, a wrong size, a crash on the UPI payment screen, and a rude reply from support. The score adds all of that together and reports one number. When the number drops, someone still has to sit and read the reviews to find out why, which is the work the report was supposed to remove.
We still compute sentiment, as a filter rather than a result. It tells you which reviews to read first and which issues people describe with the most frustration. That is a reasonable input to the pipeline and a poor thing to put on the leadership screen by itself.
Extracting issues instead of a mood
Issue extraction reads each review and returns zero or more issue mentions. Write down what a mention is before building anything. Ours is one complaint or one piece of praise about one part of the product, with a short label, the product area, the polarity, and the sentence it came from. A review that says the app is slow and the refund page is confusing produces two mentions, not one score.
| Review (excerpt) | Label | Area | Polarity |
|---|---|---|---|
| "Took ten days to reach Guwahati" | Delivery late | Logistics | Complaint |
| "Payment failed twice on UPI, then went through" | Payment failed | Checkout | Complaint |
| "Payment failed twice on UPI, then went through" | Charged twice | Checkout | Complaint |
| "Fabric is better than I expected" | Fabric quality | Product | Praise |
There are two ways to produce those rows, and most teams need both, in order.
- 01Reviewsall channels, with date and product
- 02Extract mentionslabel, area, polarity, quote
- 03Map to issue listor the unlisted bucket
- 04Count and trendshare per issue per week
- 05Hand to ownercounts and example quotes
Steps two and three use a model; the mapping is maintained by a person; the last step is a ticket.
Open extraction hands the review to a language model and asks it to name whatever problem it sees, with no list to pick from. It finds things you had not thought of. It also names the same problem twenty different ways: "late", "delivery delayed", "took ten days to reach Guwahati". Classification against a fixed issue list is consistent and countable, and it cannot see anything new.
So we start with open extraction over a sample, to discover what the list should be. Then we freeze a numbered list and classify everything against it, with an unlisted bucket for mentions that match nothing. Someone reviews the bucket on a schedule, every fortnight for instance, and new issues move from there into the list. The classification step works much like routing support tickets without reading every one, and asking the model for a fixed output shape follows the same rules as structured extraction from emails and PDFs.
Keeping issue names stable over time
"Late delivery", "took two weeks to arrive" and "still waiting" are one issue. "Battery" means one thing for a phone and another for a laptop. Embeddings and clustering produce candidate groups quickly. A person decides what each group is called and where its edges are, and that person should be someone who has handled the complaints, not only the engineer running the clustering.
Keep that decision as data, with one source of truth for the issue list. A mapping table says which extracted labels fold into which issue, and carries a version number and a date.
| Extracted label | Issue id | Issue name | List version |
|---|---|---|---|
| delivery delayed | 7 | Delivery late | v3 |
| still waiting for order | 7 | Delivery late | v3 |
| payment failed | 12 | Payment failure at checkout | v3 |
| money deducted no order | 12 | Payment failure at checkout | v3 |
That table is what makes this month's counts comparable with last month's. If the clustering is rerun from scratch every Monday, the groups shift, last week's issue 7 becomes this week's issue 12, and the trend line means nothing.
Counting as a share of reviews
A raw count misleads by itself. Review volume climbs in the Diwali week and goes quiet in a dull month, so define the measure as mentions of an issue divided by reviews received in the same period, and show that share. A new issue at low volume in the week after a release also matters more than a large old one that has sat flat for a year, because the new one is still cheap to fix.
- One score per week
- A drop needs manual reading to explain
- No owner for anything
- Cannot show whether a fix worked
- Share per issue per week
- New issues flagged after each release
- Each issue has a team and a ticket
- Share falls after the fix ships, or does not
Split the share by whatever differs in the product: app version, city or region, marketplace versus own site, plan. When the payment failure issue shows up only on one Android build released last Tuesday, it is a bug with a date on it. The same share spread evenly across every version and channel points to a design problem, and a different team should get it. Before you trust any split, reconcile the review counts against what each marketplace reports, because a channel whose export silently dropped a week will look like a channel with no complaints.
Giving every issue an owner
An issue board that nobody is responsible for turns into one more report. Route each issue to the team that can do something about it: delivery complaints to the operations head, crashes to the mobile team, confusion about pricing to product. The handoff is a ticket in whatever tracker that team already uses, Jira, a shared sheet, or a WhatsApp group if that is honestly where they work. It carries the issue's share, its trend, five representative quotes and a link back to the full set.
Then close the loop. When the fix ships, watch that issue's share over the next few weeks. If it falls, you have evidence the fix worked, in customers' own words. If it does not fall, the ticket reopens with the new quotes attached, and the conversation is much shorter the second time.
Where review analysis goes wrong
The most common failure is the sentiment line sitting on the leadership screen, read every week and acted on never. Close behind it are clusters recomputed on every run, so no issue keeps a stable identity long enough to draw a trend. Then issues with no owner, discussed in a Monday review and forgotten by Wednesday.
The counting failures are quieter. Raw counts rise and fall with marketing spend rather than with the product. One channel's export breaks and nobody notices the gap. And one long, angry, well-written review becomes "a trend" because someone read it aloud in a meeting.
A sensible order to do it in
- Pull three hundred recent reviews and read them. Write down every distinct issue you see. That is your first issue list, and it took an afternoon.
- Run open extraction over the same three hundred and compare its labels with yours. Where it found something you missed, add it.
- Freeze the list as version one, with an unlisted bucket and a mapping table.
- Classify everything from the last six months against it and chart share per issue per week.
- Send the three issues with the steepest rise to their owners, with quotes, and fix a date to look at the shares again.
We ask teams to do step one before any discussion about models. Every later decision, including which model to use and whether a model is needed at all, gets easier once someone on the team has actually read the material. If the issue list turns out to be the useful part, the extraction and classification behind it are the kind of NLP work we describe under AI solutions.
