Support Ticket Classification: How Automatic Routing Works

Most support teams already classify their tickets by hand. Someone reads each new message, decides whether it is about a refund, a delivery or a login problem, and moves it to the queue for that team. At a few hundred tickets a week one person can manage that in a morning, but after a festive sale the volume grows several times over and the first reply slips to the next day. A text classifier can do most of this sorting, and this post walks through how it works, step by step, and where it still needs a person.

What ticket classification means

Text classification is the task of reading a piece of text and assigning it to one category from a fixed list. For support tickets, the categories are your queues or tags, and the text is the subject, the body and sometimes the channel the ticket arrived on. The model learns from examples where the right answer is already known, which is called supervised learning.

The important part is that most helpdesks already hold those examples. Every closed ticket ended in some queue, so a year of closed tickets exported with their final queue is a labelled training set that costs nothing extra to produce. It also contains the way your customers actually write. For an online brand that sells direct to consumers (D2C), that often means half the messages arrive in Hinglish over WhatsApp, which no generic dataset would cover.

Before training anything, check whether you need a model at all. If a handful of keywords sort most of your tickets correctly, rules may be enough, and the trade-off is laid out in when a trained model earns its place over rules.

The parts of a routing system

A ticket router has more parts than the model. Here is what each one does.

• The categories are the queues and tags your agents already use, so the labels mean something to them from the first day.
• The training set is the export of closed tickets, each with the queue it finally landed in.
• The classifier is the model. It reads a ticket and returns a score for every category instead of a single answer.
• A threshold is the minimum score at which the system trusts a label enough to act on it. Each category gets its own.
• The helpdesk integration writes the result into Zendesk, Freshdesk or whichever tool the team already works in, through that tool's API (application programming interface).
• The other bucket is a holding queue for tickets the classifier could not place with enough confidence.

The categories need attention before any training starts. Some turn out to be a person or a date rather than a topic, such as "Priya's queue" or "Q3 migration". Some pairs get filed inconsistently: if two agents would put the same "payment deducted but order not placed" ticket in different queues, one under payments and one under orders, the model will be inconsistent too, because it learns from their choices. Merge those categories, or define them clearly and relabel a sample. Twenty ambiguous tickets labelled carefully by the team lead settle more than another thousand historical ones.

How one ticket moves through it

A classifier does not return "billing". It returns a probability for every category, and the top one may be barely ahead of the second. Treating the top label as final is where automated routing gets its bad reputation. Here is how a ticket moves through a system that respects that.

1. The ticket arrives with its subject, body and channel.
2. The classifier scores it against every category. A ticket about a failed payment might come back with a high score for payments and a lower but real score for orders.
3. The system compares the top score with the threshold for that category.
4. Depending on the result, the ticket lands in one of three places.

1. 01Ticket arrivessubject, body, channel
2. 02Classifier scoresone score per category
3. 03Above thresholdroute automatically
4. 04Middle bandagent confirms
5. 05Below allother bucket

Thresholds are set per category, so a refund can route at lower confidence than a security report.

Where the ticket lands	When it lands there	What a person does
Its queue, automatically	Top score is above the category's threshold	Works it as usual, and moves it if it is wrong
Its likely queue, with a suggested label	Top score falls in the middle band	Confirms or changes the label in one click
The other bucket	No category scores high enough	Reads it during the daily review

The thresholds differ by category because the cost of a mistake differs. A refund request sent to the wrong queue loses a few hours. A security report sent to the wrong queue can lose much more, so its threshold sits higher and more of those tickets get a human look.

Whatever the outcome, the classifier writes into the helpdesk and stays out of sight otherwise. Four things belong on the ticket record: the label, in the tag or queue field the helpdesk already has; the confidence score, as a custom field; the model version that produced it; and the original text, untouched. The model reads the text and never edits it, not even to tidy Hinglish into something neater. Agents keep the same control they always used to move a ticket, and every time they use it, the team gets a labelled misroute for free.

Why the other bucket needs an owner

Teams tend to treat the unsure bucket as a failure to minimise. It is more useful than that, because it is where new problems show up first. When a release breaks OTP (one-time password) login on a Friday evening, the first fifty tickets about it match no existing category, and they collect in the other bucket before anyone has named the incident. When a new product line launches, its questions land there too.

So the bucket needs an owner and a rhythm. Someone reads it daily while it is small and reviews it weekly for clusters of similar tickets. When a cluster appears, it becomes a new category, its tickets become the first training examples, and the model is retrained. If the bucket grows steadily, the categories or the thresholds need attention. If it is always empty, the thresholds are probably too loose and the misroutes are hiding in the normal queues.

Measuring misroutes instead of accuracy

Accuracy on a held-out test set, meaning tickets the model did not see during training, tells you the model learned something. It does not tell you whether the support team is better off, and it stops being true as soon as the product or the pricing changes. The number that matters is the misroute: a ticket the model placed in a queue and an agent later moved somewhere else.

Says something

• Misroutes per category per week
• Time to reach the final queue
• Size and age of the other bucket
• Overrides and where they went

Easy to collect, says little

• Test-set accuracy measured once
• One accuracy figure overall
• Count of tickets processed
• Average confidence score

Measure a baseline before automating anything. People misroute tickets too, and the rate at which agents already move tickets between queues is the number the model has to beat. Without that baseline, the first misroute report looks alarming and there is nothing to compare it with. The same thinking about when software should step back and pass a case to a person applies well beyond tickets, and designing the handoff from an agent to a person covers it in more depth.

Common ways ticket routing goes wrong

The mistake I have read about most often is categories that follow the org chart instead of the ticket. The model then has to guess which team owns something from text that never mentions a team. A customer writes "paisa kat gaya, order nahi aaya" (the money was deducted, the order did not arrive). Nothing in that sentence says whether it belongs to payments or to orders, and a model cannot learn a boundary that the text does not contain.

The other common ones are smaller but add up. One threshold is used for every category, even though a wrong refund route and a wrong security route cost very different amounts. Routing is built without a visible override, so agents reassign tickets quietly and the misroutes never get counted. The model is trained once on last year's data and never again, while the product and the categories change underneath it. And the other bucket ends up as a line in a report that nobody opens.

What to try first

Here is the order I would follow. The first two steps need no model at all.

1. Export a year of closed tickets with the queue each one ended in.
2. Count how often agents currently move tickets between queues. That is your baseline.
3. Train a plain baseline classifier and look at which categories it confuses with each other. Fix those categories before tuning the model.
4. Choose a threshold per category, starting conservative.
5. Write the label, confidence and model version into the helpdesk record, with the override intact.
6. Name an owner for the other bucket and put its review in the calendar.

Start with step two, because it needs no model and takes an afternoon of helpdesk reports. A routing system judged against no baseline will be judged on its worst week. Ticket classification is one of the natural language processing (NLP) tasks described on our AI solutions page, if you want to see where it sits among the others.