Write a rule two people can apply
Give the duty manager a one-sentence definition of urgent. For example, a customer’s operation has stopped and they have no workable alternative. An order tracking enquiry might qualify if the missing part has halted production. A message headed ‘URGENT’ might simply ask for a copy of yesterday’s invoice. Write down which evidence changes the priority.
Ask two experienced colleagues to label a sample independently using only the information available when each ticket arrived. Compare the disagreements. If one person used the customer’s contract and the other only read the message, decide whether the assistant should receive the relevant service commitment. Where details are missing, give staff a route to ask for clarification. Preserve any existing contractual escalation rules in the help desk.
Illustrative example with 200 tickets
Suppose a UK equipment supplier examines 200 recent tickets. Staff agree that ten needed urgent attention. A system placing every ticket in the ordinary queue would get 190 out of 200 labels right, or 95% accuracy, while missing all ten urgent cases. For the shift manager, the counts in the table say more than that percentage.
Both settings below use the same tickets and the same model scores. The team changes only the score at which the help desk raises an alert. The figures illustrate a decision and are not measured supplier or model performance.
| Result | Higher threshold | Lower threshold |
|---|---|---|
| Urgent tickets caught | 6 of 10 | 9 of 10 |
| Urgent tickets missed | 4 | 1 |
| Routine tickets flagged | 2 | 9 |
| Total alerts to review | 8 | 18 |
Translate the counts into a staffing decision
At the higher threshold, six of eight alerts deserve urgent attention. That is 75% precision. The team catches six of the ten urgent tickets, or 60% recall. At the lower threshold, precision falls to 50% and recall rises to 90%. These are the definitions used in Google’s classification guidance. Always show the underlying counts when genuinely urgent tickets are rare.
Lowering the threshold creates ten extra reviews to catch three more urgent tickets in this sample. If initial review takes four minutes per alert, that is 40 extra minutes across the sample. Check when those tickets arrived. Forty minutes spread across a week has different consequences from ten alerts landing before the same morning handover.
Ask what happened to the urgent ticket missed by both settings. If a predictable phrase or account condition should always trigger escalation, add a specific routing rule and assess the combined system again. Keep the ordinary queue staffed and keep an eye on its oldest waiting ticket.
Find out what the score means
A threshold is the cut-off where a score triggers an action. A classifier can rank tickets usefully even when its scores are poor estimates of probability. Scikit-learn’s guidance on decision thresholds treats the score and the business decision as separate choices. Ask the supplier for the score definition and whether it can be compared across tickets and model versions.
If a supplier describes a score as an 80% probability of urgency, check a group of tickets given similar scores. With good calibration, roughly eight in ten around that level should actually be urgent over a sufficiently large sample. A language model writing ‘80% confident’ in an answer gives you a number to investigate. It needs evidence from labelled cases before you can use it as a probability.
If your tool only returns labels such as ‘urgent’ and ‘routine’, you can still count misses and false alerts for that setting. Ask whether the supplier exposes a score or offers different routing policies before planning a threshold comparison.
Try the rule on a shift before changing the queue
Use one set of labelled tickets to choose candidate settings. Reserve a later set, kept out of tuning, to check the selected rule. Keep replies from the same incident together so near-identical messages do not appear on both sides of the comparison. Include quieter shifts, busy periods and the channels your team actually supports.
Run the assistant alongside the existing process first. Capture its proposed priority before staff assign theirs. Review missed urgent cases and unnecessary alerts with the duty manager. Sample ordinary tickets as well as flagged ones, otherwise the team sees the interruptions but has little evidence about what it missed.
For a limited trial, agree who watches each queue, how long an ordinary ticket can wait and what sends all work back to manual triage. Track time to first meaningful response for both queues. If flagged tickets get faster replies while ordinary tickets miss their deadlines, change the routing or staffing before expanding the trial.