Choose a count that answers the team’s question
A service manager asking which team should call a customer back may need one owner for each reply. A product manager deciding what to fix needs to see every relevant issue, including the second one in a sentence. You can store one owner and several issue labels on the same reply. Making the owner field do both jobs loses information.
Pew Research Center illustrates the difference in two published methods. For one survey its researchers asked a language model to apply every relevant category to an answer. For another they asked it to select a single category. Both exercises used a written set of category definitions and checks against human judgements. The counting rule followed the research question.
Write the unit into the report heading. If your source is a survey export with one row per submission, report replies. To report distinct customers, you also need a reliable way to recognise two submissions from the same customer. For anonymous feedback, stick to what the export can support.
Write a rule for the awkward cases
Give each label a short definition, an example that qualifies and a similar example that should stay out. For “tracking failure”, include a broken link or a tracking page that shows no updates. Exclude a working page that accurately shows a late delivery. Otherwise the tracking team can inherit a courier problem simply because the customer used the word “tracking”.
Ask the model to return the reply ID, the label and the shortest passage that supports it. Keep the original text available to the reviewer. The Government Digital Service recommends writing down what researchers observed before grouping observations into themes. The same discipline helps here. A customer saying the page stayed blank gives you evidence of a symptom. It leaves the technical cause open.
Allow a useful comment to sit outside the current labels. Keep blank answers separate from comments such as “everything worked”. The latter is an actual response with no reported issue. A blank tells you nothing about that customer’s experience.
Illustrative example — six replies to a parcel survey
Suppose six customers each submit one comment to a retailer. The fictional comments below show what goes into the tally. An adviser can mark up this small set by hand before asking a model to apply the same rules to fresh replies.
Three of the six replies report a late delivery. Two report a tracking failure. One reports damage and one reports no issue. That gives six issue labels across five replies with a problem. The customer in A appears in two issue totals, while the repetition in D counts once. Adding the issue totals would give the wrong number of dissatisfied respondents.
For the weekly report, write “Two of six respondents reported a tracking failure.” Keep the survey question and collection period beside that figure. The other customers who received parcels but never answered the survey are outside that denominator.
| Reply | Illustrative comment | Issue labels |
|---|---|---|
| A | Parcel late and tracking link broken. | Late delivery, tracking failure |
| B | Delivery a day late. The tracking page showed the delay. | Late delivery |
| C | Arrived on time but the parcel was crushed. | Damage |
| D | No tracking updates. I checked twice and still no updates. | Tracking failure |
| E | Everything worked. | None |
| F | Arrived after the promised day. | Late delivery |
Check both what the model adds and what it misses
Set aside fresh replies. Ask a colleague to label them using the same definitions without seeing the model’s choices. Compare the two sets one issue at a time. For tracking, inspect replies the model labelled incorrectly and replies where the colleague found a problem the model missed. Two equal totals can conceal both kinds of error.
Include some comments about deliveries that went well, some that mention several issues and some that fall outside your categories. Where two reviewers disagree, ask which words support the label. A dispute about whether “slow” describes the website or the courier may call for an “unclear” result. Forcing agreement by guessing will make the numbers look tidier than the evidence.
Let a spreadsheet or database total the accepted reply-and-label pairs. Have the narrative summary read those totals. If a category changes, record the definition and the date. Recheck earlier replies under the new rule before claiming that a rise or fall reflects a change in customer experience.
Take one finding to the next service meeting
Pick a single question the team can act on. For tracking failures, take the count, the number of replies reviewed and a few checked passages to the colleague who owns the tracking page. Ask whether the reported failures share a carrier, device or order type. That is a concrete investigation the comments can support.
Frequency alone cannot set priority. One report that a collection page displayed someone else’s details needs individual attention even if it never becomes a common theme. Keep the usual route for handling urgent individual cases alongside the weekly tally.