Measure the words and the booking separately
Speech recognition turns audio into text. A second step, often another model, picks details out of that text and puts them into a booking form. Keep the audio reference, the transcript and the proposed fields together during testing. If the transcript says fifteen and the form says fifty, changing the speech recogniser targets the wrong step.
Microsoft’s speech evaluation guidance compares recognition output with reference transcripts using word error rate. Count words inserted, deleted or substituted, then divide the total by the number of words in the reference. This gives a way to compare recognisers on the same recordings. A mistake in a greeting and a mistake in a unit number can each count as one word error.
Alongside that measure, count how many drafts have every required detail correct, how many go to review and how many wrong details reach dispatch. Keep the total number of calls visible. An assistant that sends every booking for manual checking may avoid dispatch errors while costing staff more time than typing the details themselves.
Illustrative example from a collection desk
Imagine a team testing two transcription settings on the same 20 invented calls, with 1,000 words in the reference transcripts altogether. For simplicity, each call contains one unit number. Both settings make ten word errors, giving each a word error rate of 1%.
Setting A gets all 20 unit numbers right. Setting B gets two wrong, so only 18 of its 20 unit numbers match. If the team looked at the overall percentage alone, it would have no reason to prefer either setting. For this collection desk, the two wrong destinations warrant investigation even if most of the transcript reads well.
Listen to those clips and compare both drafts with the reference. Did the recogniser confuse similar sounds? Did the caller correct themselves later? Did the form keep the first value after the transcript changed? Each answer points to a different fix. Twenty examples can expose a failure mode. Estimate its frequency on a larger sample drawn from the calls the desk actually handles.
Build a small recording set the desk recognises
List the fields that must be right before dispatch. For a collection these might include site, unit, collection day and job reference. Agree how to score equivalent forms such as ‘unit fifteen’ and ‘Unit 15’. Keep meaningful distinctions, including a leading zero in a reference. Decide how to mark a field that the caller never supplied.
Start with authorised recordings or staged calls using invented details. Include different speakers, phone connections, background noise, interruptions and a caller spelling out a reference. Ask a colleague to prepare the expected fields from the audio before seeing the AI output. Mark genuinely inaudible details as unclear. Reward a referral for clarification in that case and record a guessed value as an error.
Keep some recordings aside while adjusting the system. Include ordinary calls as well as difficult ones when you check the final version. A set deliberately packed with confusing numbers can reveal weaknesses, but its error rate will reflect the mixture you chose.
Wait for the caller’s correction
A live transcript can change while a person is speaking. Amazon Transcribe documents how it revises partial results as more audio arrives and marks when a speech segment is complete. In a live booking interface, show provisional words as provisional. Preserve the difference between displaying a suggestion and saving a confirmed field.
Even a completed segment can be superseded by what the caller says next. Try a staged call in which they say ‘Tuesday’, pause, then add ‘Sorry, Thursday’. Check the displayed draft, saved booking and dispatch message. All three should agree on the corrected day once a person confirms it. If the desk needs a live draft for speed, let staff edit it and protect a confirmed correction from an older transcript update.
Test what a vocabulary change gets wrong
A recogniser may struggle with local site names or a product code. Google Cloud documents phrase adaptation for supported Speech-to-Text configurations. Increasing a phrase’s weight makes the recogniser more likely to select it. Check support for your chosen language and model before making it part of the design.
Test a troublesome site name in calls that contain it and in calls that mention a similar site instead. Count both recovered names and new substitutions. Where two destinations remain plausible, let the operator see both and ask the caller to confirm. A directory match on its own proves only that the proposed destination exists.
Set a decision for the first shift
Run the draft alongside the team’s current booking process. Record incorrect fields, requests for clarification, staff corrections and handling time. Review any wrong destination individually. Include calls where no draft appeared, otherwise the trial overlooks work that staff handled entirely by hand.
Give the duty manager a specific choice at the end of the shift. They might keep AI drafts for job descriptions while requiring staff to read back every destination and reference. If the extra checking takes longer than the current method, narrow the pilot to the field where drafting actually helps. Repeat the comparison on fresh calls before giving the system more responsibility.