Start with the decision you need to make
Write down why you want the change: fewer incorrect answers, better handling of incomplete requests, shorter waiting times or lower operating cost. Then identify what must remain reliable. A cheaper response is not useful if staff spend longer correcting it.
Anthropic’s evaluation guidance recommends specific, measurable criteria tied to the application’s purpose. It covers several dimensions, including task quality, consistency, latency and price. Treat these as separate questions rather than collapsing everything into an undefined quality score.
For your own workflow, agree release conditions before viewing the candidate’s results. Name the person who can accept a trade-off. Keep serious failures, such as exposing another customer’s information or taking an unauthorised action, outside any average that could hide them.
Illustrative example: changing an enquiry-routing assistant
Imagine a UK service business using AI to classify incoming enquiries and prepare an internal summary. The team wants to try a smaller model. This is an illustrative design, not a Fewzen customer result or a claim that a particular model will save money.
A normal sales enquiry should reach sales with the requested service recorded. An existing customer reporting a fault should reach support. A message containing both needs an agreed handling rule. An unclear enquiry should be flagged for review instead of receiving a confident but unsupported label.
Suppose the candidate produces shorter summaries but omits the reported fault in a mixed enquiry. Better writing or a lower model bill would not resolve that operational failure. Record the changed outcome and decide whether to adjust the workflow, revise the prompt or keep the current version.
Collect examples that expose the real work
Create a versioned collection of inputs with expected outcomes and a short explanation of each judgement. Include ordinary enquiries, missing information, unusual wording, long messages, conflicting requests and previously observed mistakes. Remove unnecessary personal information and control access to retained examples.
As a practical starting exercise, gather a small set that the process owner can review carefully, then expand it as gaps appear. Do not treat a convenient sample size as statistical proof. A set containing only easy, recent successes cannot tell you how the system handles rare but consequential cases.
Keep some examples separate from the ones used to revise the prompt. Otherwise the team may repeatedly tune for familiar questions without learning whether the change works more broadly. When the business policy changes, update the expected outcomes and record why.
LangSmith’s documentation describes offline evaluation on curated datasets and regression testing across application versions. That provides an implementation option; a small team can begin with a controlled spreadsheet and a script if that preserves the same inputs, outputs and decisions.
Run both versions under comparable conditions
Save the prompt version, model identifier, settings, tool definitions and relevant reference material for each run. Keep those factors fixed while comparing the change you intend to make. If you change the model, prompt and retrieval system together, you cannot easily identify which change caused a failure.
Use a test environment with safe tool substitutes. An evaluation should not send real customer messages, create live invoices or repeat production actions. Check the proposed tool and arguments as part of the result when the workflow uses tools.
Run important examples more than once where outputs vary. Preserve failures as well as successes; do not select the nicest response from several attempts. Report the number of examples and runs alongside results so that a small improvement is not presented as conclusive evidence.
Use clear checks and review ambiguous results
Use code for conditions with a definite answer, such as an allowed routing label, a required identifier or a prohibited tool call. Use an operational reviewer for questions that need context, such as whether a summary preserves the customer’s actual problem. Define the rubric in language the team can apply consistently.
Google’s evaluation documentation distinguishes pointwise scoring, which grades one output against criteria, from pairwise comparison, which chooses between two outputs. For a prompt revision, comparing the old and new answers side by side can help reveal trade-offs. Include a tie option in your own review process.
If you use an AI judge, check its decisions against human-reviewed examples before relying on it. Hide version labels during human comparison where practical and vary presentation order. Ask reviewers to identify the evidence for a judgement, not simply which answer sounds more polished. Neither agreement between models nor a confident grading explanation establishes correctness.
Inspect regressions before looking at the average
Group results by business case. A candidate can improve common sales enquiries while becoming worse on complaints or incomplete messages. Compare which examples changed from acceptable to unacceptable, and investigate those failures with the owner of the affected process.
Keep a simple release record: what changed, the examples used, failures found, unresolved limitations and the decision. Record handling time and the actual usage bill where available. Include retries and review work when comparing operating cost; do not infer savings from a provider’s headline price.
The sources here describe evaluation methods and tooling, checked on 2 November 2026. They do not establish a universal passing score, prove a model is suitable for your business or replace your own acceptance criteria.
Release gradually and keep a route back
After an acceptable comparison, start with a limited rollout appropriate to the workflow’s risk. Keep the previous configuration available and name the person who can pause the change. Monitor corrections, escalations and customer outcomes as well as technical errors.
Add newly discovered failure cases to the example collection after reviewing the expected behaviour. Revisit coverage when the customer mix, source documents or business rules change. Passing yesterday’s examples is useful evidence, but it cannot cover every future input.
Your next step is to choose one proposed prompt or model change and write its release question in one sentence. Ask the operational owner for examples of an acceptable result, an unacceptable result and a case that should go to a person. Build the first comparison around those distinctions.