Test Skills Like Product Capabilities, Not Prompt Snippets
I want evidence that a skill works across realistic cases, edge cases, and failure cases. A few impressive examples in a playground are not an evaluation strategy.
- Golden cases and adversarial cases
- Deterministic assertions versus model-judged quality
- Tool-use evaluation
- Regression suites
- Routing tests
- Human review and production sampling
The situation
A skill can look excellent while it is being developed because the author naturally tests examples that match the intended design. Production users introduce incomplete requests, contradictory evidence, unusual language, stale records, tool failures, and cases the author did not imagine.
The naive solution
Read a handful of outputs, tune the prompt until they look good, and ship the skill.
The hidden constraint
Skills are probabilistic components inside deterministic business systems. A prompt edit that improves one example can regress another. A new model version can change tool selection. A clearer description can improve output quality while accidentally making the router select the skill too often. Without a repeatable evaluation set, improvement becomes opinion.
The decision
Give every important skill an evaluation suite. Tests should cover normal cases, boundary cases, missing-data cases, prohibited actions, tool failures, and realistic examples from production where appropriate.
The implementation
I combine deterministic assertions with qualitative evaluation. If an output schema requires a customer ID, I test that directly. If the skill must never fabricate an accreditation, I test for unsupported claims. If quality is inherently judgement-based, I can use rubrics and human review or carefully designed model-based graders. Tool tests should check not just final text but which actions were requested and with what parameters.
The tradeoff
Evals take time and can themselves become a maintenance burden. I focus on failures that matter to the business rather than trying to create an academic benchmark for every wording preference.
The evidence I would look for
Before releasing a skill version, I should be able to compare it with the previous version on the same cases and understand what improved or regressed. Production incidents should become new regression cases instead of disappearing into chat history.
The reusable lesson
If a capability matters enough to reuse, it matters enough to evaluate repeatedly.
The consultant's questions
- What test would fail if this skill started fabricating a critical fact?
- Do your evaluations inspect tool behaviour as well as final prose?
- Are production failures converted into regression cases?
- Can you compare two versions of the same skill on a stable evaluation set?