Observe Skills in Production
I need to know which skill ran, why it was selected, which context and tools it used, what it cost, how long it took, whether it succeeded, and what happened afterwards.
- Skill-level traces
- Routing decisions
- Tool calls and side effects
- Latency and cost
- Success and escalation rates
- Outcome feedback
The situation
When an agent system produces a bad result, the raw conversation is rarely enough to diagnose the problem. The wrong skill may have been selected, the right skill may have received stale context, a tool may have failed silently, the model may have ignored a constraint, or the final action may have been blocked correctly by policy.
The naive solution
Log the user message and final assistant response. If something looks wrong, read the transcript manually.
The hidden constraint
A final response collapses many internal decisions into one artefact. Without structured traces, I cannot distinguish a skill-quality problem from a routing, context, tool, policy, or orchestration problem. I also cannot calculate which skills create cost without value.
The decision
Instrument at the skill level. Every invocation should have an identity and connect to the wider workflow trace.
The implementation
I would capture skill name and version, routing reason, relevant input references, model/provider, tool calls, duration, token or cost data where available, output status, approval events, errors, retries, and downstream outcome. Sensitive content does not need to be copied indiscriminately into logs; references and redacted summaries are often enough for operational analysis.
The tradeoff
More telemetry creates storage, privacy, and operational design work. The alternative is operating an intelligent system through anecdotes and screenshots, which does not scale.
The evidence I would look for
I should be able to answer questions such as: which skill has the highest failure rate, which one causes the most human escalation, which tool fails most often, which skill became slower after a version change, and which capability costs more than the business value it creates.
The reusable lesson
Observability turns agent behaviour from a conversation into an operable system.
The consultant's questions
- Can you reconstruct why a particular skill was selected?
- Can you separate model failure from tool failure and policy rejection?
- What is the success, escalation, latency, and cost profile of each high-volume skill?
- Which production signals would tell you a skill should be disabled or rolled back?