Task span
Goal, user, tenant, policy version, input references, final status, and outcome score.
Production Operations
Observe the task, not just the model call
Model telemetry is only one layer of an agent system. Production observability must connect the user goal to plans, state transitions, retrieval, tools, approvals, outcomes, cost, and recovery behavior.
Answer in brief
Use a task-level trace with child spans for model calls, retrieval, tools, memory, and human approvals. Combine operational metrics with outcome evaluations, and retain enough evidence to reproduce why an action occurred.
Trace model
Use these dimensions to make the architecture decision explicit and testable.
Decision span
Plan version, selected action, alternatives, confidence, and reason codes.
Tool span
Validated parameters, permission decision, response, retries, latency, and side effects.
Memory span
Items written or retrieved, filters, ranking, provenance, and retention class.
Metrics
Use these dimensions to make the architecture decision explicit and testable.
Reliability
Completion, failure, retry, timeout, rollback, and escalation rates.
Quality
Task-specific correctness, groundedness, policy compliance, and human acceptance.
Efficiency
Cost per completed task, useful latency, tool-call count, and wasted branches.
Safety
Blocked actions, injection detections, privilege violations, and approval bypass attempts.
Operating loop
Use these dimensions to make the architecture decision explicit and testable.
Detect
Alert on outcome degradation and policy violations, not token volume alone.
Diagnose
Replay a task with the same inputs, model version, tools, and policy.
Improve
Add failures to evaluation sets and compare changes before release.
Govern
Control access to traces because they may contain sensitive prompts, data, and tool output.
Production checklist
Use this as a design-review gate before release.
- Create one trace per user goal.
- Version models, prompts, tools, and policies.
- Record validated tool inputs and side effects.
- Join traces to business outcomes.
- Protect sensitive telemetry.
- Turn production failures into regression tests.
Continue learning
Place this pattern inside a broader agent architecture and operating model.
AI agent tutorial library
Browse memory, planning, frameworks, reliability, safety, and observability.
Open the tutorial library →AI agent guide
Start with agent features, design patterns, benefits, challenges, and industry applications.
Open the guide →