Agent Observability Spec Skill

Agent Observability Spec Skill

You can't fix what you didn't record. For LLM systems the unit of observability is the trace — everything the model saw and did — because behaviour, not uptime, is what fails. This skill specifies what to capture, what to compute from it, and when to page someone.

What This Skill Produces

• A trace schema: per-request spans and the fields each must carry
• Metric definitions across health, quality, cost, and behaviour — each with a threshold and owner
• A sampling and retention policy that keeps cost sane and debugging possible
• A privacy note: what logged content contains, who can see it, and how long it lives

Required Inputs

Ask for (if not already provided):

• The system's shape — single LLM call, RAG pipeline, or multi-step tool-using agent
• Traffic volume and cost sensitivity — full tracing at 10M req/day is a budget decision
• What "misbehaving" means here — the two or three failure modes that matter most (wrong facts? wrong actions? cost? refusals?)
• Existing observability stack (Datadog, Langfuse, OTel, homegrown) — spec into it, not around it

Trace Schema

Every request produces one trace; every model call, retrieval, guardrail check, and tool execution is a span. Minimum fields:

The test of the schema: an engineer can replay any incident from its trace alone (see agent-incident-postmortem).

Metrics and Alerts

Define four families; every metric gets a threshold, a window, and an owner.

• Health — error rate, p50/p95 latency, timeout rate, provider 429/5xx rate. Page on these.
• Cost — cost per request (p50, p99), tokens per request, cache hit rate, daily spend vs. budget (pair with llm-cost-latency-budget). Alert on p99 and daily-budget burn — cost incidents are caused by the tail, not the mean.