LLM Observability & Cost Attribution

LLM Observability & Cost Attribution

You cannot debug, optimize, or cost what you cannot see. Traditional APM assumes deterministic, cheap, fast functions. LLM systems break all three assumptions: the same input yields different outputs, a single user request fans out into a dozen model and tool calls, every call costs real money proportional to tokens, and output quality silently drifts as models, prompts, and data change underneath you. Observability for LLM systems is therefore a first-class engineering discipline, not a dashboard you bolt on later — it is the only way to answer "why did this request fail," "which feature is burning our budget," and "did quality regress after the deploy."

How to use this skill: instrument every LLM call as a span inside a request-level trace (§2), attach the metadata tags that make cost and quality attributable (§5), emit the metrics in §3, watch for drift in §4, and debug from traces in §6. Treat the failure modes in §9 as a pre-ship checklist.

1. Why LLM observability is its own discipline

• Non-determinism. Temperature, sampling, and model updates mean the same prompt produces different responses. You cannot reproduce a bug from inputs alone — you need the exact request/response that was captured, including the model version that served it.
• Multi-step agents. A modern request is a tree: planner → retrieval → tool call → sub-agent → synthesis. A failure three levels deep surfaces as a vague top-level error. Without nested traces you see the symptom, never the cause.
• Cost is variable and large. Spend scales with tokens, not requests. A single runaway agent loop or a bloated context window can cost 100× a normal request. You must measure cost per call to control it.
• Quality drifts silently. No exception is thrown when answers get worse. A provider ships a new model snapshot, your RAG corpus shifts, a prompt edit regresses one intent — and nothing alerts unless you measure quality online.

Concretely, what blindness costs you. A support agent starts giving wrong answers after a Tuesday deploy. Without traces: you know error rate is flat (it didn't error, it answered confidently and wrongly), you can't reproduce the complaint (different output every run), you can't tell if it was the prompt edit, a model snapshot rollover, or stale retrieval, and you have no idea the bad path is also the most expensive one because it retries internally. With traces + tags + online evals: the faithfulness eval dropped at 14:02, every failing trace shows an empty retrieval span, the prompt-version stamp points at the deploy, and the cost-by-feature chart already flagged the spike. Same incident, minutes vs. days.

The discipline in one line: capture every call with enough structure (trace shape) and enough metadata (tags) that you can later ask why it failed and what it cost — broken down by the dimensions your business cares about.

2. Traces & spans — the backbone

A trace is one end-to-end request (a user turn, an API call, a scheduled job). A span is one unit of work inside it. Model the request as a tree of nested spans so latency, cost, and errors roll up naturally.

trace: chat-request (user=u_42, tenant=acme, feature=support-copilot, session=s_9)
├─ span: agent.plan            (llm)      120 tok in / 40 out   180ms
├─ span: retrieval.search      (vector)   k=8, 0.85 top score   45ms
├─ span: tool.get_order_status (tool)     args={id:...} ok      210ms
├─ span: agent.synthesize      (llm)      1340 tok in / 280 out 1.9s
│   └─ span: subagent.summarize (llm)     900 tok in / 120 out  900ms
└─ span: guardrail.check       (eval)     pass                  30ms

Capture on every span: model + version, prompt (messages/template + resolved variables), response/completion, token counts (input / output / cached separately — cached reads are priced lower), latency (TTFT and total), computed cost, tool args + results, error type, and the metadata block (§5). For retrieval spans also capture the query, the k documents returned, and their scores; for tool spans, the raw arguments and return value.

Nesting & correlation. Propagate a single trace_id across the whole agent run and parent each span to its caller. When a sub-agent or a separate service is invoked, pass the trace context through (HTTP header / message field) so the child spans attach to the same trace. Without this you get orphaned fragments and cannot follow one request.

Instrumentation pattern. Wrap each unit of work in a span context — a decorator/with block — that opens the span, records inputs, runs the call, records outputs + usage + cost on the way out, and closes it. The pattern: