Observability Setup — Which Backend + One-Line Autolog

> "Instrument before you need it. The cheapest debugging session is a trace you already have."

This is an ENHANCE overlay. The local skills langsmith]], phoenix]] and mlflow]] each teach one backend deeply. This skill sits one level up: it answers the question those skills cannot — which one, and how do I turn it on in a single line right now — then hands off to the chosen backend's own skill for depth.

1. 何时激活 (When to activate)

Activate at one of three moments — earlier is always cheaper:

| Trigger | Signal | |---|---| | Starting any LM project | First dspy.LM / ChatOpenAI / LlamaIndex / agent graph in the repo, and no tracing wired yet | | Before first deploy | About to ship an LM feature to real users with no trace UI — the AP-15 trap (§6) | | "Why did it do that?" with no traces | A bug surfaced, you reach for history, and there is nothing recorded — you are debugging blind | | Multi-component pipeline | Retriever + reranker + LLM + tools; per-call printing (agentsop-prompt-history-inspect]]) is no longer enough | | Cost / latency regression | Need aggregate token & latency dashboards across runs, not a single printout |

Do NOT activate when:

• You have already chosen a backend and just need its API — defer to langsmith]] / phoenix]] / mlflow]] directly.
• The task is classic ML experiment tracking with no LLM calls — that is plain MLflow, no decision needed.
• You only need to read one rendered prompt right now — that is agentsop-prompt-history-inspect]] (the cheaper first move).

The relationship to agentsop-prompt-history-inspect]]: that skill is the reactive first move (dump one prompt, no setup). This skill is the proactive layer — wire persistent tracing so the next "why" is answered by a trace that already exists, not a frantic re-run.

2. 核心心智模型 (Core mental model)

   Instrument BEFORE you need it
   ───────────────────────────────────────────────
   pick backend      one-line          verify         add eval
   by constraints →  autolog turns  →  a trace    →   hooks (judge,
   (stack/scale/      it on            actually        datasets,
    budget)           (~1 line)        landed          alerts)
   ───────────────────────────────────────────────
   cost of skipping each stage compounds: a missing trace at deploy
   becomes a multi-hour blind-debug later (AP-15, §6).

Three load-bearing ideas:

1. Choosing is the hard part, not wiring. Every modern backend offers ~one-line autolog

(langsmith env vars, px.launch_app(), mlflow.<framework>.autolog()). The real cost is picking the one that won't lock you in or under-serve you at scale (§4 decision table, §5 cases).

1. The cheapest debugging is a trace you already have. A trace recorded proactively costs

near-zero; reconstructing one after a production bug costs hours and may be impossible (no repro).

1. One backend, owned end-to-end. Don't wire two trace backends "to be safe" — pick one,

standardize on it, and let OpenTelemetry GenAI semantics keep you portable if you outgrow it (§7).

3. SOP (Standard operating procedure)

A five-step path. Each step gates the next.

Step 1 — Pick the backend by constraints

Run the decision table in §4 (OP-1). Inputs: stack (is it LangChain/LangGraph? OSS-only? already on MLflow?), scale (dev-only vs high-volume production), budget/hosting (managed-OK vs must-self-host). Output: exactly one backend.

Step 2 — One-line autolog

Wire the single call/env-var for the chosen backend (OP-2). Resist building a custom tracing layer first — autolog gets you a trace today; you can refine later.

Step 3 — Verify traces actually land

Run one real LM call, then open the trace UI and confirm the call appears with inputs, outputs, latency, and token counts (OP-3). A backend that is "configured" but shows no traces is the #1 silent failure (wrong project name, env var not exported, sampling at 0).

Step 4 — Add eval hooks

Once raw traces flow, attach what makes them actionable: an LLM-as-judge or rule evaluator, a dataset built from real traces for regression testing, and basic cost/latency alerts (OP-4). This is what turns "we have logs" into "we catch regressions before users do."

Step 5 — Hand off to the backend skill

Hand off to the chosen backend's own skill (langsmith]] / phoenix]] / mlflow]]) for the deep API once tracing and evals are flowing (OP-6).

Step gating, explicitly. Do not start Step n+1 until Step n is observably true. The most common failure is jumping from Step 2 (wired) straight to Step 4 (evals) without Step 3 (verify) — you build evaluators on top of a trace stream that was silently empty the whole time. Each step has a one-line proof: Step 1 → a backend name written down; Step 2 → an autolog call in the code; Step 3 → one trace visible in the UI with token counts; Step 4 → one evaluator firing on that trace. If any proof is missing, you are not at that step yet.

4. 操作模型 (Operational model — Trigger / Action / Output / Evidence)

OP-1 · Backend decision table (the core of this skill)

• Trigger: Stage 1 — you have an LM project and no backend chosen yet.
• Action: Match your dominant constraint to a row. First matching row wins; ties broken by the "tie-breaker" column.

| If your situation is… | Choose | Why | Tie-breaker / caveat | |---|---|---|---| | Stack is LangChain / LangGraph (or LlamaIndex) and managed SaaS is acceptable | LangSmith → langsmith]] | First-class, zero-glue integration; "same team builds both" so tracing is native (R1] swarnendu.de / langgraph AP-15) | Vendor lock-in; pricing scales with trace volume — see Case B (§5) | | OSS-only / local-first, want OpenTelemetry, no vendor lock-in | Phoenix (Arize) → phoenix]] | OTel-based, self-hosted with SQLite/Postgres, framework-agnostic (R1] phoenix SKILL frontmatter) | Self-host = you run the infra; managed Arize Cloud exists if you outgrow it | | You are an ML-shop already running MLflow (model registry, experiments) | MLflow → mlflow]] | One pane of glass: LLM traces alongside existing runs/registry; mlflow.<fw>.autolog() (R1] mlflow SKILL; dspy-sop deploy line) | LLM-trace UI is younger than purpose-built LLM tools; fine if MLflow is already your hub | | Must self-host a purpose-built LLM-observability product (data-residency, privacy) | Langfuse | Self-hostable LLM-native tracing + evals; the common DSPy/CrewAI bolt-on (R1] dspy-sop "LangFuse-style traces"; crewai obs) | Not a local skill here — install per Langfuse docs; OTel-compatible | | Mixed stack, want maximum portability, framework churn expected | OpenTelemetry GenAI semantics under any of the above | Standard span schema → swap backends without re-instrumenting | More wiring; pick a concrete backend (Phoenix is OTel-native) to actually view spans |

• Output: Exactly one backend name + its skill to hand off to.
• Evidence: backend frontmatter in langsmith]]/phoenix]]/mlflow]]; langgraph-sop AP-15; llamaindex-sop "LangSmith / Phoenix for observability"; dspy-sop MLflow + LangFuse lines. See references/R1-source-evidence.md.

OP-2 · One-line autolog per backend

• Trigger: Stage 2 — backend chosen, turn it on.
• Action: Use the minimal turn-on for the chosen backend:

python # LangSmith — env vars only, no code change (auto-traces LangChain/LangGraph): # export LANGSMITH_TRACING=true # export LANGSMITH_API_KEY=... export LANGSMITH_PROJECT=my-app # (raw SDK / non-LangChain: wrap calls with langsmith.wrappers.wrap_openai or @traceable)

# Phoenix — launch local app + auto-instrument: import phoenix as px; px.launch_app() from phoenix.otel import register register(auto_instrument=True) # picks up installed framework instrumentors

# MLflow — one autolog call per framework: import mlflow mlflow.set_tracking_uri("http://localhost:5000") mlflow.dspy.autolog() # or mlflow.langchain / mlflow.openai / mlflow.crewai

• Output: Traces start flowing on the next LM call — no per-call instrumentation written by hand.
• Evidence: langsmith]]/phoenix]]/mlflow]] quick-start sections; dspy-sop mlflow.dspy.log_model deploy line; prompt-history-inspect op-001 (mlflow.dspy.autolog).

OP-3 · Verify a trace actually landed

• Trigger: Stage 3 — autolog wired, before trusting it.
• Action: Run one real call, open the UI (LangSmith project / px localhost / MLflow :5000), confirm the run shows inputs + outputs + latency + token count. If empty: check project name, that the env var is exported in this process, and that sampling ≠ 0.
• Output: One verified trace = the backend is real, not just configured.
• Evidence: AP "configured-but-empty" is the most common silent failure; verify-before-trust is standard SOP (references/R2).

OP-4 · Add eval hooks on top of raw traces

• Trigger: Stage 4 — traces flow; make them actionable.
• Action: Attach (a) an evaluator — LLM-as-judge (Phoenix) or dataset evaluators (LangSmith) or logged metrics (MLflow); (b) a dataset built from real traces for regression testing; (c) cost/latency alerts. Defer the deep API to the chosen backend's skill.
• Output: Traces become a regression-catching system, not just a log.
• Evidence: langsmith]]/phoenix]] evaluation sections; llamaindex-sop "eval loop is what surfaces failures"; dspy-sop metric pattern.

OP-5 · What to trace (minimum payload)

• Trigger: Deciding what each span should capture.
• Action: At minimum capture, per LM call: rendered prompt, response, model id, latency, token in/out, and the upstream context (for RAG: query + retrieved_nodes + scores + index_id; for agents: tool name + args + observation).
• Output: A span that can answer "why did it do that" without a re-run.
• Evidence: llamaindex-sop line 159 ("tracing captures query + retrieved_nodes + scores + index_id + LLM prompt for every failure"); prompt-history-inspect §what-to-dump.

Quick decision: how much to capture. Default to the minimum payload above on every span, plus full request/response bodies on errors and a sampled fraction of successes. Capturing full bodies on 100% of high-volume traffic is the over-instrumenting trap (§6) and the cost driver in Case B. Capturing less than the minimum recreates AP-15 in slow motion — you have traces, but they cannot answer the "why."

OP-6 · Handoff to backend skill

• Trigger: Backend wired and verified; need depth (custom evaluators, datasets, dashboards, alerting).
• Action: Stop here and invoke the chosen backend's own skill — langsmith]], phoenix]], or mlflow]]. This overlay deliberately does not duplicate their APIs.
• Output: Clean separation — decision/wiring here, depth there.
• Evidence: overlay design; this skill's boundary (§6).

5. 困境决策案例 (Dilemma cases / worked examples)

Case A — Vendor lock-in: LangSmith vs OSS Phoenix

Situation: A team building on LangGraph wants tracing before launch. LangSmith is one env-var away and natively integrated; Phoenix is OSS/OTel but needs self-hosting. Tension: Speed-to-trace + native integration (LangSmith) vs no vendor lock-in + data control (Phoenix). The langgraph-sop is explicit that LangSmith integration is first-class — "the same team builds both" (R1] AP-15). Resolution: If you are LangChain/LangGraph-first and SaaS is acceptable, take LangSmith now — the integration tax of Phoenix is real and AP-15 warns against shipping with no tracing far more loudly than against the lock-in. If data must stay in-house or you have a mixed/OSS stack, take Phoenix and accept the self-host cost. The wrong move is to dither and ship with neither (AP-15). Either choice beats no choice; OTel semantics (§7) keep migration possible.

Case B — Trace volume at scale (the Replit problem)

Situation: A product graduates from dev to high traffic. Per-trace pricing on a managed backend turns "nice dashboards" into a line item that scales linearly with users; full-fidelity tracing of every call becomes both expensive and noisy. Tension: Full observability vs cost & signal-to-noise at production volume. High-volume LLM products (the Replit-class case — agent products generating millions of LM calls) cannot afford to trace 100% on a metered SaaS plan, but turning tracing off recreates AP-15. Resolution: Don't binary-choose between "trace all" and "trace none." (1) Sample — full-fidelity on errors and a small % of successes; (2) move to a self-hosted backend (Phoenix / Langfuse) where marginal cost is infra, not per-trace fees, once volume crosses the break-even; (3) keep eval datasets curated from sampled traces so regression coverage survives the sampling. Decide the sampling/hosting policy before the bill or the noise forces a panicked migration — instrument before you need it, but scale the instrumentation deliberately.

6. 反模式与边界 (Anti-patterns & boundaries)

Anti-patterns

• AP-15 · No observability in production (the headline trap). Shipping an LM feature with no

trace UI. "Replaying a checkpoint locally only goes so far; production needs the trace UI" (R1] langgraph-sop AP-15). Symptom: a user hits a bug, you have nothing to inspect, and you cannot reproduce it. Fix: this skill's Stage 1–3 before deploy.

• Over-instrumenting. Wiring two trace backends "to be safe," tracing every internal function

as a span, or building a bespoke tracing layer before trying one-line autolog. Cost: noise, double bills, maintenance burden. Fix: one backend, autolog first, add detail only where a real question demands it.

• Configured-but-empty. Declaring victory after wiring autolog without verifying a trace landed

(OP-3). The env var wasn't exported in the running process, the project name is wrong, or sampling is 0 — and you discover it only when you need a trace and there is none.

• Tracing the prompt but not the response / context. Truncation, refusal, and bad-retrieval

bugs live in the response and the upstream nodes, not the prompt (OP-5).

• Leaving full-fidelity tracing on at scale without a cost plan (Case B).

Boundaries (when this skill is the wrong tool)

• Backend already chosen → use langsmith]] / phoenix]] / mlflow]] directly; this overlay adds nothing.
• Need to read one prompt right now, no setup → agentsop-prompt-history-inspect]] is the cheaper move.
• Pure ML experiment tracking, no LLM calls → plain MLflow; there is no backend decision to make.
• Pre-call infra failure (auth/network/import) → fix infra; there is nothing to trace yet.

7. 跨框架对照 (Cross-backend comparison)

| Backend | Hosting | Stack fit | One-line turn-on | Lock-in | Local skill | |---|---|---|---|---|---| | LangSmith | Managed SaaS (self-host enterprise) | LangChain / LangGraph / LlamaIndex first-class | export LANGSMITH_TRACING=true (+ key + project) | High (proprietary) | langsmith]] | | Phoenix (Arize) | Self-host (SQLite/Postgres) or Arize Cloud | OSS / any framework via OTel | px.launch_app() + register(auto_instrument=True) | Low (OSS, OTel) | phoenix]] | | MLflow | Self-host / managed (Databricks) | ML-shops; framework-agnostic, registry + runs | mlflow.<framework>.autolog() | Low (Apache-2.0) | mlflow]] | | Langfuse | Self-host (LLM-native) or cloud | DSPy / CrewAI bolt-on; privacy/data-residency | from langfuse import Langfuse + framework callback | Low (OSS) | (no local skill — per docs) | | OpenTelemetry GenAI | Backend-agnostic span schema | Any — portability layer under the above | Instrument with OTel GenAI conventions, export to chosen backend | None | (standard, not a product) |

Reading the table: pick by the dominant constraint, not a feature checklist. Stack-coupling (LangSmith), OSS/lock-in aversion (Phoenix), existing-MLflow-investment (MLflow), and hosting/privacy (Langfuse) are the four forces; OpenTelemetry is the escape hatch that keeps any choice reversible. Then hand off to the matching local skill for depth.

Overlay note: this skill intentionally stops at decision + one-line wiring + verification + eval hooks. For backend-specific APIs (custom evaluators, dataset management, dashboards, alerting), defer to langsmith]], phoenix]], mlflow]]. For the reactive single-prompt dump, see agentsop-prompt-history-inspect]].