---
name: avizmarlon/observability-when-to-add
source: https://app.decimal.ai/s/avizmarlon-observability-when-to-add@1/SKILL.md
source_sha256: 6942e8d5e3b4
---

## Observability & Instrumentation — Directed, Not Preventive

**Industry principle** (Google SRE Book, Honeycomb, Charity Majors): instrumentation carries maintenance cost proportional to its value. Every metric is code to maintain, schema that can break, storage cost, and signal to interpret. **You instrument when you have confirmed pain, not "just in case."**

### When to add observability

- Performance bug you cannot locate (classic in multi-threaded, async, or distributed systems)
- External SLA or performance requirement demands metrics
- System is in production with real user traffic
- Flow analysis is part of the product itself (user journey tracking, etc.)

### When NOT to add observability

- "Good practice" or preventive coverage before pain is identified
- Small side projects in active development
- Before you have identified a concrete bottleneck or failure mode
- To satisfy an abstract rule like "instrument everything"

### What to use when you decide to instrument

- **Production / real traffic** → **OpenTelemetry** (CNCF graduated project) + collector + backend (Honeycomb, Tempo, Jaeger, or cloud equivalent)
- **Local development / single-user** → structured JSON logging + `jq`/`grep`. Add `correlation_id` if the call crosses process boundaries.
- **Spot performance hunt** → use language builtins (`console.time`/`console.timeEnd` in JavaScript, `time.perf_counter()` in Python, etc.). Discard after resolving the issue.

### Universal principles when instrumenting

**Instrument boundaries, not internals**
- Entry points (API handlers, RPC endpoints)
- Network calls (HTTP, gRPC, database queries)
- External service boundaries
- NOT every function, NOT every variable assignment

**Aggressive sampling in production**
- Default: 1-10% of requests
- 100% only in development or for high-value (error) events
- Sampling reduces noise and cost without losing signal on aggregate behavior

**Telemetry failure must not cascade**
- All log writes go in try/catch blocks (silent failure okay)
- If the observability system is down, the application runs normally
- Observability is about understanding running systems, not blocking them

**Never log PII, tokens, or raw payloads**
- Log summaries instead of bodies (e.g., "request 256 bytes to /api/v1/users" not the JSON)
- Redact or hash sensitive fields
- Assume logs are readable by ops teams and retained for weeks

**Use `verb:noun` naming for events**
- Examples: `fetch:external_api`, `query:database`, `publish:event`
- This naming schema makes `grep` work reliably months later
- Easier to discover related events and trace flows

**Use monotonic clocks for duration**
- `performance.now()` (JavaScript), `time.perf_counter()` (Python), `System.nanoTime()` (Java)
- NOT wall-clock time (`Date.now()`, `time.time()`) — wall clocks can jump backward
- Duration must be reliable across system-clock corrections

### Red flags — when you're over-instrumenting

- Observability code is more complex than the business logic it instruments
- You added metrics months ago and have never queried them
- The instrumentation is "just in case" with no corresponding monitoring/alerting
- Log volume exceeds bandwidth or storage budget
- You notice yourself filtering/ignoring metrics because they're too noisy

### Canonical references

- Google SRE Book, chapters 6 (Monitoring Distributed Systems) and 12 (Effective Troubleshooting)
- Charity Majors, "[Observability is for Unknown Unknowns](https://www.honeycomb.io/blog/observability-is-for-unknown-unknowns)"
- OpenTelemetry specification: https://opentelemetry.io/
- Honeycomb Observability Guide: https://www.honeycomb.io/