---
name: macro-inc-live-debug
description: "Debug the running local stack with traces, logs, and a shared headless browser. Use when investigating a bug in a running service, tracing a request across services, reading service logs, reproducing a frontend issue, or writing/reviewing tracing instrumentation."
license: "AGPL-3.0"
metadata:
  decimalai-skill-id: "dd6e4588-333f-478e-a945-d6991fde9723"
  decimalai-version-id: "6d74d95c-dd3e-492a-994e-daccf5e8cd14"
  decimalai-version: "3"
  decimalai-body-sha256: "53843ce4a89002338c5a3148e02958253d2b44cab205e7b6bcd6a5d3a6b550ad"
  decimalai-name: "macro-inc/live-debug"
  decimalai-format: "1"
  decimalai-invocation: "model"
  decimalai-category: "devops"
  decimalai-source: "https://app.decimal.ai/s/macro-inc-live-debug@3/SKILL.md"
---
# Live debugging

`just run_local` and `just stack up` start the LGTM collector by default and
the agent browser with `--with-chrome` (both global: one per machine, shared
across instances, left running):

| Endpoint | What |
| --- | --- |
| http://localhost:3001 | Grafana (anonymous admin) — traces + logs UI |
| http://localhost:3200 | Tempo API — traces |
| http://localhost:3100 | Loki API — service logs |
| http://localhost:9090 | Prometheus API — metrics |
| localhost:4317 / 4318 | OTLP intake (gRPC / HTTP), alias `otel-collector` |
| http://localhost:9222 | Headless Chrome (CDP) |

`--traces off` disables the collector; `--traces jaeger|datadog` swaps it.
All collectors bind 4317/4318 — exactly one runs at a time.

Every Rust service exports spans AND its `tracing` events (as correlated log
records) over OTLP. The frontend exports browser spans through the proxy and
propagates `traceparent`, so one trace covers browser → proxy → services.
Wiring happens at stack start: if you start a collector by hand, restart the
stack to pick it up.

## Query telemetry (prefer APIs over the Grafana UI)

The `grafana` MCP server (`.mcp.json` / `opencode.json` / `.cursor/mcp.json`,
Docker `mcp/grafana` on the host network) is pointed at this Grafana. Prefer its tools:

- Logs: `query_loki_logs`, `list_loki_label_values`, `find_error_pattern_logs`
- Traces: `tempo_traceql-search`, `tempo_get-trace`,
  `tempo_get-attribute-values`, `tempo_docs-traceql` (proxied from Tempo's
  own MCP server; every `tempo_*` call needs `datasourceUid: "tempo"`)
- Metrics: `query_prometheus`; plus `search_dashboards`, `generate_deeplink`

Datasource UIDs are stable: `loki`, `prometheus`, `tempo`, `pyroscope`.

Typical calls — service names are the binary names (`email_service`,
`document-storage-service`); match on route, duration, or any span attribute:

- `tempo_traceql-search` `{datasourceUid: "tempo", query: '{resource.service.name="email_service" && status=error}'}`
  (also `'{span.http.route="/documents" && duration>500ms}'`), then
  `tempo_get-trace` with the returned trace ID. Searches default to the past
  hour; widen with RFC3339 `start`/`end`.
- `query_loki_logs` `{datasourceUid: "loki", logql: '{service_name="email_service"} |= "error"'}`
  — log lines carry `trace_id`/`span_id` for correlation. Discover services
  with `list_loki_label_values` on `service_name`.

Timing quirks: Tempo's search index flushes every ~30s — a trace you just
produced is fetchable by ID immediately but may not show in search yet.

Everything above is also plain HTTP: TraceQL search at
`http://localhost:3200/api/search?q=<traceql>`, trace fetch at
`/api/traces/<id>`, LogQL at
`http://localhost:3100/loki/api/v1/query_range?query=<logql>` (`start`/`end`
are unix epoch **nanoseconds**). Use the HTTP form when there is no MCP (pi),
and for bulk retrieval you want to reduce before reading — a full trace can
be 50+ spans, so `curl /api/traces/<id> | jq` (filter to slow spans, compute
offsets) beats dumping `tempo_get-trace` output into context. `docker compose
-p macro logs -f <service>` still works for raw stdout, but Loki is queryable
and survives restarts.

Verbosity knobs (set in the shell before `just run_local`, or per service in
Doppler): `RUST_LOG` filters console + Loki output; `OTEL_TRACE_FILTER`
independently filters exported spans (default `info`). Lowering `RUST_LOG`
never silences traces.

## Drive the shared headless Chrome

`--with-chrome` runs a headless Chrome in Docker with CDP on 9222 (if 9222
doesn't answer, start it: the compose command is in the `headless-chrome`
comment in `docker/docker-compose.yml`). The `chrome-devtools` MCP server
(`.mcp.json` / `opencode.json` / `.cursor/mcp.json`) is already pointed at
it — prefer its tools (navigate, snapshot, click, evaluate, console, network) for browser
work. State (cookies, login) persists across agent sessions until the
container restarts.

- The container uses host networking, so plain localhost URLs work: the app
  is **http://localhost:3000/app**, the proxy `https://localhost:8090` (named
  instances remap these ports; read the stack summary).
- A human can watch the browser live at **http://localhost:6080/vnc.html**
  (noVNC over the Xvfb display) — work in the visible window, not isolated
  contexts, when someone may be watching.
- Login is passwordless: any email works, and the login API returns the code
  in its response (also visible in Mailpit at http://localhost:8025).
- From Playwright instead: `chromium.connectOverCDP('http://localhost:9222')`.
  For token-injection and route-interception recipes see
  `apps/web/docs/playwright-debugging.md`.

Correlate a browser repro with backend traces: note the time, then search
Tempo for that window — the browser's `traceparent` means the frontend action
and the Rust handler share one trace ID.

### chrome-devtools technique

- Snapshot-first: `take_snapshot` after every navigation or pane change; act
  only on uids from the latest snapshot (uid prefixes bump on re-render).
  Macro snapshots are huge — split panes duplicate the doc text — so save
  big ones to a file (`filePath` param) and grep them.
- `navigate_page` can time out while the SPA actually loaded (cold Vite
  compile); follow with `wait_for` on expected text instead of re-navigating.
- `wait_for` matches any text presence, including placeholders. For "AI
  finished"-style conditions, poll in `evaluate_script` for the Stop
  button's absence — the only reliable completion signal.
- `fill` works on plain inputs but NOT contenteditable: click to focus, then
  `type_text` (Enter splits paragraphs/sends). Combobox token fields need
  type → wait for the "N options available" live region → Enter to tokenize.
- If a radio/tab control won't click ("did not become interactive"), click
  its adjacent label text node instead.
- On any error dialog or blank state: `list_console_messages` +
  `list_network_requests` (filter xhr/fetch), then `get_network_request` for
  the failing request's body — pairing console error with failing request
  localizes the fault in one step.
- Verify editor state with `evaluate_script` (e.g. query
  `[contenteditable] strong` to confirm an AI edit) — cheaper and more
  precise than screenshots.

### Driving the Macro app

The condensed version is below; the full field-tested guide (routes, every
surface, keyboard model, crash recovery, trace correlation from a network
request's `traceparent`) is `docs/agent-guide/`.

- Unauthenticated users land on `/app/welcome`: "Continue with email" → fill
  the email input → "Continue". Locally this may log in with no code prompt;
  otherwise the code is in Mailpit. First login auto-creates the user.
- Documents live at `/app/md/<uuid>`; a doc-scoped AI chat at
  `/app/md/<uuid>/chat/<chatId>`; split panes give the right pane its own URL
  segment (`/app/md/<uuid>/channel/<channelId>`).
- Everything is created via the top-left "Create" button (Document D,
  Channel G, Message M, Task T, …). Sidebar buttons are named "Go to X" in
  the a11y tree. Search is the "Search" button — results appear as you type,
  no Enter.
- Editor: title field is focused on creation; type the title, Enter moves
  into the body. The contenteditable's a11y `value` exposes the full body
  text, so snapshots double as content verification.
- AI edit: "Edit with AI" button under the editor → type the instruction →
  Enter. Edits apply in place; done when the "Stop" button disappears.
  AI chat: "Ask Macro" in the doc's Actions panel (doc pre-attached), Enter
  sends, "Stop generating" disappears when the response is complete.
- Channels: "Create" → Channel → name it, tokenize invitees in the "To:"
  combobox, "Create Channel". Verify membership in the Participants tab.
  Composer: Enter sends; the "Task" switch turns a message into a task.

## Write tracing code that is debuggable

How to add spans is the `instrumentation` skill; the conventions are the
always-on [observability rule](../../rules/observability.md) and CS-22 in
[`rust/code.md`](../../rules/rust/code.md). Use the queries above to verify new
instrumentation live: run the code path and confirm the span shows up in Tempo,
connected to its parent trace, with the fields you expect. Unverified
instrumentation is the usual reason "the trace was empty" during an incident.