--- name: kunanonj/cursor-plugin-posthog-exploring-signals-scouts source: https://app.decimal.ai/s/kunanonj-cursor-plugin-posthog-exploring-signals-scouts@1/SKILL.md source_sha256: fa6c6cb10073 --- # Exploring Signals scouts A **scout** is a scheduled agent that wakes on its own interval, looks at one PostHog project, decides what's genuinely worth surfacing, and either emits it as a **finding** into the Signals inbox or closes out empty (a real, valid outcome). PostHog ships a fleet of canonical scouts — a cross-product generalist (`signals-scout-general`) plus per-surface specialists (`-error-tracking`, `-ai-observability`, `-logs`, `-revenue-analytics`, `-surveys`, `-csp-violations`, `-observability-gaps`). A project may also have **custom scouts** beyond the canonical fleet — any `signals-scout-*` skill a team authored (e.g. `-brand-mentions`, `-mcp-feedback`) shows up here too, so don't assume the roster is only the canonical set. This skill helps you **understand and explore what a project's scouts are doing and how they're performing** — entirely through read-only MCP tools. It is the observability counterpart to [`authoring-signals-scouts`](../authoring-signals-scouts/SKILL.md) (which teaches writing and tuning) and to [`inbox-exploration`](../inbox-exploration/SKILL.md) (which covers the inbox reports scouts feed into). There are four things you can observe about the fleet, each with its own tool: | What you want to know | Tool | What it tells you | | -------------------------------------------- | --------------------------------------- | ------------------------------------------------------------- | | Which scouts run, how often, in what posture | `signals-scout-config-list` | One row per scout: schedule, `enabled`, `emit`, `last_run_at` | | What the scouts actually did, run by run | `signals-scout-runs-list` / `-retrieve` | Per-run status, timing, end-of-run summary, deep-link | | What the fleet has learned across runs | `signals-scout-scratchpad-search` | Durable per-team memory (baselines, noise, allowlists) | | What the scouts surfaced to the user | `inbox-reports-list` | Findings that cleared the bar and became inbox reports | The orienting fifth is `signals-scout-project-profile-get` — the deterministic snapshot of "what's true about this project" that every scout cold-starts from. When a scout found nothing, this is usually why. ## Output handling: expect to offload to a file Two of these tools — `signals-scout-runs-list` and especially `tasks-runs-session-logs-retrieve` — routinely return payloads that **overflow an MCP client's token budget and get spilled to a file**. This is the normal path, not an error. Plan for it up front rather than discovering it after a failed call: - **Keep `limit` small** on `signals-scout-runs-list` (~10–15). Each row carries a long prose `summary`, and runs come back newest-first across the _whole_ fleet, so even a modest page is large. - **Session logs are large by nature.** A single run's log is hundreds of KB to a few MB. Fetch it with **`call --json`** (so the saved file is real JSON, not the pretty text format — `jq`-able) and read the saved file with `jq` / a script rather than inline. - **Don't hand-parse the session log.** The bundled [`scripts/`](#helper-scripts) do the reconstruction for you — see below. ## Start here: is the fleet even set up? Don't assume the project has scouts. The fleet only runs on teams enrolled via the `signals-scout` feature flag, and a project may have no configs, all-disabled scouts, or scouts stuck in dry-run. Run this first whenever a user asks about their scouts for the first time in a session. ```json signals-scout-config-list ``` Read the result against three cases: The config list is unpaginated — it comes back as `{ results: [...] }` (a bare array), with no `count` field. Read the result against three cases: - **Empty (`results: []`)** — no scouts are registered. The project isn't enrolled in the scout fleet (or hasn't ticked yet). Say so plainly; don't go fishing for runs. Point the user at the Signals scout settings / PostHog Code onboarding rather than inventing activity. - **Configs exist but all `enabled: false`** — the fleet is registered but paused. Nothing is running. Tell the user which scouts exist and that they're all off. - **At least one `enabled: true`** — the fleet is registered and that scout is allowed to run. For each enabled scout note its `run_interval_minutes` (cadence), `emit` (false = **dry-run**, runs but writes nothing to the inbox), and `last_run_at`. One caveat before reporting "it's live": runs are gated by the `signals-scout` feature flag, not by `enabled`. A project that was enrolled and later drained from the flag keeps its `enabled: true` rows, but the coordinator no longer plans runs for it — so a stale or `null` `last_run_at` on an enabled scout usually means the project is no longer enrolled, not that the scout is idle. **`last_run_at` is a _dispatch_ stamp, not proof a run executed.** The coordinator advances it the moment it _enqueues_ a child workflow for a due scout — before any worker picks the run up. Child dispatch is fire-and-forget, so if workers are saturated or down the children just queue and no run ever materializes, yet `last_run_at` keeps marching forward each tick. So a recent `last_run_at` means "dispatched this tick," **not** "a run is genuinely happening." The authoritative liveness signal is the newest actual **run row** in `signals-scout-runs-list`, not the config stamp. Cross-check them: if `last_run_at` is fresh (minutes ago) but no run row has appeared for that scout in well over its `run_interval_minutes`, the fleet is **dispatching but not running** — workers backed up / down, or runs stranded — a real reliability problem, not a live scout. Don't report "it's running" off `last_run_at` alone. A scout that is `enabled: true` but `emit: false` is the most common source of "my scout isn't doing anything" confusion: it _is_ running and reasoning every tick, it just isn't allowed to post findings yet. Always surface the `emit` posture when reporting on a scout. See [`references/scout-data-model.md`](references/scout-data-model.md) for every field on a config, run, and scratchpad entry, the run status values, and how the pieces link together. ## Workflow: survey the fleet "What scouts do I have / what are they doing?" — lead with `config-list`, then enrich with the most recent run per scout so the user sees liveness, not just configuration. 1. `signals-scout-config-list` — the roster. 2. For each enabled scout, `signals-scout-runs-list` and pick the newest run with a matching `skill_name` (runs come back newest-first across the whole fleet, so a single call usually covers everyone). Report `status` and how long ago it ran. Present it as a table the user can scan — scout, cadence, posture, last run, last outcome — and call out anything anomalous (never run, last run errored, stuck in dry-run for a long time). ## Workflow: understand one scout end to end "How does my error-tracking scout work / how is it doing?" 1. **Read its config** — find the row in `config-list` for `signals-scout-error-tracking`: schedule, posture, last run. 2. **Read its body** — `posthog:llma-skill-get {"skill_name": "signals-scout-error-tracking"}` returns the team's actual instruction set (which may be a canonical default or a diverged, hand-edited row). This is what the agent is told to do every run — its signal-vs-noise discriminator, explore patterns, and disqualifiers. To understand _why_ a scout behaves the way it does, read its body. 3. **Read its recent runs** — `runs-list` with `text` set to the skill's domain, or just scan the newest runs and filter to its `skill_name`. The end-of-run `summary` on each run is the scout's own account of what it looked at and decided. 4. **Read what it remembered** — `scratchpad-search` (see below). The memory entries a scout wrote reveal the baselines and noise it has internalized about this project. ## Workflow: read recent runs `signals-scout-runs-list` returns the most recent runs across the whole fleet, newest first (capped at 100). Use it to answer "what happened lately?" - **Scope to a window** with `date_from` / `date_to` (ISO-8601; inclusive lower, exclusive upper on `created_at`). Walk backwards by passing an earlier `date_to`. - **Search summaries** with `text` — a case-insensitive substring match on each run's end-of-run `summary`. This is how the headless scout dedupes, and it's how you find "did any run already look at the checkout error spike?" Each summary row carries `run_id`, `skill_name`, `skill_version`, `status`, `started_at`, `completed_at`, `task_url` (a deep-link into the Tasks UI for the full transcript), and the `summary` prose. Lead with the `summary` when narrating to the user — it's the scout's own plain-language close-out — and always offer the `task_url` for the full reasoning. ## Workflow: drill into a single run When the user wants the full story of one run (or pastes a run id / Tasks URL): ```json signals-scout-runs-retrieve { "id": "" } ``` Note the field name flip: `runs-list` returns each run's id as `run_id`, but `runs-retrieve` takes it as `id`. Pass the `run_id` value through as `id`. Returns the full run: `status`, `started_at` / `completed_at` (compute duration from these), `skill_name` / `skill_version` (what ran, at what body version), the end-of-run `summary`, and `task_url`. The transcript — the actual tool calls and reasoning — lives in the Tasks UI behind `task_url`, not in this payload; hand the user that link when they want to see every step. A **failed** run returns an empty `summary` and **no error field** — the payload looks the same as the list row, so to learn _why_ it failed you need the transcript. You don't have to open the UI for that: **`tasks-runs-session-logs-retrieve` returns the run's session log (every tool call, message, and reasoning step) as data** — handy when you're diagnosing a failure or want to trace exactly what a run did without leaving the conversation. Pass the run's `task_run_id` as `id` and its `task_id` (both are on the run row). The raw stream is large (hundreds of KB to a few MB) and will overflow inline, so **fetch it with `call --json` and let it spill to a file**, then run it through [`scripts/render_run_report.py`](#helper-scripts) rather than parsing it by hand. ⚠️ **Do not reach for `exclude_types: "tool_call_update,…"` to slim it down.** It is tempting — the stream is dominated by incremental `tool_call_update` chunks — but each tool's **actual input lives only in those chunks**: the base `tool_call` event carries an empty `rawInput`, and the streamed updates build the input (and the final `rawOutput`) token by token. Excluding them leaves you with tool _names_ but no idea what the scout actually queried. Fetch the **full** log and let the script reassemble each call (it groups by `toolCallId`, keeps the richest `rawInput`, and attaches the completion's `rawOutput`/`status`). **Telling whether a run emitted is not as direct as you'd hope.** The run row carries no emit flag and no finding count — the only readily-available signal is the prose `summary`, which says "EMITTED nothing" on a quiet run and names what it emitted otherwise. Read it carefully: a phrase like "already emitted P1 … did not re-emit" describes a _prior_ run and means this run emitted nothing, so substring-matching the summary for "emitted" is unreliable. Findings do carry a deterministic `source_id = run::finding:`, but it's stored in the signal's `metadata.extra` (not a top-level field) and grouping merges scout findings into the same clusters as other sources, so the `source_product: "signals_scout"` inbox filter does **not** reliably surface them. See [`references/scout-data-model.md`](references/scout-data-model.md) for the run-to-finding link and its limits. A run with `status` complete and an empty-handed summary ("surface at baseline, nothing to emit") is a **healthy** outcome, not a failure — most runs should close out empty. Treat a stream of empty close-outs as the fleet doing its job, not as the fleet being broken. ## Workflow: inspect what the fleet has learned The **scratchpad** is the fleet's durable, per-team memory — prose entries scouts write so future runs get smarter and quieter. Reading it tells you what the fleet believes about this project. ```json signals-scout-scratchpad-search { "text": "error_tracking" } ``` Returns entries newest-first (capped at 100); `text` matches `content` and `key` case-insensitively. Omit `text` to browse everything. Each entry's `key` carries a category prefix that tells you _what kind_ of learning it is: | Prefix | Meaning | | ------------- | ------------------------------------------------------------------ | | `pattern:` | A baseline — how this team's data normally shapes | | `watch:` | A live issue being tracked but still below the emit bar | | `noise:` | A pattern the fleet has decided to ignore (dev-only, single-user…) | | `addressed:` | Something the team fixed or moved on from | | `dedupe:` | A gate on re-emitting a specific issue / fingerprint / finding | | `allowlist:` | Vetted entities never to re-surface | | `not-in-use:` | A product/surface this team doesn't use (close-out memo) | | `mcp-gap:` | A tooling gap a scout noticed worth raising later | This is the common vocabulary, not a closed set — scouts coin their own prefixes and `` labels as needed (the live fleet uses `watch:` heavily, for example), so treat an unfamiliar prefix as just another category. Entries cross-reference each other with `[[key]]` wikilinks. Keys follow `::` (e.g. `dedupe:error_tracking:019e8375-…`). When a user asks "why isn't my scout flagging X anymore?", search the scratchpad for `noise:`, `addressed:`, `dedupe:`, and `allowlist:` entries — the fleet may have deliberately learned to suppress it. The canonical prefix vocabulary and the four-state dedupe classifier the fleet reasons in terms of are documented in [`../authoring-signals-scouts/references/dedupe-and-memory.md`](../authoring-signals-scouts/references/dedupe-and-memory.md). ## Workflow: see what scouts have surfaced Scout findings reach the user as inbox reports. Filter the inbox to the scout source: ```json inbox-reports-list { "source_product": "signals_scout", "limit": 20 } ``` This is the direct way to find scout-backed reports. Each finding is emitted with `source_product="signals_scout"`, that tag rides through grouping into the report's signal metadata, and the inbox filter keeps any report whose contributing signals include `signals_scout` — so the result is the set of reports the fleet has surfaced. An empty result means the fleet hasn't emitted (yet), **not** that the filter is broken. Scouts hold a high bar — most runs close out without emitting — so on a quiet or newly enrolled project zero scout-backed reports is the normal, expected state. Read it as "nothing surfaced," and fall back to each run's `summary` for the per-run record of what was (or wasn't) emitted. To browse the inbox more broadly, use the [`inbox-exploration`](../inbox-exploration/SKILL.md) skill (statuses, suggested reviewers, drilling into a report's underlying signals). The emit contract behind each finding — weight, confidence, severity, the description prose — is documented in [`../authoring-signals-scouts/references/emit-contract.md`](../authoring-signals-scouts/references/emit-contract.md). ## Workflow: assess health and performance "Is my scout actually working / earning its cost?" There's no single metric — judge a scout over a window of runs. Pull the runs (`runs-list` with a `date_from`), then reason across the dimensions below. The full playbook, including how to read each signal and the common failure modes, is in [`references/assessing-performance.md`](references/assessing-performance.md). - **Cadence adherence** — are runs landing roughly every `run_interval_minutes`? Large gaps mean the coordinator is skipping it (disabled, drained from the flag, or capped out on busy ticks) — _or_ it's dispatching but the runs aren't materializing. Tell the two apart with `last_run_at`: if the config's `last_run_at` is also stale, the coordinator stopped planning it; if `last_run_at` is fresh but the newest run row is hours old, it's the dispatch-vs-execution divergence above (workers backed up / down, or runs stranded), which `runs-list` alone hides. - **Success rate** — how many runs reach a clean `status` vs. error out? A run of errors is a broken scout, not a quiet one. - **Emit rate** — what fraction of runs emitted vs. closed out empty. Near-zero over a long window on a live surface can mean the discriminator is too strict (or the surface really is quiet); near-100% usually means it's too noisy. Most healthy scouts emit rarely. - **Signal-to-noise** — of what it emitted, how much became actionable inbox reports vs. got suppressed? Cross-check emitted findings against `inbox-reports-list` report states. - **Memory growth** — a healthy scout accumulates `pattern:` / `noise:` / `dedupe:` entries over time. A scout with an empty scratchpad after many runs isn't learning. ## Helper scripts The skill bundles three **pure formatters** under [`scripts/`](scripts/) for the most common asks. They do **no network I/O** — they are the back half of an "agent fetches, script formats" split. The pattern is always the same: 1. Fetch each payload with the MCP using **`call --json`** (raw JSON, not the pretty text format) and save it to a file. For the big ones (`runs-list`, `tasks-runs-session-logs-retrieve`) this is mandatory anyway — they overflow inline and spill to a file you can point the script at. 2. Run the script over those files. All three are stdlib-only Python 3.11+ and print **plain text** to stdout (or `--out`) — designed to read well in a terminal, so save them as `.txt`. ### `scripts/render_run_report.py` — drill into one run Produces the kind of detailed write-up you'd want when inspecting a single run: header (status, duration, posture), a **narrated timeline that interleaves the agent's narration with each tool call _and its real input_**, the end-of-run summary, and any scratchpad memory. ```bash # fetch (note --json), saving each to a file: # call --json signals-scout-runs-retrieve { "id": "" } -> run.json # call --json tasks-runs-session-logs-retrieve { "id": "", "task_id": "", "offset": 0 } -> log.json (FULL — no exclude_types) # (optional) call --json signals-scout-scratchpad-search { ... } -> mem.json # (optional) call --json signals-scout-config-list {} -> cfg.json python scripts/render_run_report.py --run run.json --log log.json \ --scratchpad mem.json --config cfg.json --out report.txt ``` Modes (`--mode`, default `detailed`): | Mode | Contains | `--log` needed? | | ---------- | ------------------------------------------------------------------ | --------------- | | `summary` | header + posture + close-out prose | no | | `detailed` | + narrated timeline with tool **inputs** + tool tally + scratchpad | yes | | `full` | + each tool call's (truncated) **output** inline | yes | Other flags: `--show-output` (outputs in detailed mode), `--input-width` / `--output-width` (truncation), `--no-art` (skip the hedgehog banner), `--base-url` (defaults to `us.posthog.com`). ### `scripts/fleet_survey.py` — survey the whole fleet One scannable table — scout, enabled, posture, cadence, last run, last outcome — with a "worth a look" section that flags never-run, stuck-in-dry-run, and last-run-failed scouts. ```bash # call --json signals-scout-config-list {} -> cfg.json # (optional) call --json signals-scout-runs-list { "limit": 30 } -> runs.json (small limit!) python scripts/fleet_survey.py --config cfg.json --runs runs.json --now ``` Pass `--now` (the current time, ISO-8601) to get relative "ago" columns; the emit/quiet column is a **heuristic** on each run's summary prose — confirm against the summary before trusting it. ### `scripts/assess_health.py` — health over a window of runs Implements the "assess health and performance" workflow above: a per-scout table (runs, success %, emit %, cadence gap vs interval, adherence, median duration, memory growth) plus a "worth a look" section flagging all-failed scouts, timeout-shaped failures, cadence stalls, staleness, and empty scratchpads. ```bash # call --json signals-scout-runs-list { "limit": 100, "date_from": "" } -> runs.json # (optional) call --json signals-scout-config-list {} -> cfg.json # (optional) call --json signals-scout-scratchpad-search {} -> mem.json python scripts/assess_health.py --runs runs.json --config cfg.json \ --scratchpad mem.json --now [--skill signals-scout-general] ``` `--config` is what lets it score cadence adherence (the expected interval) and staleness (the authoritative `last_run_at`, which the windowed runs can miss when the 100-row cap truncates the newest runs). Without `--scratchpad` the memory column shows `n/a` and no memory flags fire. The emit % is the same summary-prose heuristic — cross-check signal-to-noise against `inbox-reports-list`. ## Tips - **Always surface the `emit` posture.** "Running but in dry-run" is the single most common reason a user thinks a scout is broken when it isn't. - **An empty close-out is success.** Most runs should find nothing. Don't report a wall of clean, empty runs as a problem. - **There's no emit flag to filter on.** Neither the run row nor the inbox exposes a clean "scout-emitted" filter — judge emit-vs-quiet from each run's `summary`, and don't read an empty `source_product: "signals_scout"` inbox result as "the fleet emitted nothing." - **A ~30-min run that `failed` is usually a timeout, not a broken scout.** Completed runs finish in a couple of minutes. Most often the scout over-investigated and ran the full budget (the fleet self-corrects by writing "tight-run recipe" scratchpad entries) — but some are false timeouts where the scout actually finished in a few minutes and the run then hung on a dropped close-out. The session log (above) tells them apart: real over-investigation shows tool calls right up to the wall; a false timeout goes silent long before it. Don't assume over-investigation from duration alone. - **Lead with the run `summary`**, then offer `task_url` for the full transcript — don't dump raw run rows at the user. - **`last_run_at: null`** means a scout has never fired — check it's enabled and the project is enrolled before digging further. - **To explain a quiet scout, read the project profile.** `signals-scout-project-profile-get` shows whether the surface it watches is even in use — a logs scout on a project with no logs has nothing to do. - **This skill is read-only.** To change a scout's schedule, posture, or body, hand off to [`authoring-signals-scouts`](../authoring-signals-scouts/SKILL.md) — it covers `signals-scout-config-update` and the skills-store edit path.