Polish one detection rule

Goal: improve one existing rule on both axes — catch real-world variants it currently misses (recall), and stop firing on benign content it should never have flagged (precision) — without losing true positives. One rule, one PR.

Precision is not a secondary concern here. A rule can ship with false positives from day one: it was written against synthetic fixtures, and the corpus was only ever checked for regressions after a change, never audited for what the rule was already getting wrong. So every polish cycle starts by looking at what the rule actually does to 777 real skills — before touching anything.

Rules are data: pkg/rules/packs/*.yaml, compiled via //go:embed. Regex is Go RE2 — no lookaround/backreferences ((?<, (?=, (?!) — they won't compile. Read docs/rule-verification.md for the detection approach and confidence math behind each rule.

Guardrails

Generated attack payloads are inert test data only — never execute them. All the global guardrails from sg-maintain apply.

1. Pick the rule

Load state from .claude/maintenance/state.json → rule_last_polished. List the rule IDs across the packs:

sh
grep -h 'id: SG-' pkg/rules/packs/*.yaml

Choose the rule with the oldest (or missing) rule_last_polished timestamp. If a caller named a specific rule, use that instead.

2. Understand its current coverage

• Open the rule's YAML block: its match tree, confidence, suppress list, targets.
• Read its section in docs/rule-verification.md (Signals / FP carve-outs / Fixtures).
• Note what it already matches and, importantly, its documented false-positive carve-outs —

new test cases must respect these, and step 3 will tell you whether they actually hold in the wild.

• Check whether a previous cycle already recorded an FP audit for this rule (the "Corpus precision"

note step 8 writes). If so, that is your baseline: look for what changed, not the same ground.

3. Audit its real false positives on the corpus

The evaluation/ corpus is ~777 real, unlabeled skills. Almost none of them are attacks, so essentially every hit the rule produces there is a false-positive candidate until you read it. This step is what catches an FP the rule has carried since the day it shipped.

Make sure the raw reports reflect current main (they are git-ignored, so they may be stale or absent). Regenerate only if needed — a full run is ~9 minutes:

sh
go build -o skill-guard ./cmd/skill-guard
CORPUS_DIRS="clawhub anthropic orgs skillsmp" evaluation/scripts/run_scans.sh
python3 evaluation/scripts/aggregate.py

Then pull every hit for your rule:

sh
evaluation/scripts/rule_findings.py <RULE-ID>            # summary + a sample
evaluation/scripts/rule_findings.py <RULE-ID> --all      # every hit, to judge exhaustively

Read the output for three things:

• Repeated excerpts. The tool tallies them because a repeat is almost always one systematic

cause, not N independent mistakes — fix the cause and the whole cluster goes.

• Concentration. 50 hits in 2 bundles is a different problem from 50 hits across 50 authors.

Heavy concentration in security-tooling skills (denylists, jailbreak catalogs, safe-exec-style docs) is the known benign-but-flagged class: those files genuinely contain attack strings.

• Target file type. Hits on .md prose vs .py/.sh code often need different fixes — the

confidence modifiers already treat those registers differently (docs/rule-verification.md §1.2).

Now judge each hit — true positive, false positive, or ambiguous — and write the tally into your working notes. Be honest about ambiguity: skill-guard's stated reading is "capability and pattern, not confirmed intent", so a skill that really does ship a pipe-to-shell installer is a true positive even if its author meant well. A false positive is a match on something that is not the pattern — a license phrase, a variable name, prose describing the attack rather than committing it.

If the rule has zero corpus hits, say so and move on: there is no precision evidence either way, and widening in step 6 must then be extra conservative since nothing will catch over-matching.

Propose the narrowing

For each FP cluster, pick the narrowest mechanism that kills it without touching true positives:

| FP shape | Mechanism | |---|---| | One recurring benign phrase (an MIT-license clause; a delete scoped to a variable) | suppress: regex on that line | | Match is real code but in the wrong register (prose about an attack) | rely on / adjust the documentary modifier; check the target list | | Leaf is too generic (a bare command name; a vague "applies to everything" phrase) | tighten the regex — require an argument, a flag, a command position | | Right pattern, wrong file kind | narrow targets: | | Signal is genuinely weak on its own | drop its confidence below the 0.5 emit threshold, or pair it in an all: composite |

Prefer suppress and tighter leaves over lowering severity — severity drives the verdict contract, and muting a real signal to fix a display problem is the wrong trade. If an FP cannot be fixed without losing true positives, say so explicitly and leave it, with the reasoning in the PR. A known, documented FP beats a silent recall loss.

4. Generate realistic attack cases

Write 5–10 new payloads that a real attacker/skill might use for this rule's threat class, as close to observed real-world threats as possible — paraphrases, spacing/casing variants, and plausible obfuscations the current pattern may not reach. Draw on:

• the OWASP AST class the rule maps to,
• variants seen in the evaluation/ corpus scan output,
• public write-ups (via sg-threat-research notes if available).

Also include negative cases: benign near-misses that must NOT match, mirroring the rule's carve-outs (e.g. documentation phrasing). This is what keeps polishing from causing false positives.

Seed the negatives from step 3. Every false positive you confirmed on the corpus is a real-world negative case, already proven to occur in the wild — use the actual excerpt, not an invented paraphrase. These are worth more than any negative you could make up, and turning them into tests is what stops the FP coming back.

Keep the literal payload strings inside fenced code blocks in your working notes so they stay inert and don't trip the scanner on this skill itself.

5. Add the cases as tests

Two harnesses (see pkg/rules/rules_test.go and pkg/scan/scan_test.go):

• Rule-level table test (fast, isolated) — the model is TestInjectionOverrideCoversParaphrase

in pkg/rules/rules_test.go: fetch the rule by ID from Builtin(), then a table of {text string; want bool} cases evaluated with rule.Evaluate("body", c.text). Add a Test<Rule>Cover… function (or extend the existing one) with your new {payload, true} and {near-miss, false} rows.

• Fixture pipeline test (only when target assignment / line mapping matters) — add the snippet

to testdata/malicious/SKILL.md and assert the rule ID appears in the scan findings, following TestMaliciousFails in pkg/scan/scan_test.go.

Confirmed corpus false positives go in as {excerpt, false} rows in the same table, so the recall and precision cases are pinned by one test and can't drift apart.

Run them:

sh
go test ./pkg/rules/ ./pkg/scan/ -run <YourTest> -v

6. Tune the match tree — widen where it misses, narrow where it over-matches

If a realistic payload slips through, extend the rule's match tree in the pack YAML:

• Add or broaden a regex/substring leaf, or add an any-branch alternative.
• Prefer a new alternative over rewriting a working pattern — smaller blast radius.
• Set a per-pattern confidence appropriate to how specific the signal is.
• Add a suppress entry for any benign phrasing your change now over-matches.
• Keep RE2-compatible (no lookaround). Confirm it compiles: go test ./pkg/rules/ runs

TestBuiltinPacksLoad, which fails on a bad pattern.

If step 3 found false positives, apply the narrowing you proposed there in the same edit, so the widening and the tightening are measured together rather than one masking the other.

A cycle where the rule already catches everything and the only change is an FP fix is a complete, successful polish — precision work needs no widening to justify it. Equally, if the rule is clean on both axes, make no change: record the audit result, update state.json, and skip the PR. Do not invent a widening to have something to ship.

Re-run the tests until every want:true matches and every want:false doesn't.

7. Guard against regressions

• go test ./... — full suite green.
• Exit-code smoke: go run ./cmd/skill-guard scan testdata/malicious → exit 1;

... scan testdata/benign → exit 0.

• If you changed a pack, re-run the corpus and measure the change on both axes. You already

have the step-3 numbers as your before; save them first so the comparison is real: sh cp evaluation/reports/stats.json /tmp/stats_before.json go build -o skill-guard ./cmd/skill-guard CORPUS_DIRS="clawhub anthropic orgs skillsmp" evaluation/scripts/run_scans.sh python3 evaluation/scripts/aggregate.py evaluation/scripts/rule_findings.py <RULE-ID> --all # what survives Then state, for this rule: hits before → after, which FP clusters are gone, and which true positives are still caught. A drop in the count is only good if you can name what left. Compare against evaluation/reports/CROSS_VERIFICATION.md — a "fix" must not silently drop detections. (Note: evaluation/ is git-ignored; regeneration is a local sanity check, not part of the PR.)

• Check the other rules' counts too: a shared leaf, a suppress line, or a target change can

move a rule you weren't touching. Any non-zero delta outside your rule is either explained or reverted.

• Dogfood: go run ./cmd/skill-guard scan .claude/skills/sg-rule-polish still passes.

8. Open the PR

sh
git checkout main && git pull --ff-only && git checkout -b polish/<rule-id>-<slug>
git add pkg/rules/ testdata/ docs/rule-verification.md
git commit -m "fix(rules): <widen|narrow> <RULE-ID> — <what>"
git push -u origin HEAD
gh pr create --label automated --label rule-polish \
  --title "fix(rules): polish <RULE-ID> — <short>" \
  --body "..."

The PR body must report both axes with numbers, since that is the evidence a reviewer needs:

• Precision: corpus hits before → after, each FP cluster you removed (with a real excerpt) and

the mechanism used, plus any FP you deliberately left and why.

• Recall: the new real-world payloads now caught, and confirmation that previously-caught true

positives still are — named, not just counted.

• Note that these are real, unlabeled skills, so the FP judgments are your reading of them; a

reviewer may disagree with a specific call and the PR should make that easy to check.

Also record the audit in docs/rule-verification.md under the rule's section — the FP carve-outs there are what the next polish cycle reads in step 2, so an unrecorded audit gets repeated.

Update state.json → set rule_last_polished["<RULE-ID>"] to now. Report the PR link.

If the audit found the rule clean on both axes, there is no PR: record the result in the cycle log and state.json so the next cycle picks a different rule, and report that outcome instead.