Install any skill in seconds. Free to start, no credit card required.
Get Started Free →A single agent asked to "find vulnerabilities" will hallucinate. The pipeline decomposes the task into focused steps, with external state carrying the synthesis between steps. All agents run on `model: "inherit"` except flow-tracing and gap-analysis, which pin opus for cross-file reasoning depth.
.claude/skills/sast-methodology/SKILL.md| Test case | Without → With | Effect | Δ tokens | Δ turns |
|---|---|---|---|---|
| case-12 | ✗→✓ | ▲ Improved | — | — |
| case-04 | ✗→✓ | ▲ Improved | — | — |
| case-01 | ✗→✓ | ▲ Improved | — | — |
| case-22 | ✗→✓ | ▲ Improved | — | — |
| case-03 | ✗→✓ | ▲ Improved | — | — |
A single agent asked to "find vulnerabilities" will hallucinate. The pipeline decomposes the task into focused steps, with external state carrying the synthesis between steps. All agents run on model: "inherit" except flow-tracing and gap-analysis, which pin opus for cross-file reasoning depth.
File Ranking (inherit) → comprehension
↓
Entry Point Mapping (inherit) → reading + listing
↓
Dangerous Op Mapping (inherit) → pattern matching
↓
Flow Tracing (pinned Opus) → cross-file reasoning
↓
Gap Analysis (pinned Opus) → interaction reasoning
↓
Devil's Advocate (inherit) → adversarial checking
↓
PoC Confirmation (inherit) → targeted coding + ASan
↓
Exploit Development (inherit) → exploitation techniquesEach agent does ONE thing well. The pipeline does the synthesis.
The SACK bug requires simultaneously understanding: (1) missing bounds check on sack_start, (2) signed integer arithmetic in SEQ_LEQ, (3) linked list behavior when only node deleted, (4) how they interact. No current model reliably holds all four in synthesis.
Decomposed:
Each step is shallow. The pipeline depth is structural.
Deterministic tools (CodeQL, Semgrep, Cppcheck) catch mechanically-detectable patterns. AI catches subtle interaction bugs they miss. Best results come from combining both:
The devil's advocate agent exists because AI hallucination is the #1 cost sink. Hallucinated findings look plausible, reference real functions, describe coherent root causes — and are completely wrong.
The disproval checklist catches:
On score-5 files, running multiple independent hunter instances catches more bugs and filters hallucinations:
| Tool | Use | |---|---| | ASan | Heap/stack overflow, UAF, double-free, OOB | | UBSan | Integer overflow, null deref, alignment | | MSan | Uninitialized memory reads | | TSan | Data races, deadlocks | | Valgrind | When sanitizers unavailable | | GDB | Debugging, register inspection |
bash# C/C++: CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g -O1" # Rust (nightly): RUSTFLAGS="-Z sanitizer=address" cargo +nightly build # Go: go build -race
Opus 4.6 can: rank files, map entry points and dangerous operations, trace simple data flows, find missing bounds checks, write PoCs for straightforward bugs.
Opus 4.6 struggles with: multi-step interaction bugs, novel exploitation techniques, heap feng shui, complex ROP chains, bugs requiring understanding of compiler optimization behavior.
The decomposed pipeline helps because: it turns "find an interaction bug" (hard, fails ~95% of the time) into "map entries" + "map dangers" + "connect them" + "check gaps" (each succeeds ~80% of the time). Pipeline success rate: 0.8^4 ≈ 40% — better than 5%, still not 76% (Mythos).
| Phase | Model | Cost/file | Skip when | |---|---|---|---| | File ranking | inherit | ~$0.05 | Never (cheapest, most important) | | Entry mapping | inherit | ~$0.10 | Score < 3 | | Danger mapping | inherit | ~$0.10 | Score < 3 | | Flow tracing | Opus (pinned) | ~$0.50 | Score < 4 | | Gap analysis | Opus (pinned) | ~$0.50 | No hot/warm flows found | | Devil's advocate | inherit | ~$0.10 | Never (cheapest hallucination filter) | | PoC building | inherit | ~$0.20 | No surviving candidates | | Exploit dev | inherit | ~$0.30 | Severity < medium |
Budget 30 files at min-score 4: scales with the orchestrator model — opus end-to-end runs higher than a mixed pipeline, but removes model-tier handoffs between phases.
| Case | Status | Duration (ms) | Turns | Tokens | Tool calls | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Without | With | Δ | Without | With | Δ | Without | With | Δ | Without | With | Δ | ||
case-17 | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
case-06 | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
case-16 | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
case-21 | pass→pass | — | — | — | — | — | — | — | — | — | — | — | — |
case-15 | pass→pass | — | — | — | — | — | — | — | — | — | — | — | — |
case-11 | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
case-07 | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
case-08 | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
case-12 | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
case-19 | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
case-14 | pass→pass | — | — | — | — | — | — | — | — | — | — | — | — |
case-02 | pass→pass | — | — | — | — | — | — | — | — | — | — | — | — |
case-20 | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
case-04 | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
case-01 | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
case-22 | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
case-18 | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
case-03 | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
case-05 | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
case-09 | fail→fail | — | — | — | — | — | — | — | — | — | — | — | — |
case-10 | pass→pass | — | — | — | — | — | — | — | — | — | — | — | — |
case-13 | fail→pass | — | — | — | — | — | — | — | — | — | — | — | — |
DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted, and 20 counted toward the lift figure. The other 2 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +27 percentage points is the difference between those two pass rates over the 20 comparable cases. 2 cases got worse with the skill loaded, and they are included in that figure.
The per-case answers from this run were removed by the retention sweep, so the case table below shows the verdicts without the text either arm produced. The counts above were recorded at the time and are unaffected. Answers are now kept for 180 days.
Other measured skills in the registry, with their headline benchmark lift.