Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →Audit/regression checks for the evidence-first survey pipeline: citation health, per-section coverage, placeholder leakage, and template repetition. **Trigger**: auditor, audit, regression test, quality report, 审计, 回归测试. **Use when**: `output/DRAFT.md` exists and you want a deterministic PASS/FAIL report before LaTeX/PDF.
.claude/skills/willoscar-pipeline-auditor/SKILL.md| Test case | Without → With | Effect | Δ tokens | Δ turns |
|---|---|---|---|---|
| case-04 | ✗→✓ | ▲ Improved | 17% | 0% |
| case-05 | ✗→✓ | ▲ Improved | 219% | 0% |
| case-07 | ✗→✓ | ▲ Improved | 18% | 0% |
| case-09 | ✗→✓ | ▲ Improved | 158% | 0% |
| case-10 | ✗→✓ | ▲ Improved | 43% | 0% |
Purpose: a deterministic “regression test” for the writing stage.
It answers:
This skill is analysis-only. It does not edit content. For all survey-family profiles, style/citation-shape violations are blocking by default.
output/DRAFT.mdoutline/outline.ymloutline/evidence_bindings.jsonlcitations/ref.biboutput/AUDIT_REPORT.mdoutput/TEMPLATE_RESIDUE_SCORECARD.jsonA150++ citation targets (used by the auditor):
Course-paper targets:
..., …), TODO markers, scaffold tags.outline/outline.yml.course_paper, >=2 for survey/deep (inserted by section-merger from outline/tables_appendix.md; index tables remain internal).This subsection ..., In this subsection ...)Next, we move ..., We now turn to ...)this run, this workspace)this pipeline, this stage, and quality gateblock only when the same sentence contains a Harness anchor such as a checkpoint, Unit ID, Harness lock, attempt ledger, or template residue; ordinary subject-matter uses remain non-blocking warnings
survey synthesis/comparisons should ....Taken together, ... and similar high-signal generator stems.citations/ref.bib exists): undefined keys, duplicates, basic formatting red flags.[@a] [@b]) and no duplicate keys inside one block ([@a; @a]). Mid-sentence citation ratio is >=20% for course_paper and >=30% for survey/deep.outline/evidence_bindings.jsonl exists): citations used per H3 should stay within the bound evidence set.output/FRONT_MATTER_CONTEXT.json to record the selected front-matter assets and hashes, then require the three template-owning Skill implementations to match .harness/harness.lock.json; missing provenance, legacy locks, and repository drift block acceptance.The JSON scorecard records the measured ratio, counts, threshold, selected asset hashes, heading-aware examples, and implementation-lock result. During normal Harness execution, Completion projects its verdict and dimensions into .harness/evaluations/ledger.jsonl, including failed Attempts, so Run Audit can expose the latest measurement instead of reducing it to PASS/FAIL. The scorecard file remains the complete evidence object; the ledger is intentionally smaller. The current 10% limit is an initial policy target. The published Survey replay completes the current 31-check contract at 0/226 residue, establishing attainability for one retained Artifact set. Clean from-scratch execution, unrelated topics, and cross-profile calibration remain open.
Treat output/AUDIT_REPORT.md as a “what to fix next” router.
Common FAIL families -> responsible stage/skill:
subsection-briefs / evidence-draft / writer-context-pack, then rewrite affected sections.table-schema + appendix-table-writer produced outline/tables_appendix.md (>=1 course-paper table; >=2 survey/deep tables; citation-backed, no placeholders), then rerun section-merger.transition-weaver (and ensure briefs include bridge_terms / contrast_hook), then re-merge.sections/S*.md via writer-selfloop (local, section-level) or subsection-polisher.draft-polisher or local section rewrites).section-mapper → evidence-binder) and regenerate packs.citation-diversifier → citation-injector (NO NEW FACTS), then draft-polisher.sections/S<sec_id>.md front-matter file via writer-selfloop (front-matter path) using dense positioning + method paragraph.If you want the auditor to PASS without a heavy polish loop:
uv run python .codex/skills/pipeline-auditor/scripts/run.py --helpuv run python .codex/skills/pipeline-auditor/scripts/run.py --workspace <workspace>--workspace <dir>--unit-id <U###> (optional; for logs)--inputs <semicolon-separated> (rare override; prefer defaults)--outputs <semicolon-separated> (rare override; defaults write output/AUDIT_REPORT.md and output/TEMPLATE_RESIDUE_SCORECARD.json)--checkpoint <C#> (optional)global-reviewer and before LaTeX/PDF:uv run python .codex/skills/pipeline-auditor/scripts/run.py --workspace <workspace>Fix:
citation-verifier and ensure citations/ref.bib contains every cited key.Fix:
sections/* files, then re-merge.Fix:
citation-diversifier to produce output/CITATION_BUDGET_REPORT.md.citation-injector (edits output/DRAFT.md, writes output/CITATION_INJECTION_REPORT.md).draft-polisher → global-reviewer → auditor.| Case | Status | Duration (ms) | Turns | Tokens | Tool calls | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Without | With | Δ | Without | With | Δ | Without | With | Δ | Without | With | Δ | ||
case-01 | fail→fail | 19,867 | 5,034 | -75% | 1 | 1 | 0% | 4,334 | 2,292 | -47% | 0 | 0 | — |
case-02 | fail→fail | 23,368 | 4,922 | -79% | 1 | 1 | 0% | 4,534 | 2,244 | -51% | 0 | 0 | — |
case-03 | fail→fail | 16,991 | 5,483 | -68% | 1 | 1 | 0% | 3,364 | 2,152 | -36% | 0 | 0 | — |
case-04 | fail→pass | 17,739 | 12,548 | -29% | 1 | 1 | 0% | 3,231 | 3,793 | +17% | 0 | 0 | — |
case-05 | fail→pass | 6,357 | 10,771 | +69% | 1 | 1 | 0% | 1,101 | 3,507 | +219% | 0 | 0 | — |
case-06 | fail→fail | 5,165 | 5,872 | +14% | 1 | 1 | 0% | 248 | 2,123 | +756% | 0 | 0 | — |
case-07 | fail→pass | 10,409 | 2,904 | -72% | 1 | 1 | 0% | 2,010 | 2,377 | +18% | 0 | 0 | — |
case-08 | pass→pass | 5,777 | 2,104 | -64% | 1 | 1 | 0% | 826 | 2,374 | +187% | 0 | 0 | — |
case-09 | fail→pass | 4,848 | 2,204 | -55% | 1 | 1 | 0% | 900 | 2,321 | +158% | 0 | 0 | — |
case-10 | fail→pass | 7,870 | 3,027 | -62% | 1 | 1 | 0% | 1,570 | 2,250 | +43% | 0 | 0 | — |
case-11 | fail→pass | 12,751 | 5,510 | -57% | 1 | 1 | 0% | 1,707 | 2,862 | +68% | 0 | 0 | — |
case-12 | fail→pass | 4,321 | 3,045 | -30% | 1 | 1 | 0% | 732 | 2,472 | +238% | 0 | 0 | — |
case-13 | fail→pass | 8,158 | 3,285 | -60% | 1 | 1 | 0% | 1,187 | 2,526 | +113% | 0 | 0 | — |
case-14 | pass→pass | 8,495 | 1,937 | -77% | 1 | 1 | 0% | 1,468 | 2,254 | +54% | 0 | 0 | — |
case-15 | fail→pass | 11,464 | 2,398 | -79% | 1 | 1 | 0% | 1,699 | 2,300 | +35% | 0 | 0 | — |
case-16 | fail→pass | 11,054 | 2,290 | -79% | 1 | 1 | 0% | 1,861 | 2,326 | +25% | 0 | 0 | — |
case-17 | fail→pass | 11,860 | 2,879 | -76% | 1 | 1 | 0% | 1,530 | 2,406 | +57% | 0 | 0 | — |
case-18 | fail→pass | 27,975 | 1,889 | -93% | 1 | 1 | 0% | 2,323 | 2,260 | -3% | 0 | 0 | — |
case-19 | fail→pass | 8,052 | 2,546 | -68% | 1 | 1 | 0% | 1,232 | 2,346 | +90% | 0 | 0 | — |
case-20 | pass→pass | 9,734 | 5,129 | -47% | 1 | 1 | 0% | 1,495 | 2,772 | +85% | 0 | 0 | — |
case-21 | fail→pass | 13,172 | 7,323 | -44% | 1 | 1 | 0% | 2,095 | 3,261 | +56% | 0 | 0 | — |
case-22 | fail→pass | 8,870 | 2,128 | -76% | 1 | 1 | 0% | 1,381 | 2,288 | +66% | 0 | 0 | — |
DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted, and 18 counted toward the lift figure. The other 4 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +68 percentage points is the difference between those two pass rates over the 18 comparable cases.
Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.
Other measured skills in the registry, with their headline benchmark lift.