Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Documentation Audit · aiskillstore | DecimalAI

aiskillstore/documentation-audit

SkillScore74Maintained

This skill should be used when verifying documentation claims against codebase reality. Triggers on "audit docs", "verify documentation", "check docs", "docs accurate", "documentation drift", "before release", "after refactor", "docs don't match". Uses two-pass extraction with pattern expansion for comprehensive detection.

Documentationby aiskillstore

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

Doc Ops pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/aiskillstore-documentation-audit/SKILL.md

decimalai skills pull aiskillstore-documentation-audit --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/aiskillstore-documentation-audit/SKILL.md https://app.decimal.ai/s/aiskillstore-documentation-audit/SKILL.md

Raw file: /s/aiskillstore-documentation-audit/SKILL.md ↗ · machine-readable summary at /s/aiskillstore-documentation-audit.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

23 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 8d ago

Yes — +30% lift+4% tokens0% turns74% of whole cases pass with skill (17/23)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-10	✗→✓	▲ Improved	-39%	0%
case-11	✗→✓	▲ Improved	-21%	0%
case-15	✗→✓	▲ Improved	-8%	0%
case-18	✗→✓	▲ Improved	38%	0%
case-19	✗→✓	▲ Improved	-29%	0%

Showing 5 of 23 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

aiskillstore / marketplace

skills/2389-research/documentation-audit/SKILL.md

View on GitHub ↗

Imported 8/3/2026Version v1

Skill contents

SKILL.md

Documentation Audit

Systematically verify claims in documentation against the actual codebase using a two-pass approach.

Overview

Core principle: Low recall is worse than false positives—missed claims stay invisible.

Two-pass process:

Pass 1: Extract and verify claims directly from docs
Pass 2A: Expand patterns from false claims to find similar issues
Pass 2B: Compare codebase inventory vs documented items (gap detection)

Quick Start

Identify target docs (user-facing only, skip plans/, audits/)
Note current git commit for report header
Run Pass 1 extraction using parallel agents (one per doc)
Analyze false claims for patterns
Run Pass 2 expansion searches
Generate docs/audits/AUDIT_REPORT_YYYY-MM-DD.md

Claim Types

| Type | Example | Verification | |------|---------|--------------| | file_ref | scripts/foo.py | File exists? | | config_default | "defaults to 'AI Radio'" | Check schema/code | | env_var | STATION_NAME | In .env.example + code? | | cli_command | --normalize flag | Script supports it? | | behavior | "runs every 2 minutes" | Check timers/code |

Verification confidence:

Tier 1 (auto): file_ref, config_default, env_var, cli_command
Tier 2 (semi-auto): symbol_ref, version_req
Tier 3 (human review): behavior, constraint

Pass 2 Pattern Expansion

After Pass 1, analyze false claims and search for similar patterns:

Dead script found: diagnose_track_selection.py
  → Search: all script references → Found 8 more dead scripts

Wrong interval: "every 10 seconds"
  → Search: "every \d+ (seconds?|minutes?)" → Found 3 more

Wrong service name: ai-radio-break-gen.service
  → Search: service/timer names → Found naming inconsistencies

Common patterns to always check:

Dead scripts: scripts/*.py references
Timer intervals: every \d+ (seconds?|minutes?)
Service names: ai-radio-*.service, *.timer
Config vars: RADIO_* environment variables
CLI flags: --flag patterns in bash blocks

Output Format

Generate docs/audits/AUDIT_REPORT_YYYY-MM-DD.md:

markdown
# Documentation Audit Report
Generated: YYYY-MM-DD | Commit: abc123

## Executive Summary
| Metric | Count |
|--------|-------|
| Documents scanned | 12 |
| Claims verified | ~180 |
| Verified TRUE | ~145 (81%) |
| **Verified FALSE** | **31 (17%)** |

## False Claims Requiring Fixes
### CONFIGURATION.md
| Line | Claim | Reality | Fix |
|------|-------|---------|-----|
| 135 | `claude-sonnet-4-5` | Actual: `claude-3-5-sonnet-latest` | Update |

## Pattern Summary
| Pattern | Count | Root Cause |
|---------|-------|------------|
| Dead scripts | 9 | Scripts deleted, docs not updated |

## Human Review Queue
- [ ] Line 436: behavior claim needs verification

Detailed References

For execution checklist and anti-patterns: checklist.md For claim extraction patterns: extraction-patterns.md

Want to see if aiskillstore/documentation-audit would activate on your phrasing? Try it before you install — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/21/2026, 18:59 UTC · 23 cases

Lift

+30%

74% vs 43%

Agent turns

1.0 vs 1.0

Tokens

+4%

1,931.91 vs 1,853.41

fail → pass · 7pass → pass · 10fail → fail · 6

Per-case results (23)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 Can you check our user-facing markdown documentation against the repo's actual codebase? We need to find any inaccurate claims like broken file references, outdated default configuration values, or invalid environment variable names. Please generate a full audit report detailing verified facts, false statements with their line numbers and suggested corrections, and a section for items requiring manual review.	fail→fail	1,909	5,730	+200%	1	1	0%	238	1,620	+581%	0	0	—
case-02 We've had several refactors recently and I suspect our documentation is out of sync with our code. Please scan the docs to verify all CLI flags, timer intervals, and script path references against the codebase. If you encounter any broken references or wrong defaults, search across the docs for similar recurring patterns. Put together a markdown report summarizing the document scan results, listing false claims alongside true code realities, and flagging questionable behavior claims for human review.	fail→fail	4,189	4,944	+18%	1	1	0%	251	1,066	+325%	0	0	—
case-03 I need a comprehensive documentation audit before our upcoming release. Please cross-reference all claims in our user guides—such as script usages, default parameters, and environment settings—against the actual implementation in the repository. Skip internal planning docs. Produce a structured audit report summarizing total claims scanned, false claim breakdowns with line numbers and fixes, root cause pattern summaries, and an unverified human review queue.	fail→fail	21,275	5,368	-75%	1	1	0%	3,767	1,212	-68%	0	0	—
case-04 We need to generate HTML API documentation for our Python web application 'FastData' from docstrings in the source files. Should we run an audit verification tool or configure Sphinx with autodoc extensions to build the HTML pages?	pass→pass	10,419	8,010	-23%	1	1	0%	1,896	2,165	+14%	0	0	—
case-05 Our codebase for 'PaymentGateway' was recently refactored to remove the legacy SOAP endpoint, but the docs still mention SOAP support. Can you refactor the Python source code in src/gateway/ to add back the SOAP endpoint so it matches the documentation?	fail→fail	11,303	14,803	+31%	1	1	0%	2,013	2,795	+39%	0	0	—
case-06 We want to publish our static documentation site for the project 'OpenCloud' whenever a push hits main. Please write a GitHub Actions workflow YAML file that builds the documentation and deploys it to GitHub Pages.	pass→pass	10,225	9,689	-5%	1	1	0%	1,878	2,660	+42%	0	0	—
case-07 We are preparing version 2.4.0 of our database tool 'DBMigrate'. Please review our git commit log from the past month and generate a user-facing CHANGELOG.md highlighting new features, bug fixes, and breaking changes.	fail→fail	7,654	9,028	+18%	1	1	0%	1,346	2,302	+71%	0	0	—
case-08 We are preparing to audit docs in the 'AppCore' repository. The repository contains `README.md`, `docs/USER_GUIDE.md`, `plans/roadmap.md`, and `docs/audits/AUDIT_REPORT_2024-01-01.md`. Should we scan all markdown files equally, or filter certain directories during the document selection step?	pass→pass	7,902	3,850	-51%	1	1	0%	1,376	1,558	+13%	0	0	—
case-09 When verifying claims in 'AppCore' user guides, I have four specific types of claims: broken file paths like `scripts/clean.py`, default config settings like `port: 8080`, environment variable names like `DATABASE_URL`, and command flags like `--verbose`. Should these be processed via human review or automated checks?	pass→pass	11,772	4,634	-61%	1	1	0%	1,866	1,755	-6%	0	0	—
case-10 During our documentation audit of the 'AppCore' repository, we identified claims mentioning function names like `process_payload()` and minimum Python versions like `>= 3.10`. How should these symbol references and version requirements be categorized for verification confidence?	fail→pass	14,804	3,312	-78%	1	1	0%	2,435	1,491	-39%	0	0	—
case-11 In `docs/USER_GUIDE.md` for 'AppCore', we found statements like "the worker process retries failed jobs every 3 minutes" and "maximum payload size must not exceed 10MB". A reviewer suggests writing quick automated scripts to mark these true or false immediately. How should behavior and constraint statements be categorized?	fail→pass	16,727	7,742	-54%	1	1	0%	2,676	2,127	-21%	0	0	—
case-12 While auditing `docs/SETUP.md` in 'AppCore', Pass 1 found that `scripts/run_backup.py` no longer exists in the codebase. I am tempted to just fix that single line in `docs/SETUP.md` and move on. What follow-up action should be taken across the documentation before writing the final report?	pass→pass	10,882	8,171	-25%	1	1	0%	1,741	2,316	+33%	0	0	—
case-13 In Pass 1 of our 'AppCore' doc audit, we found a claim stating "syncs every 10 seconds" which turned out to be false because the code uses 60 seconds. What search pattern should be expanded across all docs to find similar timing discrepancies?	pass→pass	12,703	6,752	-47%	1	1	0%	2,197	1,967	-10%	0	0	—
case-14 We finished Pass 1 direct claim extraction for the 'AppCore' repo and verified all explicit documentation statements. Is the audit complete after Pass 1 and pattern expansion, or is there another pass step needed regarding unmentioned codebase elements?	fail→fail	9,257	5,698	-38%	1	1	0%	1,425	1,811	+27%	0	0	—
case-15 We need to run Pass 1 claim extraction on 15 user guide documents in the 'AppCore' project. Should a single sequential process extract claims one document at a time, or how should Pass 1 extraction be structured for efficiency?	fail→pass	14,049	6,653	-53%	1	1	0%	2,115	1,938	-8%	0	0	—
case-16 We are verifying a claim in `docs/DEPLOYMENT.md` stating `STATION_NAME` is used for node identification. To mark this environment variable claim as true during Tier 1 verification, what specific repo files must be checked?	pass→pass	8,405	3,443	-59%	1	1	0%	1,316	1,484	+13%	0	0	—
case-17 In `docs/CLI.md`, a code snippet shows running `python main.py --normalize`. How should this CLI command claim be verified against the codebase?	pass→pass	10,082	6,825	-32%	1	1	0%	1,680	2,058	+23%	0	0	—
case-18 During Pass 1 audit of systemd service docs for 'AppCore', we found `ai-radio-break-gen.service` was misnamed. What pattern search should be executed in Pass 2A to catch related system service discrepancies across the docs?	fail→pass	9,048	6,479	-28%	1	1	0%	1,436	1,985	+38%	0	0	—
case-19 We are running a doc audit on the 'AppCore' repository on October 24, 2024. What exact file path and filename must the audit report be saved to?	fail→pass	10,391	1,980	-81%	1	1	0%	1,657	1,184	-29%	0	0	—
case-20 When building the Executive Summary section for our 'AppCore' audit report, what specific quantitative metrics must be summarized in the summary table?	fail→pass	13,433	4,565	-66%	1	1	0%	2,125	1,538	-28%	0	0	—
case-21 We found recurring root causes across our audit false claims, such as 9 deleted scripts still referenced in docs. Where and how should these recurring root causes be reported in the output document?	fail→pass	12,413	9,018	-27%	1	1	0%	1,842	2,271	+23%	0	0	—
case-22 In `docs/CONFIG.md`, a claim states that the parameter `timeout` "defaults to 300". How should this configuration default claim be verified in Tier 1 auto verification?	pass→pass	11,760	7,900	-33%	1	1	0%	2,005	2,222	+11%	0	0	—
case-23 In the 'AppCore' audit report, how should Tier 3 unverified claims (such as complex behavioral statements) be listed in the human review queue section?	pass→pass	11,779	7,786	-34%	1	1	0%	1,745	2,043	+17%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 23 cases were attempted, and 22 counted toward the lift figure. The other 1 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +30 percentage points is the difference between those two pass rates over the 22 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 43.5%
Pass rate with the skill: 73.9%
Cases attempted: 23
Cases counted toward lift: 22
Cases that regressed: 0
Skill version measured: v1
Run completed: 8/21/2026, 18:59 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-10

Prompt

During our documentation audit of the 'AppCore' repository, we identified claims mentioning function names like `process_payload()` and minimum Python versions like `>= 3.10`. How should these symbol references and version requirements be categorized for verification confidence?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run74

Live evalNeeds ≥20 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+14% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 7 days ago

Other measured skills in the registry, with their headline benchmark lift.