--- name: schizo16/ai-claim-audit source: https://app.decimal.ai/s/schizo16-ai-claim-audit@1/SKILL.md source_sha256: 0ccff5cb2dfc --- # ai-claim-audit --- ```yaml name: ai-claim-audit description: Audit AI-generated answers for unsupported claims, overconfident language, and weak evidence compatibility: opencode >= 1.0.0 ``` --- ## Purpose This skill reviews AI-generated text and identifies when the AI makes claims without supporting evidence, or sounds overconfident despite weak justification. It produces a structured claim ledger that separates what the AI asserts from what it assumes, infers, or treats as hypothetical — without consulting external sources or labeling claims as true or false. --- ## When to Activate Activate when the user asks to audit, review, or check an AI answer for evidence quality, unsupported claims, or overconfident language: - "audit this answer for evidence quality" - "check this AI response for unsupported claims" - "review confidence vs evidence in this answer" - "identify assumptions and weakly supported claims" - "analyze the claim types in this text" Do **not** activate for: - Requests to fact-check against external sources or the web - Requests to verify whether a claim is true or false - Requests to rewrite or improve the text --- ## Core Rule **Do not verify claims against external sources.** This skill analyzes the text itself — it assesses whether evidence is present in the provided text, not whether the claim is factually correct. --- ## Workflow ### 1. Ingest Text Accept input as inline text, a file path, or a block identified in the conversation. **Output format:** ``` ## Input **Source:** **Text length:** ``` ### 2. Segment into Claims Parse the input text into individual claim units — sentences or clauses that assert something. Group closely related claims where useful. **Output format:** ``` ## Segmented Claims | # | Claim text | Source context | |---|------------|---------------| | 1 | | | | 2 | | | ``` ### 3. Classify Claim Type Assign one of six categories to each claim: - `factual_claim` — asserts a verifiable fact (dates, statistics, definitions, events, historical statements) - `reasoning_claim` — asserts a logical or causal relationship ("X leads to Y", "because of Z") - `assumption` — takes something for granted without justification - `inference` — draws a conclusion from premises - `opinion` — subjective statement, not verifiable - `hypothetical_or_rhetorical` — "what if", "imagine", rhetorical questions — flag separately, do not grade **Output format:** ``` ## Claim Types | # | Claim text | Type | |---|------------|------| | 1 | | factual_claim | | 2 | | inference | ``` ### 4. Grade Evidence Status Assign one of five evidence statuses to each claim. Only grade `factual_claim`, `reasoning_claim`, `assumption`, and `inference`. Opinions and hypotheticals receive `not_applicable`. - `supported_in_text` — the text provides direct evidence, quoted source, data, concrete example, or explicit reasoning that supports the claim - `weak_support` — the text gives some reasoning or example, but it is incomplete, indirect, or not enough for the strength of the claim - `needs_evidence` — the claim may be plausible, but the text provides no citation, data, example, source, or explicit support - `unsupported` — the claim is presented as established or certain, but the text provides neither evidence nor reasoning, or the claim uses broad/absolute language without support - `not_applicable` — used for opinions, hypotheticals, rhetorical questions, or subjective preferences **Output format:** ``` ## Evidence Status | # | Claim text | Claim type | Evidence status | |---|------------|------------|----------------| | 1 | | factual_claim | supported_in_text | | 2 | | inference | weak_support | ``` ### 5. Flag Overconfidence Scan each claim for strong certainty markers: "definitely", "always", "proven", "undoubtedly", "it is certain that", "without a doubt", "clearly", "obviously". Cross-reference against evidence status. Flag a claim as overconfident only when strong certainty language is paired with `weak_support`, `needs_evidence`, or `unsupported`. Do **not** flag appropriately hedged language ("may", "suggests", "could", "might", "possibly"). **Output format:** ``` ## Overconfidence Flags | # | Claim text | Confidence language | Evidence status | Flagged | |---|------------|---------------------|----------------|---------| | 1 | | "undoubtedly" | unsupported | YES | | 2 | | "suggests" | weak_support | no | ``` ### 6. Compile Claim Ledger Produce a consolidated claim ledger with all findings. **Output format:** ``` ## ai-claim-audit-ledger | # | Claim text | Claim type | Evidence status | Confidence language | Overconfidence flag | Recommendation | |---|------------|------------|----------------|---------------------|--------------------|----------------| | 1 | | factual_claim | supported_in_text | none | no | — | | 2 | | inference | weak_support | "clearly" | YES | strengthen with direct evidence | ### Summary - **Total claims:** - **By type:** factual_claim: , reasoning_claim: , assumption: , inference: , opinion: , hypothetical_or_rhetorical: - **Unsupported:** | **Needs evidence:** | **Weak support:** - **Overconfidence flags:** - **Risk level:** low / moderate / high ``` --- ## Skill Name Rules - Use lowercase with hyphens: `ai-claim-audit` - Match the directory name to the skill name - Use a noun phrase describing what the skill does - Avoid verbs as the first word - Maximum 4 words - Must be unique within the project's skill directory --- ## Quality Audit Rubric | Criterion | Definition | |-----------|------------| | **Metadata present** | YAML or metadata block with name, description, compatibility | | **Purpose clear** | Single paragraph explaining why the skill exists and when it activates | | **Workflow complete** | All workflow steps documented with clear transition conditions | | **Output formats defined** | Every workflow step has a documented output format | | **Do-not rules present** | Explicit prohibitions on dangerous or unwanted behaviors | | **Trigger clarity** | Trigger pattern precise enough to avoid false positives | | **No assumptions as facts** | Skill does not assert untested assumptions as established facts | | **Safety constraints** | Limits on destructive operations, external writes, sensitive data exposure | | **Naming convention followed** | Name follows the Skill Name Rules | A skill must pass all criteria to be considered complete. --- ## Do-Not Rules - Do not verify claims against external sources or browse the web - Do not mark claims as "true" or "false" - Do not mark claims as "verified" — only assess whether evidence is present in the text - Do not rewrite or modify the input text - Do not apply domain-specific knowledge not configured by the user - Do not treat hypotheticals or rhetorical questions as evidence-gradable claims — grade as `not_applicable` - Do not flag overconfidence for appropriately hedged language ("may", "suggests", "could", "might", "possibly") - Do not invent evidence that is not present in the text - Do not output CLI exit codes - Do not install packages or modify project configuration --- ## Safety / Boundaries **Read access:** - The input text provided by the user (inline, paste, or file path) **Write access:** - Only the report file: `ai-claim-audit-ledger.md` — written to the current working directory or user-specified path **Prohibited:** - Network calls or web browsing - Accessing external APIs or databases - Modifying source files or input text - Installing packages or dependencies **Gating:** - Any operation beyond static text analysis requires user confirmation --- ## Response Style - Be direct and concise - Label each workflow step clearly - Use structured output formats (headings, tables, lists) — not prose paragraphs - When grading evidence, include the specific text that constitutes the evidence (or state that none was found) - When flagging overconfidence, quote the certainty language that triggered the flag - When uncertain, state the limitation rather than guessing