Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →Before asserting any factual claim about external tool behavior, API surface, library version, framework feature, current event, third-party service, or anything outside the agent's training cutoff or the immediate session context, web-research first. The trained default to "assume what feels right" is one of the most consistent sources of confident fabrication; the structural defense is a verification step at the claim boundary. Use when about to assert anything where being wrong has cost great
| Test case | Without → With | Effect | Δ tokens | Δ turns |
|---|---|---|---|---|
| case-20 | ✗→✓ | ▲ Improved | 137% | 0% |
| case-25 | ✗→✓ | ▲ Improved | 372% | 0% |
| case-01 | ✓→✗ | ▼ Worse | 26% | 0% |
| case-12 | ✓→✗ | ▼ Worse | 50% | 0% |
| case-03 | ✓→✗ | ▼ Worse | 139% | 0% |
The structural defense against confident AI fabrication. When about to assert a factual claim with material consequences if wrong, verify first via web research. The trained default to "assume what feels right" produces consistent fabrication at exactly the moments confidence feels highest — that confidence is the signal to verify, not the signal to proceed.
Trigger conditions:
Skip verification when:
Heuristic: if being wrong would cost more than ~30 seconds, verify. ~30 seconds is the typical verification cost; if backtracking would cost more, the math favors verification by default.
Modern AI models are trained on data that rewards confident-sounding answers. The training signal weakly punishes "I don't know" responses. Result: models hallucinate confidently at exactly the moments they shouldn't — when training data was thin, ambiguous, or stale.
The pattern: the more uncertain the underlying knowledge, the MORE confident the surface phrasing tends to be (compensation effect). When you feel the pull to assert with strong confidence on a factual claim, that's often the signal to verify, not the signal to proceed.
Common fabrication categories:
The structural defense: verify at the claim boundary, before assertion enters the conversation. Once an unverified claim is asserted, downstream work assumes it; backtracking cost compounds.
Notice the moment you're about to assert a factual claim. The notice itself is the architectural work — most fabrication happens because the model didn't pause at the claim boundary. Common surfacing language:
If any of these surface AND the claim is non-trivial, pause before asserting.
Three depths, choose based on stakes:
Light (1-2 minutes): single web search + spot-check authoritative source. Use for routine factual claims where you mostly know but want to confirm.
Medium (3-5 minutes): web search + read official docs / primary source. Use for claims that will inform downstream work (architecture decisions, configuration choices, API integrations).
Deep (5+ minutes): multi-source research, cross-check between sources, possibly test empirically. Use for high-stakes claims that will affect commit / deploy / public artifact.
Execute the search. Read what comes back. Update your understanding based on what's actually true, not what you initially believed.
Common adjustments at this step:
Once verified, assert the claim with appropriate confidence. If verification was partial, qualify with what you actually know vs. what's inferred.
Critical: don't compensate for verification time by over-asserting. "I just spent 3 minutes verifying" is not a license to drop honest uncertainty markers. The verification calibrates confidence, doesn't manufacture it.
The fabrication patterns this catches:
Confidently wrong CLI flags / API parameters. "Use --flag X" when the actual flag is --flag-x or doesn't exist. Cost: user runs the command, gets confusing error, has to debug what you said wrong.
Stale version claims. "Library X version 2.5 supports feature Y" when actually feature Y arrived in version 3.0 or never existed. Cost: user wastes time integrating against wrong expectations.
Plausible-sounding framework features. "Framework X has built-in support for Y" when actually it doesn't (just looks like it could). Cost: user goes hunting for nonexistent documentation.
Hallucinated URLs. Citing docs.example.com/specific-page that doesn't exist. Cost: user clicks broken link.
Hallucinated authors / sources. "According to Jane Smith's 2024 paper on X..." when no such paper exists. Cost: user can't find the cited work; trust degraded.
Wrong organizational facts. "Company X was acquired by Y" when actually the other direction or didn't happen. Cost: user makes business decisions on wrong facts.
The economic argument: ~30 seconds of search at the claim boundary saves minutes-to-hours of downstream backtracking. The expected value is overwhelmingly positive even when most claims would have been right anyway.
Skipping verification because "I'm confident." The confidence is the symptom, not the signal. Highest-confidence assertions are often the most-likely-fabricated (compensation effect from thin training data). Verify confident-feeling claims, especially.
Verification theater. Doing a token search that doesn't actually inform the claim. "Let me verify..." → search "X" → return generic top result → assert original belief. The verification has to actually change behavior when it surfaces contradicting evidence.
Verifying then ignoring. Search returns "actually feature X arrived in version 3.0 not 2.5" — and the response still says "in version 2.5." Read what the search returned. Update.
Over-verification. Verifying claims that are genuinely known from training (Python uses indentation, water boils at 100°C, etc.) wastes time. The heuristic is "would being wrong cost > 30 seconds?" — most fundamental facts don't pass this gate.
Hedging instead of verifying. "I think X but I'm not sure" with no follow-up verification is a worse pattern than either verifying-then-asserting OR honestly saying "I don't know; here's how to find out." Either commit to verification or commit to uncertainty; don't half-do both.
Asserting from search snippets without reading. Top search results may be wrong, outdated, or about a different thing. For non-trivial claims, click through to authoritative source.
If the user has already verified a claim in the current session (e.g., "X library version 3.0 has feature Y, I just checked the docs"), trust that — re-verifying would be insulting and waste time. The session context IS the verification surface for facts the user has already established.
When in doubt: ask the user "have you confirmed this?" rather than silently re-verifying.
This skill is part of a family of patterns that explicitly override RLHF-trained defaults. The trained default rewards:
The override:
This is what makes the skill load-bearing for partnership-quality AI work. The trained default produces confident-fabrication; the override produces verified-truth-or-honest-uncertainty. The partnership compounds on the latter; the trained default degrades it.
This skill is self-contained — mechanics are conceptually simple, value lies in the discipline of actually doing it. Related skills:
flow-cold-read — broader cross-substrate evaluation that includes "verify factual claims" as one dimension among many. Use cold-read when the entire artifact needs review; use verify-before-claiming when a specific factual claim needs verification.flow-drift-detection — same pattern applied to accumulated state rather than fresh claims. Continuity files drift; web-verify catches stale facts during scheduled review.flow-forever-rules — for agents that need to maintain "I verify before claiming" as a forever-rule rather than a context-sensitive heuristic.This pattern is one of FLOW methodology's anti-fragility operations (Op 20 from the Three Layers Underneath Agent Orchestration paper). Graduated to Proven in FLOW after 9+ operational firings across heterogeneous claim classes (library versions, API surfaces, CLI flags, organizational facts, technical documentation, current events). The verification discipline is one of the load-bearing patterns that makes FLOW-style partnership-work substantially more reliable than default-AI-assistance over multi-session work bodies.
Other measured skills in the registry, with their headline benchmark lift.