Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Plumbing Verified ≠ Behavior Verified · jpoindexter | DecimalAI

testingby jpoindexter

Plumbing Verified ≠ Behavior Verified

jpoindexter/vd-plumbing-vs-behavior

Source jpoindexter/verified-done ↗License MIT

A passing lower layer or adjacent entry point does not prove the user-facing behavior. Verifying a direct tools/call over stdio is not the agent deciding to call the tool with its persona; a green unit test is not the feature working; a reachable endpoint is not the flow succeeding.

SkillScore91Maintained

Automatic

QA & Test Engineering pack

Last verified 8/3/2026

What it changes

Passed 20 of 22 test cases with the skill, and 9 of 22 without it.

Without skill41%

With skill91%

Checks it passed that the model alone missed

Classifies direct gRPC tool testing as plumbing validation
Classifies query parser unit tests as plumbing verification
States that direct terminal execution measures plumbing only

and 18 more in the full report

11 fixed9 passed either way2 still failing0 broke

Pass-rate lift: +50 pp
Tokens: -10%
Agent turns: 0%
Response time: -31%

Model gemini-3.6-flashMeasured 8/3/2026v120 of 22 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/jpoindexter-vd-plumbing-vs-behavior/SKILL.md https://app.decimal.ai/s/jpoindexter-vd-plumbing-vs-behavior/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add jpoindexter-vd-plumbing-vs-behavior, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/jpoindexter-vd-plumbing-vs-behavior/ and pins it in .decimal/skills.lock

decimalai skills add jpoindexter-vd-plumbing-vs-behavior

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/jpoindexter-vd-plumbing-vs-behavior/SKILL.md https://app.decimal.ai/s/jpoindexter-vd-plumbing-vs-behavior/SKILL.md

Raw file: /s/jpoindexter-vd-plumbing-vs-behavior/SKILL.md ↗ · machine-readable summary at /s/jpoindexter-vd-plumbing-vs-behavior.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 11 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 11

Prompt

Direct calls to the image generation REST API successfully produce PNG files. Is the design assistant's image generation feature verified?

What was checked, without and with the skill
Checked	Without skill	With skill
Identifies REST API direct calls as plumbing verification	Passed	Passed
Requires running the agent prompt loop where the persona decides to generate the image	Missed	Passed

case-17 · gemini-3.6-flash · 8/3/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md26 linessingle file

Plumbing Verified ≠ Behavior Verified

The trap: you prove a component or an adjacent path works and quietly upgrade that to "the feature works." They are different claims with different evidence.

The canonical example

The MCP server's callTool executes the tool directly — no system prompt, no persona, no "agent decides to call the tool."

Hand-sending tools/call audit_site over stdio proves the engine is exposed over MCP. It does not prove the Done criterion — "say 'audit stripe.com', it round-trips and returns the graded report" — because that criterion is the agent taking an instruction and choosing the tool. Different path, different proof.

The layers you might confuse

Unit passes ≠ the units work together (integration).
Endpoint reachable ≠ the flow that calls it succeeds end-to-end.
Tool callable directly ≠ the agent loads its persona and chooses to call it.
Function returns right in isolation ≠ it's wired into the screen the user touches.
Happy path runs ≠ the error paths behave.

How to apply

Write the real user path as a sentence. "User says X → system does Y → user sees Z." That sentence, executed, is the only proof.
Locate where your test entered. If it entered below or beside that path (direct call, mock, lower layer), you tested plumbing — name it as plumbing.
Find the one-shot that runs the whole path. (e.g. vanta run "<instruction>" invokes the real agent loop with persona + tools + provider — that's the true Done test, not the MCP direct call.)
Run that. Then grade it ✅ in the ledger ([[vd-claim-ledger]]); until then it stays ◐.

Governed by [[vd-done-is-executed]]. Pairs with [[vd-run-the-real-path]].

Track record

Benchmark

20 / 22

test cases passed, vs 9 / 22 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/1/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

jpoindexter / verified-done

skills/vd-plumbing-vs-behavior/SKILL.md

View on GitHub ↗

License MITImported 8/1/2026Version v1

Other skills for the same job

Published skills whose descriptions closely match this one, with their headline benchmark lift.

jpoindexter/vd-done-is-executedThe always-on governor against over-claimed completion. Never call work done, fixed, passing, working, wired, or complete unless you RAN the actual Done criterion end-to-end and observed the result with your own eyes. Code that would work, a traced code path, a passing adjacent layer, or "looks right" are NOT done. Applies silently to every completion, success, or "it works" claim; dispatches the sibling vd-* skills.Close match+36 pp pass-rate lift

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

Plumbing Verified ≠ Behavior Verified

The canonical example

The layers you might confuse

How to apply

Track record

Benchmark

Adoption

SkillSafety

Source

Other skills for the same job

Related skills