Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Benchmark methodology · jgravelle | DecimalAI

testingby jgravelle

Benchmark methodology

jgravelle/benchmark-methodology

Source jgravelle/jcodemunch-mcp ↗License NOASSERTION

Rules for measuring or quoting any benchmark number in this repo. Load before running the bench tier, writing a delta into a PR or CHANGELOG, or touching anything under benchmarks/.

SkillScore82Maintained

Automatic

QA & Test Engineering pack

Last verified 10/6/2026

What it changes

Passed 16 of 19 test cases with the skill, and 3 of 19 without it.

Without skill14%

With skill82%

Checks it passed that the model alone missed

Formats missing or absent values as lowercase n/a.
Names tests/test_provenance.py as the test gate file.
Names benchmarks/tasks.json as the pinned corpus file.

and 22 more in the full report

15 fixed3 passed either way4 still failing0 broke

Pass-rate lift: +68 pp
Tokens: -14%
Agent turns: 0%
Response time: -55%

Model gemini-3.6-flashMeasured 10/6/2026v118 of 22 cases passed with skill · 19 comparable

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/jgravelle-benchmark-methodology/SKILL.md https://app.decimal.ai/s/jgravelle-benchmark-methodology/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add jgravelle-benchmark-methodology, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/jgravelle-benchmark-methodology/ and pins it in .decimal/skills.lock

decimalai skills add jgravelle-benchmark-methodology

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/jgravelle-benchmark-methodology/SKILL.md https://app.decimal.ai/s/jgravelle-benchmark-methodology/SKILL.md

Raw file: /s/jgravelle-benchmark-methodology/SKILL.md ↗ · machine-readable summary at /s/jgravelle-benchmark-methodology.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 14 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 14

Prompt

I am looking for the authoritative document that defines the specific benchmark row specifications and findings in this project. Which file in the harness documentation serves as the primary authority for benchmark rows?

What was checked, without and with the skill
Checked	Without skill	With skill
Identifies docs/harness/ARCHAEOLOGY.md as the authority for benchmark rows.	Missed	Passed

case-18 · gemini-3.6-flash · 10/6/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md27 linessingle file

Benchmark methodology

Authority: docs/harness/ARCHAEOLOGY.md (the benchmark rows), benchmarks/METHODOLOGY.md, benchmarks/REPRODUCING.md, CLAUDE.md Practice 4, docs/harness/FINDINGS.md F-10, F-13, F-17.

Rules, each a pointer, none restated:

Never hand-type a number. Every figure comes from a run in THIS session or the CI-captured artifact benchmarks/jcm_reference.json (Practice 4). The comparators read that file; there is no estimator.
Per row, never per total (F-13): the total hid one cause behind another. Diff threshold ids one by one.
Five mirrors move together (Practice 4): results.md, METHODOLOGY.md, README, provenance/measured.json, REPRODUCING.md; tests/test_provenance.py is the gate and --reference rewrites them.
The reference is captured where the gate runs (F-13): benchmark.yml dispatch with reference=true; a dev-box number differs by CRLF, walk order and a HOME ledger (F-17).
Deterministic configuration is the bench tier itself: --offline, pinned corpora (benchmarks/tasks.json), the no-network fixture. Add no flags.
A refusal is not a zero: an absent value prints n/a.
Floors live only in harness/thresholds.json; read one with uv run python -m harness threshold <id>.
benchmarks/schema_baseline.json is the only source for schema-token figures (CLAUDE.md "Tier-switch pricing"); the harness counts a different payload and the two never agree digit for digit.

Track record

Benchmark

16 / 19

test cases passed, vs 3 / 19 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v5 — 9/9/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

jgravelle / jcodemunch-mcp

.claude/skills/benchmark-methodology/SKILL.md

View on GitHub ↗

License NOASSERTIONImported 9/9/2026Version v1

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

Benchmark methodology

Track record

Benchmark

Adoption

SkillSafety

Source

Related skills