---
name: jgravelle-benchmark-methodology
description: "Rules for measuring or quoting any benchmark number in this repo. Load before running the bench tier, writing a delta into a PR or CHANGELOG, or touching anything under benchmarks/."
license: "NOASSERTION"
metadata:
  decimalai-skill-id: "8ba789e0-6854-4198-a95b-8aba0a3062b7"
  decimalai-version-id: "ff9836d2-9b90-4e81-93ca-4477f0f28108"
  decimalai-version: "1"
  decimalai-body-sha256: "da1e2f2360194fa9ada913d8cc1044738e948c8a276ebe559d7ee9a0e0b1c3c5"
  decimalai-name: "jgravelle/benchmark-methodology"
  decimalai-format: "1"
  decimalai-invocation: "model"
  decimalai-category: "testing"
  decimalai-source: "https://app.decimal.ai/s/jgravelle-benchmark-methodology@1/SKILL.md"
---
# Benchmark methodology

Authority: `docs/harness/ARCHAEOLOGY.md` (the benchmark rows),
`benchmarks/METHODOLOGY.md`, `benchmarks/REPRODUCING.md`, CLAUDE.md
Practice 4, `docs/harness/FINDINGS.md` F-10, F-13, F-17.

Rules, each a pointer, none restated:
1. **Never hand-type a number.** Every figure comes from a run in THIS
   session or the CI-captured artifact `benchmarks/jcm_reference.json`
   (Practice 4). The comparators read that file; there is no estimator.
2. **Per row, never per total** (F-13): the total hid one cause behind
   another. Diff threshold ids one by one.
3. **Five mirrors move together** (Practice 4): `results.md`,
   `METHODOLOGY.md`, README, `provenance/measured.json`, `REPRODUCING.md`;
   `tests/test_provenance.py` is the gate and `--reference` rewrites them.
4. **The reference is captured where the gate runs** (F-13):
   `benchmark.yml` dispatch with `reference=true`; a dev-box number differs
   by CRLF, walk order and a HOME ledger (F-17).
5. **Deterministic configuration** is the bench tier itself: `--offline`,
   pinned corpora (`benchmarks/tasks.json`), the no-network fixture. Add no
   flags.
6. **A refusal is not a zero**: an absent value prints `n/a`.
7. **Floors live only in `harness/thresholds.json`**; read one with
   `uv run python -m harness threshold <id>`.
8. `benchmarks/schema_baseline.json` is the only source for schema-token
   figures (CLAUDE.md "Tier-switch pricing"); the harness counts a
   different payload and the two never agree digit for digit.