Trust Me, Bro

Measure the marginal value of one skill. Keep the task and environment fixed; change only the skill condition.

“Trust me” is not a control condition.

Design

1. Name one capability claim. Reject vague claims such as "works better."
2. Pin task inputs, repository revision, model, harness, permissions, tools,

reasoning setting, verifier, and timeout.

1. Select real trigger cases and nearby failures. Start with three to five

tasks for a smoke trial. Require repeated runs before publishing a performance claim.

1. Define conditions:

• baseline: no candidate skill content;
• candidate: the selected skill revision;
• previous: optional prior revision.

1. Read references/trial-format.md for the result

record and blind comparator contract.

Run

1. Randomize condition order.
2. Use a fresh context for each task-condition-trial. Do not let baseline

workers discover the candidate skill or prior conclusions.

1. Give every worker the same task-local context, tools, permissions, and

budget. Give only candidate workers the candidate skill.

1. Capture raw outputs, diffs, logs, timings, token counts when available, and

user interventions. Do not grade from worker self-reports.

1. Prevent live external writes unless the user explicitly authorizes the

exact target. Prefer fixtures, sandboxes, or drafts.

Judge

1. Use deterministic acceptance checks first.
2. For subjective output, anonymize and randomize artifacts before giving

them to a fresh comparator. The comparator receives the original task and rubric, but no condition labels, expected winner, or implementer analysis.

1. Record failures as well as successes. Do not repair candidate output before

grading it.

1. Aggregate JSONL results with:

bash node scripts/summarize-trials.mjs results.jsonl node scripts/summarize-trials.mjs results.jsonl --reference previous --comparison candidate

Decide

• Keep when the gain or risk reduction justifies instruction, token, latency,

and intervention cost.

• Trim and rerun when only part of the skill appears causal.
• Delete or disable when baseline is equal and cheaper.
• Block release when the candidate introduces a safety or correctness

regression.

One run can catch a bug; it cannot establish a stable uplift. State sample size and uncertainty without manufacturing statistical confidence.

Return

Return the pinned configuration, task matrix, condition-level results, paired improvements/regressions, cost delta, raw artifact locations, decision, and known limits. Never market a smoke test as a benchmark.