Eval Harness First

Eval Harness First

The Phase 0 gate for the whole plugin: finetuning-method-selection and every downstream skill assume this harness exists before a training config gets written. The harness is not a run-end side artifact — it is the data-curation engine. The same labeled traces that build the goldens feed training data, minus an explicit holdout.

Input: production/agent traces if they exist, or a task spec if they don't, plus labelers willing to grade ≥100 examples. Output format: the eval/ directory below — goldens, graders, drift suite, and the base-model baseline that later phases gate on.

The Gate

No eval harness, no fine-tune. Skip to a training config and there is nothing to measure against, nothing to catch regressions, and no labeled data to train on. The flywheel:

1. Collect traces — production/agent spans, or synthetic tasks if none exist yet.
2. Error analysis — open coding on ≥100 traces, axial coding into 4–8 failure buckets.
3. One grader per bucket — deterministic first; calibrated LLM-judge only for genuinely subjective criteria.
4. Prioritize by frequency × severity × value.
5. The labeled traces feed dataset curation, minus an explicit holdout. Every eval/goldens.jsonl ID stays excluded from training data by ID.
6. Train.
7. Re-run the same harness on the checkpoint — not a different, looser one.
8. Drift detection feeds back to step 2 — new production failure modes re-open error analysis.