new-golden — decide the split, then build a fixture that can fail

new-golden — decide the split, then build a fixture that can fail

Two different tools, constantly confused:

• Unit test — pure logic, no network, no DB, no model. Fast, runs in CI on every PR, asserts exact behaviour.
• Golden test — an LLM surface (routing, extraction, synthesis, an enrichment pass) against a real model. Opt-in, excluded from CI, slow, asserts structure, not text.

The split

Put in a unit test everything that does not need a model: parsing, tree building, prompt construction (not its answer), digests and cache keys, triviality gates, ordering, rendering, pruning. Most of a pipeline is this, and it should be — pure functions are where cheap certainty lives.

Put in a golden test only what a model's behaviour decides: does retrieval rank the right thing, does the pass produce a well-formed artifact, does the router pick the right agent, does the JSON parse.

If you find yourself wanting a golden to check something deterministic, extract the deterministic part into a pure function and unit-test that. A golden is the most expensive test you own — spend it only where a real model is the thing under test.

The fixture is the test

A fixture that only contains shapes the code already handles proves nothing. This is the single most common way a suite passes while the code is broken: the sample repo/document/payload is small, flat and friendly, and every hard case lives only in production data.

Real example: a documentation pipeline had a small flat fixture repo. Three defects shipped anyway and only surfaced on a 148-class codebase — an aggregate step ran on the wrong size budget (the fixture's inputs were never big enough to overflow), a model swap silently skipped regeneration (the fixture was never re-run under a second model), and a deep nesting chain cost N redundant calls (the fixture was two levels deep).