--- name: test-oracle-design description: "Use when you need to test code whose correct output is hard to state directly — a large or complex result, a value that varies by input, a computation with no easy-to-write expected answer — and the reflex is to paste whatever the code returned once as the expected value. Chooses a test-oracle strategy (golden-master/snapshot, metamorphic relation, property/invariant, or cross-reference to a trusted implementation) by reasoning about WHY the output is hard to state. Not for choosing which input values to test, writing the test framework wiring, or triaging a flaky test." metadata: decimalai-skill-id: "33d1a45b-1805-4234-a0ec-df9428cf421c" decimalai-version-id: "2d3d7ff9-6f10-481c-96f2-9595bcf55d4a" decimalai-version: "1" decimalai-body-sha256: "75ebb55bb7b6de456a8af1f16a16544e6732d4e57b5c5282b9a8b49e9f182202" decimalai-name: "test-oracle-design" decimalai-format: "1" decimalai-invocation: "model" decimalai-category: "testing" decimalai-source: "https://app.decimal.ai/s/test-oracle-design@1/SKILL.md" --- # Test-oracle design A test needs an **oracle**: a way to decide whether the output is right. When the correct output is easy to write down (`add(2, 3) == 5`), the oracle is trivial and there's nothing to design. This skill is for the other case — when stating the expected output directly is impractical: the result is a 300-line rendered report, a value that shifts with every input, an optimized computation whose answer you can't work out by hand, output that includes volatile fields. The failure mode this replaces is **the pasted expected value**: run the code once, copy whatever it returned into `assert result == `, and move on. That assertion is a snapshot nobody reviewed. It passes because the code produced it, not because it's correct — it bakes in whatever bugs already existed, and when it breaks it says only "output changed," never "output is wrong." A value you cannot independently justify is not an oracle. The move is to pick a real oracle strategy by asking **why** the output is hard to state, then assert the thing you actually know to be true instead of a value you merely observed. ## Pick the oracle from why the output is hard to state | Why you can't just write the expected output | Oracle strategy | |---|---| | The output is large/complex but stable, and you trust the current behavior (e.g. refactoring) | **Golden-master / snapshot** | | You have no trustworthy reference, but you know how the output must change when the input changes | **Metamorphic relation** | | The exact value varies, but there are structural truths every valid output must satisfy | **Property / invariant** | | A slower or independent implementation you trust already computes the answer | **Cross-reference / differential** | These combine. A hard-to-test function often gets a few invariants, one or two metamorphic relations, and a differential check against a naive version — each catches a different class of bug. ## Golden-master / snapshot Capture the current output as a reference, then assert future runs match it byte-for-byte. Use it when the output is too big to hand-write but you trust the behavior you have now — the classic case is **refactoring**: you're changing how the code works and want proof the result didn't change. Do it right, or it becomes the pasted-value anti-pattern with tooling: - **Review the master once.** A snapshot preserves behavior; it does not verify correctness. If you approve it without reading it, you've frozen whatever bugs are in there. Read the first capture. - **Normalize volatile fields before comparing.** Timestamps, random ids, ordering that isn't guaranteed, absolute paths — mask or sort them, or the test fails every run for reasons that aren't the code. A snapshot that's red half the time trains people to re-approve blindly. - **Regenerate deliberately, never reflexively.** When a snapshot breaks, the question is "is this change intended?" — not "update it so CI goes green." Re-approving on autopilot deletes the oracle. ## Metamorphic relation When you can't say what `f(x)` is, you can often say how `f` of a *transformed* input relates to `f(x)`. Assert that relation instead of the value. This gives you an oracle with no reference output at all — you never state a single correct answer, only a rule two runs must jointly obey. Examples of the shape: - `sort(shuffle(xs))` equals `sort(xs)` — permuting the input doesn't change the sorted result. - A search with an added filter returns a **subset** of the unfiltered results. - Scaling every input price by 1.1 scales the computed total by 1.1. - `translate(translate(text, en→fr), fr→en)` is close to the original — a round-trip relation. - Adding a strongly-matching document to a ranked search never pushes it *below* a weak one. Choose relations the code doesn't satisfy trivially — one that would actually break if the logic were wrong. "The output is still a list" is not a metamorphic relation worth its line. ## Property / invariant The exact output varies, but every valid output obeys structural truths regardless of input. Assert those. Common invariants: - **Round-trip:** `decode(encode(x)) == x`, `deserialize(serialize(obj)) == obj`. Tests the pair without ever stating the intermediate form. - **Idempotence:** `f(f(x)) == f(x)` — dedup, normalize, canonicalize. - **Conservation / bounds:** output is a permutation of the input (same elements); a summary's mean lies between the min and max; a total equals the sum of its parts; nothing is negative. - **Postconditions:** the result is sorted; no duplicates remain; no PII appears in the output. An invariant that pins the exact bytes is over-specified — if `decompress(compress(x)) == x` holds, don't *also* assert `compress(x)` equals a specific captured byte string, or a library upgrade that changes the encoding (while round-tripping fine) breaks a test for no real defect. Test the truth you care about, not an incidental encoding of it. Pair properties with **generated/varied inputs** — a property asserted on one hand-picked case is barely stronger than a single example. ## Cross-reference / differential When a second implementation you trust already produces the answer, run both and compare. The trusted oracle can be too slow, too simple, or too indirect to ship — it only has to be correct: - A **brute-force** version checks an optimized algorithm (naive O(n²) vs the fast one; BFS vs a tuned pathfinder). - A **reference library** or a previous, trusted version of your own code checks a port or rewrite. - Another language's standard library checks your reimplementation. The one rule: the reference must be **genuinely independent**. Diffing the code against a copy of itself, or against an oracle derived from the same buggy assumption, proves nothing — both agree on the wrong answer. ## The through-line Assert what you can independently justify: a reviewed reference, a relation between runs, an invariant that must hold, or agreement with a trusted second implementation. Never assert a value whose only credential is that the code under test produced it.