--- name: romainsimon/learning-from-outcomes source: https://app.decimal.ai/s/romainsimon-learning-from-outcomes@1/SKILL.md source_sha256: 9abd35e2a87d --- # Learning from outcomes Experience does not automatically produce judgment. Without a written prediction and a scoring step, a review produces a story in which the good outcomes were foreseen and the bad ones were unforeseeable. That story feels like learning and teaches nothing. Three things make a review compound: 1. **Score the predictions**, not the outcomes. 2. **Assign credit across time**, because results arrive long after the decisions that caused them. 3. **Replay old decisions deliberately**, because recent experience otherwise overwrites older lessons entirely. ## Workflow ``` - [ ] 1. Collect the predictions that came due - [ ] 2. Score them - [ ] 3. Separate decision quality from outcome quality - [ ] 4. Assign credit for delayed results - [ ] 5. Replay two old decisions that are not due - [ ] 6. Update the priors that were wrong ``` ### 1. Collect what came due Pull every prediction from the decision log whose date has passed. Record the outcome as a plain binary before discussing anything, and before anyone explains why it happened. If there is nothing to collect, the problem is upstream: decisions are being recorded without falsifiable predictions. Fix that in **framing-decisions** first; nothing here works without it. ### 2. Score them **Run the calculator. Do not compute these by hand.** ```bash node scripts/calc.js calibrate predictions.json ``` Brier decomposition, bin assignment and the base-rate benchmark are exactly the kind of arithmetic that comes out subtly wrong when done in prose, and a miscalibration verdict derived from a wrong number is worse than no verdict. Paste the real output into the review. Input shape: `examples/calibrate.json`. Four numbers matter: - **Brier score** - mean squared error of your probabilities. Lower is better. - **Always-base-rate** - what you would have scored by ignoring specifics and always quoting the base rate. **If your Brier score is not beating this, your judgment is adding nothing** and the honest response is to quote base rates until it does. - **Verdict** - overconfident, underconfident, or well calibrated. - **The bin table** - of the things you called 80%, how many happened? Overconfidence concentrates in the high bins, which is where it does the most damage. Twelve predictions is enough to see a pattern; five is not. Score quarterly at minimum, and keep the whole history rather than the current window. ### 3. Decision quality is not outcome quality For each item, place it in the grid before discussing what to change: | | Good outcome | Bad outcome | |---|---|---| | **Good process** | earned it | bad luck - change nothing | | **Bad process** | got away with it - fix anyway | the only cell that deserves blame | The two diagonal cells are where reviews go wrong. Punishing bad luck teaches people to avoid variance rather than to decide well. Rewarding a lucky call teaches the room to copy a bad process. The decision record makes this decidable rather than rhetorical: if the record shows the advantage of second place was under a percent, the decision was a coin flip and the outcome carries almost no information about the process. ### 4. Assign credit for delayed results Results arrive months after the decisions that caused them, and the discussion systematically credits whatever shipped most recently. ```bash node scripts/calc.js credit result.json --lambda 0.7 ``` Input shape: `examples/credit.json`. It decays credit by `lambda` per period back through the decisions that preceded the result. `lambda = 0.7` is a reasonable default; use higher when effects are slow (SEO, brand, hiring) and lower when they are fast (a pricing change, an ad creative). The output is a prompt for a better argument, not an answer. Its job is to put the six-month-old decision back on the table so someone has to argue about it explicitly rather than forget it. ### 5. Replay two old decisions Pick two decisions from six or more months ago that are **not** due for review, and re-read them. Ask: does the reasoning still look right, knowing what you know now? This is deliberate replay of old experience, and it exists to counter a specific failure: learning purely from recent events overwrites older lessons, so a system that only reviews what is current will confidently repeat a mistake it already learned about. Almost every long-running organisation has a lesson it has learned three times. Keep a short list of replayed decisions so you rotate through the log rather than re-reading the same memorable ones. ### 6. Update the priors that were wrong **A lesson that does not change a number will not change a decision.** State that, then act on it: the output of a review is not a list of lessons, it is a set of **changed numbers**, each written into a named file that will actually be read next time. "We should be more careful about X" survives one quarter. "The prior on X is 0.3, in `runbooks/traffic-drops.md`" survives indefinitely. Name the destination for every number. A revised estimate with no home is a lesson wearing a number's clothes. Numbers worth changing: - The prior in a diagnosis set (**tracking-beliefs**) - The reference-class estimate for how long something takes - The likelihood assigned to a study method (**valuing-information**) - Your calibration adjustment: if you are systematically overconfident at 80%, shade future 80% claims to 70% until the bin table says otherwise A lesson that does not change a number will not change a decision. Write the number. ## Output template The review's output is a diff, not minutes. If something does not fit one of these sections, it did not need a meeting. ```markdown ## Predictions that came due | Prediction | stated P | outcome | |---|---|---| ## Calibration Skill versus base rate: ## Process versus outcome | Decision | process | outcome | cell | action | |---|---|---|---|---| ## Credit for delayed results ## Decisions replayed ## Numbers changed | number | was | now | lives in | evidence | |---|---|---|---|---| ## Scenarios added - -> robustness matrix for ``` ## Gotchas - **Score predictions, not outcomes.** A review of outcomes is a review of luck. - **Never rewrite a prediction after the fact.** Append the outcome. The value of the log is that it preserves what you actually believed, including the embarrassing parts. - **Overconfidence is the normal finding.** If your 90% claims come true 70% of the time, that is typical, correctable, and worth more than any single lesson in the review. - **Watch the proxy metric.** Any metric used as a target stops measuring what it measured. If activation rate became a goal and then rose without revenue following, the metric was optimised rather than the outcome. Check the downstream number. - **A result with one obvious cause is suspicious.** Real results have several contributing decisions. If the room agrees instantly on a single cause, run the credit assignment and see who else is on the list. - **Do not review only failures.** Successes with bad process are the most dangerous cell in the grid, because nobody wants to examine them and the process survives. - **Small samples justify simple explanations.** Three data points do not support an elaborate causal story, however satisfying it is. - **Attribution windows lie.** A 30-day attribution window credits whatever the customer touched last. The credit tool exists precisely because that is not causation. ## Reference - `references/review-format.md` - the monthly and quarterly review agendas, with time boxes - `references/calibration.md` - reading the bin table, fixing overconfidence, how many predictions you need ## Related skills - **framing-decisions** writes the records this skill grades - **tracking-beliefs** receives the corrected priors - **stress-testing-plans** converts a past failure into a reusable scenario - **valuing-information** receives the corrected likelihoods for study methods