---
name: schema-evolution-compat
source: https://app.decimal.ai/s/schema-evolution-compat@1/SKILL.md
source_sha256: 18cf99450f66
---

# Schema Evolution Compatibility

Reviews a proposed change to a schema that other people's code already reads — a warehouse
table queried by dashboards, an event on a shared topic, a message format between services,
a dataset in a lake. The goal is a change that ships without breaking any existing consumer,
and that lets each consumer upgrade on its own schedule instead of being forced to redeploy
in lockstep with the producer.

A schema is a **contract**, not private state. Once anything else reads it, its shape is a
promise. The base model knows compatibility rules but does not apply them by default: asked
to "add a field" or "rename a column" it tends to hand back the direct one-step change. This
skill makes the compatibility checks mandatory before the change ships.

This is about the **reader-vs-writer contract**, not the database operation. Table locks,
downtime, and rollback safety of the migration itself are a separate review; so is semver on
a function or REST signature. Here the only question is: **does an existing consumer break?**

## When to Activate

- We're adding a field to an event / message / topic that other teams consume
- Is it safe to rename this column that dashboards and jobs query?
- Will changing this field's type break the services reading it?
- We stopped writing this column — can we drop it now?
- How do we roll out this payload change without breaking older consumers?
- What compatibility mode should our schema registry enforce on this subject?

## The One Default: Additive-Only

Prefer changes that only **add**, and make every added field **optional** — nullable, or
carrying a non-volatile default. An optional added field is the one change that is safe in
every direction: old consumers ignore it, old producers omit it, old records simply lack it.
Everything else needs the checks below.

## The Compatibility Checks

Run the change through all of these. A "no" is a finding.

1. **Additive, and optional?** Is this only adding fields, each optional? Adding a
   **required** field is breaking: old producers won't populate it and old records don't
   have it, so any reader validating against the new schema fails on existing data. Making an
   existing optional field required is the same break. Fix: add it optional; require it only
   after every producer emits it and history is backfilled.

2. **Any rename?** A rename is a drop **plus** an add to every consumer — the instant the
   name changes, all readers of the old name break. Never rename in place. Add the new name
   alongside, populate both (dual-write), move readers over, then retire the old name later.

3. **Any narrowing of a type or its domain?** Widening is usually safe; narrowing breaks
   readers. Narrowing includes: shrinking a type (string→int, long→int), making a nullable
   field non-null, removing an allowed enum value, tightening a length/range/precision, and
   splitting one field into several. **Same type but changed meaning or units is also a
   silent break** — dollars→cents, local-time→UTC, an id that starts pointing at a different
   entity. Treat a meaning change as a new field, not an edit to the old one.

4. **Reusing a retired identifier?** Never recycle the slot of a removed field for new
   semantics. In Protobuf, never reuse a field **number** (or tag) — `reserved` it so no one
   can. In Avro, field name and position carry meaning; don't repoint them. In JSON or
   columnar data, don't give an old key a new meaning. A stale consumer will read new data
   into the old field and be silently wrong.

5. **Removing without a deprecation window?** Don't delete a field the moment you stop
   writing it — a consumer may still read it. Mark it deprecated, keep emitting/keeping it,
   confirm from telemetry that no consumer reads it across a communicated window, **then**
   remove. Consumers upgrade on their timeline; a same-day drop breaks the slow ones.

6. **Is the contract versioned, with a compatibility mode?** A shared schema needs an
   explicit version — an envelope `schemaVersion`, a versioned topic/subject, a registry
   entry — and a declared compatibility mode the registry enforces (below). Changing a shape
   with no version and no mode is how a silent break ships.

7. **Do consumers read tolerantly?** The reason additive changes are safe is that consumers
   **ignore unknown fields** instead of failing closed. If a consumer crashes on a new field,
   that is the consumer's bug (fix it to skip unknowns) — but until every consumer reads
   tolerantly, treat even additive changes as risky and stage them.

## Direction Decides Who Breaks

The same change is safe or breaking depending on who upgrades first. Name the direction.

| Change | Who breaks | Safe rollout |
|---|---|---|
| Add **optional** field | no one | ship in any order |
| Add **required** field | old producers + old data on a strict reader | backfill + upgrade all producers first, then require |
| Remove a field | consumers still reading it | stop all consumers reading it first, then remove (deprecate-then-remove) |
| Rename | every reader of the old name, immediately | add-new + dual-write + migrate readers + drop-old |
| Narrow type / domain | consumers relying on the old range | add a new field with the new type; deprecate the old |
| Change meaning/units, same type | every consumer, silently | new field; never edit in place |

## Compatibility Mode ↔ Safe Change

A registry compatibility mode is a promise about upgrade order. Pick it from who upgrades
independently, then only allow the changes it permits.

- **BACKWARD** — new schema can read data written by the old schema. Lets **consumers**
  upgrade after producers. Allows: add optional field, remove a field. Default for a stream
  where consumers lag.
- **FORWARD** — old schema can read data written by the new schema. Lets **producers**
  upgrade first, consumers later. Allows: add a field, remove an optional field.
- **FULL** — both. Allows only: add/remove **optional** fields. Use for a widely shared
  contract with many independent consumers.

For a shared topic with consumers you don't control, default to **BACKWARD** or **FULL** and
reject anything the mode forbids.

Exact per-format rules — Protobuf field-number and `reserved` discipline, Avro default and
type-promotion requirements, JSON-Schema `required`/`additionalProperties`, Parquet/columnar
add-at-end and by-name reads — are catalogued in **references/format-evolution-rules.md**.

## Output

Present findings as a table, then a verdict.

| # | Severity | Check failed | Finding | Safe alternative |
|---|----------|--------------|---------|------------------|
| 1 | 🔴 Block | Rename | In-place column rename breaks every dashboard selecting the old name at once | Add new column, dual-write, migrate readers, drop old later |
| 2 | 🟡 Warn | Deprecation window | Field dropped the same week writes stopped; a reader may remain | Deprecate, confirm no reads over a window, then remove |

**Verdict:** COMPATIBLE / STAGE FIRST / BREAKING — one line naming the single biggest reason.

## Edge Cases

- **No existing consumers yet** (new schema, or a private one only you read): compatibility
  doesn't apply — change it freely. Don't over-stage a schema nobody else consumes.
- **Adding an enum value** to a field consumers switch on: safe only if consumers have a
  default/unknown branch (tolerant reader). Otherwise the new value breaks an exhaustive
  switch — gate on how consumers handle unknowns.
- **Widening is not always free**: int→long is safe for a long-schema reader reading int
  data, but a producer writing long breaks an int-schema reader. Check the direction, not
  just "wider."
- **Coordinated single-consumer upgrade**: if you own the only consumer and can deploy both
  sides atomically, some staging collapses — but a change to a shared/streamed contract still
  needs the full discipline because replayed history and lagging consumers persist.

## Evaluation Criteria

A good execution of this skill should:
- [ ] Default to additive-only with optional/defaulted new fields, and say so explicitly
- [ ] Catch in-place renames as immediate breaks and give the add-new/dual-write/drop-old staging
- [ ] Flag type narrowing AND same-type meaning/unit changes as breaking, not just obvious type swaps
- [ ] Require deprecate-then-remove over a window instead of an immediate drop
- [ ] Name the upgrade direction (who breaks, who upgrades first) rather than a flat safe/unsafe
- [ ] Call for an explicit contract version and a compatibility mode on a shared schema
- [ ] Refuse to reuse a retired field number/identifier