Schema Evolution Compatibility

Reviews a proposed change to a schema that other people's code already reads — a warehouse table queried by dashboards, an event on a shared topic, a message format between services, a dataset in a lake. The goal is a change that ships without breaking any existing consumer, and that lets each consumer upgrade on its own schedule instead of being forced to redeploy in lockstep with the producer.

A schema is a contract, not private state. Once anything else reads it, its shape is a promise. The base model knows compatibility rules but does not apply them by default: asked to "add a field" or "rename a column" it tends to hand back the direct one-step change. This skill makes the compatibility checks mandatory before the change ships.

This is about the reader-vs-writer contract, not the database operation. Table locks, downtime, and rollback safety of the migration itself are a separate review; so is semver on a function or REST signature. Here the only question is: does an existing consumer break?

When to Activate

• We're adding a field to an event / message / topic that other teams consume
• Is it safe to rename this column that dashboards and jobs query?
• Will changing this field's type break the services reading it?
• We stopped writing this column — can we drop it now?
• How do we roll out this payload change without breaking older consumers?
• What compatibility mode should our schema registry enforce on this subject?

The One Default: Additive-Only

Prefer changes that only add, and make every added field optional — nullable, or carrying a non-volatile default. An optional added field is the one change that is safe in every direction: old consumers ignore it, old producers omit it, old records simply lack it. Everything else needs the checks below.

The Compatibility Checks

Run the change through all of these. A "no" is a finding.

1. Additive, and optional? Is this only adding fields, each optional? Adding a

required field is breaking: old producers won't populate it and old records don't have it, so any reader validating against the new schema fails on existing data. Making an existing optional field required is the same break. Fix: add it optional; require it only after every producer emits it and history is backfilled.

1. Any rename? A rename is a drop plus an add to every consumer — the instant the

name changes, all readers of the old name break. Never rename in place. Add the new name alongside, populate both (dual-write), move readers over, then retire the old name later.

1. Any narrowing of a type or its domain? Widening is usually safe; narrowing breaks

readers. Narrowing includes: shrinking a type (string→int, long→int), making a nullable field non-null, removing an allowed enum value, tightening a length/range/precision, and splitting one field into several. Same type but changed meaning or units is also a silent break — dollars→cents, local-time→UTC, an id that starts pointing at a different entity. Treat a meaning change as a new field, not an edit to the old one.

1. Reusing a retired identifier? Never recycle the slot of a removed field for new

semantics. In Protobuf, never reuse a field number (or tag) — reserved it so no one can. In Avro, field name and position carry meaning; don't repoint them. In JSON or columnar data, don't give an old key a new meaning. A stale consumer will read new data into the old field and be silently wrong.

1. Removing without a deprecation window? Don't delete a field the moment you stop

writing it — a consumer may still read it. Mark it deprecated, keep emitting/keeping it, confirm from telemetry that no consumer reads it across a communicated window, then remove. Consumers upgrade on their timeline; a same-day drop breaks the slow ones.

1. Is the contract versioned, with a compatibility mode? A shared schema needs an

explicit version — an envelope schemaVersion, a versioned topic/subject, a registry entry — and a declared compatibility mode the registry enforces (below). Changing a shape with no version and no mode is how a silent break ships.

1. Do consumers read tolerantly? The reason additive changes are safe is that consumers

ignore unknown fields instead of failing closed. If a consumer crashes on a new field, that is the consumer's bug (fix it to skip unknowns) — but until every consumer reads tolerantly, treat even additive changes as risky and stage them.

Direction Decides Who Breaks

The same change is safe or breaking depending on who upgrades first. Name the direction.

| Change | Who breaks | Safe rollout | |---|---|---| | Add optional field | no one | ship in any order | | Add required field | old producers + old data on a strict reader | backfill + upgrade all producers first, then require | | Remove a field | consumers still reading it | stop all consumers reading it first, then remove (deprecate-then-remove) | | Rename | every reader of the old name, immediately | add-new + dual-write + migrate readers + drop-old | | Narrow type / domain | consumers relying on the old range | add a new field with the new type; deprecate the old | | Change meaning/units, same type | every consumer, silently | new field; never edit in place |

Compatibility Mode ↔ Safe Change

A registry compatibility mode is a promise about upgrade order. Pick it from who upgrades independently, then only allow the changes it permits.

• BACKWARD — new schema can read data written by the old schema. Lets consumers

upgrade after producers. Allows: add optional field, remove a field. Default for a stream where consumers lag.

• FORWARD — old schema can read data written by the new schema. Lets producers

upgrade first, consumers later. Allows: add a field, remove an optional field.

• FULL — both. Allows only: add/remove optional fields. Use for a widely shared

contract with many independent consumers.

For a shared topic with consumers you don't control, default to BACKWARD or FULL and reject anything the mode forbids.

Exact per-format rules — Protobuf field-number and reserved discipline, Avro default and type-promotion requirements, JSON-Schema required/additionalProperties, Parquet/columnar add-at-end and by-name reads — are catalogued in references/format-evolution-rules.md.

Output

Present findings as a table, then a verdict.

| # | Severity | Check failed | Finding | Safe alternative | |---|----------|--------------|---------|------------------| | 1 | 🔴 Block | Rename | In-place column rename breaks every dashboard selecting the old name at once | Add new column, dual-write, migrate readers, drop old later | | 2 | 🟡 Warn | Deprecation window | Field dropped the same week writes stopped; a reader may remain | Deprecate, confirm no reads over a window, then remove |

Verdict: COMPATIBLE / STAGE FIRST / BREAKING — one line naming the single biggest reason.

Edge Cases

• No existing consumers yet (new schema, or a private one only you read): compatibility

doesn't apply — change it freely. Don't over-stage a schema nobody else consumes.

• Adding an enum value to a field consumers switch on: safe only if consumers have a

default/unknown branch (tolerant reader). Otherwise the new value breaks an exhaustive switch — gate on how consumers handle unknowns.

• Widening is not always free: int→long is safe for a long-schema reader reading int

data, but a producer writing long breaks an int-schema reader. Check the direction, not just "wider."

• Coordinated single-consumer upgrade: if you own the only consumer and can deploy both

sides atomically, some staging collapses — but a change to a shared/streamed contract still needs the full discipline because replayed history and lagging consumers persist.

Evaluation Criteria

A good execution of this skill should:

• ] Default to additive-only with optional/defaulted new fields, and say so explicitly
• ] Catch in-place renames as immediate breaks and give the add-new/dual-write/drop-old staging
• ] Flag type narrowing AND same-type meaning/unit changes as breaking, not just obvious type swaps
• ] Require deprecate-then-remove over a window instead of an immediate drop
• ] Name the upgrade direction (who breaks, who upgrades first) rather than a flat safe/unsafe
• ] Call for an explicit contract version and a compatibility mode on a shared schema
• ] Refuse to reuse a retired field number/identifier