· 8 min read FundamentalsArchitecture

Schema Evolution: The Field You Removed Is Still in Someone's Queue

Backward, forward and full compatibility — what each permits, why required fields are a trap, and the rules that keep a rolling deploy safe.

Schema evolution during a rolling deploy: old and new service versions run simultaneously, so a message written by a new writer is read by an old reader and vice versa, which is why only full compatibility is safe mid-rollout

The deploy went out at 14:20. Nothing broke. The alert fired at 02:40 the next morning, from a consumer that had been down for maintenance and had just come back to a queue with eleven hours of backlog in it.

That backlog was written by the new code. The consumer was still the old code. And somewhere in the middle of it was a message it could not parse, which it retried, and failed, and retried.

The schema change had been reviewed. It looked harmless. It was harmless at the moment it deployed, which is the part that fools people — because a schema change isn’t a change to your code. It’s a change to a contract with every piece of data that already exists and every process that hasn’t caught up yet.

Two directions, and you need both

Compatibility has a direction, and the naming confuses nearly everyone the first time. The direction is stated from the new schema’s point of view.

Backward compatible — new code can read old data. You can deploy readers first. This is the one people check, because it’s the one that matters for the data sitting in your database right now.

Forward compatible — old code can read new data. You can deploy writers first. This is the one people forget, and it’s the one that bit the consumer at 02:40.

Full compatibility is both.

Here’s why you need both, even though it feels like overkill. During a rolling deploy there is a window — minutes if you’re lucky, hours if a canary is baking, days if some consumer is paused — where v1 and v2 are both running. Both are writing. Both are reading. A message written by v2 will be read by v1, and a message written by v1 will be read by v2, and you do not control which.

So “is this change backward compatible?” is the wrong question. The question is: can old and new run side by side, in both directions, for as long as the slowest consumer takes to catch up? If the answer is no, you don’t have a schema change. You have a coordinated flag day, and you should know that before you merge, not after.

The specific moves, and what each one costs

Let’s use Protocol Buffers for the examples since the field-number model makes the hazards visible. Avro and JSON Schema have the same shape with different mechanics; I’ll note where they diverge.

Adding an optional field — safe

message Order {
  string id    = 1;
  int64  cents = 2;
  string currency = 3;   // new
}

Old readers see an unknown field number and skip it. New readers see an absent field and get the default — empty string here. Both directions work.

The catch is that the default is doing real work and nobody writes it down. An empty currency is not the same as USD, but half your code will treat it that way because every message it has seen so far was USD. Pick the default deliberately and say what it means in a comment, because that comment is the only place the semantics will live.

One caution specific to proto3: a scalar set to its zero value is indistinguishable from an unset field on the wire. cents = 0 and “no cents field” decode identically. If you need to tell “zero” from “not provided” — and for money you almost always do — use optional (which generates a presence bit) or a wrapper type. This is a genuine foot-gun and it has caused real financial bugs.

Removing a field — safe only if you reserve it

message Order {
  string id    = 1;
  int64  cents = 2;
  string currency = 3;
  reserved 4;
  reserved "legacy_total";
}

Deleting the line is not enough. Six months from now someone adds a field, the next free number looks like 4, and now every archived message and every replayed event that carried the old legacy_total gets decoded as whatever type the new field claims to be. Sometimes that throws. Worse, sometimes it doesn’t.

reserved makes the compiler refuse. It costs one line and removes an entire category of incident. Reserve the name too — Avro and JSON-shaped formats key on names, and someone will eventually regenerate your schema from a JSON sample.

Renaming a field — not what you think it is

In Protobuf, a rename is free on the wire, because the wire format only carries numbers. It is not free anywhere else: your JSON mapping changes, your BigQuery column changes, your logs change, anything doing reflection changes.

In Avro and JSON Schema, a rename is a remove plus an add, and you need an alias or a dual-write period to survive it.

So the honest answer to “can I rename this field?” is: it depends on which part of the stack you mean, and the answer is probably different for the wire, the warehouse, and the dashboard.

Changing a type — you can’t, really

There is no safe in-place type change. Not int to string, not int32 to int64 in every case, not enum to string.

What you do instead is a four-step migration, and the reason it’s four steps is that every one of them is independently reversible:

  1. Add currency_code (the new field) alongside currency (the old one).
  2. Write both. Readers still prefer the old one. Deploy. Let it soak.
  3. Switch readers to prefer the new field, falling back to the old. Deploy. Soak again.
  4. Stop writing the old field. Wait a full retention period — not a sprint, a retention period — then reserve the number.

Step 4 is where people get impatient, and it’s the step that determines whether you’ve actually migrated or just deferred the breakage. If your Kafka topic retains seven days and your S3 archive retains seven years, those are two different clocks and the archive wins.

Required fields: don’t

Proto3 removed required entirely, which was the right call and is worth understanding rather than just accepting.

A required field is permanent in both directions. You can’t remove it, because every reader demands it. You can’t add one, because every writer that hasn’t been upgraded doesn’t send it. It is a one-way door disguised as a type annotation, and the moment it ships you have given up the ability to evolve that message without a coordinated restart of everything that touches it.

Optional field plus documented default gets you the same validation with none of the lock-in — just enforce the requirement in your application logic, where you can change it, rather than in the serialization format, where you can’t.

JSON Schema has the same trap in its required array, and because JSON Schema is easy to edit, people add to that array far too casually. Adding a name to required breaks every existing producer. It is a breaking change that looks like a one-word diff.

The registry is the enforcement point

All of this is just discipline until something checks it, and discipline does not survive a Friday afternoon.

A schema registry — Confluent’s, Apicurio, Buf’s registry for Protobuf — stores schema versions and refuses a registration that violates the compatibility mode you’ve configured. That turns “we agreed to be careful” into a build failure, which is the only form of agreement that holds.

Two things worth getting right when you set one up:

Set the mode to FULL (or FULL_TRANSITIVE) rather than the usual BACKWARD default. BACKWARD only checks that new readers can read old data, which leaves the 02:40 failure wide open. FULL_TRANSITIVE checks against every historical version rather than just the previous one, which is what you want if you ever replay from an archive.

And put the check in CI, not just at publish time. A developer should find out that their change breaks compatibility while they’re still in the pull request, not when the deploy pipeline rejects it twenty minutes into a release.

If you’re working in a context with no registry — an internal HTTP API, say — the equivalent is a contract test that runs the previous version’s client against the new version’s server and vice versa. It’s less elegant and it catches the same class of bug.

Where this bites in an AI platform specifically

Two places, and both are newer than most of the tooling around them.

Tool schemas for agents are schemas. When you change a tool’s parameter — rename it, make it stricter, change an enum — you’ve changed a contract that is being consumed by a model that was never recompiled and doesn’t read your changelog. Worse, the model may have been evaluated against the old schema, which means your eval suite is now measuring something that no longer exists. Version tool definitions and treat a schema change as a change that invalidates evals, the same way you’d treat a model swap. The agent skill supply chain problem has this inside it.

Event payloads feeding a vector index or a warehouse have a retention clock measured in years, not days. If you’re embedding documents and storing the vectors, a schema change upstream means your index now contains two populations with different field semantics, and nothing about the vectors will tell you which is which. That’s a correctness problem that shows up as “retrieval got worse” six weeks later, with no deploy to correlate it against.

The short version

Default to additive. Adding optional fields is the only change that is reliably safe in both directions, and most changes can be expressed that way if you’re willing to carry two fields for a while.

Reserve everything you retire — numbers and names, every time, no exceptions, because the exception is what you’ll be debugging.

Never reuse an identifier. Old data still means what it meant.

Set the registry to full transitive compatibility and run it in CI, so the rule is enforced by a machine rather than remembered by a person.

And measure the migration clock in retention periods, not sprints. The field you removed is still sitting in someone’s queue, in an archive, in a replay buffer, in a consumer that’s been paused since Tuesday. It will come back. The only question is whether the code that reads it then still knows what it was.

Related: Idempotency Keys: The Thing That Makes a Retry Safe · Queues and Message Brokers · REST vs GraphQL APIs

Frequently asked questions

What is the difference between backward and forward compatibility?

Backward compatible means new code can read data written by old code — you can upgrade readers first. Forward compatible means old code can read data written by new code — you can upgrade writers first. The direction is named from the perspective of the new schema looking at existing data. Full compatibility is both at once, and it is what a rolling deploy actually requires, because during the rollout old and new versions are both reading and both writing at the same time.

Why are required fields a problem in a schema?

Because required is permanent. A required field can never be removed without breaking every reader that still expects it, and it can never be added without breaking every writer that does not yet send it. Protocol Buffers removed the keyword in proto3 for exactly this reason. The practical replacement is an optional field plus a documented default, which lets you deprecate it later by simply ceasing to write it, and lets you introduce it without a coordinated flag day.

Why should field numbers or names never be reused?

Because old data still carries the old meaning. In Protocol Buffers the wire format identifies fields by number, not name, so assigning number 4 to a new string when it used to be an integer means any stored message or queued event written under the old schema will be decoded as the wrong type or silently misread. Reserve retired numbers and names explicitly so the compiler refuses to reuse them. Avro has the same hazard keyed on names and aliases.

How do I change a field's type safely?

You generally cannot change it in place. The safe pattern is to add a new field with the new type, write both fields for a full retention period while readers are migrated to prefer the new one, then stop writing the old field, and only reserve its number once no stored or in-flight data could still contain it. Each step is independently reversible, which is the property that makes it safe; a type change in place is a single irreversible step applied to data you have already written.

Comments