Encoding and Evolution

4.1Chapter Thesis

Data written today will be read by code deployed next month, and that future code did not exist when the write happened. Encoding is the contract that makes this work anyway - it defines what a reader is allowed to assume about a writer it has never met and never will.

The engineering problem is not "how do we serialize an object" - that part is easy. It's how a schema is allowed to change over time without the old data becoming unreadable and without every reader and writer having to deploy simultaneously, which in a system with more than one process is never actually possible.

4.2First Principles

Rolling upgrades force compatibility in both directions

In any system with more than one process, deploys are staggered - some nodes run the new code while others still run the old one, for minutes or hours. Backward compatibility means new code can read data written by old code. Forward compatibility means old code can read data written by new code. A real system needs both, simultaneously, for the entire rollout window.

Field identity must survive renaming and reordering

If a schema identifies fields by name and position, any reorder silently breaks readers that assume position. If fields are identified by an explicit, permanent numeric tag, the field's name in code can change freely - the tag is the actual contract, the name is just documentation for humans.

Required vs. optional is a compatibility decision, not a data-modeling one

Marking a field required means every future writer must always supply it - a promise no schema evolution can safely make, since you cannot force every already-running writer to add a field simultaneously. Treating new fields as optional by default and giving them either no default or an explicit backward-safe default is what makes evolution possible at all.

4.3Mechanism

A tagged binary encoding writes each field as [tag][length][value bytes] instead of relying on field order or names. A reader that encounters a tag it doesn't recognize can skip length bytes and move on - this single mechanism is what gives forward compatibility: new fields written by new code are silently and safely ignored by old readers, because the reader never needed to understand the tag to skip past it correctly.

Backward compatibility follows from never reusing a retired tag number and never changing what an existing tag means - an old field can be dropped from the schema, but the tag number itself is permanently retired, not recycled for something else.

4.4Build It Yourself

type Field = { tag: number; value: string };

encode(fields: Field[]): string {
  return fields
    .map((f) => `${f.tag}:${f.value.length}:${f.value}`)
    .join("|");
}

// unknown tags are skipped, not rejected -
// this is the entire forward-compatibility mechanism.
decode(
  data: string,
  knownTags: Set<number>
): Record<number, string> {
  const result: Record<number, string> = {};
  for (const chunk of data.split("|")) {
    const [tagStr, lenStr, ...rest] = chunk.split(":");
    const tag = Number(tagStr);
    if (knownTags.has(tag)) {
      result[tag] = rest.join(":");
    }
    // else: silently skip - the reader doesn't need to
    // understand this field to correctly ignore it.
  }
  return result;
}

4.5Failure Modes

4.6Tradeoffs

4.7Production Perspective

Production systems formalize this with a schema registry: every writer registers its schema version, every reader fetches the schema it needs to decode a given message, and compatibility rules (backward, forward, full) are checked automatically before a new schema version is allowed to be published at all - turning this chapter's manual tag discipline into an enforced policy.

4.8Related Concepts