The Future of Data Systems, Designing Data-Intensive Applications Notes

12.1Chapter Thesis

Almost no real system is one database. A typical application has a system-of-record store, a search index, a cache, an analytics warehouse, and a handful of denormalized read views - each holding a different shape of the same underlying facts. The engineering problem this creates is keeping all of those derived copies in sync with the source of truth, and the dominant failure mode in practice is not any single store being wrong, but the copies silently disagreeing with each other.

12.2First Principles

System of record vs. derived data

The system of record holds the authoritative version of a fact - if it disagrees with anything else, the system of record is right by definition. Derived data - a cache, an index, a materialized view - is produced entirely from the system of record and can, in principle, always be rebuilt from it. Confusing which store is which is what makes outages permanent instead of recoverable.

Dual writes cannot be made atomic without help

Writing to two different stores directly from application code ("update the database, then update the search index") cannot be made atomic without a distributed transaction across heterogeneous systems, which most of the involved stores don't support. Any crash between the two writes leaves them permanently disagreeing, with no log of the fact that they diverged.

Change data capture turns storage into a stream

Instead of writing to two stores directly, write to one system of record and treat every change to it as an event on a stream - every derived store then subscribes to that stream and updates itself independently. This reframes multi-store consistency as the stream-processing problem from the previous chapter: it inherits the same at-least-once/idempotency discipline, applied to keeping stores in sync instead of computing aggregates.

12.3Mechanism

A change stream emits one event per write to the system of record, in the order the writes occurred. A materializer subscribes to that stream and applies each change to a derived index, tracking its own consumed offset exactly like a stream consumer. Because the materializer can always replay from offset zero, the derived view can always be discarded and rebuilt from scratch - this replayability is what makes derived data safe to treat as disposable, unlike the system of record itself.

12.4Build It Yourself

derived-view-materializer.ts

interface ChangeEvent { key: string; field: string; value: string; }

class ChangeStream {
  private events: ChangeEvent[] = [];

  emit(e: ChangeEvent): void {
    this.events.push(e);
  }

  since(offset: number): ChangeEvent[] {
    return this.events.slice(offset);
  }
}

class DerivedIndexMaterializer {
  private index = new Map<string, Record<string, string>>();
  private offset = 0;

  constructor(private stream: ChangeStream) {}

  // idempotent by construction: replaying the same event
  // twice sets the same field to the same value both times -
  // safe to rebuild from offset 0 at any time.
  catchUp(): void {
    for (const e of this.stream.since(this.offset)) {
      const record = this.index.get(e.key) ?? {};
      record[e.field] = e.value;
      this.index.set(e.key, record);
      this.offset++;
    }
  }

  rebuild(): void {
    this.index.clear();
    this.offset = 0;
    this.catchUp();
  }

  get(key: string) {
    return this.index.get(key);
  }
}

12.5Failure Modes

Condition	Resulting state
A dual write to two stores fails between the first and second write	The two stores now permanently disagree, with no record anywhere of which write succeeded - this is the exact failure change data capture is designed to eliminate by having only one authoritative write path.
Events are applied out of order (e.g. from parallel consumers on the same key)	A field can end up holding an older value than a previously applied, now-overwritten one, since the materializer has no way to detect ordering violations without an explicit sequence check.
The materializer's offset is lost without the underlying data being cleared	A naive restart from offset 0 without clearing the index first double-applies every event; the code shown handles this correctly by clearing state in rebuild(), but a real system must remember to do so explicitly.

12.6Tradeoffs

Approach	Benefit	Cost	When it makes sense
Dual writes from application code	Simple to implement for a first version	No atomicity across stores; divergence is silent and permanent	Prototypes, or when the second store is purely advisory and can tolerate drift
Change data capture + derived materializer	Single source of truth; derived stores are always rebuildable	Added infrastructure (a change stream) and eventual - not immediate - consistency between stores	Any system where multiple stores must reflect the same underlying facts reliably

12.7Production Perspective

Production change data capture systems (Debezium, database replication logs repurposed as event streams) tap directly into a database's own write-ahead log rather than requiring application code to explicitly emit events - meaning every write is captured by construction, closing the exact gap that makes hand-rolled dual writes unreliable in the first place.

The Future of Data Systems