Replication, Designing Data-Intensive Applications Notes

5.1Chapter Thesis

Replication exists to answer one question: what happens to your data, and your ability to serve it, when one machine disappears? Copying data onto multiple machines buys durability and availability, but it introduces a problem that a single machine never has - the copies can, and eventually will, briefly disagree.

Every replication scheme is a specific answer to how much disagreement between replicas is tolerable, for how long, and who finds out about it first.

5.2First Principles

Leader-based replication picks a single order

If writes can land on any replica independently, replicas can receive conflicting writes to the same key with no inherent way to agree on which one happened "first." Electing a single leader to accept all writes and stream them, in order, to followers sidesteps this by construction - there is only ever one order, because there is only ever one writer.

Synchronous vs. asynchronous replication is a latency/durability tradeoff

A synchronous follower must confirm receipt before the leader acknowledges the write to the client - this guarantees the follower has the data, at the cost of the write's latency being bound by the slowest synchronous follower. An asynchronous follower confirms nothing before the leader responds - the write is fast, but a leader crash before replication completes loses that write permanently.

Replication lag is a real, observable delay, not an edge case

Because followers apply the replication log in order but not instantly, there's an always-present window during which a follower's data is strictly older than the leader's. Any read from that follower during the window is a stale read - this isn't a bug, it's the direct consequence of choosing asynchronous replication for its latency benefit.

5.3Mechanism

The leader appends every write to a local, ordered replication log. Each follower maintains a cursor into that log - the offset of the last entry it has applied - and continuously requests the next batch of entries after its cursor, applying them to its own copy of the data in the same order the leader wrote them. This is the same append-only log mechanism from Storage and Retrieval, reused as a communication channel instead of purely a storage structure.

5.4Build It Yourself

leader-follower-log.ts

interface LogEntry { key: string; value: string; }

class Leader {
  private log: LogEntry[] = [];

  write(key: string, value: string): number {
    this.log.push({ key, value });
    return this.log.length - 1; // offset
  }

  entriesSince(offset: number): LogEntry[] {
    return this.log.slice(offset);
  }
}

class Follower {
  private state = new Map<string, string>();
  private cursor = 0;

  constructor(private leader: Leader, private replicationDelayMs = 0) {}

  // polling with an injectable delay is what makes
  // replication lag something you can actually observe.
  async sync(): Promise<void> {
    if (this.replicationDelayMs > 0) {
      await new Promise((r) => setTimeout(r, this.replicationDelayMs));
    }
    const entries = this.leader.entriesSince(this.cursor);
    for (const e of entries) this.state.set(e.key, e.value);
    this.cursor += entries.length;
  }

  read(key: string): string | undefined {
    return this.state.get(key);
  }
}

Tip

Reading from the follower immediately after a leader write, before sync() has run, reproduces a stale read on demand - this is the experiment listed for this chapter.

5.5Failure Modes

Condition	Resulting state
Leader crashes before an async write replicates	The acknowledged write is permanently lost when a follower is promoted to leader - the client believes the write succeeded.
Network partition isolates the leader from all followers	If the isolated leader keeps accepting writes while the followers elect a new leader, two leaders now exist accepting divergent writes - split brain.
A follower falls far enough behind	Reads from it can return data old enough to violate application-level expectations (e.g. a user not seeing their own just-submitted comment) - read-your-writes consistency has to be engineered on top, it isn't automatic.

5.6Tradeoffs

Decision	Benefit	Cost	When it makes sense
Synchronous replication	No acknowledged write is ever lost on leader failure	Write latency bound by the slowest synchronous follower; that follower becoming unavailable can block all writes	Data where losing an acknowledged write is unacceptable (financial ledgers)
Asynchronous replication	Low, leader-only write latency	Acknowledged writes can be lost on leader failure; followers can serve stale reads	High write-throughput systems that can tolerate rare, small data loss on failover

5.7Production Perspective

Production databases usually combine both: one synchronous follower (guaranteeing at least one up-to-date copy survives a leader crash) plus several asynchronous followers for read scaling and additional durability - a middle point on the same latency/durability line this chapter's two-node example sits at the extremes of.

Replication