The Trouble with Distributed Systems, Designing Data-Intensive Applications Notes

8.1Chapter Thesis

A single machine either runs correctly or it doesn't, and when it fails, it fails all at once and tells you so, immediately - a stack trace, a crash, a nonzero exit code. A distributed system doesn't get that courtesy: a remote node can be slow, partially failed, disconnected but still running, or gone entirely, and from where you're standing, all four look identical for an unbounded amount of time.

Every distributed systems technique in the rest of this book exists to manage that ambiguity, because the ambiguity itself cannot be eliminated - only bounded, worked around, or made explicit.

8.2First Principles

The network cannot promise delivery, order, or timing

Packets can be delayed arbitrarily, dropped silently, duplicated, or reordered - and the sender has no reliable way to distinguish "the packet was lost" from "the reply was lost" from "the recipient is just slow." A timeout is not a detection of failure; it is a guess, calibrated by picking a duration and accepting the false-positive rate that comes with it.

Clocks drift, and network time sync is itself unreliable

Every machine's local clock drifts from true time at its own rate, and even with NTP synchronization, clock readings across machines can disagree by tens to hundreds of milliseconds, or jump discontinuously when corrected. Using wall-clock timestamps to decide "which write happened first" across machines is therefore not a comparison of when things happened - it's a comparison of two independently drifting guesses.

A process can pause for arbitrarily long without crashing

Garbage collection pauses, virtual machine live migration, disk I/O stalls, and OS scheduling delays can all suspend a process for seconds at a time. From the outside, a paused process is indistinguishable from a crashed one until it resumes - which means any protocol that assumes "no response within timeout X means the node is dead" can be wrong even when the node never actually crashed.

8.3Mechanism

A failure detector works by sending periodic heartbeats and declaring a peer "suspected failed" if no heartbeat arrives within a timeout window. The timeout is the entire mechanism, and it embeds a direct tradeoff: too short, and normal network jitter or a GC pause triggers false failure detections; too long, and a genuinely failed node stays "alive" in the eyes of the system for longer, delaying failover.

8.4Build It Yourself

unreliable-network.ts

class UnreliableNetwork {
  constructor(
    private dropRate = 0.1,
    private maxDelayMs = 500
  ) {}

  async send(deliver: () => void): Promise<void> {
    if (Math.random() < this.dropRate) return; // silently dropped
    const delay = Math.random() * this.maxDelayMs;
    await new Promise((r) => setTimeout(r, delay));
    deliver();
  }
}

class FailureDetector {
  private lastHeartbeat = Date.now();

  constructor(private timeoutMs: number) {}

  onHeartbeat(): void {
    this.lastHeartbeat = Date.now();
  }

  // "suspected" is the honest word here - this can never
  // truthfully return "confirmed dead".
  isSuspectedFailed(): boolean {
    return Date.now() - this.lastHeartbeat > this.timeoutMs;
  }
}

Tip

Feeding heartbeats through UnreliableNetwork with dropRate = 0 but a nonzero maxDelayMs that occasionally exceeds the detector's timeout reproduces a false failure suspicion on a node that never actually went down - the experiment for this chapter.

8.5Failure Modes

Condition	Resulting state
Heartbeat delayed past the timeout, node is actually alive	False positive: the system suspects and potentially fails over a healthy node, which can trigger unnecessary and disruptive leader elections.
Node experiences a long GC pause	Indistinguishable from a crash to any external observer relying on timeouts; the node resumes mid-election or mid-failover with stale assumptions about its own role.
Two nodes' clocks disagree and a protocol uses wall-clock time to order events	Events can be ordered incorrectly - a write with an earlier wall-clock timestamp can have actually happened after one with a later timestamp, silently corrupting any logic that depends on timestamp order.

8.6Tradeoffs

Timeout choice	Benefit	Cost	When it makes sense
Short timeout	Fast failover when a node genuinely fails	High false-positive rate under normal jitter or GC pauses	Latency-critical systems that can tolerate occasional unnecessary failovers
Long timeout	Fewer false-positive failure suspicions	Genuine failures take longer to detect and act on	Systems where an unnecessary failover is more disruptive than a slower recovery

8.7Production Perspective

Production systems mitigate this with adaptive timeouts (adjusted based on observed network conditions rather than a fixed constant) and lease-based mechanisms, where a node holding a lease must proactively and safely stop acting as leader once the lease expires, rather than relying purely on other nodes to detect its failure - shifting part of the correctness burden onto the node that might be paused, not just its observers.

The Trouble with Distributed Systems