Reliability, Scalability, Maintainability

1.1Chapter Thesis

Software systems fail. Disks die mid-write, networks drop packets in the middle of a request, and the person deploying the change makes a typo in a config file at 2am. None of this is exceptional - it is the baseline operating condition of any system that runs for more than a few hours on real hardware, serving real traffic, maintained by real people.

The engineering problem this chapter addresses is not "how do we prevent failure" - that is not achievable - but how do we define, in advance, precisely what kind of correctness we are promising, under what conditions that promise can be broken, and how the system behaves once it is broken. Reliability, scalability, and maintainability are the three axes engineers use to make that definition concrete instead of aspirational.

1.2First Principles

Fault vs. failure

A fault is one component deviating from its specification - a disk sector that can't be read, a process that receives a malformed packet, a developer who pushes bad config. A failure is the system as a whole no longer providing the required service to its users. Reliability engineering is the discipline of building systems where individual, expected faults do not compose into a user-visible failure.

This distinction is an invariant you can test for directly: if a component fails and no client observes incorrect behavior, the system was fault-tolerant with respect to that fault class. If a client does observe it, either the fault class wasn't anticipated, or the tolerance mechanism itself failed.

Fault classes

• Hardware faults - independent, statistically well-understood (disk MTBF, RAM bit flips). Solved historically with redundancy: RAID, dual power supplies, hot-swappable CPUs.
• Software faults - systematic and correlated. A bug in a library triggered by a specific input can bring down every process that imported it, simultaneously. Redundancy does not help here because the fault isn't independent across replicas.
• Human faults - configuration mistakes are the dominant cause of outages in practice, precisely because humans are the least reliable component and the hardest to add redundancy for.

Load is not one number

"How much traffic can this handle" is underspecified until you pick a load parameter: requests/second to an API, ratio of reads to writes in a database, number of simultaneously active users in a chat room, hit rate on a cache. The right load parameter is the one whose growth actually degrades the specific bottleneck you're worried about - everything else is a vanity metric.

Performance is a distribution, not a number

A single "average response time" hides the shape of user experience almost entirely, because response times for the same operation vary run to run - due to context switches, GC pauses, page faults, network jitter, queueing behind other requests. The right unit of description is a percentile: p50 (median), p95, p99, p999. The tail matters disproportionately because a user's page often depends on many backend calls in sequence - if any one of twenty parallel calls is a p99 outlier, roughly one in five page loads hits it.

1.3Mechanism

Describing performance correctly requires computing percentiles over a rolling window of observed response times, then re-deriving them continuously as new requests complete. The naive mechanism:

1. Every completed request appends its duration to a buffer.
2. On a fixed interval, sort the buffer and index into it: p95 is the value at index floor(0.95 * n).
3. Discard the buffer, or slide the window forward.

This is correct but O(n log n) per window and holds every raw sample in memory - fine at low volume, unusable once a single service handles tens of thousands of requests per second across many windows tracked simultaneously (per endpoint, per customer, per region). Production systems replace exact sorting with forward decay or t-digest structures: fixed-memory sketches that approximate the percentile within a bounded error, updated in O(log n) per observation instead of resorting everything.

1.4Build It Yourself

A minimal streaming percentile tracker, small enough to reason about completely, using bounded reservoir sampling instead of a full sketch - good enough to observe the underlying idea before reaching for a production library.

class ReservoirPercentile {
  private samples: number[] = [];
  private seen = 0;

  constructor(private capacity: number = 2000) {}

  observe(durationMs: number): void {
    this.seen++;
    if (this.samples.length < this.capacity) {
      this.samples.push(durationMs);
      return;
    }
    // Algorithm R: replace a uniformly random existing sample
    // so every observation has equal probability of surviving,
    // regardless of stream length.
    const j = Math.floor(Math.random() * this.seen);
    if (j < this.capacity) this.samples[j] = durationMs;
  }

  percentile(p: number): number {
    const sorted = [...this.samples].sort((a, b) => a - b);
    const idx = Math.min(
      sorted.length - 1,
      Math.floor(p * sorted.length)
    );
    return sorted[idx];
  }
}

1.5Failure Modes

1.6Tradeoffs

1.7Production Perspective

Real observability stacks (Prometheus histograms, Datadog distributions) implement variants of the sketch discussed above, and additionally have to solve problems this minimal version ignores entirely: percentiles must be computed per endpoint and often per customer, because aggregating across a fast internal endpoint and a slow external one produces a percentile that describes neither. They also have to survive process restarts by periodically flushing digest state, and reconcile digests computed independently on different hosts before reporting a single fleet-wide number - which is itself a form of the merge problem covered later in Replication.

1.8Related Concepts