Designing Data-Intensive Applications

Every data system design decision is a bet against a specific way it will eventually break.

A data model is a claim about which relationships are cheap to query and which ones you pay for later.

A database is a promise about which reads and writes are fast, made concrete by one data structure.

Data outlives the code that wrote it. Encoding is the contract between versions of a program that will never run at the same time.

Copying the same data onto multiple machines trades durability and availability for a new kind of lie: the copies disagree, briefly, on purpose.

Splitting data across machines is a hashing problem wearing an operations costume.

A transaction is a promise to make a set of changes look atomic to everyone else, at a cost you get to choose.

Distributed systems are hard because the network and the clock both lie to you, and you cannot tell when they're doing it.

Consensus is the act of getting machines that can't fully trust each other, the network, or time itself, to agree on one thing anyway.

Batch processing turns a dataset too large for one machine's memory into a sequence of operations small enough for any one machine to run.

A stream is a batch that never ends, which breaks every assumption batch processing was allowed to make about time.

Most real systems are not one database: they are several, kept in sync by derivation, and correctness is an end-to-end property, not a per-component one.