Stream Processing, Designing Data-Intensive Applications Notes

11.1Chapter Thesis

Batch processing assumes the input dataset has an end - a file you can finish reading. A stream has no end, only a continuously growing history, and that single difference invalidates almost every assumption batch processing was allowed to make: there is no final answer to compute, only a continuously updated one, and "done" has to be redefined as "caught up to right now," a moving target.

11.2First Principles

An event log is an append-only sequence with durable position

A stream is materialized as an ordered, append-only sequence of events, each identified by a monotonically increasing offset. Unlike a message queue that deletes a message once consumed, a log retains history, letting multiple independent consumers each track their own position and replay from any earlier offset - this is what allows a new consumer to be added later without disrupting existing ones.

Event time and processing time are different clocks

Event time is when something actually happened; processing time is when the system observed it. Network delay, retries, and consumer backlog mean these two clocks diverge, sometimes by a lot - any windowed computation ("events per minute") has to explicitly choose which clock it's windowing by, because the two produce different, both-valid answers.

Delivery semantics are a spectrum, not a binary

At-most-once delivery can silently drop messages on failure. At-least-once can redeliver the same message after a failure, requiring the consumer to handle duplicates. Exactly-once processing semantics (not delivery - delivery exactly-once is not achievable over an unreliable network) require the consumer's side effects to be idempotent, so that a redelivered message produces the same end state as processing it once.

11.3Mechanism

A partitioned log splits the event stream across multiple partitions (usually by a key, so all events for the same entity land in the same partition and preserve order relative to each other). Each consumer tracks a committed offset per partition - the position up to which it has durably finished processing. Whether that offset is committed before or after processing determines which delivery guarantee the consumer gets.

11.4Build It Yourself

partitioned-log.ts

interface Event { key: string; payload: string; }

class PartitionedLog {
  private partitions: Event[][];

  constructor(private numPartitions: number) {
    this.partitions = Array.from({ length: numPartitions }, () => []);
  }

  private partitionFor(key: string): number {
    let h = 0;
    for (let i = 0; i < key.length; i++) h = (h * 31 + key.charCodeAt(i)) >>> 0;
    return h % this.numPartitions;
  }

  produce(event: Event): void {
    this.partitions[this.partitionFor(event.key)].push(event);
  }

  read(partition: number, fromOffset: number): Event[] {
    return this.partitions[partition].slice(fromOffset);
  }

  latestOffset(partition: number): number {
    return this.partitions[partition].length;
  }
}

class Consumer {
  private committedOffset = 0;

  constructor(private log: PartitionedLog, private partition: number) {}

  // committing AFTER processing gives at-least-once:
  // a crash between processing and commit replays events.
  poll(processFn: (e: Event) => void): void {
    const events = this.log.read(this.partition, this.committedOffset);
    for (const e of events) {
      processFn(e);
      this.committedOffset++;
    }
  }

  lag(): number {
    return this.log.latestOffset(this.partition) - this.committedOffset;
  }
}

11.5Failure Modes

Condition	Resulting state
Consumer crashes after processing but before incrementing committedOffset	At-least-once semantics: on restart, the same events are read and processed again - safe only if processFn is idempotent.
Offset is committed before processing (inverse order)	At-most-once semantics: a crash after commit but before processing permanently skips those events with no error raised.
Producer bursts events faster than the consumer can poll	Consumer lag (latestOffset − committedOffset) grows unbounded; downstream effects of those events fall further and further behind real time with no automatic backpressure in this minimal design.

11.6Tradeoffs

Semantics	Benefit	Cost	When it makes sense
At-most-once	Simplest implementation, no duplicate handling needed	Silent data loss on any failure	Metrics or logs where an occasional dropped data point is acceptable
At-least-once	No data loss	Consumer must handle duplicate processing correctly	Most production pipelines, paired with idempotent processing
Exactly-once processing	No loss, no duplicate side effects	Requires transactional or idempotent sinks; higher implementation complexity	Financial or otherwise duplicate-intolerant processing

11.7Production Perspective

Kafka implements this exact partitioned-log-plus-consumer-offset model at scale, and layers exactly-once processing on top by making offset commits part of the same transaction as the produced output, rather than by trying to prevent redelivery at the network level - accepting at-least-once delivery and canceling out duplicates at the processing layer instead.

Stream Processing