Module 12 11 lessons 140 min

Problems in distributed systems

After the split, new problems appear. A payment request times out and nobody knows whether it succeeded, the last 3 items in stock are sold 5 times, and one slow service makes the whole app slow.

In this module

10 Read · 1 Lab

  1. 12.1Partial failure and timeoutsYou can't tell a slow machine from a crashed one. After a timeout, you don't know whether the payment went through.Read12 min
  2. 12.2Retry storms and cascading failuresWhen the payment service slows down, every caller retries, and retries multiply at each layer. Backoff, jitter and retry budgets.Read12 min
  3. 12.3Circuit breakers and load sheddingStop calling a failing service and return a fallback. Under overload, reject some requests so the rest still succeed.Read12 min
  4. 12.4Tail latencyA page that calls ten services hits at least one service's p99 on most requests. How fan-out makes tail latency matter.Read10 min
  5. 12.5Clocks and orderingClocks on different machines drift. Last-write-wins can lose updates, and timestamps can't reliably order events.Read12 min
  6. 12.6Duplicate and out-of-order messages“Order cancelled” can arrive before “order placed”, and “order paid” can arrive twice. Why this happens, and why handlers must be idempotent.Read10 min
  7. 12.7Inconsistent dataThe order is saved but the “paid” event is never sent. A customer updates their address and the next page shows the old one. Two writes in two places, with no shared transaction.Read12 min
  8. 12.8Counters and limits across machinesStock counts that oversell, per-server rate limits that allow too much traffic, and auto-increment IDs that collide.Read12 min
  9. 12.9Locks and leadersA lock in one process doesn't protect anything on another machine. A lease can expire during a pause, leaving two machines that both think they're the leader.Read10 min
  10. 12.10Proof: one slow serviceSlow down the payment service under load and measure what retries do to the app. Then add backoff, a circuit breaker and load shedding, and measure again.Read8 min
  11. 12.11Lab: Inject failures into the clinicAdd latency, stop a machine, skew a clock and split the network. Predict the effect on bookings, then test it.Lab30 min
Start with 12.1 · Partial failure and timeouts
Then · Module 13Replication in depth