Designing Data-Intensive Applications
Ch. 8

Unreliable Networks

Network packets get lost, delayed, and reordered — and you often cannot tell which happened.

The network is not reliable. Packets drop, switches fail, and congestion causes unbounded delays. Distributed protocols must handle lost messages, duplicate messages, and the inability to distinguish slow responses from dead nodes.

The two generals problem

Two armies must agree to attack simultaneously via messengers who may be captured. Proves that consensus is impossible with unreliable communication.

In practice

gRPC clients set deadlines; Envoy or nginx reverse proxies configure retry policies with backoff. Istio circuit breakers stop hammering a failing service. AWS ALB health checks eject unhealthy targets. Mobile clients on flaky networks must use idempotent APIs (Stripe idempotency keys) because TCP reliability does not survive application-level timeouts.

Uber at scale

Driver matching RPCs run over mobile networks with unpredictable latency. gRPC deadlines abort hung calls after 3 seconds; idempotent retries prevent duplicate trip assignments when packets are lost.

typescript — gRPC client deadline
// Uber microservices — gRPC deadlines detect slow/dead peers
const deadline = Date.now() + 3_000;
const trip = await tripClient.getTrip(
  { tripId },
  { deadline }, // client aborts after 3s — cannot distinguish slow vs dead
);
// Envoy retry policy + circuit breaker stops cascading timeouts
Diagram
Key Takeaways
  • TCP provides reliable delivery within a connection but cannot detect peer crashes.
  • Timeouts are the only way to detect failure — but choosing the right timeout is hard.
  • Network partitions split the cluster into islands that cannot communicate.
  • Synchronous networks (e.g., internal datacenter) are more predictable than the public internet.
  • You must design for messages that never arrive or arrive twice.
  • gRPC deadlines, Envoy retries, and circuit breakers handle unreliable networks in microservices.
network partitiongRPCEnvoytimeoutTCPcircuit breaker
← Faults and Partial FailuresUnreliable Clocks →