Designing Data-Intensive Applications
Ch. 4

Modes of Dataflow

Data moves between processes via databases, service calls, and message queues — each with different trade-offs.

When one process needs to share data with another, the dataflow pattern matters. Writing to a shared database is the oldest integration pattern. RPC calls offer synchronous coupling. Message queues add buffering and decouple availability.

In practice

A typical AWS microservice stack: services call each other via gRPC behind an Envoy or nginx reverse proxy, share state in PostgreSQL, cache hot reads in Redis, and publish domain events to Kafka or AWS SQS. WebSockets and Server-Sent Events (SSE) push real-time updates to browsers — another dataflow mode for live dashboards and notifications.

Airbnb at scale

A booking spans listings, calendar, payments, and notifications — each a separate service. Synchronous gRPC checks availability; Kafka events notify hosts asynchronously when a reservation confirms.

typescript — Synchronous gRPC between services
// Airbnb microservices — synchronous gRPC between booking services
const listing = await listingsClient.getListing({ id: listingId });
const availability = await calendarClient.checkDates({
  listingId,
  checkIn,
  checkOut,
});
// REST/GraphQL for browsers; gRPC for internal service-to-service calls
Diagram
  • Database: simple but creates tight schema coupling.
  • REST/RPC: real-time but requires both sides to be available.
  • Message queue: resilient to spikes but adds delivery guarantees complexity.
Key Takeaways
  • Database-as-mediator: processes write and read shared tables asynchronously.
  • REST and RPC: synchronous request/response between services.
  • Message queues: asynchronous, buffered delivery with at-least-once semantics.
  • Event-driven architectures decouple producers and consumers in time.
  • The encoding format must be agreed upon across all participants.
  • nginx/Envoy reverse proxies front gRPC services; Kafka and SQS carry async events.
RESTgRPCKafkaRedisreverse proxyWebSocketSSEmessage queue
← Formats for Encoding Data