Designing Data-Intensive Applications
Case 9

Storage and Real-Time Transport

Split blob storage, metadata DB, collab server, and CDN for assets.

Diagram

Step-by-step walkthrough

Real-time path — live editing

  • ① Send/receive ops — User A's edits flow over WebSocket to the collab server (OT or CRDT ordering).
  • ② Send/receive ops — Server broadcasts transformed ops to User B and other connected editors.
  • ③ Heartbeat + cursors — Presence (who is online, cursor position) stored in Redis with TTL.

Persistence path — durable state

  • ④ Persist ops + snapshots — Collab server batches operations into Postgres; periodic snapshots bound replay time.
  • ⑤ Asset metadata — File pointers and ACL metadata in Postgres; blobs are not inlined in rows.
  • ⑥ Signed URL upload — Large images upload direct-to-S3; collab server never proxies multi-MB files.
In practice

Google Docs historically used centralized OT servers at scale. Notion shards documents across cells; Miro uses regional collab routers for EU data residency.

Why these technologies?

Why WebSocket collab server?

Sub-100 ms operation broadcast for cursors and edits. HTTP polling would feel laggy and waste bandwidth.

Why OT or CRDT library (Yjs, Automerge)?

Merge concurrent edits without locking the whole document. OT needs server ordering; CRDTs can peer-sync offline then merge.

Why PostgreSQL (metadata & ACL)?

Document titles, share permissions, folder hierarchy — relational model fits. Not for multi-MB canvas blobs.

Why S3 / GCS (assets)?

Images, fonts, video layers are large immutable files — cheap object storage with CDN in front for export/download.

Why Redis (presence & pub/sub)?

Who is online, cursor color, typing — ephemeral with TTL. Pub/sub bridges collab server instances in multi-region deploys.

Why Sticky routing by document ID?

OT requires consistent operation ordering per document — route all ops for doc X to the same server or use CRDT to avoid stickiness.

Why Not storing every keystroke in Postgres forever?

Op log compacted to periodic snapshots — unbounded insert rate would saturate OLTP.

Key Takeaways
  • Object storage (S3) for images/fonts; Postgres for doc metadata and ACL.
  • Collab server cluster sticky by document ID for OT ordering.
  • WebSocket gateway scales with connection count; Redis pub/sub bridges regions.
  • Export/render jobs async — separate from live editing path.
  • Version history: snapshot + op log compaction weekly.
  • Same architecture pattern as Canva, Google Docs, Notion — different CRDT/OT choices.
WebSocketS3PostgreSQLsnapshotcompactionRedisCRDTops/s
← OT, CRDTs, and Conflict-Free Editing