Designing Data-Intensive Applications
Case 8

Durable Execution and Retries

Workflow state must survive process crashes. Durable timers and replayable history are the answer.

Uber at scale

A trip saga spans matching, driver en route, payment capture, and receipt email. Temporal replays from history if the worker crashes mid-step.

typescript — Durable workflow stub
// Temporal-style durable step (conceptual)
async function onboardMerchant(merchantId: string) {
  await activities.verifyIdentity(merchantId); // retried automatically
  await activities.linkBankAccount(merchantId);
  await sleep("7 days"); // durable timer — survives worker crash
  await activities.enablePayouts(merchantId);
}
Outbox + inbox

Combine sagas with transactional outbox so 'charge card' and 'mark order paid' never diverge.

Key Takeaways
  • Workflow engine persists event history before calling external APIs.
  • Activities retry with exponential backoff; non-retryable errors branch to compensate.
  • Exactly-once illusion via idempotent activity implementations.
  • Human tasks (KYC review) park workflow for days without holding threads.
  • Visibility APIs show ops which step failed and with what payload.
  • Version workflows carefully — running instances use old definition.
Temporaldurable executionretryidempotencyoutbox