Designing Data-Intensive Applications
Case 10

Backend for Long-Running Applications

API design for partial saves, file uploads, and async verification steps.

Diagram

Step-by-step walkthrough

Synchronous path — autosave while editing

  • ① PATCH /draft every 30s — Client debounced autosave; dominates write QPS vs final submit.
  • ② UPSERT JSON — Draft API persists partial answers in Postgres keyed by user + form version.
  • ③ PUT via signed URL — Large documents (PDF, scans) upload direct-to-S3; draft stores pointer only.

Async path — submit and verify

  • ④ POST /submit — Idempotent final submit; draft status moves to PENDING_VERIFICATION.
  • ⑤ Start verification — API calls external KYC/credit vendor; HTTP returns 202 immediately.
  • ⑥ Webhook result — Vendor calls back hours later with pass/fail; no long-polling from browser.
  • ⑦ status = APPROVED — Draft row updated; user notified via email or in-app poll.
Diagram
In practice

TurboTax and government visa portals use identical patterns: session-less drafts in DB, step validators as pure functions, async steps wired to mainframe or vendor APIs.

typescript — Step validation handler
// Validate only fields for current step
const stepSchemas = {
  1: z.object({ legalName: z.string().min(2), dob: z.string().date() }),
  2: z.object({ ssn: z.string().regex(/^\d{9}$/) }),
  3: z.object({ income: z.number().positive() }),
};

function validateStep(step: number, payload: unknown) {
  return stepSchemas[step as keyof typeof stepSchemas].parse(payload);
}

Why these technologies?

Why PostgreSQL (draft storage)?

Multi-day forms need durable JSON blobs keyed by user + form version — survives browser close and device switch. Session cookies expire; DB drafts don't.

Why S3 pre-signed URLs?

Large documents (PDFs, scans) upload directly to object storage — API servers don't proxy gigabyte files. Draft row stores S3 key pointer only.

Why Step-scoped validation API?

Validate only fields for current step — faster UX and clearer errors than validating entire 50-field form on every keystroke.

Why Async verification vendor + webhook?

Credit checks and identity verification take seconds to days — cannot block HTTP request. Webhook transitions draft from PENDING to APPROVED.

Why Idempotent final submit?

Double-click submit must not create two loan applications — same pattern as Stripe idempotency keys.

Why Field-level encryption?

SSN and tax IDs encrypted at rest in Postgres — defense in depth beyond disk encryption.

Why Not a workflow engine for simple 3-step forms?

Temporal is overkill for a checkout wizard — Postgres draft + state machine in application code suffices until human review gates appear.

Key Takeaways
  • POST /drafts upserts JSON blob; PATCH /steps/:id validates slice only.
  • Large uploads direct-to-S3 with signed URLs; draft stores pointer.
  • Async steps (credit check) transition draft to PENDING_VERIFICATION.
  • Webhook or poll updates UI when vendor returns.
  • PII encrypted at rest; field-level encryption for SSN/tax IDs.
  • Analytics on funnel drop-off per step drives product iteration.
draft APIS3webhookPIIfunnel analyticsPostgreSQLautosave QPS
← Steps, Drafts, and Validation