DDIA Learn — Designing Data-Intensive Applications

MapReduce's rigid map-shuffle-reduce cycle forces disk writes between every stage. Modern engines materialize less, push computation to data, and expose declarative APIs — while retaining fault tolerance through lineage graphs.

In practice

Databricks runs Apache Spark for ETL and ML training over terabytes in memory. Apache Flink unifies batch and stream processing in one runtime. Snowflake separates storage (S3/Azure) from elastic compute warehouses — you write SQL, not MapReduce jobs. dbt layers transformations on top of Snowflake or BigQuery with version-controlled SQL models.

Netflix at scale

Recommendation model training runs on Spark over viewing-history Parquet in S3 — iterative in-memory stages replace MapReduce's disk-bound shuffle between every step.

typescript — Declarative warehouse SQL

// Google BigQuery / Snowflake — declarative OLAP over columnar storage
const revenueByRegion = await snowflake.execute(`
  SELECT d.region, SUM(f.revenue_cents) AS total
  FROM fact_orders f
  JOIN dim_date d ON f.order_date = d.date_key
  WHERE d.year = 2025
  GROUP BY d.region
  ORDER BY total DESC
`);
// Optimizer reads only region + revenue columns — not full rows

Beyond MapReduce