An interactive study companion for Martin Kleppmann's classic on reliable, scalable, and maintainable systems.
Secondary indexes on partitioned data require either local indexes per partition or a global index.
Primary key partitioning is straightforward — route by key hash. Secondary indexes (search by email, filter by date) complicate routing because the indexed value may live on a different partition than the record.
MongoDB sharded clusters maintain local indexes per shard — a query on an unsharded field must scatter-gather to every shard. Elasticsearch distributes inverted indexes across nodes; a search hits all shards and merges results. PostgreSQL on Citus uses co-location and reference tables to reduce scatter-gather overhead.
Search by city or neighborhood hits secondary indexes scattered across listing shards. Each shard returns partial results; the router merges and ranks them — scatter-gather is the cost of partitioning by listing_id.
// Airbnb search — secondary index scatter-gather across shards
async function searchByCity(city: string): Promise<Listing[]> {
const shards = await router.allShards();
const partial = await Promise.all(
shards.map((s) => s.query("SELECT * FROM listings WHERE city = $1", [city]))
);
return partial.flat().sort((a, b) => b.rating - a.rating).slice(0, 50);
}