Database Sharding

Sharding is horizontal partitioning of a database into smaller, independent pieces (shards), each holding a disjoint subset of rows and running on its own hardware. You shard when a single database genuinely cannot handle the write volume or dataset size — not before.

The Shard Key

The shard key decides how data is split and where each row lives. Choosing it is the most consequential decision in sharding:

• A good key spreads load evenly.
• A bad key creates hot shards — a few shards take most of the traffic while others idle.

Example: sharding by user_id spreads users evenly; sharding by created_at puts all current traffic on one shard (everyone writes to “today”).

Common Strategies

Strategy	How it works	Trade-off
Range-based	Shard 1: IDs 1–1M, Shard 2: 1M–2M…	Simple range queries; uneven load risk
Hash-based	hash(key) % n picks the shard	Even distribution; range scans hit all shards

For dynamic shard counts, consistent hashing avoids mass reassignment when shards are added (see Consistent Hashing).

Routing

A routing layer — application logic or a proxy — maps each request to the correct shard. This layer must know the partitioning scheme and adds one more component that can fail or become a bottleneck if not made highly available.

Replication Per Shard

Each shard is itself replicated for availability:

• One master handles all writes; slaves serve reads.
• If the master fails, a slave is promoted (failover).

This gives you both scalability (data spread across machines) and availability (no single machine’s loss loses data).

Strengths

• Improves scalability (write capacity grows with shards) and availability (blast radius shrinks per shard).
• Shards can be split further only when they grow too large.

Costs & Complexity

• Cross-shard queries are expensive — joins and transactions spanning shards need application-level coordination.
• Rebalancing is difficult — moving data between shards while serving traffic requires careful tooling.
• System complexity — routing, monitoring per shard, and debugging all get harder. Only pay this price when you truly need it.

---

Part of the Databases series.