Load Balancing & Consistent Hashing

A load balancer sits between clients and your server pool, distributing requests so no single machine is overwhelmed. It is mandatory equipment the moment you horizontally scale — multiple servers are useless if all traffic still lands on one of them.

The Naive Approach: Simple Hashing

The simplest distribution scheme is hashing:

server = hash(request_id) % n_servers

Same request ID always maps to the same server. Cheap, deterministic, and gives you cache/session locality for free.

The Drawback

n_servers changes whenever you add or remove a server — which is exactly what horizontal scaling is for. When n = 10 becomes n = 11, almost every key remaps to a different server:

• Sessions stored in-memory are lost.
• Caches go cold (most requests now miss).
• Upstream connections churn.

At scale this disruption is unacceptable.

Consistent Hashing

Consistent hashing uses a fixed hash space (conceptually a ring, e.g. hash(key) % M where M is a huge constant like 2³²):

1. Servers are hashed onto the ring at specific positions.
2. Keys are hashed onto the same ring.
3. A key belongs to the next clockwise server.

Because the hash space never changes, adding or removing a server only reassigns the keys sitting between the old and new position — roughly 1/n of keys move instead of nearly all of them.

	Naive hashing	Consistent hashing
Keys moved when scaling	~all keys	~1/n of keys
Cache/session locality	Broken on every change	Mostly preserved
Complexity	Trivial	Ring + node lookup logic

Virtual Nodes

Physical servers don’t sit evenly on the ring by chance — some get huge ranges, others tiny ones (hot spots).

Virtual nodes fix this: each physical server is placed on the ring many times (e.g. 100–200 replicas under different hash offsets). With enough virtual nodes, ranges even out statistically, and a failing server sheds its small slices to many neighbors instead of dumping one giant range onto one unlucky peer.

Where You’ll See It

• Distributed caches and stores: Redis Cluster, Cassandra, DynamoDB
• Load balancers with sticky routing
• Database sharding layers (see Sharding)

Interview Framing

When asked “how would you route jobs to workers?”, walk through the progression: round-robin → hashing for stickiness → why % n breaks → hash ring → virtual nodes for balance. The story of why each step exists matters more than naming them.

---

Part of the System Design Fundamentals series.