Keys map to the next node clockwise. Remove a node and only its keys move — not the whole dataset.
Here’s a deceptively hard problem. You have data spread across N servers, and you decide which server owns a key with hash(key) mod N. Simple, fast, balanced. Then you add one server. Now it’s mod N+1, and almost every key maps to a different server than before — a near-total reshuffle, every cache invalidated at once, data stampeding across the network. Consistent hashing is the fix, and it’s one of those ideas that once you see it, you notice it holding up half your infrastructure.
Why hash mod N betrays you
The problem with mod N is that N is baked into the answer. Change the number of nodes and you change the divisor, and the remainder for nearly every key changes with it. Add a node to a 10-node cluster and roughly 90% of your keys get reassigned. For a cache, that means a near-total cache miss the instant you scale — the exact moment you were adding capacity because you were under load. It’s a self-inflicted thundering herd.
You want the opposite property: when membership changes, only a small, proportional fraction of keys should move.
The ring
Consistent hashing gets there with one shift in perspective. Instead of mapping keys to N slots, map both keys and nodes onto the same large circular hash space — the ring in the diagram. A key belongs to the first node you hit going clockwise from where the key lands.
Now watch what happens when Node B leaves. Only the keys that lived in the arc between B and the node before it need a new home — they slide to the next node clockwise (C). Every other key is completely undisturbed. Add a node and it’s the mirror image: the new node lifts a slice of keys off exactly one neighbor, and nobody else notices. On any membership change, only about 1/N of the keys move — not all of them.
Virtual nodes, because the basic ring is lumpy
The basic ring has a flaw: with only a few nodes placed randomly, the arcs between them are uneven, so some nodes own huge spans and others own slivers. Load is unbalanced, and when a node dies, its entire load dumps onto one unlucky neighbor.
The fix is virtual nodes: each physical node is hashed onto the ring many times (say 100–200 tokens). Now each physical server owns many small arcs scattered around the ring. Load evens out, and when a node fails, its share is redistributed across many neighbors instead of crushing one. Every production consistent-hashing implementation uses virtual nodes; the plain ring is a teaching tool.
Where it’s quietly running
- Distributed caches. Memcached client libraries and Redis Cluster use consistent hashing so scaling the cache tier doesn’t nuke the hit rate.
- Databases. Cassandra and DynamoDB partition data around a ring. It’s why they scale horizontally by just… adding nodes.
- CDNs and load balancers. Consistent hashing keeps a given user or object routed to the same edge node, preserving cache locality even as the fleet changes.
- Sharded anything. Any time you split data across a changing set of machines and want stability, this is the tool.
The AI/cloud angle
This isn’t a museum piece — it’s load-bearing in modern AI infrastructure.
Sharded vector indexes. A billion-vector index doesn’t fit on one machine, so it’s partitioned across many. When you scale that fleet up (bigger corpus) or a node fails, consistent hashing is what keeps you from re-sharding and re-indexing the entire dataset — an operation that could take hours and tank query availability. The stability property directly bounds your re-indexing pain.
Session and cache affinity for stateful AI. Stateful agent sessions, KV-cache reuse, and prompt caches all benefit from routing the same conversation to the same worker so warm state gets reused. Consistent hashing gives you that stickiness while still letting the worker pool scale and heal. Route a conversation to a random worker each turn and you throw away cache you already paid for.
Model and GPU pools. As you autoscale inference workers, you want request routing that’s stable under membership churn so you’re not constantly cold-starting and re-warming. Same ring, same benefit.
The mental model to keep
The one-sentence version: consistent hashing makes the cost of a membership change proportional to the size of the change, not the size of the system. Add one node out of a thousand and ~0.1% of data moves, not 99.9%. That single property is why distributed systems can grow, shrink, and heal without a coordinated global reshuffle — and why a 1997 paper is still holding up the newest, most GPU-hungry systems we build. When you’re designing anything that shards data across a fleet that will change size, reach for the ring before you reach for mod N.