Consensus sounds like a topic for papers, not pagers. Then etcd loses quorum, the Kubernetes API server goes read-only, and every controller in your cluster quietly stops making progress. That is a consensus failure, and it is worth understanding the machine underneath it.
Every system that has to keep one authoritative copy of some critical state — which node is the leader, what the current config is, who holds a lock — faces the same problem: you want more than one machine holding that state so a single crash doesn’t lose it, but the moment you have more than one copy, they can disagree. Consensus is the discipline of making several machines agree on an ordered sequence of decisions as if they were one machine that never crashes.
Paxos solved this in the 1990s and was famously hard to understand. Raft was designed in 2014 with understandability as an explicit goal, and it won: etcd, Consul, TiKV, CockroachDB, and a long list of others run on it. If you operate anything cloud-native, Raft is running underneath you right now.
The one guarantee
Strip away the mechanism and Raft promises exactly one thing: every node applies the same commands in the same order. That is called a replicated state machine. Feed identical, ordered inputs to identical state machines and they end up in identical states — so any node you read from gives the same answer, and a crashed node can rebuild itself by replaying the log.
So the whole problem reduces to agreeing on an ordered log of entries. Not the state — the log. Get every node to hold the same log in the same order and consistency falls out for free.
Raft splits that into two subproblems that it deliberately keeps separate: elect one leader to order writes, and replicate the log from that leader to everyone else.
Leader election, and the role of terms
At any moment each node is a follower, a candidate, or a leader. Normal operation has exactly one leader and everyone else a follower.
The leader sends periodic heartbeats. As long as followers hear them, they stay followers. When a follower hears nothing for its election timeout — a randomized interval on the order of hundreds of milliseconds (the Raft paper suggests 150–300 ms; etcd defaults to 1 s) — it assumes the leader is gone, becomes a candidate, increments the term, votes for itself, and asks everyone else for their vote.
The term is the idea that makes this safe. It is a logical clock — a number that only ever increases, incremented on every election. Every message carries its sender’s term, and the rule is absolute: a node that sees a higher term than its own immediately steps down to follower and adopts that term. This is how a partitioned old leader that reappears after the cluster has moved on discovers, in a single message, that it has been deposed — and stops acting like a leader before it can do any damage.
A candidate wins by collecting votes from a majority. Each node grants at most one vote per term, so at most one candidate can win — you cannot get two majorities in the same term to vote for different candidates, because the two majorities must overlap in at least one node, and that node only voted once.
Randomized timeouts are what break ties. If two followers time out together and split the vote, no one gets a majority, the term ends with no leader, and they time out again — but at different randomized intervals, so one almost always goes first next time. Split votes are possible but self-correcting.
Log replication and the commit rule
Once elected, the leader is the sole entry point for writes. A client sends a command; the leader appends it to its own log and sends AppendEntries to the followers.
Here is the entire heart of the algorithm, the rule worth memorizing:
An entry is committed once the leader has stored it on a majority of nodes. Only committed entries are applied to the state machine and acknowledged to the client.
Walk through why the majority is non-negotiable. Suppose a write lives on a majority. Any future leader must also have been elected by a majority. Two majorities of the same cluster always share at least one member. Therefore any future leader’s voters include at least one node that already has your committed write — and Raft’s election rules refuse to elect a candidate whose log is missing committed entries that a voter holds. The consequence: a committed entry can never be lost and can never be contradicted. The overlap of majorities is the safety proof, compressed to one sentence.
The diagram above shows the state mid-flight: entries 1–3 are on three of five servers, so they’re committed (green); entries 4–5 are on the leader and one follower but haven’t reached a majority yet (amber), so they are durably stored but not yet acknowledged. If the leader crashed this instant, 1–3 would survive any election; 4–5 might not, which is exactly why the client hasn’t been told they succeeded.
Why the node count is always odd
Fault tolerance is a function of majority size, and this table is the entire reason production clusters are 3, 5, or 7:
| Nodes | Majority | Failures tolerated |
|---|---|---|
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
| 6 | 4 | 2 |
| 7 | 4 | 3 |
Four nodes tolerate the same single failure as three, while costing more and being slightly more prone to split votes. Every even size is dominated by the odd size below it. So you never run an even cluster on purpose — the extra machine adds coordination cost and zero resilience.
Losing quorum: the failure you will actually meet
This is where consensus stops being theory and becomes an incident.
Raft is a CP system in CAP terms: it chooses consistency over availability. If the cluster cannot assemble a majority — three nodes down in a five-node cluster, or a network partition that isolates the leader with only a minority — then nothing commits. No writes, no new leader, no progress. The system freezes rather than risk two sides each believing they’re authoritative and diverging. Unavailable-but-correct beats available-but-split-brained, and Raft makes that choice for you.
Operationally this shows up as: etcd loses quorum → the Kubernetes API server can’t persist changes → the reconciliation loops that keep your cluster in its desired state stop reconciling. Pods that are already running keep running, because the data plane doesn’t need etcd for steady state — but nothing new schedules, no config takes effect, and every kubectl apply hangs. The cluster isn’t down, exactly. It’s stuck, which is its own kind of frightening the first time you see it.
Three implications for anyone who operates a Raft-backed system:
Spread nodes across failure domains, but mind latency. Every commit needs a majority round-trip, so the slowest node in the fastest majority sets your write latency. Three availability zones in one region is the sweet spot; a member on another continent drags every write toward it. Consensus writes are bounded by the network round-trip you can’t wire around.
Never lose a majority at once. Rolling upgrades take members down one at a time and wait for the cluster to be healthy between each. Pulling two nodes from a five-node cluster together is fine; pulling three is an outage you caused.
Back up the log/snapshot out of band. Quorum protects against crashes, not against a bad write, a corrupt snapshot, or rm -rf. Consensus keeps replicas identical — which means it faithfully replicates your mistake to all of them.
Where you already rely on this
Leader election via a consensus store is the standard way to get exactly one active instance of something — one active controller, one holder of a distributed lock — without two nodes ever both believing they’re it. etcd and Consul exist largely to sell you Raft as a service so you don’t implement it yourself. And “at most one leader per term,” enforced by majority voting, is precisely the property that prevents the split-brain a naive heartbeat-based failover would hand you.
The rule worth remembering
Agreement reduces to an ordered log; the log is safe because every commit lives on a majority, and any two majorities overlap. Everything else — terms, randomized timeouts, heartbeats — is machinery in service of that one invariant. When your cluster freezes, don’t ask “why is it down.” Ask “can it still form a majority.” That question is the whole game.
Comments