Raft consensus

Why a 5-server cluster keeps working after 2 servers die

Raft lets a group of servers agree on one ordered log of commands, so every copy applies the same writes in the same order. One server is elected leader. Every write goes through it, and it copies each write to the followers.

A write counts as committed, meaning safe forever, once a majority of servers have stored it. With 5 servers that is 3, so 2 can crash or drop off the network and the other 3 still accept and commit writes.

Time runs in numbered terms, each with at most one leader. A follower that hears no leader for a random timeout (about 150–300 ms) starts a new term and asks for votes. Each server votes once per term, and a majority wins.

Raft consensus

Kubernetes keeps its whole cluster's state in etcd, which runs Raft

Kubernetes, the tool that runs containers across many machines, stores which apps exist, where they run and their settings in etcd, a key-value store that uses Raft. Production setups usually run 3 or 5 etcd servers.

If one etcd machine dies, the rest elect a new leader, typically within a second or so, and Kubernetes carries on. Databases such as CockroachDB and TiKV, and HashiCorp's Consul, use Raft in the same way to keep their copies identical.

Raft consensus

Any two majorities share a server, so a committed write can't be lost

Raft is safe because any two majorities of the same cluster overlap. Among 5 servers, any 3 that stored a write and any 3 that vote in an election share at least one server, since 3 + 3 > 5.

A server refuses to vote for a candidate whose log is less up to date than its own. So a candidate missing a committed write can't win a majority, and every new leader already holds every committed entry.

An old leader cut off on a minority side can't reach a majority, so nothing it accepts commits. Once it sees a higher term, it steps down and its uncommitted entries are overwritten: Raft picks consistency.