Load balancing
The conceptWhy one web address can be served by hundreds of machines
A load balancer sits in front of a group of identical servers. Every request reaches it first, and it forwards each one to a server, so users see one address while many machines share the work.
Say a shop gets 1,200 requests per second and one server handles 400. With round robin, the balancer sends request 1 to server A, 2 to B, 3 to C, and so on. With four servers, each gets about 300 per second.
Smarter rules check how busy servers are. Least connections picks the server with the fewest requests in progress. Power of two choices samples two servers at random and takes the less busy one, so balancers don't all pile onto one.
What it’s forTells you how busy each server is once the balancer spreads traffic evenly.
ρ = λ / (N · μ)
- ρ
- utilization: the fraction of each server's capacity in use, from 0 to 1
- λ
- incoming requests per second, e.g. 1,200
- N
- number of healthy servers behind the balancer
- μ
- requests per second one server can handle, e.g. 400
What it givesHow close each server is to its limit; near 1 means long queues and slow replies.
- λ = 1,200, μ = 400, N = 4
- ρ = 1,200 / (4 × 400) = 1,200 / 1,600 = 0.75
- Each server runs at 75% of its capacity