Load Balancing Explained: How One URL Serves Millions of Users
How a load balancer spreads traffic across many servers so no single one is overwhelmed and none is a single point of failure. Covers round robin vs least connections, health checks, sticky sessions, L4 vs L7, and load balancer redundancy — with a clear diagram.
🔵 Fundamentals · Fresher-friendly → 🟠 senior nuance
Once you decide to run more than one server (and every serious system does), a new question appears: when a request arrives, which server should handle it? The answer is a load balancer — arguably the most important box in all of system design, because it's what turns a pile of separate servers into one smooth, reliable service. This guide explains what it does, the algorithms it uses to decide, and the senior-level details interviewers love.
What a load balancer actually is
A load balancer is a piece of software (or dedicated hardware) that receives all incoming traffic and forwards each request to one of several backend servers. To the outside world there's a single address; behind it, many identical servers share the work. Users never know — and never need to know — how many servers exist.
Analogy: it's the host at a busy restaurant. Guests (requests) arrive at one door; the host decides which free table (server) each party goes to, keeping every waiter evenly busy. If a table's out of service, the host just stops seating there.
Why you can't live without one
Two problems disappear the moment you add a load balancer:
- No overload: instead of one server drowning under all traffic, the work is spread so each handles a fair share. This is what makes horizontal scaling real — add a server, and the balancer starts using it.
- No single point of failure: if a server crashes, the balancer notices and routes around it. Users feel nothing. Without this, one dead server = your whole site down.
How it decides: the balancing algorithms
🟠 "Spread the load" sounds simple, but how you spread it matters. These are the algorithms you should be able to name and compare:
| Algorithm | How it picks a server | Best when |
|---|---|---|
| Round robin | Next server in turn, 1→2→3→1… | Servers are equal, requests similar |
| Least connections | The server with the fewest active requests | Requests vary in length |
| Weighted | Stronger servers get a bigger share | Servers have different power |
| IP hash | Same client always maps to the same server | You need "stickiness" (see below) |
Round robin and least connections cover the vast majority of cases. In an interview, "I'd use least connections since request durations vary" is a great, specific answer.
The picture
Notice Server 3 is down — the balancer simply stops sending it traffic. That detection comes from health checks.
Health checks: how it knows a server is alive
🟠 A load balancer regularly pings each server (e.g. requests a /health endpoint every few seconds). If a server stops answering or returns errors, the balancer marks it unhealthy and removes it from rotation — automatically, in seconds, with no human involved. When the server recovers and starts passing checks again, it's added back. This quiet mechanism is a huge part of why big sites feel "always up."
The stateless catch: sticky sessions
🟠 Here's where load balancing meets the golden rule of scaling. If your app stores a user's session in one server's memory, sending their next request to a different server logs them out. Two ways to handle it:
- Sticky sessions: tell the balancer to keep sending the same user to the same server (via IP hash or a cookie). Works, but it unbalances load and breaks if that server dies — a band-aid, not a cure.
- Stateless + shared store (better): keep no session in server memory; store it in a shared cache like Redis. Now any server can serve any user, and you never need stickiness. This is the approach real systems prefer, and exactly why statelessness keeps mattering.
L4 vs L7: two levels of smart
🔴 One senior distinction. A Layer 4 (L4) load balancer works at the network level — it just forwards packets by IP and port, blazing fast but "dumb" (it can't read the request). A Layer 7 (L7) balancer understands HTTP — it can route based on the URL path (send /images to one pool, /api to another), terminate HTTPS, and do smart routing. Most web systems use L7 for its flexibility; L4 shows up where raw speed matters most. Mentioning this split signals depth.
Wait — what if the load balancer itself dies?
Good instinct: the balancer could become the new single point of failure. In production you run at least two load balancers in a failover pair, so if one fails the other takes over (often sharing a floating IP). The principle never stops: remove every single point of failure, one layer at a time.
What to read next
- Caching Strategies — the next big lever for speed and scale
- Scalability: Vertical vs Horizontal — why the LB exists
- Reverse Proxy vs Load Balancer — two things people confuse
- ← The complete System Design guide (hub)
Frequently Asked Questions
What does a load balancer do?
A load balancer sits in front of several servers and forwards each incoming request to one of them, so no single server is overwhelmed. If a server fails, it routes around it. To users there is one address; behind it, many servers share the work.
What is the difference between round robin and least connections?
Round robin sends requests to servers in turn (1,2,3,1...), which works when servers and requests are similar. Least connections sends each request to the server currently handling the fewest active requests, which is better when request durations vary.
What is the difference between L4 and L7 load balancing?
A Layer 4 load balancer forwards packets by IP and port, very fast but it cannot read the request. A Layer 7 load balancer understands HTTP, so it can route by URL path, terminate HTTPS and make smarter decisions. Most web systems use L7.
