Load Balancing

In the previous page, we put ShortLink on 5 servers behind a load balancer. We said what it does, but not how it works or how it picks a server. In this page, we open the box and look inside.

Definition

Load balancing means spreading incoming requests across several servers, so that no single server gets overloaded.

A load balancer is the machine or software that does this job. It sits in front of the servers and decides where each request goes.

Think of a supermarket with 5 checkout counters. A staff member stands at the entrance and points each customer to a counter with a short line. The staff member doesn't scan any items. They only keep the lines even.

Click to zoom

Let's see what breaks without one.

Need for Load Balancing

Here is the problem. ShortLink has 5 servers, but users only know one address, sl.io. If all requests go to Server 1, the other 4 servers do nothing.

Click to zoom

A load balancer gives us three things:

• Even load. No server is overloaded while others are idle.
• Higher availability. If a server dies, the load balancer stops sending traffic to it.
• Easy scaling. We can add or remove servers, and users never notice.

Next, let's follow one request through the load balancer.

How a Load Balancer Works

A user opens sl.io/abc123. Here is what happens:

Click to zoom

The load balancer keeps two things in memory:

• A list of servers it can send traffic to.
• A health status for each server: healthy or not.

For every request, it does two simple steps:

1. Remove unhealthy servers from the list.
2. Use a rule to pick one server from the healthy ones.

The client only sees the load balancer's address. It never knows how many servers are behind it, or which one answered.

Load balancers come in different kinds. Let's group them.

Types of Load Balancers

Load balancers can be grouped by where the balancing happens. Each type has its own page:

• Server-Side Load Balancer

  A separate machine sits in front of the servers and picks one for each request. Clients only know its address. This is what ShortLink uses now.

• Client-Side Load Balancer

  The client keeps a list of servers and picks one itself. No middle machine is needed. It is common when one internal service calls another service.

• DNS-Based Load Balancer

  DNS is the system that turns a name like sl.io into a server address. Here, DNS gives different addresses to different users, so the traffic is split before it reaches any machine.

• Global Load Balancer

  Sends each user to the nearest or healthiest data center (a building full of servers) in a different region. It is used when ShortLink runs in more than one country.

1Server-side:  Client → Load Balancer → Servers
2Client-side:  Client (picks) → Server 1 / Server 2 / Server 3
3DNS-based:    Client → DNS → address of Server 1 or Server 2
4Global:       Client → Global LB → nearest Data Center

These types can be used together. For example, a global load balancer picks a data center, and a server-side load balancer inside it picks a server.

No matter the type, every load balancer needs a rule to pick a server. Let's look at the common rules.

Load Balancing Algorithms

An algorithm here is just the rule the load balancer uses to pick a server. Here are the four most common ones.

Round Robin

Round robin is the simplest rule. The load balancer keeps a list of servers and sends each new request to the next server in the list. When it reaches the end, it goes back to the start.

Analogy: a teacher dealing cards to 3 students. One card to student 1, one to student 2, one to student 3, then back to student 1. Nobody is skipped, and everyone gets the same number of cards.

Click to zoom

Here is the same idea in code:

public class RoundRobinBalancer {
    private final List<String> servers;
    private final AtomicInteger counter = new AtomicInteger(0);

    public RoundRobinBalancer(List<String> servers) {
        this.servers = servers;
    }

    public String pickServer() {
        // AtomicInteger is safe when many requests arrive at the same time
        int index = Math.floorMod(counter.getAndIncrement(), servers.size());
        return servers.get(index);
    }
}

For ShortLink, every redirect is fast and similar. With 3,000 RPS and 5 servers, round robin gives each server 600 RPS. The load is even, and there is almost nothing to track.

Here is what breaks. Round robin counts requests, but it does not check how busy each server is. Say Server 2 gets a few slow requests and gets stuck. Round robin still sends it the next request, because it is "its turn".

Choose it when servers are equal and requests take about the same time.

If servers have different power, we need a smarter rule.

Weighted Round Robin

Weighted round robin is round robin with one change: stronger servers get more turns. Each server gets a number called a weight. A server with weight 2 gets twice as many requests as a server with weight 1.

Analogy: a teacher dealing cards, but one student has a bigger desk. That student gets 2 cards for every 1 card the others get.

Here is what breaks with plain round robin. Say ShortLink has two different servers. Server A is big and can handle 1,000 RPS. Server B is small and can handle 500 RPS. Peak traffic is 900 RPS.

Click to zoom

A simple way to build it: put each server in the list as many times as its weight, then use plain round robin on that list.

public class WeightedRoundRobinBalancer {
    private final List<String> slots = new ArrayList<>();
    private final AtomicInteger counter = new AtomicInteger(0);

    public WeightedRoundRobinBalancer(Map<String, Integer> weights) {
        // Weight 2 means the server appears 2 times in the list
        for (Map.Entry<String, Integer> entry : weights.entrySet()) {
            for (int i = 0; i < entry.getValue(); i++) {
                slots.add(entry.getKey());
            }
        }
    }

    public String pickServer() {
        int index = Math.floorMod(counter.getAndIncrement(), slots.size());
        return slots.get(index);
    }
}

// Usage:
// Map<String, Integer> weights = new LinkedHashMap<>();
// weights.put("ServerA", 2);
// weights.put("ServerB", 1);
// WeightedRoundRobinBalancer lb = new WeightedRoundRobinBalancer(weights);

Real load balancers spread the turns more smoothly, like A, B, A, A, B, A, so a strong server does not get many requests in a row. The idea is the same.

Choose it when your servers have different power, but requests take about the same time.

Weights are fixed, but real load changes every second. What if we pick the server that is least busy right now?

Least Connections

Least connections sends each new request to the server that has the fewest active connections right now. A connection is one open conversation between a client and a server. A server with many open connections is probably busy. A server with few is probably free.

Analogy: choosing a line at a supermarket. You do not pick by turn. You look and join the shortest line.

Here is what breaks with round robin. ShortLink adds a link analytics page. Most requests take 20 ms, but some analytics requests take 5 seconds. A server that gets a few slow requests fills up with open connections.

Click to zoom

To do this, the load balancer must track connections. It adds 1 when a request starts and subtracts 1 when it ends.

public class LeastConnectionsBalancer {
    private final Map<String, AtomicInteger> active = new ConcurrentHashMap<>();

    public LeastConnectionsBalancer(List<String> servers) {
        for (String server : servers) {
            active.put(server, new AtomicInteger(0));
        }
    }

    public String pickServer() {
        String best = null;
        int fewest = Integer.MAX_VALUE;
        for (Map.Entry<String, AtomicInteger> entry : active.entrySet()) {
            int count = entry.getValue().get();
            if (count < fewest) {
                fewest = count;
                best = entry.getKey();
            }
        }
        active.get(best).incrementAndGet();   // a new connection starts
        return best;
    }

    public void requestFinished(String server) {
        active.get(server).decrementAndGet(); // the connection ended
    }
}

This code is a simple version. In real load balancers, picking and counting happen together safely, so two requests at the same moment do not both pick the same server.

Note that a server with few connections is not always the least loaded. One heavy request can use more power than ten light ones. Least connections is a good guess, not a perfect measure.

Choose it when requests take very different amounts of time, such as a mix of fast redirects and slow analytics.

Both rules spread load. But sometimes we want the opposite: the same client should always reach the same server.

IP Hash

IP hash picks a server using the client's address. An IP address is the number that identifies a client on the network, like 203.0.113.7. The load balancer runs this address through a hash function, a formula that turns any input into a number. That number picks the server.

Analogy: a school assigns students to classrooms by roll number. The same roll number always gives the same classroom.

1server = hash(client IP) mod number of servers
2Same IP → same result → same server

Here is what breaks with round robin. Alice's first request goes to Server 1 and her next one goes to Server 2. If Server 1 kept her data in memory, Server 2 does not have it. With IP hash, Alice always lands on the same server, so no cookie is needed.

public class IpHashBalancer {
    private final List<String> servers;

    public IpHashBalancer(List<String> servers) {
        this.servers = servers;
    }

    public String pickServer(String clientIp) {
        int index = Math.floorMod(clientIp.hashCode(), servers.size());
        return servers.get(index);
    }
}

Here is the catch. The result depends on the number of servers. When we add or remove a server, the formula changes, and many clients move to a different server.

1Say the hash numbers are: Alice = 10, Bob = 11, Carol = 12
2 
33 servers:  Alice → 10 mod 3 = 1   Bob → 11 mod 3 = 2   Carol → 12 mod 3 = 0
44 servers:  Alice → 10 mod 4 = 2   Bob → 11 mod 4 = 3   Carol → 12 mod 4 = 0
5 
6Alice and Bob moved. Only Carol stayed. ✗

Moved clients lose whatever the old server remembered about them. There is a smarter method called consistent hashing that moves far fewer clients. We cover it in a later page.

Choose it when the same client should reach the same server and you cannot use cookies. It is a lighter cousin of sticky sessions, which we cover shortly.

All these rules assume the chosen server is alive. But what if it is dead? That is where health checks come in.

Here is when to use each:

For ShortLink, a redirect is very fast and every request is similar. So round robin is a good start. If we add a slow feature like link analytics, least connections becomes a better fit.

But what if the chosen server is dead? That is where health checks come in.

Health Checks and Failover

Here is what breaks. Server 3 crashes. If the load balancer keeps sending requests to it, 1 out of every 5 users gets an error.

The fix is a health check: the load balancer regularly asks each server, "Are you alive?"

1Every 5 seconds:
2Load Balancer → Server 3   "GET /health"
3Server 3 → Load Balancer   "OK"        ← healthy
4                           (no answer) ← counts as a failure
5 
63 failures in a row → mark Server 3 as unhealthy

There are two ways to check:

• Active check. The load balancer sends test requests on its own, like the one above.
• Passive check. The load balancer watches real user requests. If a server keeps returning errors or timing out, it is marked unhealthy.

Failover means moving the work from a failed server to the healthy ones.

1Before: 5 servers × 600 RPS = 3,000 RPS
2Crash:  Server 3 dies → 4 servers left
3After:  3,000 / 4 = 750 RPS each → 75% busy, still fine

When Server 3 is fixed and passes the health check again, the load balancer puts it back into the list.

A catch: detection takes time. With checks every 5 seconds and 3 failures needed, it can take up to about 15 seconds to notice a dead server. During that time, some requests may fail.

Next, let's see how smart the load balancer can be about what it reads in each request.

Layer 4 vs. Layer 7 Load Balancing

Network traffic is handled in layers. Each layer looks at a different part of the data. Two layers matter for load balancing:

• Layer 4 (transport layer): sees only the IP address and port number. It does not read what is inside the request.
• Layer 7 (application layer): reads the actual HTTP request, like the URL, headers, and cookies.

Analogy: a post office sorter. A Layer 4 sorter only reads the city on the envelope. A Layer 7 sorter opens the envelope, reads the letter, and decides based on what it says.

Here is how Layer 7 helps ShortLink. We have two kinds of work: creating links (writes) and opening links (reads). With Layer 7, we can send each kind to its own group of servers:

1                          ┌→ /api/create  → Create Servers
2Clients → Load Balancer → ┤
3                          └→ /abc123      → Redirect Servers

A Layer 4 load balancer cannot do this. It does not see the URL.

Load balancers also differ in what they run on:

Most new systems start with a cloud or software load balancer.

Now let's revisit a problem from the last page: login state.

Sticky Sessions

Remember the login problem. Alice logs in on Server 1, then her next request goes to Server 2, which doesn't know her.

Sticky sessions (also called session affinity) fix this by making the load balancer send the same user to the same server every time. It does this using a cookie (a small piece of data the browser stores) or the client's IP address.

1Alice's 1st request → Server 1  (load balancer remembers: Alice = Server 1)
2Alice's 2nd request → Server 1  ✓ same server
3Alice's 3rd request → Server 1  ✓

It works, but it brings problems back:

• Uneven load. If Alice sends many requests, Server 1 stays busy while others are free.
• Lost sessions. If Server 1 dies, Alice loses her login.
• Harder scaling. Removing a server breaks the sessions on it.

Sticky sessions are a temporary patch. The better fix is stateless servers.

Now, a bigger worry. We added a load balancer to remove single points of failure. But what about the load balancer itself?

Load Balancer as a Single Point of Failure

Here is what breaks. All traffic goes through one load balancer. If it dies, ShortLink is down, no matter how many healthy servers sit behind it.

1Clients → [ Load Balancer ] → 5 healthy servers
2                 ✗
3          Load balancer crashes → ShortLink is down

We solved the problem for the app servers, and created it again one level up.

The fix is to run two load balancers. There are two common ways:

Active-passive. One load balancer works. The other waits as a standby and checks on the first one with regular "I'm alive" signals, called heartbeats. If the heartbeats stop, the standby takes over, using the same address.

1Normal:    Clients → LB 1 (active) → Servers
2                     LB 2 (standby, waiting)
3 
4LB 1 dies: Clients → LB 2 (now active) → Servers

Active-active. Both load balancers work at the same time, and each takes part of the traffic. If one dies, the other takes everything.

The risk in the last row means each load balancer must be able to handle the full load alone.

Cloud load balancers are usually built on several machines already, so the provider handles this for you.

Even with backups, a load balancer has costs. Let's list them.

Limitations and Trade-offs of Load Balancing

1. Extra hop. Every request now goes through one more machine. This adds a small delay.

2. Extra cost and effort. You pay for the load balancer, its backup, and the work to set it up and watch it.

3. It can become a bottleneck. The load balancer has its own capacity. If traffic grows past it, everything slows down, even if the servers are fine.

4. Mistakes affect everyone. A wrong setting can break every request at once.

5. It does not fix everything. It spreads load, but it cannot fix a slow database or code that keeps state in memory.

6. Rules are not perfect. Round robin ignores how busy a server is. Smarter rules need more tracking, which costs more.

For a tiny site with one server, a load balancer is not needed. Add it when you have two or more servers.

There is one more confusion to clear. People mix up three similar terms.

Load Balancer vs. Reverse Proxy vs. API Gateway

All three sit in front of your servers. They differ in what they do.

Reverse proxy. It receives requests on behalf of your servers and forwards them. Clients talk to the proxy, never to the servers directly.

Load balancer. It is a reverse proxy whose main job is spreading requests across many servers.

API gateway. It is a reverse proxy built for APIs. It sits in front of many different services and adds extra rules: login checks, rate limiting (blocking users who send too many requests), and routing each request to the right service.

Analogy: an office building.

• A reverse proxy is the front desk. Visitors talk to the desk, not to the employees.
• A load balancer is the front desk sending each visitor to a free employee.
• An API gateway is the front desk with a security check. It checks your badge, limits how often you visit, and sends you to the right department.

1Reverse proxy:  Client → Proxy → Server
2Load balancer:  Client → Load Balancer → Server 1 / Server 2 / Server 3
3API gateway:    Client → Gateway (login, limits) → Create Service / Redirect Service

In real tools, these roles overlap. NGINX, for example, can act as all three. Think of them as jobs, not separate products.

Summary

1Load balancing      →  spread requests across servers so none is overloaded
2Need                →  extra servers are useless if one gets all the traffic
3How it works        →  one address → pick a healthy server using a rule → forward
4Types               →  server-side, client-side, DNS-based, global
5Algorithms          →  round robin, weighted, least connections, IP hash
6Health checks       →  find dead servers and send traffic only to healthy ones
7Layer 4 vs Layer 7  →  L4 is fast (IP and port), L7 is smart (URL and headers)
8Sticky sessions     →  same user, same server; a patch, not a fix
9Load balancer SPOF  →  run two (active-passive or active-active)
10Limitations         →  extra hop, extra cost, can be a bottleneck

ShortLink now spreads 3,000 RPS across 5 servers, and every request still reads from the database. Since ShortLink is read-heavy and the same popular links are opened again and again, we can avoid most database reads by keeping them in memory, which is caching, our next topic.