Horizontal Scaling

In the previous page, we made ShortLink's single server bigger. It worked, but we hit a ceiling. The server is still one machine, so it has a size limit, it goes down during upgrades, and if it fails, every short link breaks. In this page, we fix that by using more than one server.

Definition of Horizontal Scaling

Horizontal scaling means handling more load by adding more machines and sharing the work between them.

You don't make one server bigger. You add more servers of the same kind.

Think of the restaurant again. Vertical scaling gave the one cook a bigger kitchen. Horizontal scaling hires more cooks. Each cook is normal-sized, but together they serve many more customers.

1Before:  [ Server ]
2After:   [ Server ] [ Server ] [ Server ]   ← same size each, more of them

Because we add machines "sideways", horizontal scaling is also called scaling out.

Let's see how this works for ShortLink.

How Horizontal Scaling Works

Here is what breaks first. ShortLink's peak traffic is now 3,000 requests per second (RPS). One server handles about 1,000 RPS. Even the biggest machine is not an option anymore.

Here's how we fix it:

1. Run copies of the same app. Each copy is identical. Any copy can answer any request.
2. Put something in front that splits incoming requests between the copies.
3. Add more copies when traffic grows.

1                      ┌→ App Server 1
2Clients → Splitter →  ├→ App Server 2
3                      └→ App Server 3

Back-of-envelope math:

1Peak traffic:           3,000 RPS
2One server handles:     1,000 RPS
3Servers needed:         3,000 / 1,000 = 3
4 
5But we don't want servers at 100%. Keep them near 70%:
6Each server should take: 1,000 × 0.7 = 700 RPS
7Servers needed:          3,000 / 700 ≈ 4.3 → round up to 5

So ShortLink needs about 5 small servers, not one giant one.

Next, let's see how we add and remove servers over time.

Scale Out vs. Scale In

Horizontal scaling works in both directions, just like vertical scaling did:

• Scale out: add more servers when load goes up.
• Scale in: remove servers when load goes down, to save money.

The big difference from vertical scaling: no restart of existing servers. New servers join while the old ones keep working. Users never notice.

Two things must be done carefully:

• Adding: a new server must be fully started and healthy before it receives traffic. Otherwise users get errors.
• Removing: a server should finish the requests it is already handling before it shuts down. This is called draining.

Many cloud platforms do this automatically. It is called auto scaling: the platform watches a number like CPU usage, adds servers when it's high, and removes them when it's low.

1CPU above 70% for 5 minutes  → add 2 servers
2CPU below 30% for 15 minutes → remove 1 server

But who decides which server gets each request? That is the load balancer.

Load Balancing in Horizontal Scaling

Here is what breaks without it. If we have 5 servers but users all connect to Server 1, the other 4 do nothing. Extra servers are useless unless the work is shared.

A load balancer is a machine that sits in front of the servers and spreads requests between them.

Analogy: the host at a busy restaurant. The host doesn't cook. They greet each customer and send them to a table that has space.

1                          ┌→ App Server 1
2Clients → Load Balancer → ├→ App Server 2
3                          ├→ App Server 3
4                          ├→ App Server 4
5                          └→ App Server 5

The load balancer does three simple jobs:

• Spread requests so no single server gets overloaded. The simplest way is taking turns: Server 1, then 2, then 3, and so on.
• Check health. It regularly asks each server "are you alive?". If a server doesn't answer, it stops sending traffic there.
• Hide the servers. Users only know one address. They don't know there are 5 servers behind it.

Because of the load balancer, we can add or remove servers without users noticing. We will study load balancers in detail in a later page.

Now there's a catch. Any server can get any request, so servers must not depend on remembering things.

Stateless vs. Stateful Applications

Here is what breaks. ShortLink now has a login page. Users sign in to see their own links.

Server 1 remembers that Alice is logged in, by keeping it in its own memory. Alice's next click goes to Server 2. Server 2 has never heard of Alice, so it says "please log in again."

1Request 1:  Alice → Load Balancer → Server 1   "Login OK" (Server 1 remembers)
2Request 2:  Alice → Load Balancer → Server 2   "Who are you?" ✗

This happens because Server 1 is stateful.

• State means data that the server remembers between requests, like who is logged in.
• A stateful server keeps that data inside itself.
• A stateless server keeps nothing. Each request carries what it needs, or the server fetches it from a shared place.

Analogy: a waiter who remembers your order in his head (stateful) vs. a waiter who writes it on a shared order slip that any waiter can read (stateless). If your waiter goes on break, the slip still works.

Here is the same idea in code:

// Stateful: the session lives inside this one server
public class StatefulLogin {
    private Map<String, String> sessions = new HashMap<>();

    public void login(String sessionId, String userName) {
        sessions.put(sessionId, userName);   // only THIS server knows
    }

    public String whoIs(String sessionId) {
        return sessions.get(sessionId);      // null on any other server
    }
}

// Stateless: the session lives in a shared store
public class StatelessLogin {
    private SessionStore sharedStore;        // all servers use the same one

    public void login(String sessionId, String userName) {
        sharedStore.save(sessionId, userName);
    }

    public String whoIs(String sessionId) {
        return sharedStore.get(sessionId);   // works on every server
    }
}

There are two common ways to make servers stateless:

There is also a quick fix called sticky sessions: the load balancer always sends Alice to the same server. It works, but it brings back the old problems. Load becomes uneven, and if that server dies, Alice loses her session. So we prefer stateless servers.

The rule for horizontal scaling: keep app servers stateless. Then any server can be added or removed freely.

But the data has to live somewhere. That brings us to storage.

Shared Storage and Distributed Databases

Here is what breaks. Suppose each of the 5 servers keeps its own copy of the data.

Alice creates sl.io/abc123 on Server 1. Bob opens that link, and his request lands on Server 3. Server 3 has no record of abc123, so Bob gets "link not found."

1Alice → Server 1 → saves abc123 in its own storage
2Bob   → Server 3 → "abc123 not found" ✗

The fix: all servers share one database.

1                          ┌→ App Server 1 ─┐
2Clients → Load Balancer → ├→ App Server 2 ─┼→ Database
3                          └→ App Server 3 ─┘

Now every server sees the same links. App servers are stateless, and the data lives in one place.

But this creates a new problem. We scaled the app servers to 5, and all 5 now send requests to one database. The database becomes the new bottleneck, the slowest part that limits everything else.

Copying app servers is easy because they hold no data. Copying a database is hard because every copy must stay correct. To scale it, we use a distributed database, which means the data is spread over several machines. Two common ideas:

• Replication: keep extra copies of the data, so more machines can answer read requests.
• Sharding: split the data into parts, so each machine holds only a part.

ShortLink is read-heavy, so replication helps a lot. We cover both ideas in later pages. For now, remember that scaling the app is easy, and scaling the data is hard.

Now let's see what we gained from all this work.

Advantages of Horizontal Scaling

1. No hard ceiling. Need more power? Add more servers. You are not limited by the size of one machine.

2. Better availability. If one server dies, the others keep working. We cover this in the next section.

3. No downtime to scale. New servers join while the old ones keep serving users.

4. Cost grows smoothly. You use many ordinary machines, which are cheaper than one giant machine. You also pay only for what you run: 10 servers at peak, 5 at night.

5. Handles spikes. When a link goes viral, you add servers quickly. When it's over, you remove them.

For ShortLink, this means we can survive a viral tweet without any panic.

The biggest gain deserves a closer look.

Fault Tolerance and High Availability

Remember the single point of failure from the last page? One server crashed, and everything stopped.

Fault tolerance means the system keeps working even when some parts fail. High availability means the system is up almost all the time.

With 5 servers, losing one is not a disaster:

1Before:  5 servers × 700 RPS = 3,500 RPS capacity
2Crash:   Server 3 dies. 4 servers left.
3After:   3,000 RPS / 4 servers = 750 RPS each → 75% busy, still fine

The load balancer notices the dead server through its health checks and stops sending traffic to it. Users see nothing.

Here is a simple way to see the availability gain. Say one server is up 99% of the time, which means it is down about 3.6 days a year.

1One server down chance:         1% = 0.01
2Two servers, both down at once: 0.01 × 0.01 = 0.0001 = 0.01%
3Availability with two servers:  99.99%

This math assumes the servers fail independently. In real life they can fail together, for example if they share the same power or the same bad update. So teams place servers in different locations too.

Be careful, though. If there is only one load balancer and one database, they are still single points of failure. High availability needs backups for every important part, not just the app servers.

So horizontal scaling is great. But it isn't free.

Limitations of Horizontal Scaling

Here is what we pay for.

1. More complexity. With 1 server, there is 1 thing to watch. With 5, we need a load balancer, health checks, monitoring, and a plan for failures.

2. State must be handled. Servers must be stateless, as we saw. Old apps that keep data in memory need changes first.

3. The database is the next bottleneck. The app can scale to 50 servers, but if they all hit one database, the database limits everything. Fixing this needs replication or sharding, which is hard.

4. Data consistency is harder. When data has many copies, they can briefly disagree. Keeping them in sync is a big topic we cover later.

5. Network delay. Servers talk to each other over a network, which is slower than talking inside one machine.

6. Gains are not perfect. Doubling the servers rarely doubles the speed, because shared parts like the database and load balancer don't double.

7. Cost and effort. More machines mean a bigger bill, more tools, and more people needed to run it. At small scale, one server is cheaper and simpler.

Now let's put both scaling styles side by side.

Horizontal Scaling vs. Vertical Scaling

When to choose which:

• Start vertical while the system is small. It is the quickest and cheapest fix.
• Go horizontal when you hit the size limit, or when you need high availability.
• Most real systems use both: scale the database up first, and scale the app servers out.

Summary

1Horizontal scaling  →  add more machines instead of a bigger one (scale out)
2Scale out / in      →  add servers at peak, remove them when quiet
3Load balancer       →  spreads requests and skips dead servers
4Stateless servers   →  keep no user data inside, so any server can handle any request
5Shared data         →  all servers use one database, which becomes the next bottleneck
6Advantages          →  no ceiling, high availability, no downtime, handles spikes
7Limitations         →  complexity, state, database bottleneck, higher cost at small scale
8Best practice       →  scale up first, scale out when needed, often use both

ShortLink now runs on 5 servers behind a load balancer, but we only said what a load balancer does, not how it decides where each request goes. That is the job of the load balancer, our next topic.