Auto Scaling

In the previous pages, we put ShortLink on 5 servers behind a load balancer. We added and removed servers by hand. In this page, we make the system do that job by itself. This is auto scaling.

Definition

Auto scaling means the system adds servers when load goes up and removes servers when load goes down, automatically, using rules you set once.

You do not log in and click "add server". A program watches the load and does it for you.

In ShortLink, this means the number of app servers follows the number of clicks.

Let's see why doing it by hand is not enough.

Need for Auto Scaling

Here is what breaks with manual scaling. ShortLink's traffic changes during the day:

1Night   (8 hours):   800 RPS
2Day    (10 hours): 2,000 RPS
3Evening (6 hours): 3,000 RPS
4Viral spike:      10,000 RPS (nobody knows when)

One server handles about 1,000 requests per second (RPS). We keep servers near 70% busy, so each server takes about 700 RPS.

With manual scaling, we have two bad choices:

• Run for the peak all day. We pay for servers that sit idle at night.
• Run for the average. Users suffer during the evening, and a viral spike crashes us.

Here is the cost in numbers:

1Servers needed:
2Night:   800 / 700 = 1.1 → 2 servers
3Day:   2,000 / 700 = 2.9 → 3 servers
4Evening: 3,000 / 700 = 4.3 → 5 servers
5 
6Fixed at peak:  5 servers × 24 hours = 120 server-hours
7Auto scaling:   (2 × 8) + (3 × 10) + (5 × 6) = 16 + 30 + 30 = 76 server-hours
8 
9Saved: 120 - 76 = 44 server-hours → about 37% less

And a human cannot react to a viral tweet at 3 AM. Auto scaling gives us three things:

• Lower cost. We pay only for what we need.
• Better user experience. Capacity grows before users suffer.
• Less manual work. Nobody watches graphs all day.

Next, let's see how the system actually does this.

How Auto Scaling Works

Auto scaling is a simple loop that repeats every minute or so:

11. Watch    → collect numbers (for example, average CPU of all servers)
22. Decide   → compare them with the rules
33. Act      → add servers, remove servers, or do nothing
44. Wait     → give the change time to take effect, then repeat

1App Servers → send numbers → Monitoring → Auto Scaler → add / remove servers
2     ↑                                                          │
3     └──────────────────────────────────────────────────────────┘

Here is a very simple version of the loop in code:

public class AutoScalerLoop {
    private final MetricsClient metrics;          // gives us the numbers
    private final Fleet fleet;                    // starts and stops servers
    private final TargetTrackingScaler scaler;    // decides how many we need

    public AutoScalerLoop(MetricsClient metrics, Fleet fleet, TargetTrackingScaler scaler) {
        this.metrics = metrics;
        this.fleet = fleet;
        this.scaler = scaler;
    }

    // Called once every minute
    public void runOnce() {
        double averageCpu = metrics.averageCpu();
        int current = fleet.runningCount();
        int desired = scaler.desiredServers(current, averageCpu);

        if (desired > current) {
            fleet.addServers(desired - current);
        } else if (desired < current) {
            fleet.removeServers(current - desired);
        }
    }
}

Cloud platforms such as AWS, Google Cloud, and Azure provide this loop for you. You only set the numbers and the rules.

Now, what numbers should we watch?

Scaling Metrics

A metric is a number that tells us how the system is doing. Auto scaling decisions are only as good as the metric we pick.

For ShortLink's redirect servers, requests per server or CPU is a good start. A redirect is small, so more requests mean more CPU.

A good metric follows two rules:

• It moves with load. When traffic doubles, the number should clearly rise.
• It drops when we add servers. If we add servers and the number does not fall, it is the wrong metric.

Here is what breaks with a wrong metric. Say the database is the bottleneck. App servers sit at 20% CPU, waiting for slow database answers. Users see slow pages. Auto scaling looks at CPU, sees "all fine", and adds nothing. Even if it did add servers, the database would get worse.

Next, we need rules that turn a metric into an action.

Scaling Policies

A scaling policy is the rule that says what to do when a metric changes. There are four common kinds. Each has its own page:

• Target Tracking

  Pick a target for one number, like "keep CPU near 60%". The system works out how many servers keep it there. This is the easiest and most common choice.

• Step Scaling

  Bigger problems get bigger actions. For example, add 1 server at 70% CPU, add 3 servers at 85%, add 6 servers at 95%.

• Scheduled Scaling

  Scale at known times. For example, go to 5 servers at 6 PM every day, because ShortLink is always busy in the evening.

• Predictive Scaling

  Learn from past traffic and add servers before the rise. It works only when traffic follows a repeating pattern.

Here is how target tracking decides. The idea is one formula:

1desired servers = current servers × (current metric / target metric)
2 
3Example: 5 servers, average CPU = 90%, target = 60%
4desired = 5 × (90 / 60) = 7.5 → round up to 8 servers

After adding 3 servers, the same work is shared by 8 machines, so each is about 56% busy. Here is the same formula in code:

public class TargetTrackingScaler {
    private final int minServers;
    private final int maxServers;
    private final double targetCpu;   // for example 60.0

    public TargetTrackingScaler(int minServers, int maxServers, double targetCpu) {
        this.minServers = minServers;
        this.maxServers = maxServers;
        this.targetCpu = targetCpu;
    }

    public int desiredServers(int currentServers, double averageCpu) {
        double raw = currentServers * (averageCpu / targetCpu);
        int desired = (int) Math.ceil(raw);
        // Never go below the minimum or above the maximum
        return Math.max(minServers, Math.min(maxServers, desired));
    }

Most real systems mix them. Scheduled or predictive handles the expected rises. Target tracking handles the surprises.

Notice the code clamps the answer with a minimum and a maximum. Let's look at those limits.

Min, Max, and Desired Capacity

Auto scaling needs three numbers:

• Minimum: the fewest servers we ever run.
• Maximum: the most servers we ever run.
• Desired capacity: how many servers we want right now. Auto scaling changes this number, always between the minimum and the maximum.

1Minimum = 2     Maximum = 20     Desired = 5 (changes over time)
2 
32 ◄──────────── 5 ────────────────────────► 20
4min          desired now                    max

Why a minimum? One server is a single point of failure. With a minimum of 2, ShortLink survives one server dying, even at 3 AM. The minimum is also our ready capacity for a sudden spike.

Why a maximum? It protects us in two ways:

• Cost. A bug or an attack can make load look huge. Without a cap, the bill grows without limit.
• Database. Every server opens connections to the database. Too many servers can break it.

1Each server opens 20 database connections.
2Database allows 500 connections.
3Maximum safe servers: 500 / 20 = 25 → we set the max to 20 to leave room.

Here is how to pick the maximum for a spike:

1Viral spike: 10,000 RPS
2Servers needed: 10,000 / 700 = 14.3 → 15 servers
3 
4Max = 20 → covers the spike ✓
5Max = 10 → capacity is 10 × 700 = 7,000 RPS → still overloaded ✗

So the maximum must be high enough for the worst spike we plan for, but low enough to protect the database and the budget.

Now, a common problem appears when the rules are too eager.

Cooldown and Flapping

Here is what breaks. We set two rules:

• Add a server when CPU is above 70%.
• Remove a server when CPU is below 60%.

Watch what happens with 3 servers and a steady load:

110:00  3 servers, CPU 72% → above 70%, add 1 server
210:01  4 servers, CPU 54% → below 60%, remove 1 server
310:02  3 servers, CPU 72% → above 70%, add 1 server
410:03  4 servers, CPU 54% → below 60%, remove 1 server
5...repeats forever

This is called flapping. The system keeps adding and removing servers for no reason. It wastes money, and users can lose connections every time a server is removed.

The cause: adding a server lowers the average CPU, which crosses the other rule right away. The system forgets that its own action changed the number.

There are three fixes:

• Cooldown. After any change, wait a fixed time (for example 5 minutes) before changing again. This gives the new numbers time to settle.
• A bigger gap between the rules. Add at 70%, but remove only below 40%. Then 4 servers at 54% is "fine, do nothing".
• Scale in slowly. Remove one server at a time, and only after the load is low for a long time (for example 15 minutes).

1Scale out: fast       → add servers soon, because users are suffering
2Scale in:  slow       → remove servers late, because waiting costs only a little money

The rule of thumb: scale out fast, scale in slow. A late scale-out hurts users. A late scale-in only costs a little extra.

But even a perfect rule is not instant. New servers need time to start.

Startup Time and Warm-up

Here is what breaks. A viral tweet sends ShortLink from 3,000 RPS to 10,000 RPS within one minute. Auto scaling reacts, but a new server is not ready right away:

1Metrics show the high load:          ~1 min (the system checks once a minute)
2Cloud creates the machine:           ~1 min
3App starts and passes health check:  ~1 min
4New server is ready:                 ~3 min after the spike began
5 
6During these 3 minutes:
7Capacity: 5 servers × 700 = 3,500 RPS
8Traffic:  10,000 RPS
9→ users see errors and timeouts ✗

Auto scaling is fast for humans, but slow for a sudden spike. There is also a second delay called warm-up. A fresh server is "cold". Its memory caches are empty, and the app code runs slower at first. If the load balancer sends it a full share on the first second, it may struggle.

How to reduce the problem:

Remember: auto scaling can never save you from a spike that is much bigger than your spare room. It reduces the problem, but spare capacity is still our first defense.

New servers must also join the load balancer correctly. Let's connect the two.

Auto Scaling with a Load Balancer

Auto scaling adds the machines. The load balancer sends them traffic. They must work together:

1Scale out:
21. Auto scaler starts a new server
32. Server boots and starts the app
43. Load balancer runs health checks on it
54. Health check passes → load balancer starts sending traffic ✓
6 
7Scale in:
81. Auto scaler picks a server to remove
92. Load balancer stops sending it NEW requests
103. Server finishes the requests it is already handling   (this is draining)
114. Server shuts down ✓

Two details matter:

• Health check grace period. A new server needs time to start. For the first minute or two, the auto scaler should not mark it as "failed" just because it does not answer yet. Otherwise it kills the server and starts another one, again and again.
• Draining. If we shut a server down in the middle of a request, the user gets an error. Draining lets requests finish first, with a time limit, for example 30 seconds.

The load balancer also gives a very good metric. Requests per server is measured at the load balancer, and it moves directly with the work.

1Load balancer sees: 3,000 RPS across 5 servers
2Requests per server = 3,000 / 5 = 600
3Target: 700 per server → fine, do nothing
4 
5Traffic rises to 6,000 RPS → 1,200 per server
6Desired servers = 6,000 / 700 = 8.6 → 9 servers

Not every app can be scaled this way. Let's see what the app must look like.

Requirements for Auto Scaling

Auto scaling creates and destroys servers all the time. So a server must be easy to throw away and replace. Here is what ShortLink needs:

A good test: can you delete any server right now, with no user noticing? If yes, the app is ready for auto scaling. If a server holds a login or a file that exists nowhere else, it is not ready.

Even a ready app has limits. Let's list them.

Limitations and Trade-offs of Auto Scaling

1. It is not instant. Detection and startup take minutes, so very sudden spikes still hurt.

2. It needs the right metric. A wrong metric means wrong decisions, or no decisions.

3. It only scales what you tell it to. It scales app servers. It does not fix a slow database, a slow external service, or a bug in the code.

1App servers: 5 → 20 (auto scaled)
2Database: still 1 machine → now it gets 4× the connections → slower ✗

4. The bill can surprise you. A bug, a retry storm, or an attack can make auto scaling add many servers. The maximum limit and cost alerts protect us.

5. Tuning takes effort. Targets, cooldowns, and limits need testing. Wrong settings cause flapping or late reactions.

6. Removing servers can hurt. Without draining, users lose requests when a server is removed.

7. Hard to debug. Servers appear and vanish. You need central logs and monitoring to see what happened.

So when do we still choose manual scaling? Let's compare them directly.

Auto Scaling vs. Manual Scaling

When to choose which:

• Use manual when traffic is small and steady, or when the system is not ready for it (for example, servers keep state).
• Use auto scaling when traffic goes up and down, or when you cannot predict spikes.
• In real systems, combine them. Use scheduled scaling for events you know, such as a sale, and auto scaling for surprises.

Summary

1Auto scaling → add and remove servers automatically using rules
2→ Needed because traffic changes and humans react too slowly
3→ Loop: watch metrics → decide → add or remove servers → wait
4→ Metrics: CPU, requests per server, latency, queue length
5→ Policies: target tracking, step, scheduled, predictive
6→ Limits: minimum for safety, maximum for cost and database protection
7→ Cooldown stops flapping: scale out fast, scale in slow
8→ New servers need startup time, so keep spare capacity
9→ Works with a load balancer: health checks to join, draining to leave
10→ Needs stateless servers and a same-template, fully automatic setup
11→ Limits: not instant, can raise the bill, does not fix the database

ShortLink's app servers now grow and shrink by themselves, but every redirect still reads from the database, and the same popular links are opened again and again. Keeping those hot links in memory is caching, our next topic.