Scalability Fundamentals

We'll keep using ShortLink, our URL shortener. Right now, it runs on a single server, which works fine for a few users. In this module, we'll see what breaks as traffic grows, and how to fix it step by step.

Definition of Scalability

Think of a small tea stall. One person makes tea for 20 customers an hour. Life is good.

Now 200 customers show up. The stall breaks down. Long lines. Angry customers.

Scalability is the ability of a system to handle more work by adding resources, without breaking and without a big redesign.

In ShortLink, "more work" can mean more users, more short links, or more clicks.

1Few users  → 1 server → works fine
2Many users → 1 server → slow, then crashes
3Many users → more resources → still works   ← scalable

A scalable system does not need to be big today. It needs a clear way to grow tomorrow.

Click to zoom

Next, let's see why we need this at all.

Need for Scalability

Here is what breaks without it.

ShortLink launches. Day 1 has 1,000 users, and one server is enough. A famous person tweets a ShortLink link. Suddenly 1 million people click it in one hour.

1Without scalability:
2Clients → 1 Server (overloaded) → timeouts → users leave
3 
4With scalability:
5Clients → many Servers → everyone gets a fast redirect

We need scalability because:

• Growth is unpredictable. One viral moment can multiply traffic by 100.
• Slow systems lose users. People leave if a page takes a few seconds.
• Redesigning in a panic is expensive. Planning early is cheaper than fixing during a crash.

But we should not over-build either. A tiny app does not need 100 servers. We build only for the growth we can reasonably expect.

Scalability is often confused with other words, so let's separate them.

Scalability vs. Performance

Performance means how fast the system is for one user or a few users. Scalability means whether it stays fast as users grow.

Analogy: a sports car is fast for one driver. A bus is slower, but carries 50 people. Performance is the car. Scalability is the bus.

Quick test:

• Slow for one user → performance problem.
• Fast for one, slow for many → scalability problem.

Fixing one does not always fix the other. A faster server helps performance a little, but it will not save you from 1 million users.

Next, another word people mix up: availability.

Scalability vs. Availability

Availability means the system is up and answering requests when users need it. We usually measure it as a percentage of time, like 99.9%.

Analogy: a shop is "available" if the door is open when you arrive. A shop is "scalable" if it can serve a big crowd once you are inside.

They are different, but they help each other. If ShortLink has 5 servers and one dies, the other 4 keep working. Scaling out gave us availability for free.

The reverse is not true. One huge server can handle a lot of load, but if it dies, everything is down.

Now the third neighbour: reliability.

Scalability vs. Reliability

Reliability means the system gives the correct result, every time, even when things go wrong.

Analogy: a postman who is fast (scalable) but delivers letters to the wrong house is not reliable.

For ShortLink, a reliable system means sl.io/abc123 always goes to the right long link. A redirect to the wrong website is a reliability failure, even if the server is up and fast.

1Scalable    → can handle many users
2Available   → is up when users come
3Reliable    → gives the right answer

A good system needs all three. On this page we focus on scalability. To scale well, we first need to understand the load we are scaling for.

Workload and Traffic Patterns

Workload is the kind of work the system does. Traffic is how much of that work arrives, and when.

ShortLink has two kinds of work:

• Write: create a short link. Rare.
• Read: open a short link (redirect). Very common.

This is a read-heavy system. Say the ratio is 100 reads for every 1 write. That one fact will shape our design later, because reads can be sped up in ways writes cannot.

Traffic also changes over time. Common patterns:

Analogy: a restaurant is empty at 4 PM and packed at 8 PM. You must plan for the 8 PM crowd, not the 4 PM crowd.

Click to zoom

Now we need numbers to describe this load. That is where throughput and latency come in.

Throughput and Latency

Two numbers describe almost every system.

Latency is how long one request takes. Example: 20 ms for one redirect. Throughput is how many requests the system finishes per second. Example: 500 requests per second (RPS).

Analogy: a water pipe. Latency is how long one drop takes to travel through. Throughput is how much water comes out per second. A wider pipe gives more throughput, but a drop still takes the same time.

1Latency    = time for ONE request       (milliseconds)
2Throughput = requests finished / second (RPS)

Back-of-envelope math. Say one worker on a server takes 20 ms per redirect.

11 second = 1000 ms
2One worker: 1000 ms / 20 ms = 50 requests per second
310 workers: 10 × 50 = 500 requests per second

Adding workers raised throughput to 500 RPS. Each request still takes 20 ms, so latency did not change.

Now ShortLink's real load:

110 million redirects per day
21 day = 24 × 60 × 60 = 86,400 seconds
310,000,000 / 86,400 ≈ 116 requests per second (average)
4 
5Peak is often about 3× the average:
6116 × 3 ≈ 350 requests per second (peak)

We design for the peak, not the average. The average hides the dangerous moments.

Scaling aims to keep latency low while throughput goes up.

Click to zoom

But every system has a limit on throughput. Let's find it.

System Capacity and Bottlenecks

Capacity is the maximum load a system can handle before it slows down or fails.

A system has many parts. Each part has its own capacity. ShortLink looks like this:

1Client → App Server → Database
2         500 RPS      300 RPS

The app server can do 500 RPS, but the database can only do 300 RPS. So the whole system handles only 300 RPS.

The slowest part is the bottleneck. Analogy: a highway with 4 lanes that narrows to 1 lane at a bridge. Traffic speed is decided by the bridge, not the 4 lanes.

This leads to a very important rule:

Adding capacity to a part that is not the bottleneck changes nothing.

If we upgrade the app server to 1000 RPS, ShortLink still handles only 300 RPS. We must fix the database first. After that, a new bottleneck will appear somewhere else. Finding and fixing bottlenecks one by one is the main job of scaling.

How do we find the bottleneck? By watching resources.

Resource Utilization (CPU, RAM, Disk, Network)

Utilization is how much of a resource is being used, shown as a percentage. A bottleneck is usually a resource that is close to 100%.

Every server has four main resources:

Analogy: a kitchen. CPU is the chef. RAM is the counter space. Disk is the storage room. Network is the delivery door.

ShortLink examples:

• Many redirects per second → CPU gets busy.
• Millions of links cached in memory → RAM fills up.
• Millions of links saved in the database → Disk grows and gets slow.
• Huge viral traffic → Network gets full.

1CPU     90%  ← bottleneck
2RAM     40%
3Disk    30%
4Network 20%

Here, more RAM would not help. The CPU is the problem.

A common rule of thumb: keep utilization under about 70% at normal times. The rest is a safety cushion for spikes. A server running at 95% every day has no room for a surprise.

Once we know which resource is the problem, we need a way to add more of it.

Vertical Scaling vs. Horizontal Scaling

There are two ways to add resources.

Vertical scaling (scale up) means making one server bigger: more CPU, more RAM. Horizontal scaling (scale out) means adding more servers and sharing the work.

Analogy: a restaurant is crowded. Vertical is making the restaurant bigger. Horizontal is opening more branches.

1Vertical:    [ Server 4 CPU ]  →  [ BIG Server 32 CPU ]
2 
3Horizontal:  [ Server ]  →  [ Server ] [ Server ] [ Server ]

For ShortLink, going from 1 server to 3 servers means three machines each handling about 1/3 of the clicks. Something must decide which server gets each request. That job is done by a load balancer, which we cover on the next page.

When to choose which:

• Start with vertical when the system is small. It is the quickest fix.
• Move to horizontal when you hit the ceiling, or when you need availability.
• Big systems almost always end up horizontal.

Horizontal scaling sounds perfect, but it has a price.

Scaling Limitations and Trade-offs

Scaling is never free. Here is what we lose.

More servers means more complexity. With 1 server, there is 1 thing to watch. With 50, you need load balancing, monitoring, and a plan for when some of them fail.

State becomes a problem. If a user's data lives inside one server's memory, the next request may land on a different server that does not know it. So servers should be stateless, meaning they keep no user data themselves. Data goes to a shared place instead.

The database is hard to scale. App servers are easy to copy. A database holds data, and copying data means keeping every copy correct. This is usually the hardest part. We will look at it in later pages.

Some work cannot be split. If one step must happen one at a time, adding servers does not make it faster. Doubling servers rarely doubles speed.

Cost grows. More machines means a bigger bill, more things to maintain, and more things that can break.

The goal is not to scale as much as possible. The goal is to scale as much as needed, at a sensible cost.

Interview angle. Interviewers expect you to:

• Ask about load first (users, reads vs writes, peak RPS) before drawing anything.
• Find the bottleneck before choosing a fix.
• Explain trade-offs, not just say "add more servers".

Common mistakes:

• Saying "horizontal is always better".
• Mixing up scalability, availability, and performance.
• Scaling the app server while the database is the real bottleneck.

Summary

1Scalability = handle more load by adding resources
2→ Needed because traffic grows and spikes
3→ Not the same as performance (speed), availability (uptime), or reliability (correctness)
4→ Know the workload: read-heavy vs write-heavy, daily peaks, spikes
5→ Measure with latency (time per request) and throughput (requests per second)
6→ Capacity is limited by the bottleneck, the slowest part
7→ Watch CPU, RAM, Disk, Network utilization (keep under ~70%)
8→ Scale up (bigger machine) or scale out (more machines)
9→ Scaling costs complexity, state problems, and money

ShortLink still runs on a single server, and traffic keeps growing. The simplest fix is to make that one server more powerful: more CPU, more memory, faster disks. That is vertical scaling, our next topic.