Fault Tolerance
Hardware dies, networks drop, deployments go wrong, and dependencies slow down. At scale, something is always failing. Fault tolerance means designing the system so that these failures do not reach the user.
What Is Fault Tolerance?
A fault is a part that breaks, such as a server, a disk, or a network link. A failure is when users see the problem. Fault tolerance is about making sure faults do not turn into failures.
It is related to, but not the same as, high availability:
- High availability is the goal: the system is up for as much time as possible.
- Fault tolerance is how we get there: redundancy, failover, and safe handling of errors.
Availability Numbers
Availability is the percentage of time the system works. Every extra "9" is much harder and more expensive to reach.
| Availability | Name | Downtime per year |
|---|---|---|
| 99% | Two nines | about 3.65 days |
| 99.9% | Three nines | about 8.76 hours |
| 99.99% | Four nines | about 52.6 minutes |
| 99.999% | Five nines | about 5.26 minutes |
Two simple formulas show why design matters:
- Services in a chain multiply. If a request needs three services that are each 99.9%, the whole request is only about 99.7% available. More dependencies means less availability.
- Redundant copies improve it. Two independent servers that are each 99% available give about 99.99%, because both must fail at the same time.
Interview Tip: Say "independent". The math for redundancy only works when failures are not shared. Two servers in the same rack or the same AZ can fail together.
RTO and RPO
Two numbers decide how much fault tolerance you must pay for:
| Term | Question it answers | Example |
|---|---|---|
| RTO (Recovery Time Objective) | How long can we be down? | "Back within 5 minutes" |
| RPO (Recovery Point Objective) | How much data can we lose? | "At most the last 1 minute of writes" |
A payment system may need RPO of zero. A view counter can lose a few seconds of data. The stricter the numbers, the higher the cost.
No single point of failure and redundancy
A single point of failure (SPOF) is any part whose failure takes the whole system down.
The method is simple. Take your architecture diagram and, for every box, ask: "What happens if this dies?" If the answer is "everything stops", you have a SPOF. The fix is redundancy: run more than one copy.
| Layer | Possible SPOF | How to remove it |
|---|---|---|
| Entry point | One load balancer | Multiple load balancers, or a managed one |
| Application | One server | Many stateless instances behind the load balancer |
| Database | One database | Primary plus replicas with automatic failover |
| Cache | One cache node | A replicated cache cluster |
| Queue | One broker | A replicated broker |
| Data center | One Availability Zone | Deploy across multiple AZs |
| Region | One region | Deploy across multiple regions |
| DNS | One DNS provider | A DNS service with global redundancy |
Stateless Services Make It Easy
If a server keeps no user data in its own memory, any instance can handle any request. When one dies, the load balancer simply sends traffic to the others. Keep state in a database, cache, or session store instead of in the server.
Multi-AZ vs Multi-Region
| Multi-AZ | Multi-Region | |
|---|---|---|
| Protects against | A data center or AZ failure | A whole region failure |
| Latency between copies | Low (same region) | High (far apart) |
| Data consistency | Easier | Harder |
| Cost and effort | Moderate | High |
| Common choice | Default for most systems | For critical or global systems |
Failure detection and failover
You cannot react to a failure you do not notice. So the system must detect failures and then switch to a healthy copy.
Detecting Failures
- Health checks: the load balancer or orchestrator calls an endpoint like
/healthevery few seconds. - Heartbeats: a node sends "I am alive" messages. If they stop, the node is treated as dead.
- Timeouts: if a call gets no answer in time, treat it as failed.
There are two kinds of health check, and they do different things:
| Check | Question | If it fails |
|---|---|---|
| Liveness | Is the process alive? | Restart it |
| Readiness | Can it handle traffic right now? | Stop sending traffic to it, but do not restart |
The trade-off: if you detect failures very fast, you get false alarms from short network blips, and servers flap in and out. If you detect slowly, users suffer longer. A common middle path is to mark a node unhealthy only after several failed checks in a row.
Failover Modes
Failover means moving the work from a failed component to a healthy one.
| Active-passive | Active-active | |
|---|---|---|
| How it works | One node serves. A standby waits and takes over | All nodes serve traffic at the same time |
| When one fails | The standby is promoted (takes time) | The others absorb its load immediately |
| Cost | Standby sits mostly idle | Needs spare capacity on every node |
| Complexity | Simpler | Harder, especially for data conflicts |
| Typical use | Databases | Stateless services |
A passive standby can be hot (always in sync and ready), warm (running but smaller), or cold (must be started first). The colder it is, the longer the failover and the higher the RTO.
Split-Brain
A dangerous case: the network breaks, and two nodes both think they are the primary. Both accept writes, and the data diverges. To prevent this, systems use leader election with a quorum (a majority must agree on who the leader is) and fencing (the old leader is cut off from writing).
Resilience patterns
In a system of many services, a slow dependency is often worse than a dead one. A dead service fails fast. A slow one makes callers wait, their threads pile up, and then they become slow too. This chain reaction is called a cascading failure. The patterns below stop it from spreading.
Timeouts
Always set a timeout on every network call. Without one, a single stuck call can hold a thread forever.
The timeout of a downstream call must be shorter than the timeout of the caller. If the user waits 3 seconds, the gateway should wait less, and each service behind it even less. This is called a timeout budget.
Retries with Exponential Backoff and Jitter
Many failures are temporary, so trying again often works. But careless retries can make things worse. Follow these rules:
- Retry only temporary errors (timeouts, 503). Do not retry errors like 400 or 404.
- Back off: wait longer after each failure (for example 100 ms, 200 ms, 400 ms, and so on, up to a limit).
- Add jitter: add randomness to the wait. Without it, all clients retry at the same moment, causing a retry storm (also called a thundering herd) that hits the recovering service again.
- Limit the attempts. Three or four tries is usually enough.
- Retry at one layer only. If each of 3 layers retries 3 times, the bottom service sees 27 requests for one user request.
| Attempt | Wait before retry (maximum) | With jitter, the real wait is |
|---|---|---|
| 1 | 100 ms | random between 0 and 100 ms |
| 2 | 200 ms | random between 0 and 200 ms |
| 3 | 400 ms | random between 0 and 400 ms |
| 4 | 800 ms | random between 0 and 800 ms |
public <T> T callWithRetry(Supplier<T> call) throws InterruptedException {int maxAttempts = 4;long baseDelayMs = 100;long maxDelayMs = 5000;for (int attempt = 1; attempt <= maxAttempts; attempt++) {try {return call.get();} catch (TransientException error) {if (attempt == maxAttempts) {throw error; // out of attempts, give up}long backoff = Math.min(maxDelayMs, baseDelayMs * (1L << (attempt - 1)));long delay = ThreadLocalRandom.current().nextLong(0, backoff + 1); // jitterThread.sleep(delay);}}throw new IllegalStateException("unreachable");}
Idempotency
A retry is only safe if repeating the request does not repeat the effect. Retrying "charge the customer" could charge them twice.
The fix is an idempotency key: the client sends a unique ID with the request, and the server remembers it. If the same ID arrives again, the server returns the saved result instead of doing the work twice. Payment APIs work this way.
Circuit Breaker
Like the breaker in your house, it cuts the connection when something is wrong, so the damage does not spread. It watches the calls to a dependency and works in three states:
| State | What happens |
|---|---|
| Closed | Normal. Requests pass through, and failures are counted. |
| Open | Too many failures. Calls fail immediately (or use a fallback) for a set time, so the broken service gets time to recover and your threads are not wasted. |
| Half-open | After the wait, a few test requests are allowed. If they succeed, go back to closed. If they fail, go back to open. |
A library such as Resilience4j (Java) provides this, so you do not write it yourself.
Bulkhead
Ships have sealed compartments, so a leak in one does not sink the whole ship. A bulkhead does the same for your system: it gives each dependency its own limited pool of threads or connections.
If the payment service becomes slow, only the payment pool fills up. Calls to the catalog and search services keep working, because they use different pools.
Fallback
When a call fails, return a safe alternative instead of an error:
- A cached or older value
- A default value
- A simpler result
For example, if the recommendation service is down, show popular products instead of a broken page.
Putting Them Together
| Pattern | Problem it solves |
|---|---|
| Timeout | A slow call holds resources forever |
| Retry + backoff + jitter | Temporary failures, without causing a retry storm |
| Idempotency | Retries repeating the same action |
| Circuit breaker | Hammering a dependency that is already down |
| Bulkhead | One bad dependency using up all resources |
| Fallback | Showing the user an error when a safe default exists |
A typical call combines them: it goes through a bulkhead, then a circuit breaker, then a retry, with a timeout on each attempt, and a fallback if all of it fails.
Data replication and backups
Servers are easy to replace. Data is not. To survive failures without losing data, we keep copies.
Leader-Follower Replication
All writes go to the leader. The leader sends the changes to followers, which keep their own copies and can also serve reads. If the leader dies, a follower is promoted to be the new leader.
Synchronous vs Asynchronous
| Synchronous | Asynchronous | |
|---|---|---|
| When the write is confirmed | After the follower has saved it too | As soon as the leader has saved it |
| Data loss on leader crash | None (RPO is 0) | The last few writes can be lost (replication lag) |
| Write speed | Slower | Faster |
| If a follower is down | Writes can block | Writes continue |
Many systems pick the middle way: wait for one follower (semi-synchronous), or use a quorum.
Quorum
With N copies, a write must reach W of them, and a read must ask R of them. If W + R is greater than N, every read overlaps with the latest write, so it sees fresh data. For example, with N = 3, W = 2, R = 2, the system keeps working with one node down.
Replication Is Not a Backup
Replication copies everything instantly, including mistakes. If someone runs a bad DELETE or a buggy deploy corrupts data, all replicas get the same damage.
So you also need:
- Regular backups, stored in a separate place
- Point-in-time recovery, so you can restore to "10 minutes before the mistake"
- Restore tests. A backup that was never restored is not proven to work.
Consistency or Availability
During a network partition, a system must choose (CAP theorem): stay consistent and refuse some requests, or stay available and risk stale data. A bank usually picks consistency. A social feed usually picks availability.
Graceful degradation and load shedding
Sometimes you cannot keep everything running. A fault-tolerant system gives up the less important parts first, and protects the core.
Graceful Degradation
When something fails or the load is too high, turn off non-essential features and keep the main function working.
| Feature | Importance | If it fails |
|---|---|---|
| Checkout and payment | Critical | Protect it at any cost |
| Product search | Important | Show a simpler or cached result |
| Recommendations | Nice to have | Show popular items |
| Reviews | Nice to have | Hide the section |
| Analytics events | Can wait | Drop them or queue them for later |
Feature flags act as kill switches: you can turn off an expensive feature in seconds, without a deploy.
Load Shedding
When traffic is more than the system can handle, trying to serve everyone makes everyone slow, and then everything fails. Load shedding means rejecting some requests early, with a quick 503 or 429 response, so the rest still succeed.
- Reject by priority: keep checkout, drop low-value traffic.
- Reject early, before the request uses expensive resources.
- Send a
Retry-Afterhint so clients do not retry at once.
Backpressure
Backpressure means telling the sender to slow down instead of piling up work. Examples are bounded queues that reject new items when full, and rate limits at the gateway. Without a limit, a growing backlog eats memory until the system crashes.
Note: Auto-scaling helps, but new servers take minutes to start. Load shedding and rate limiting protect you during those minutes.
Summary
- Faults are normal. The goal is that they do not become failures users can see.
- Availability is counted in nines. Chained dependencies lower it, and independent redundancy raises it. RTO is how long you can be down, and RPO is how much data you can lose.
- Remove every single point of failure. Ask "what if this dies?" for each box, and use stateless instances, multiple AZs, and multiple regions.
- Detect failures with health checks, heartbeats, and timeouts, then fail over. Use active-active for stateless services and active-passive for databases. Prevent split-brain with a quorum.
- Resilience patterns: timeouts, retries with backoff and jitter, idempotency, circuit breaker, bulkhead, and fallback. They stop one slow dependency from causing a cascading failure.
- Data: use leader-follower replication (sync for no loss, async for speed) and real backups, because replication also copies mistakes.
- Under overload: degrade gracefully, shed load by priority, and apply backpressure.
Interview Answer: How Do You Make a System Fault Tolerant?
I start by accepting that failures will happen, and I design so that they do not reach the user.
First, I remove single points of failure. For every component in the design, I ask what happens if it dies, and I add redundancy: stateless instances behind a load balancer, a database with replicas, and deployment across multiple Availability Zones, and multiple regions if the requirements need it.
Second, I detect and recover. Health checks and heartbeats find failed nodes, and automatic failover moves traffic to a healthy copy. I use a quorum to avoid split-brain.
Third, I protect against cascading failures with resilience patterns. I set timeouts, use retries with exponential backoff and jitter on idempotent calls, add a circuit breaker so I stop calling a broken dependency, use bulkheads to isolate resources, and provide fallbacks such as cached data.
Fourth, I protect data with replication, choosing sync or async based on the RPO, plus backups and point-in-time recovery, because replication copies mistakes too.
Finally, under overload I degrade gracefully by turning off non-critical features, and I shed load by priority. The RTO and RPO requirements decide how much of this is worth paying for.