API Gateway
API Gateway is one of the most important components in a microservices architecture. The client should not need to know about every backend service, so we put an API Gateway in front of them as the entry point.
What is an API Gateway?
An API Gateway is a single entry point between the client and the backend microservices. Its main job is:
Accept the client's API request and route it to the correct backend service based on the API endpoint.
For example, GET /api/invoice is routed to the Invoice Microservice.
Why do we need it?
Without a gateway, the client talks directly to every service (Order, Invoice, Sales). This tightly couples the client to the backend architecture.
With a gateway, the client talks only to the gateway, and the gateway handles the routing.
API Gateway vs Load Balancer
Both route traffic, so how are they different? This is a very common interview question. The difference is what they route between.
| API Gateway | Load Balancer |
|---|---|
| Routes requests between different services | Distributes traffic between instances |
| Understands API endpoints | Primarily distributes traffic |
/order → Order Service | Order Service → Order Instance 1/2/3 |
| Can provide authentication, rate limiting, etc. | Mainly handles traffic distribution |
API Gateway decides WHICH service should handle the request.
Load Balancer decides WHICH INSTANCE of that service should handle the request.
API Composition
A single page may need data from multiple services. For example, a My Orders page may need data from the Product, Invoice, Review, and Recommendation services. If the client calls each one, it has to coordinate all the calls and combine the responses itself.
API Composition solves this. The gateway (or a dedicated service) calls the required services, combines their responses, and returns one response to the client.
For example, the client calls:
1GET /api/my-ordersThe gateway gathers the data from the services and returns:
1{2 "product": {},3 "invoice": {},4 "reviews": [],5 "recommendations": []6}The client makes only one API call.
API Composition = Multiple backend API calls → One client response
Authentication
The gateway already sits between the client and all the microservices, so we can authenticate at the gateway instead of in every microservice.
For example, using an OAuth 2.0-style flow:
Without this, each microservice repeats the same authentication logic. With the gateway, the request is verified at one place, and invalid or unauthorized requests are rejected before they reach any service.
Authenticate at the entry point, then allow valid requests to continue.
Rate Limiting
Authentication controls who can access the API. But what if a valid client sends too many requests? For example, 10,000 requests at once can overload the backend.
So the gateway also provides rate limiting and throttling, using these mechanisms:
- Burst limits
- API throttling
- IP-based blocking
- API queues
Burst Limit
A burst limit controls how much traffic can be handled during a sudden spike.
1Burst Limit = 500If incoming traffic goes above the limit, the gateway rejects the extra requests before they reach the backend, with:
1HTTP 4292Too Many Requests1 Request Request Request2 | | |3 +----------+----------+4 |5 API Gateway6 |7 Burst Limit8 |9 Limit exceeded10 |11 ▼12 HTTP 429API Throttling
Burst limits handle sudden spikes. Throttling answers a more specific question:
How many requests is a particular user or application allowed to make?
1/api/invoice2 3Maximum:410 requests / minute / user1Request 1 ✓2Request 2 ✓3...4Request 10 ✓5Request 11 ✗The 11th request is blocked because the user went over the allowed rate. Throttling can be applied per API, user, application, or request rate.
IP-Based Blocking
The gateway can block requests from a specific IP address. This is another layer of traffic protection.
1Client (IP: X.X.X.X)2 |3 ▼4API Gateway5 |6 | IP blocked7 ▼8Request rejectedAPI Queues
When a huge number of requests arrive together, we do not let them all hit the backend at once. The rate limiter controls how many requests enter the system. The accepted requests wait in a queue, and the queue releases them gradually based on the capacity of the backend service.
For example, in an e-commerce flash sale, thousands of users try to place an order at the same time. The queue feeds the Order Service at the speed it can handle, so the spike does not overwhelm it.
Service Discovery
The gateway now knows which microservice should get the request. But where is that microservice running?
Services scale up and down, so instances are created and removed, and their IP addresses and ports change. Service Discovery (also called the Service Registry) keeps track of the current location of every service instance.
How it works
- The client sends
/api/order. - The gateway decides this belongs to the Order Service.
- The gateway asks the registry for the healthy instances of the Order Service.
- The gateway forwards the request to one of them, usually through a Load Balancer.
How the registry stays up to date
Approach 1: Self-registration. Whenever a microservice starts, it registers itself with Service Discovery.
Approach 2: Health checks. Service Discovery keeps checking the services. If a service stops responding, that instance is removed from the active list, so only healthy locations remain.
Common technologies: Eureka, Consul, ZooKeeper, etcd, and Kubernetes Service Discovery. In Spring Cloud, Eureka acts as the service registry.
Other API Gateway Responsibilities
Besides routing, composition, authentication, rate limiting, and service discovery, a gateway can also handle:
- Request and response transformation
- Caching
- Centralized logging
If the Gateway Is a Single Entry Point, How Does It Handle Millions of Requests?
Won't the API Gateway itself become a bottleneck or a single point of failure?
No. "Single entry point" is a logical idea, not a single server. In production, the gateway runs on many instances, across many Availability Zones, and often across many regions.
Multiple Gateway Instances
Many gateway instances run in parallel behind a load balancer. The load balancer spreads requests across them and stops sending traffic to unhealthy ones. When traffic grows, auto-scaling adds more instances.
Multiple Availability Zones
A region contains several isolated Availability Zones (AZs). Each AZ runs its own gateway and service instances, and traffic is spread across the AZs.
If AZ1 fails, traffic is redirected to AZ2, and the application keeps serving requests. This gives high availability and fault tolerance inside a region.
Multiple Regions
Multi-AZ protects against failures inside a region. If the entire region goes down, we need another level of redundancy.
The application is deployed in several regions. Each region has its own gateway, load balancers, service discovery, and microservice instances. If one region fails, traffic is redirected to another healthy region.
DNS-Based Traffic Distribution
With many regions, who decides which region gets the client's request? DNS and traffic-routing services such as AWS Route 53 and Azure Traffic Manager do. They choose a region using rules based on latency, geographic location, or compliance requirements. The request then goes to the gateway in that region.
Complete API Gateway Architecture
| Component | Main Responsibility |
|---|---|
| DNS / Traffic Manager | Choose the appropriate region |
| API Gateway | Route request to the correct service |
| Authentication | Verify the client/token |
| Rate Limiting | Control request rate |
| Service Discovery | Find current service locations |
| Load Balancer | Distribute traffic across service instances |
| Microservice | Process the actual business request |
One-Line Mental Model
DNS chooses the region → API Gateway chooses the service → Service Discovery finds its location → Load Balancer chooses the instance → Microservice processes the request.