How Grab Designs Resilient Systems Beyond Retries (Part 1: Rate Limiting)
Original post Designing Resilient Systems Beyond Retries (Part 1: Rate Limiting)
Article Summary
Grab's engineering team learned the hard way: retries and circuit breakers aren't enough when you're running hundreds of microservices at scale.
This is part one of a three-part series from Grab's engineering team on building resilient distributed systems. Michael Cartmell digs into why rate limiting is your critical second line of defense when retry storms threaten to take down your backend.
Key Takeaways
- Retry storms can overwhelm servers during failures, making problems worse instead of better
- Four-tier rate limiting: per-client-per-endpoint, per-client, per-endpoint, and server-wide
- Global rate limiting beats local: prevents downstream resource exhaustion as you scale
- Client-side limiting reduces server overhead but adds complexity to SDK implementations
Rate limiting protects your servers when client-side circuit breakers fail or are misconfigured, preventing cascading failures across your microservices architecture.
Continue reading
What This Article Covers
Grab's microservices architecture has trouble keeping circuit-breaker configurations consistent across hundreds of services. When clients misconfigure their retry logic, it becomes hard to prevent cascading failures.
Grab built a multi-tiered rate-limiting strategy with per-client-per-endpoint, per-client, per-endpoint, and server-wide thresholds. A centralized global rate-limiting service like Grab's Quotas service enforces these limits across the entire backend pool.
Global rate-limiting stops downstream resource exhaustion by monitoring thresholds across all server instances instead of just individual hardware. The system stays responsive as the number of instances grows.
This page summarizes “Designing Resilient Systems Beyond Retries (Part 1: Rate Limiting)” by Grab, published Sep 23, 2021. All credit for the original work goes to the authors.
