Rate Limiting · Request Throttling
Rate Limiting
What Is Rate Limiting?
- Hourly request limits
- Prevents overload, abuse, and runaway costs
- Blocked with a 429 error
The moment you turn on rate limiting, the assumptions behind running the service change. Before, you accepted every incoming request, held on, and then scaled up or restarted when the server fell over. Now, deliberately rejecting requests beyond a set line is normal behavior. A rejection is not an outage. It is design. Even when traffic surges, the server handles only what it can carry, the rest simply waits a little longer, and the service itself stays up. The key is to break the pattern where a small number of runaway requests drags down the many normal users with them.
There are four main approaches. A fixed window resets its counter at the start of every minute, which is easy to build but lets double the traffic through at the boundary. A sliding window removes that boundary problem. A token bucket lets you spend saved-up capacity on short bursts, and a leaky bucket keeps the processing rate steady. What you count against matters just as much. Applying the limit per IP, API key, user ID, or organization produces completely different results. A per-minute call limit, a monthly quota, and a concurrency limit are three separate mechanisms. Even with 800,000 calls left on a 1-million monthly quota, you will still get a 429 if you hit the per-second limit.
The most common mistake in practice is applying limits by IP address. If 200 of a client’s employees go out through one public IP behind a corporate NAT, all 200 get blocked at once the moment the limit is hit. The second is retries. Code that calls again 0.1 seconds after receiving a 429 does not get around the limit. It effectively launches an attack on your own system. You need exponential backoff with random jitter, and you need to read the Retry-After header. The third is internal batch jobs. If an overnight settlement job uses up the whole limit, customer requests are blocked in the morning, so internal traffic should be split onto a separate key with its own limit.
Seungtae ParkHands-on AX instructor who teaches AI business development and marketing automation by building real systems
Ilyoung JungExpert in smart factories, equipment management, and Industry 4.0A technique for managing network and API traffic that became a standard defense as web services and open APIs spread.
Consider a hypothetical case: Company A, a B2B online training platform with 140 employees. Five weeks after connecting an external LLM API to its lecture summary and quiz generation features, its monthly API bill jumped from KRW 2.1 million (≈ US$1,570) to KRW 16.8 million (≈ US$12,600). On the first Monday of every month, when course enrollment opened, its servers went down for 20 minutes. The CTO stepped in, saying, 'We don't need a bigger plan. We need a narrower door.'
- When Team Lead Kim on the backend went through two weeks of call logs, 61% of all calls came from just three accounts. One of them was an automation script hitting the API 40 times per second for two hours straight.
- The CTO and Team Lead Kim chose the token bucket method and changed the gateway settings to allow 60 calls per minute per API key, with bursts of up to 20. Anything over that would be returned with a 429.
- Senior Engineer Park on the frontend set 429 responses to return a Retry-After of 30 seconds, and added 2-, 4-, and 8-second exponential backoff with random jitter to the client so that retries would not snowball into another surge.
- On the second day after rollout, 90 headquarters employees were blocked all at once. Because of the company’s internal NAT, everyone appeared under the same IP, so Team Lead Kim summed it up: “We’re switching the basis from IP to user ID.” The early-morning settlement batch was moved to a separate key.
- Six weeks after rollout, monthly API costs fell from KRW 16.8 million (≈ US$12,600) to KRW 2.9 million (≈ US$2,170), enrollment-day downtime dropped from 20 minutes to zero, and average response time improved from 3.4 seconds to 1.1 seconds.