Plain Backoff Is a Thundering Herd
Your exponential backoff is a thundering herd aimed at your own API. When a dependency hiccups, every client that failed at the same instant backs off by the same fixed delay — and then retries at the same instant. Plain backoff doesn't spread the load. It synchronizes it, and the synchronized wave re-DDoSes the very service that was trying to recover.
Why plain backoff makes it worse
Exponential backoff was supposed to be the fix: on each failure, wait base × 2^attempt, capped at some ceiling. The waiting part works. The problem is that a fleet of clients that failed together share a clock. They all sleep 1s, then 2s, then 4s — in lockstep. Each retry window becomes a coordinated pulse. Google's SRE book calls this out directly: if retries aren't randomly distributed over the retry window, a small perturbation like a network blip can schedule retry ripples at the same time, and they amplify themselves.
The fix: full jitter
Don't sleep for the backoff delay. Sleep for a random amount between zero and the backoff delay. That's full jitter, from AWS's "Exponential Backoff And Jitter" study:
# Plain exponential backoff (the herd)
sleep = min(cap, base * 2 ** attempt)
# Full jitter (the trickle)
sleep = random_between(0, min(cap, base * 2 ** attempt))The ceiling still grows exponentially, so a struggling service still gets exponentially more breathing room. But because each client draws its own random wait, the retries smear across the whole window instead of stacking on one tick.
What the numbers say
AWS simulated 100 contending clients against a shared resource and measured two things: total calls made (server load / contention) and time to complete all work.
| Strategy | Sleep formula | Vs. no jitter |
|---|---|---|
| No jitter | min(cap, base·2^n) | baseline: high contention |
| Full jitter | rand(0, min(cap, base·2^n)) | ~½ the calls, substantially faster completion |
| Equal jitter | t/2 + rand(0, t/2), t = min(cap, base·2^n) | similar call count, but slower than full |
| Decorrelated | min(cap, rand(base, last·3)) | more calls than full, slightly faster |
Full jitter had the lowest client work and competitive completion time. AWS's conclusion: jittered backoff is huge and should be a standard approach for remote clients.
Power user notes
- Cap the ceiling, not the jitter. The
capbounds the maximum backoff so a client doesn't sleep for minutes. Jitter samples below the cap — keep the full[0, cap]range; shrinking it back toward the deterministic value re-synchronizes the herd. - Decorrelated jitter for long-lived clients.
sleep = min(cap, random(base, last_sleep × 3))walks the delay up based on the previous wait instead of the attempt count. Slightly more aggressive, still de-synchronized — handy when clients don't reset attempt counters cleanly. - Add a retry budget and a circuit breaker. Jitter spreads retries; it doesn't reduce their total. Cap retries per request and stop retrying entirely when a downstream is clearly down, so you don't jitter a firehose.
- Every RPC client, not just the flaky one. The SRE guidance is that every client making a call implements exponential backoff with jitter, because error amplification is a fleet-wide property.
Resources
- Exponential Backoff And Jitter — AWS Architecture Blog — the simulation, formulas, and the full/equal/decorrelated comparison.
- Jitter: Making Things Better With Randomness — Marc Brooker — the author's broader take on jitter in distributed systems.
- Addressing Cascading Failures — Google SRE Book, Ch. 22 — why randomized exponential backoff dampens retry ripples.
<div class="cta"> <p><strong>Building resilient AI systems?</strong> Yeda AI designs, audits, and ships production LLM and backend infrastructure that stays up under load.</p> <p><a href="/contact">Talk to us</a> · <a href="/blog">Read the blog</a></p> </div>