The retry logic looked correct in code review. Every failed call to the LLM provider waited a bit longer than the last, doubling each time, capped at a sane maximum. Then a provider outage hit for ninety seconds, and every one of forty workers that had a request in flight retried at exactly the same moment, twice, in near-perfect sync, turning a ninety-second blip into a five-minute pileup.
Exponential backoff jitter API retries work by doubling the delay after each failed call, capping it at a maximum, and adding a random amount of time on top so that many failed workers do not retry at the exact same instant. Backoff alone still produces synchronized waves of retries; jitter is the part that breaks the wave apart, and it is what actually protects the provider, and your own application, under load.
This matters more with LLM providers than with most APIs, because a single outage or rate-limit window commonly stalls dozens of concurrent requests at once, and a slow response holds a PHP worker or a Node event-loop tick open the whole time it waits.
The Thundering Herd, With Workers Instead of Cattle
The term comes from an old scheduling problem: many processes woken by the same event all compete for the same resource at once. In an API client, the trigger isn’t a scheduler waking processes, it’s a shared failure. If your provider returns a wave of 500s or 429s during a two-second blip, every request that failed in that window is now on the same retry clock. If they all wait the same fixed or exponential delay, they all try again in the same narrow window, which is the least helpful moment to add load to a service that is already struggling.
This isn’t a theoretical concern specific to one vendor. It’s the exact problem AWS’s Architecture Blog (read 2026-09-18) describes in its widely cited “Exponential Backoff And Jitter” post: with many clients contending for the same resource, synchronized retries multiply the work the backend has to do rather than reducing it.
Backoff Alone Still Produces Retry Waves
Plain capped exponential backoff, no randomness, looks like it’s working. Fewer calls arrive over time as attempts space out. But AWS’s own analysis of the pattern makes the flaw explicit: instead of reducing how many clients compete in a given round, backoff alone just introduces stretches with no competition at all, followed by another cluster when the delays line back up. The clients are still moving in lockstep, just less often.
Jitter breaks the lockstep. Instead of every failed worker waiting exactly base times 2 to the power of the attempt number in milliseconds, each one waits a randomized amount related to that value, so the retries land spread across a window instead of a single instant.
Exponential Backoff Jitter API Formulas Compared
There are three formulas worth knowing, all traceable to the same AWS post:
| Strategy | Formula | Behavior |
|---|---|---|
| No jitter (baseline) | delay = min(cap, base * 2^attempt) | Correct spacing, but every client still moves in lockstep |
| Full jitter | delay = random(0, min(cap, base * 2^attempt)) | Widest spread, lowest total load on the provider, but more variance in how long any one request waits |
| Equal jitter | delay = half + random(0, half), where half = min(cap, base * 2^attempt) / 2 | A compromise; less spread than full jitter, less predictable timing than none |
| Decorrelated jitter | delay = random(base, previous_delay * 3), capped | Uses the previous delay rather than the attempt count, spreading clients out as a random walk |
For most LLM integrations, full jitter is the sane default. It’s the simplest to implement, it does the least total work against a struggling provider, and the small extra variance in wait time rarely matters compared to the outage or rate limit that triggered the retry in the first place. Reach for decorrelated jitter only when you’re tuning for tighter latency variance and are willing to reason about a random walk instead of a fixed formula.
Which Errors Are Actually Safe to Retry
Backoff and jitter only help when you’re retrying the right thing. A 429 with a Retry-After header is a temporary rate limit, worth retrying. A 429 that represents an exhausted spend cap or usage limit is not; it will fail identically on every attempt no matter how long you wait or how you randomize it. OpenAI’s error codes guide (read 2026-09-18) draws this line directly: when a Retry-After header is present, wait at least as long as it specifies before retrying, and when it’s absent, widen the delay between attempts and add a small random delay rather than guessing at a fixed number.
A short table of what to retry:
| Response | Retry with backoff + jitter? |
|---|---|
429 with Retry-After | Yes, honoring the header as a minimum wait |
| 500 / 503, or a provider “overloaded” error | Yes, bounded by a maximum attempt count |
| 429 identified as a spend, quota or usage limit | No; it will not resolve by waiting |
| 400 / 401 / 403 / 404 | No; the request, key or permissions are wrong |
| A timeout with no response | Only if the underlying operation is safe to repeat |
Never retry on status code alone. Read the error type or message in the response body, since both major LLM providers document a typed error format specifically so callers can tell a temporary limit from a permanent one apart from the raw 429.
A PHP Example That Doesn’t Block a Worker for Minutes
A capped exponential backoff with full jitter, written so it fails loudly rather than looping forever:
function callLlmWithBackoff(callable $request, int $maxAttempts = 5, int $baseMs = 500, int $capMs = 20000): mixed
{
$attempt = 0;
while (true) {
try {
return $request();
} catch (RetryableApiException $e) {
$attempt++;
if ($attempt >= $maxAttempts) {
throw $e;
}
$exponential = min($capMs, $baseMs * (2 ** $attempt));
$delayMs = random_int(0, (int) $exponential); // full jitter
usleep($delayMs * 1000);
}
}
}$request is a closure that calls the provider with the API key read from an environment variable, never a literal string in the file. RetryableApiException is something your calling code throws only for the rate-limit and server-error cases from the table above, not for every failure.
The usleep() call is the part that catches people out on a WordPress site. Sleeping inside a request that a visitor is waiting on ties up a PHP-FPM worker for the whole delay, and with a five-attempt loop capped at twenty seconds, that’s a request that can run for the better part of a minute. Retry loops like this belong in a background job or a queued task, not directly inside a template or, worse, functions.php, where a slow or fatal branch turns into the kind of blank page covered in why editing functions.php keeps destroying your site. Give the retry logic its own file inside a proper plugin structure, the way Your First Real Plugin lays out, so it can be tested, disabled and reasoned about separately from the theme.
A JavaScript Example for Client-Side or Node Retries
The same shape, without the blocking concern, because await yields the event loop instead of holding a worker hostage:
async function callLlmWithBackoff(request, { maxAttempts = 5, baseMs = 500, capMs = 20000 } = {}) {
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
try {
return await request();
} catch (err) {
if (!err.retryable || attempt === maxAttempts) throw err;
const exponential = Math.min(capMs, baseMs * 2 ** attempt);
const delayMs = Math.random() * exponential; // full jitter
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
}
}err.retryable is a flag your error-handling code sets after inspecting the response, the same distinction the PHP version makes with a dedicated exception type. If this function fires from a WordPress admin-ajax handler or a REST route, when it fires matters as much as how it retries; picking the wrong moment in the request lifecycle is its own category of bug, covered in choosing the right hook for the job.
Retry Budgets: When to Stop
Backoff and jitter answer “how long to wait.” They don’t answer “how long to keep trying,” and that needs its own limit. Two caps, used together, work better than either alone:
- A maximum attempt count, typically three to six, so a persistently failing dependency doesn’t retry indefinitely.
- A maximum elapsed time for the whole retry sequence, so a user-facing request doesn’t sit open for minutes waiting through several capped delays even if the attempt count hasn’t been reached yet.
If retries are still landing in visible clusters after adding jitter, the fastest way to find out why is to strip the failing call down to the smallest script that reproduces it, outside the full application, the approach in building a minimal reproduction before you ask anyone. A minimal reproduction outside a queue or a cron schedule quickly shows whether the clustering is coming from your own code or from several independent triggers hitting the provider at the same wall-clock moment, which jitter alone can’t fix if the trigger itself is scheduled, not random.
FAQ
What’s the actual difference between backoff and jitter? Backoff decides how much longer to wait after each failed attempt, growing exponentially up to a cap. Jitter adds randomness on top of that delay so that multiple clients failing at the same time don’t all retry at the same instant. Backoff without jitter still produces synchronized waves; jitter is the part that spreads them out.
Should I use full jitter or decorrelated jitter? Full jitter is the reasonable default for most LLM integrations: it’s simple, and AWS’s own analysis found it does the least total work against a struggling backend. Decorrelated jitter is worth the extra complexity only when you’re specifically tuning for latency consistency over server-side load.
Does the official OpenAI SDK (or another provider’s SDK) already retry for me?
Often yes, for a limited number of attempts, honoring the Retry-After header when present. Check your installed SDK version’s documented retry behavior before wrapping your own retry loop around it, since stacking two retry layers multiplies the total number of attempts.
How many retries should a call to an LLM provider get before giving up? There’s no universal number; it depends on how long the calling request can stay open. A common pattern is three to six attempts with a capped exponential delay, plus a hard ceiling on total elapsed time so a chat feature or an editorial workflow doesn’t leave a user or a queued job waiting far longer than the interaction can tolerate.
Is exponential backoff with jitter only useful for LLM APIs? No. It’s a general pattern for any client retrying against a shared, rate-limited or occasionally overloaded service. LLM APIs make the problem more visible because a single slow model response holds a request open far longer than a typical REST call, so the retry storm that follows an outage is bigger and easier to notice.
