Most microservice outages start with a dependency that slows down, not one that dies. The first few retries look harmless. Then every client retries at once, the dependency receives several times its normal load, and the failure becomes self-sustaining. A retry budget is the control that stops that spiral by capping retries as a fraction of original requests.

Retries Turn Small Failures Into Cascades

A service that calls a dependency with a 500 ms timeout will retry when the dependency takes 700 ms. That is reasonable for one client. When a thousand clients do it, the dependency sees roughly twice the request volume exactly when it can least afford it. The added load pushes latency higher, which triggers more retries, and the loop feeds itself. Cascading failure is the standard name for this pattern: one part failing raises the probability that others fail.

Timeouts alone cannot stop the spiral. A timeout decides when a single call gives up. It says nothing about how many calls the system as a whole will make. Teams often tighten timeouts and add retries in the same change, which makes the amplification worse. The fix has to operate at the level of aggregate retry volume, not individual call duration.

The failure mode is common enough that it shows up across layers. A saturated database connection pool, a slow authentication service, and a throttled third-party API all produce the same shape: rising latency, rising retry rate, and a dependency that looks healthy in isolation because it is still answering some requests. The system fails while every component reports partial success.

Circuit Breakers Help But Do Not Budget

Circuit breakers are the usual first defense, and they earn their place. A breaker tracks failures per client and trips when the error rate crosses a threshold, sending subsequent calls to a fallback instead of the dependency. That prevents one client from hammering a dead service. It does not prevent many clients from each sending a modest number of retries.

If fifty service instances each hold a breaker with a threshold of twenty failures, the dependency can still receive a thousand failed probes before every breaker opens. Half-open probes add traffic too: each breaker periodically lets a test request through, and those probes accumulate across the fleet.

There is a fairness problem as well. A breaker that trips on error rate can shed healthy calls when a dependency returns errors for one tenant or one query shape. The breaker cannot tell a bad request from a bad dependency.

What a Retry Budget Actually Caps

A retry budget caps the ratio of retries to original requests over a rolling window. If a client sends 10,000 original requests per minute and the budget is 10 percent, it may send at most 1,000 retries in that window. Once the budget is exhausted, further retries fail fast instead of queueing. The dependency sees a bounded increase in load rather than an unbounded one.

Typical budgets hover near 10 percent, though the right number depends on the dependency and the cost of a failed call. A read against a replicated cache might tolerate a higher budget; a write against a primary database usually warrants a lower one. The budget is a ceiling on amplification, not a target. Setting it at the observed retry rate leaves no headroom for the next incident.

Budgets are enforced at the client or at the mesh layer. Client-side enforcement is simpler to reason about and travels with the code. Mesh enforcement covers services that share a sidecar or proxy, including ones written in languages where retry logic is inconsistent. The trade-off is visibility: a budget enforced in a proxy is invisible to application code unless the proxy exports the relevant metrics.

Budgets in Envoy and Service Meshes

Envoy supports retry budgets as a route-level setting. The budget is expressed as a percentage of active requests, and Envoy tracks retries against that allowance per route. When the budget is exhausted, Envoy stops retrying and returns the last response to the caller. That behavior is deliberate: failing fast is cheaper than queueing retries behind a slow dependency.

Istio exposes the same mechanism through its VirtualService and DestinationRule resources, so a budget can be set per route without changing application code. The retry policy itself is driven by fields on the route: attempts sets how many times a request may be retried, perTryTimeout bounds each individual attempt, and retryOn lists the conditions that trigger a retry, such as connection resets or gateway errors. Teams running Istio often discover retry amplification only after an incident, because those fields are easy to leave at whatever a template or a copy-pasted manifest supplied. Auditing the mesh configuration for retry settings is a useful first step.

gRPC has retry throttling with a similar intent, and the mechanism is a token bucket rather than a plain ratio. In the service config, retryThrottling takes maxTokens and tokenRatio. Each failed RPC subtracts one token; each successful RPC adds tokenRatio back, up to maxTokens. When the token count falls to half of maxTokens, the client stops retrying until successes refill the bucket. That design ties retry allowance to observed health rather than to a fixed window. Finagle popularized the retry budget concept in its client stack, and much of the later work in Envoy and Istio follows the same model. The implementations differ in details, but the core idea is consistent: bound the aggregate, not the individual.

The Trade-Off: Budgets Convert Amplification Into Errors

A budget is not free. If retries are capped, a dependency that is failing intermittently will produce more user-visible errors than it would with unlimited retries. That is the intended trade, and it is the whole point: the system sheds retries instead of collapsing, and the errors land on the callers that can report them rather than on a dependency that cannot answer at all.

The cost needs to be visible. A budget without alerting on exhausted retries converts a silent amplification problem into a silent error problem, and the second is harder to notice because dashboards show healthy latency on the dependency side. Emit a counter for retries dropped by the budget, alert when it moves, and treat a sustained rise as a dependency-health signal rather than as noise from the budget itself.

There is a second cost worth naming. A budget that is too tight sheds retries during healthy periods, so transient blips that would have recovered invisibly become user-facing failures. The budget has to sit above the normal retry ratio with enough margin that ordinary jitter does not exhaust it.

Choosing a Budget Without Guesswork

Start by measuring the retry ratio under normal load. Most clients and proxies can export retry counts alongside request counts, and the ratio between them is the baseline. If normal traffic produces a retry ratio near 2 percent, a budget of 10 percent leaves room for a partial failure without allowing a storm.

Tie the budget to the dependency's criticality. A dependency on the critical path of a checkout flow deserves a tighter budget than an analytics call that can fail silently. The cost of a failed call matters as much as the cost of added load. For dependencies with their own rate limits, the budget should stay well under the limit so retries cannot consume the headroom reserved for original traffic.

Review budgets after each incident. The retry ratio observed during the incident is the most useful data point for the next setting. A budget that was never approached did nothing; one that was exhausted early may have been too tight. The same discipline applies to any unmonitored limit in the stack, where the setting quietly becomes the failure because nobody watches it.

Two Actions for the Next Incident

• Instrument retry counts per dependency before the next incident, broken down by client and by route. Without that data, no budget can be set or reviewed, and the post-incident review has nothing to reason from.
• Load-test with injected dependency latency, not just injected errors. Watch the retry ratio climb as latency rises. A test that only injects errors misses the slow-dependency case that produces most cascades, because errors trip breakers while latency does not.

Set the budget on the clients that carry the most traffic to a shared dependency first, and pair it with a breaker on the same route. The breaker stops one client from hammering a dead dependency; the budget stops the fleet from amplifying a slow one, and neither covers the other's scope.