Stop ignoring API failures. Use Polly
Your app makes an API call. It fails. You return the error... but it might be temporary. Instead, give it another chance with Polly.
In this tutorial, we look at how to add Polly to an ASP.NET Core Web API and configure retries, circuit breakers, and timeouts to make your app more resilient when downstream API calls fail.
The problem with a single API attempt
Take a downstream API endpoint that is currently returning an error. If you are calling this from your app, you might think the right approach is to accept that it is down and return the error to the caller. But the issue might be temporary. If you try it a couple more times, you might find it's suddenly working.
Rather than accepting that the downstream API is down on the first attempt, it makes more sense to retry it a couple of times before giving up. That is exactly what Polly allows you to do.
Installing Polly
Polly is a .NET resilience library. It handles retries, timeouts, circuit breakers, and more. The NuGet package you need is Microsoft.Extensions.Http.Resilience.
This package integrates Polly directly into the HttpClient pipeline via AddResilienceHandler, which keeps all your resilience configuration in one place when registering your HTTP clients in Program.cs.
Adding a retry with constant backoff
When you register an HttpClient, you can chain a call to AddResilienceHandler and configure a retry policy inside it. Here is an example using a constant backoff type:
// Program.cs
builder.Services.AddHttpClient("DownstreamConstant",
configureClient: static client =>
{
client.BaseAddress = new("https://localhost:7276");
})
.AddResilienceHandler("retry", (builder, context) =>
{
builder.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 3,
Delay = TimeSpan.FromSeconds(2),
BackoffType = DelayBackoffType.Constant,
UseJitter = false,
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.HandleResult(r => !r.IsSuccessStatusCode),
});
});There are a few things to note here. MaxRetryAttempts sets how many times it will retry before giving up. Delay sets the wait time between each attempt. BackoffType is set to Constant, which means every retry waits the same amount of time. ShouldHandle tells Polly which responses should trigger a retry. In this case, it retries if the response is not a successful status code.
With this in place, the endpoint will retry three times every 2 seconds unless there is a successful status code returned.
Other backoff types
Constant backoff is straightforward, but you can configure longer intervals between retries by changing the BackoffType.
Exponential
With exponential backoff, the delay doubles on every retry attempt. With a base delay of 2 seconds, the first retry waits 2 seconds, the second waits 4 seconds, and the third waits 8 seconds.
// Program.cs
builder.Services.AddHttpClient("DownstreamExponential",
configureClient: static client =>
{
client.BaseAddress = new("https://localhost:7276");
})
.AddResilienceHandler("retry", (builder, context) =>
{
builder.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 4,
Delay = TimeSpan.FromSeconds(2),
BackoffType = DelayBackoffType.Exponential,
UseJitter = false,
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.HandleResult(r => !r.IsSuccessStatusCode)
});
});This is a better approach when the downstream API needs more time to recover. Rather than flooding it with requests at regular intervals, you are giving it progressively more breathing room between each attempt.
Linear
With linear backoff, an extra delay is added for each retry attempt based on the delay interval. With a base delay of 2 seconds, the first retry waits 2 seconds, the second waits 4 seconds, and the third waits 6 seconds.
// Program.cs
builder.Services.AddHttpClient("DownstreamLinear",
configureClient: static client =>
{
client.BaseAddress = new("https://localhost:7276");
})
.AddResilienceHandler("retry", (builder, context) =>
{
builder.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 4,
Delay = TimeSpan.FromSeconds(2),
BackoffType = DelayBackoffType.Linear,
UseJitter = false,
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.HandleResult(r => !r.IsSuccessStatusCode)
});
});Linear is a middle ground between constant and exponential. It grows steadily rather than aggressively, which works well when you want some breathing room between retries without the long waits that exponential backoff can produce at higher attempt counts.
Jitter
A problem with fixed delay intervals is that multiple clients could be hitting a struggling server all at the same time, which makes things worse. Jitter solves this by adding a small random delay to each retry attempt, spreading the load out rather than sending all clients at exactly the same moment.
// Program.cs
builder.Services.AddHttpClient("DownstreamJitter",
configureClient: static client =>
{
client.BaseAddress = new("https://localhost:7276");
})
.AddResilienceHandler("retry", (builder, context) =>
{
builder.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 4,
BackoffType = DelayBackoffType.Exponential,
UseJitter = true,
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.HandleResult(r => !r.IsSuccessStatusCode)
});
});You no longer need to set the Delay property when Jitter is enabled. Polly handles the random delay intervals for each retry attempt automatically.
Adding a circuit breaker
Retries are great, but they have a flaw. If the downstream service is completely down, retrying repeatedly makes things worse. You are delaying the response and adding unnecessary load to a service that is already struggling.
That is where the circuit breaker comes in. When the failure rate crosses a threshold, it opens the circuit and stops sending requests entirely for a set period. This protects the downstream service and avoids wasting resources on requests that are almost certainly going to fail.
// Program.cs
builder.Services.AddHttpClient("DownstreamCircuitBreaker",
configureClient: static client =>
{
client.BaseAddress = new("https://localhost:7276");
})
.AddResilienceHandler("circuit-breaker", (builder, context) =>
{
builder.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 5,
Delay = TimeSpan.FromSeconds(1),
BackoffType = DelayBackoffType.Constant,
UseJitter = false,
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.HandleResult(r => !r.IsSuccessStatusCode)
});
builder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
FailureRatio = 0.5,
SamplingDuration = TimeSpan.FromSeconds(20),
MinimumThroughput = 5,
BreakDuration = TimeSpan.FromSeconds(20),
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.HandleResult(r => !r.IsSuccessStatusCode)
});
});The circuit breaker is configured with a FailureRatio of 0.5, which means if 50% of requests fail within the SamplingDuration window of 20 seconds and at least 5 requests have been made (MinimumThroughput), the circuit opens. While open, it fast-fails all requests immediately with a BrokenCircuitException rather than going through the retry pipeline. After the BreakDuration of 20 seconds passes, it allows requests through. If it succeeds, the circuit closes and normal traffic resumes.
Adding a timeout
There is a failure mode that retries and circuit breakers do not cover. A slow API. If the downstream service hangs and never responds, your request just sits there waiting indefinitely, tying up resources.
You can add a timeout to the resilience pipeline to handle this:
// Program.cs
builder.Services.AddHttpClient("DownstreamTimeout",
configureClient: static client =>
{
client.BaseAddress = new("https://localhost:7276");
})
.AddResilienceHandler("timeout", (builder, context) =>
{
builder.AddTimeout(new HttpTimeoutStrategyOptions
{
Timeout = TimeSpan.FromSeconds(5),
});
});After 5 seconds, if a response has not been received, it will throw a timeout exception. One important thing to be aware of. A timeout on your end does not mean the downstream API call did not complete. Set the timeout interval so it gives enough time to get a response from the API. Otherwise, you might find it completed on the downstream side, but you received an error.
Targeting specific status codes
Up until now, we have been triggering a retry for any response that does not return a successful status code. But it makes more sense to be specific. A 400 or 404 is not going to fix itself on a retry. You should only retry on transient failures.
Update the ShouldHandle predicate to target specific status codes:
ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
.HandleResult(r => r.StatusCode is
HttpStatusCode.InternalServerError or
HttpStatusCode.GatewayTimeout)This limits retries to 500 and 504 responses, which are common transient failures from a downstream API.
Adding logging
Without logging, you will not know retries are happening in production. Polly provides callbacks for each strategy that you can use to wire up your own logging.
For retries, use OnRetry:
OnRetry = args =>
{
var loggerFactory = context.ServiceProvider.GetRequiredService<ILoggerFactory>();
var logger = loggerFactory.CreateLogger(typeof(ResilienceEndpoints).FullName!);
logger.LogWarning("Retry {attempt} after {delay}ms",
args.AttemptNumber + 1, args.RetryDelay.TotalMilliseconds);
return ValueTask.CompletedTask;
}For circuit breakers, use OnClosed to log when the circuit recovers:
OnClosed = args =>
{
var loggerFactory = context.ServiceProvider.GetRequiredService<ILoggerFactory>();
var logger = loggerFactory.CreateLogger(typeof(ResilienceEndpoints).FullName!);
logger.LogWarning("Circuit breaker closed");
return ValueTask.CompletedTask;
}For timeouts, use OnTimeout:
OnTimeout = args =>
{
var loggerFactory = context.ServiceProvider.GetRequiredService<ILoggerFactory>();
var logger = loggerFactory.CreateLogger(typeof(ResilienceEndpoints).FullName!);
logger.LogWarning("Timeout after {0}ms", args.Timeout.TotalMilliseconds);
return ValueTask.CompletedTask;
}Watch the video
Watch the video where we show you how to add Polly to an ASP.NET Core Web API and demonstrate retries, circuit breakers, timeouts, and more in action.
Related pages
