I have an Azure function that makes a http call to a webapi endpoint. I'm following this example GitHub Polly RetryPolicy so my code has a similar structure. So in Startup.cs i have:
builder.Services.AddPollyPolicies(config); // extension methods setting up Polly retry policies
builder.Services.AddHttpClient("MySender", client =>
{
client.BaseAddress = config.SenderUrl;
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
});
My retry policy looks like this:
public static class PollyRegistryExtensions
{
public static IPolicyRegistry<string> AddBasicRetryPolicy(this IPolicyRegistry<string> policyRegistry, IMyConfig config)
{
var retryPolicy = Policy
.Handle<Exception>()
.OrResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
.WaitAndRetryAsync(config.ServiceRetryAttempts, retryCount => TimeSpan.FromMilliseconds(config.ServiceRetryBackOffMilliSeconds), (result, timeSpan, retryCount, context) =>
{
if (!context.TryGetLogger(out var logger)) return;
logger.LogWarning(
$"Service delivery attempt {retryCount} failed, next attempt in {timeSpan.TotalMilliseconds} ms.");
})
.WithPolicyKey(PolicyNames.BasicRetry);
policyRegistry.Add(PolicyNames.BasicRetry, retryPolicy);
return policyRegistry;
}
}
My client sender service receives IReadOnlyPolicyRegistry<string> policyRegistry and IHttpClientFactory clientFactory in its constructor. My code calling the client is the following:
var jsonContent = new StringContent(JsonSerializer.Serialize(contentObj),
Encoding.UTF8,
"application/json");
HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Post, "SendEndpoint")
{
Content = jsonContent
};
requestMessage.Headers.Authorization = new AuthenticationHeaderValue("Bearer", authToken);
requestMessage.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var retryPolicy = _policyRegistry.Get<IAsyncPolicy<HttpResponseMessage>>(PolicyNames.BasicRetry)
?? Policy.NoOpAsync<HttpResponseMessage>();
var context = new Context($"GetSomeData-{Guid.NewGuid()}", new Dictionary<string, object>
{
{ PolicyContextItems.Logger, _logger }
});
var httpClient = _clientFactory.CreateClient("MySender");
var response = await retryPolicy.ExecuteAsync(ctx =>
httpClient.SendAsync(requestMessage), context);
When I attempt to test this with no endpoint service running then for the first retry attempt, the retry handler is fired and my logger records this first attempt. However, on the second retry attempt i get a error message saying:
The request message was already sent. Cannot send the same request
message multiple times
I know that other people have encountered a similar problem (see Retrying HttpClient Unsuccessful Requests and the solution seems to be do what i'm doing (i.e. use HttpClientFactory). However, i DON'T get this problem if i define my retry policy as part of the configuration in Startup as so:
builder.Services.AddHttpClient("MyService", client =>
{
client.BaseAddress = config.SenderUrl;
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
}).AddPolicyHandler(GetRetryPolicy());
static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.OrResult(msg => msg.StatusCode == System.Net.HttpStatusCode.NotFound)
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromMilliseconds(1000));
}
and simply call my service as so:
var response = await httpClient.SendAsync(requestMessage);
BUT doing it this way i lose the ability to pass my logger in the retry policy context (which is the whole reason i'm injecting in IReadOnlyPolicyRegistry<string> policyRegistry - I can not do this at startup). Another benefit is for unit testing - i can simply inject in the same collection with the same policy without copying and pasting a whole bunch of code and making the unit test redundant since i'm no longer testing my service. Having the policy defined in the startup makes this impossible. So my question is, is there a way to not get this duplicate request error using this approach ?
Here's an alternative solution (which I prefer).
The PolicyHttpMessageHandler added by AddPolicyHandler will create a Polly Context if one isn't already attached. So you can add a MessageHandler that creates a Context and attaches the logger:
public sealed class LoggerProviderMessageHandler<T> : DelegatingHandler
{
private readonly ILogger _logger;
public LoggerProviderMessageHandler(ILogger<T> logger) => _logger = logger;
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
var httpClientRequestId = $"GetSomeData-{Guid.NewGuid()}";
var context = new Context(httpClientRequestId);
context[PolicyContextItems.Logger] = _logger;
request.SetPolicyExecutionContext(context);
return await base.SendAsync(request, cancellationToken);
}
}
A little extension method for registration makes it nice:
public static IHttpClientBuilder AddLoggerProvider<T>(this IHttpClientBuilder builder)
{
if (!services.Any(x => x.ServiceType == typeof(LoggerProviderMessageHandler<T>)))
services.AddTransient<LoggerProviderMessageHandler<T>>();
return builder.AddHttpMessageHandler<LoggerProviderMessageHandler<T>>();
}
And then you can use it as such (note that it must be before the AddPolicyHandler so that it creates the Context first):
builder.Services.AddHttpClient("MyService", client =>
{
client.BaseAddress = config.SenderUrl;
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
})
.AddLoggerProvider<MyService>()
.AddPolicyHandler(GetRetryPolicy());
At runtime, LoggerProviderMessageHandler<MyService> gets an ILogger<MyService>, creates a Polly Context containing that logger, and then invokes PolicyHttpMessageHandler, which uses the existing Polly Context, so your retry policy can successfully use context.TryGetLogger.
You are getting a bit too much caught up in Polly and how to configure it, and forgetting a few basic aspects. Don't worry, this is too easy to do!
First, you cannot send the same HttpRequestMessage more than once. See this extensive Q&A on the subject. It's also documented officially, though the documentation is a bit opaque on the reason.
Second, as you have it coded, the request you created was captured once by the lambda, and then reused over and over again.
For your particular case, I would move the creation of the request inside the lambda that you are passing to ExecuteAsync. This gives you a new request each time.
Modifying your code,
var jsonContent = new StringContent(
JsonSerializer.Serialize(contentObj),
Encoding.UTF8,
"application/json");
var retryPolicy = _policyRegistry.Get<IAsyncPolicy<HttpResponseMessage>>PolicyNames.BasicRetry)
?? Policy.NoOpAsync<HttpResponseMessage>();
var context = new Context(
$"GetSomeData-{Guid.NewGuid()}",
new Dictionary<string, object>
{
{ PolicyContextItems.Logger, _logger }
});
var httpClient = _clientFactory.CreateClient("MySender");
var response = await retryPolicy.ExecuteAsync(ctx =>
{
var requestMessage = new HttpRequestMessage(HttpMethod.Post, "SendEndpoint")
{
Content = jsonContent
};
requestMessage.Headers.Authorization = new AuthenticationHeaderValue("Bearer", authToken);
requestMessage.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
httpClient.SendAsync(requestMessage), context);
}
The other captures: logger, authToken, might be OK if they don't change request over request, but you may need to move other variables inside the lambda as well.
Not using Polly at all makes most of this thought process unnecessary, but with Polly, you have to remember retries and policy are occurring across time and context.
Related
I have a .Net Core API app deployed to Azure App Service. There is only one endpoint, and all it does is call another API (inside MyCLass) and return the response.
I have the following in my Startup:
services.AddScoped<IOAuthService, OAuthService>();
services.AddHttpClient<IMyClass, MyClass>(client =>
{
var authConfig = config.Get<OAuthConfig>();
var oAuthService = services.BuildServiceProvider().GetRequiredService<IOAuthService>();
var token = oAuthService.GetTokenAsync(authConfig).GetAwaiter().GetResult();
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
});
And here is the OAuthService.GetTokenAsync method:
private IConfidentialClientApplication app;
public async Task<string> GetTokenAsync(OAuthConfig config)
{
app = ConfidentialClientApplicationBuilder.Create(config.ClientId)
.WithTenantId(config.TenantId)
.WithClientSecret(config.ClientSecret)
.WithLegacyCacheCompatibility(false) // No need to share with ADAL.NET; increases performance
.Build();
var scopes = new string[] { $"{config.Resource}/.default" };
var authResult = await app.AcquireTokenForClient(scopes)
.ExecuteAsync()
.ConfigureAwait(false);
return authResult.AccessToken;
}
A HttpClient is injected into MyClass where it calls another API and returns the response. The above code results in socket handle leaks and SNAT port exhaustion. However, changing the above code to the following would solve the problem:
// In Startup
services.AddSingleton<IOAuthService, OAuthService>();
services.AddHttpClient<IHdcApiDataConnector, HdcApiDataConnector>();
// In MyClass, whenever making the http call, make a call to `OAuthService.GetTokenAsync`
// and append the token to the request on the fly
My question is why exactly the second code solves the socket handle leak problem?
I trying to implement token base authentication in Blazor webassembly web application with Prerendering enabled.
The steps I have done so far:
Created a sample Blazor Webassembly application
Followed the Official MS doc: Prerender and integrate ASP.NET Core Razor components
Checked if application works
Added token base authentication (custom) changes - ref.
It gives an error at AuthStateProvider
public class AuthStateProvider : AuthenticationStateProvider
{
public override async Task<AuthenticationState> GetAuthenticationStateAsync()
{
var token = await _localStorage.GetItemAsync<string>("authToken");
if (string.IsNullOrWhiteSpace(token))
return _anonymous;
_httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("bearer", token);
return new AuthenticationState(new ClaimsPrincipal(new ClaimsIdentity(JwtParser.ParseClaimsFromJwt(token), "jwtAuthType")));
return _anonymous;
}
}
It gives an error
InvalidOperationException: JavaScript interop calls cannot be issued
during server-side prerendering, because the page has not yet loaded
in the browser. Prerendered components must wrap any JavaScript
interop calls in conditional logic to ensure those interop calls are
not attempted during prerendering.
at this line
var token = await _localStorage.GetItemAsync<string>("authToken");
Now that is obvious; this line should be wrapped in a condition to check for prerendering like:
public class AuthStateProvider : AuthenticationStateProvider
{
public override async Task<AuthenticationState> GetAuthenticationStateAsync()
{
var isNotPreRendering = this._httpContextAccessor.HttpContext.Response.HasStarted;
if(isNotPreRendering)
{
var token = await _localStorage.GetItemAsync<string>("authToken");
if (string.IsNullOrWhiteSpace(token))
return _anonymous;
_httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("bearer", token);
return new AuthenticationState(new ClaimsPrincipal(new ClaimsIdentity(JwtParser.ParseClaimsFromJwt(token), "jwtAuthType")));
}
return _anonymous;
}
}
But the value of isNotPreRendering always false. Is there any other way or work around to make it work?
I have fixed this issue in this way:
Once the app hits GetAuthenticationStateAsync() will find a try-catch, withing the catch I will return an AnonymousState.
AuthenticationState anonymousState = new(new ClaimsPrincipal(new ClaimsIdentity()));
try
{
var token = await _localStorageService.GetItemAsync<string>("authToken");
var memberId = await _localStorageService.GetItemAsync<int>("memberId");
if (string.IsNullOrEmpty(token) || memberId == 0)
return anonymousState;
_httpClient.DefaultRequestHeaders.Authorization = new BasicAuthenticationHeaderValue("bearer", token);
var authState = new AuthenticationState(
new ClaimsPrincipal(
new ClaimsIdentity(JwtParser.ParseClaimFromJwt(token), "jwtAuthType")));
NotifyAuthenticationStateChanged(Task.FromResult(authState));
return authState;
}
catch
{
return await Task.FromResult(anonymousState);
}
In my MainLayout.cs I call the AuthenticationState again. This is not the cleanest solution but it does the work
I successfully used this article to Secure my WASM app and the default WeatherForecastController. What I want to do now is extend the RemoteUserAccount to add the ClientId so it can be added to the User's Claims and always accessible.
What I am just now realizing is that the WeatherForecastController authorizes without issue but if I make any other Controllers and try to call an endpoint, I always get 401.
When a user is Authenticated, I have attempted to the do the below, but it only works if I remove the [RequiredScope(RequiredScopesConfigurationKey = "AzureAd:Scopes")] from the controller.
var initialUser = await base.CreateUserAsync(account, options);
if (initialUser.Identity.IsAuthenticated)
{
var userIdentity = (ClaimsIdentity)initialUser.Identity;
var httpClient = _httpClientFactory.CreateClient();
httpClient.BaseAddress = new Uri("https://localhost:5001/");
var httpResponseMessage = await httpClient.GetAsync($"AdUser?ADObjectId=xxxxx");
if (httpResponseMessage.IsSuccessStatusCode)
{
using var contentStream = await httpResponseMessage.Content.ReadAsStreamAsync();
var adUser = await JsonSerializer.DeserializeAsync<AduserModel>(contentStream);
}
foreach (var role in account.Roles)
{
userIdentity.AddClaim(new Claim("appRole", role));
}
}
I think I need to reuse the httpClient from my Program.cs but I've never done any of this before, I'm usually the UI dev.
builder.Services.AddHttpClient("BlazorWasmHostedAAD.ServerAPI", client => client.BaseAddress = new Uri(builder.HostEnvironment.BaseAddress))
.AddHttpMessageHandler<BaseAddressAuthorizationMessageHandler>();
// Supply HttpClient instances that include access tokens when making requests to the server project
builder.Services.AddScoped(sp => sp.GetRequiredService<IHttpClientFactory>().CreateClient("BlazorWasmHostedAAD.ServerAPI"));
builder.Services.AddMsalAuthentication<RemoteAuthenticationState, SecureUserAccount>(options =>
{
builder.Configuration.Bind("AzureAd", options.ProviderOptions.Authentication);
options.ProviderOptions.DefaultAccessTokenScopes.Add("api://xxx/BlazorHostedAPI.Access");
options.UserOptions.RoleClaim = "appRole";
})
.AddAccountClaimsPrincipalFactory<RemoteAuthenticationState, SecureUserAccount, SecureAccountFactory>();
Add HTTP client like ...
// The API URL
builder.Services.AddScoped(sp => new HttpClient { BaseAddress = new Uri(builder.HostEnvironment.BaseAddress) });
Inject HttpClient
#inject HttpClient _apiClient
To talk to your API the url will be something like
var response = await _apiClient.GetAsync("/api/hello");
...
WHAT DO I HAVE NOW?
Currently, I have a client configured with a RetryAsync policy that uses a primary address and on failure switches to a failover address. The connection details are read from a secrets manager.
services
.AddHttpClient ("MyClient", client => client.BaseAddress = PlaceholderUri)
.ConfigureHttpMessageHandlerBuilder (builder => {
// loads settings from secret manager
var settings = configLoader.LoadSettings().Result;
builder.PrimaryHandler = new HttpClientHandler {
Credentials = new NetworkCredential (settings.Username, settings.Password),
AutomaticDecompression = DecompressionMethods.GZip
};
var primaryBaseAddress = new Uri (settings.Host);
var failoverBaseAddress = new Uri (settings.DrHost);
builder.AdditionalHandlers.Add (new PolicyHttpMessageHandler (requestMessage => {
var relativeAddress = PlaceholderUri.MakeRelativeUri (requestMessage.RequestUri);
requestMessage.RequestUri = new Uri (primaryBaseAddress, relativeAddress);
return HttpPolicyExtensions.HandleTransientHttpError ()
.RetryAsync ((result, retryCount) =>
requestMessage.RequestUri = new Uri (failoverBaseAddress, relativeAddress));
}));
});
WHAT AM I TRYING TO ACHIEVE?
In general
My client can use a primary or failover service. When the primary is down, use failover till the primary is back up. When both are down, we get alerted and can change the service addresses dynamically via secrets manager.
In code
Now I would like to introduce also a CircuitBreakerPolicy and chain those 2 policies together. I am looking for a configuration that is encapsulated and faults are handled on the client level and not on the class consuming that client.
Scenario explained
Let's assume that there is a circuit breaker policy wrapped in a retry policy with a single client.
The circuit breaker is configured to break the circuit for 60 seconds after 3 failed attempts on transient errors on the primary base address. OnBreak - the address changes from primary to failover.
The retry policy is configured to handle BrokenCircuitException, and retry once with the address changed from primary to failover to continue.
Request on primary address - 500 code
Request on primary address - 500 code
Request on primary address - 500 code (3 consecutive failures reached)
Circuit broken for 60 seconds
Request on primary address - BrokenCircuitException caught by retry policy, call failover
Request on primary address - BrokenCircuitException caught by retry policy, call failover
Request on primary address - BrokenCircuitException caught by retry policy, call failover
Request on primary address - BrokenCircuitException caught by retry policy, call failover
(after 60 secs) Circuit half-open - (here can be broken for another 60 secs or is open - assume open)
Request on primary address - 200 code
As described in this articles, there is a solution to this using a breaker wrapped in a fallback, but as you can see there, the logic for default and fallback are implemented in class and not on client level.
I would like
public class OpenExchangeRatesClient
{
private readonly HttpClient _client;
private readonly Policy _policy;
public OpenExchangeRatesClient(string apiUrl)
{
_client = new HttpClient
{
BaseAddress = new Uri(apiUrl),
};
var circuitBreaker = Policy
.Handle<Exception>()
.CircuitBreakerAsync(
exceptionsAllowedBeforeBreaking: 2,
durationOfBreak: TimeSpan.FromMinutes(1)
);
_policy = Policy
.Handle<Exception>()
.FallbackAsync(() => GetFallbackRates())
.Wrap(circuitBreaker);
}
public Task<ExchangeRates> GetLatestRates()
{
return _policy
.ExecuteAsync(() => CallRatesApi());
}
public Task<ExchangeRates> CallRatesApi()
{
//call the API, parse the results
}
public Task<ExchangeRates> GetFallbackRates()
{
// load the rates from the embedded file and parse them
}
}
to be rewritten as
public class OpenExchangeRatesClient
{
private readonly HttpClient _client;
public OpenExchangeRatesClient (IHttpClientFactory clientFactory) {
_client = clientFactory.CreateClient ("MyClient");
}
public Task<ExchangeRates> GetLatestRates () {
return _client.GetAsync ("/rates-gbp-usd");
}
}
WHAT HAVE I READ?
How CircutBreakerWorks and what is there for
Policies can be wrapped and there is a recommender order for wrapping
Microsoft's example of Circuit breaker
Other example of Circuit breaker
Fallback Policy
WHAT HAVE I TRIED?
I have tried few different scenarios to chain and combine circuit breaker policy with a retry policy to achieve the desired goal on a client lever in the Startup file. The last state was the below. The policies are wrapped in the order where retry would be able to catch a BrokenCircuitException, but this has not been the case. The Exception is thrown on the consumer class, which is not the desired result. Although RetryPolicy is triggered, the exception on the consumer class is still thrown.
var retryPolicy = GetRetryPolicy();
var circuitBreaker = GetCircuitBreakerPolicy();
var policyWraper = Policy.WrapAsync(retryPolicy, circuitBreaker);
services
.AddHttpClient("TestClient", client => client.BaseAddress = GetPrimaryUri())
.AddPolicyHandler(policyWraper);
static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(
3,
TimeSpan.FromSeconds(45),
OnBreak,
OnReset,
OnHalfOpen);
}
static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
return Policy<HttpResponseMessage>
.Handle<Exception>()
.RetryAsync(1, (response, retryCount) =>
{
Debug.WriteLine("Retries on broken circuit");
});
}
I have left out the methods OnBreak, OnReset and OnHalfOpen since they are just printing some messages.
UPDATE: Added Logs from Console.
Circuit broken (after 3 attempts)
Retries on broken
Exception thrown: 'System.AggregateException' in System.Private.CoreLib.dll
Retries on broken circuit
Exception thrown: 'System.AggregateException' in System.Private.CoreLib.dll
'CircuitBreakerPolicy.exe' (CoreCLR: clrhost): Loaded 'C:\Program
Retries on broken circuit
Exception thrown: 'System.AggregateException' in System.Private.CoreLib.dll
UPDATE 2: Added reference URL to the class making use of the client with policies configured
UPDATE 3: The project has been updated so that implementation of WeatherService2.Get works in the desired way: When primary service is unavailable the circuit is broken, falover service is used till circuit is closed. That would be the answer to this question, however I would like to explore a solution, where same outcome is achieved using WeatherService.Get with the appropriate policy and client setup on the Startup.
Reference to class using the client.
Reference to project using the class.
On the above logs can be seen Exception thrown: 'System.AggregateException' in System.Private.CoreLib.dll which thrown by the circuitbreaker - that is not expected since there is retry wrapping the circuit breaker.
I've downloaded your project and played with it, so here are my observations:
Blocking vs Non-blocking
Because your code uses blocking async call (.Result) that's why you see AggregateException
public IEnumerable<WeatherForecast> Get()
{
HttpResponseMessage response = null;
try
{
response = _client.GetAsync(string.Empty).Result; //AggregateException
}
catch (Exception e)
{
Debug.WriteLine($"{e.Message}");
}
...
}
In order to unwrap the InnerException of the AggregateException you need to use await
public async Task<IEnumerable<WeatherForecast>> Get()
{
HttpResponseMessage response = null;
try
{
response = await _client.GetAsync(string.Empty); //BrokenCircuitException
}
catch (Exception e)
{
Debug.WriteLine($"{e.Message}");
}
...
}
Escalation
Whenever you wrap a policy into another then escalation might happen. That means if the inner can't handle the problem then it will propagate the same problem to the outer, which may or may not be able to handle it. If the outermost is not handling the problem then (most of the time) the original exception will be thrown to the consumer of the resilience strategy (which is a combination of policies).
Here you can find more details about escalation.
Let's review this concept in your case:
var policyWrapper = Policy.WrapAsync(retryPolicy, circuitBreaker);
static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(3, TimeSpan.FromSeconds(45), ...);
}
static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
return Policy<HttpResponseMessage>
.Handle<Exception>()
.RetryAsync(1, ...);
}
Initial request (1. attempt) is issued against https://httpstat.us/500
It returns 500 which will increase the consecutive transient failure from 0 to 1
CB escalates the problem to retry
Retry is not handling status 500, so retry is not triggered
httpClient returns a HttpResponseMessage with InternalServerError status code.
Let's modify the retry policy to handle transient http errors as well:
static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(3, TimeSpan.FromSeconds(45), ...);
}
static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.Or<Exception>()
.RetryAsync(1, ...);
}
Initial request (1. attempt) is issued against https://httpstat.us/500
It returns 500 which will increase the consecutive transient failure from 0 to 1
CB escalates the problem to retry
Retry is handling status 500, so retry issues another attempt immediately
1st retry request (2. attempt) is issued against https://httpstat.us/500
It returns 500 which will increase the consecutive transient failure from 1 to 2
CB escalates the problem to retry
Even though Retry is handling status 500 it will not trigger because it reached its retrycount (1)
httpClient returns a HttpResponseMessage with InternalServerError StatusCode.
Now, let's lower the consecutive failure count from 3 to 1 and handle BrokenCircuitException explicitly:
static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(1, TimeSpan.FromSeconds(45), ...);
}
static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.Or<BrokenCircuitException>()
.RetryAsync(1, ...);
}
Initial request (1. attempt) is issued against https://httpstat.us/500
It returns 500 which will increase the consecutive transient failure from 0 to 1
Circuit Breaker opens because it reaches the predefined threshold
CB escalates the problem to retry
Retry is handling status 500, so retry issues another attempt immediately
1st retry request (2. attempt) is issued against https://httpstat.us/500
CB prevents this call because it is broken
CB throws a BrokenCircuitException
Even though Retry is handling BrokenCircuitException it will not trigger because it reached its retrycount (1)
Retry throws the original exception (BrokenCircuitException) so httpClient's GetAsync will throw that one.
Finally let's increase the retryCount from 1 to 2:
static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(1, TimeSpan.FromSeconds(45), ...);
}
static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
return HttpPolicyExtensions
.HandleTransientHttpError()
.Or<BrokenCircuitException>()
.RetryAsync(2, ...);
}
Initial request (1. attempt) is issued against https://httpstat.us/500
It returns 500 which will increase the consecutive transient failure from 0 to 1
Circuit Breaker opens because it reaches the predefined threshold
CB escalates the problem to retry
Retry is handling status 500, so retry issues another attempt immediately
1st retry request (2. attempt) is issued against https://httpstat.us/500
CB prevents this call because it is broken
CB throws a BrokenCircuitException
Retry is handling BrokenCircuitException and it did not exceed its retryCount so it issues another attempt immediately
2nd retry request (3. attempt) is issued against https://httpstat.us/500
CB prevents this call because it is broken
CB throws a BrokenCircuitException
Even though Retry is handling BrokenCircuitException it will not trigger because it reached its retrycount (2)
Retry will throw the original exception (BrokenCircuitException) so httpClient's GetAsync will throw that one.
I hope this exercise helped you to better understand how to create a resilience strategy, where you combine multiple policies by escalating the problem.
I've reviewed your alternative solution which has the same design issue as it was discussed in my previous post.
public WeatherService2(IHttpClientFactory clientFactory, IEnumerable<IAsyncPolicy<HttpResponseMessage>> policies)
{
_primaryClient = clientFactory.CreateClient("PrimaryClient");
_failoverClient = clientFactory.CreateClient("FailoverClient");
_circuitBreaker = policies.First(p => p.PolicyKey == "CircuitBreaker");
_policy = Policy<HttpResponseMessage>
.Handle<Exception>()
.FallbackAsync(_ => CallFallbackForecastApi())
.WrapAsync(_circuitBreaker);
}
public async Task<string> Get()
{
var response = await _policy.ExecuteAsync(async () => await CallForecastApi());
if (response.IsSuccessStatusCode)
return response.StatusCode.ToString();
response = await CallFallbackForecastApi();
return response.StatusCode.ToString();
}
Your Fallback policy is never triggered.
HttpClient receives a response with statusCode 500
CircuitBreaker breaks
CB propagates the HttpResponseMessage with statusCode 500 to the outer policy
Fallback does not trigger because it was setup for exceptions Handle<Exception>()
Policy returns the HttpResponseMessage with statusCode 500
Your code manually examines the response and then manually calls the fallback.
If you change your policy to this:
_policy = Policy
.HandleResult<HttpResponseMessage>(response => response != null && !response.IsSuccessStatusCode)
.Or<Exception>()
.FallbackAsync(_ => CallFallbackForecastApi())
.WrapAsync(_circuitBreaker);
then there is no need for manual fallback.
HttpClient receives a response with statusCode 500
CircuitBreaker breaks
CB propagates the HttpResponseMessage with statusCode 500 to the outer policy
Fallback triggers because it was setup for unsuccessful status codes as well
HttpClient receives a response with statusCode 200
Policy returns the HttpResponseMessage with statusCode 500
There is one more important thing that you need to understand. The previous code only works because you have registered the HttpClients without the circuitbreaker policy.
That means the CB is not attached to the HttpClient. So, if you change the code like this:
public async Task<HttpResponseMessage> CallForecastApi()
=> await _primaryClient.GetAsync("https://httpstat.us/500/");
public async Task<HttpResponseMessage> CallFallbackForecastApi()
=> await _primaryClient.GetAsync("https://httpstat.us/200/");
then even though the CircuitBreaker will be Open after the first attempt the CallFallbackForecastApi will not throw a BrokenCircuitException.
BUT if you attach the CB to the HttpClient like this:
services
.AddHttpClient("PrimaryClient", client => client.BaseAddress = PlaceholderUri)
...
.AddPolicyHandler(GetCircuitBreakerPolicy());
and then you simplify the WeatherService2 like this:
private readonly HttpClient _primaryClient;
private readonly IAsyncPolicy<HttpResponseMessage> _policy;
public WeatherService2(IHttpClientFactory clientFactory)
{
_primaryClient = clientFactory.CreateClient("PrimaryClient");
_policy = Policy
.HandleResult<HttpResponseMessage>(response => response != null && !response.IsSuccessStatusCode)
.Or<Exception>()
.FallbackAsync(_ => CallFallbackForecastApi());
}
then it will miserably fail with a BrokenCircuitException.
If your WeatherService2 would look like this:
public class WeatherService2 : IWeatherService2
{
private readonly HttpClient _primaryClient;
private readonly HttpClient _secondaryClient;
private readonly IAsyncPolicy<HttpResponseMessage> _policy;
public WeatherService2(IHttpClientFactory clientFactory)
{
_primaryClient = clientFactory.CreateClient("PrimaryClient");
_secondaryClient = clientFactory.CreateClient("FailoverClient");
_policy = Policy
.HandleResult<HttpResponseMessage>(response => response != null && !response.IsSuccessStatusCode)
.Or<Exception>()
.FallbackAsync(_ => CallFallbackForecastApi());
}
public async Task<string> Get()
{
var response = await _policy.ExecuteAsync(async () => await CallForecastApi());
return response.StatusCode.ToString();
}
public async Task<HttpResponseMessage> CallForecastApi()
=> await _primaryClient.GetAsync("https://httpstat.us/500/");
public async Task<HttpResponseMessage> CallFallbackForecastApi()
=> await _secondaryClient.GetAsync("https://httpstat.us/200/");
}
then it could work fine only if the PrimaryClient and FailoverClient have different circuit breakers.
services
.AddHttpClient("PrimaryClient", client => client.BaseAddress = PlaceholderUri)
...
.AddPolicyHandler(GetCircuitBreakerPolicy());
services
.AddHttpClient("FailoverClient", client => client.BaseAddress = PlaceholderUri)
...
.AddPolicyHandler(GetCircuitBreakerPolicy());
if they would share the same Circuit Breaker then the second call would fail again with a BrokenCircuitException.
var cbPolicy = GetCircuitBreakerPolicy();
services
.AddHttpClient("PrimaryClient", client => client.BaseAddress = PlaceholderUri)
...
.AddPolicyHandler(cbPolicy);
services
.AddHttpClient("FailoverClient", client => client.BaseAddress = PlaceholderUri)
...
.AddPolicyHandler(cbPolicy);
Recently I wanted to upgrade from dot.net netcoreapp2.2 to netcoreapp3.1. I am using https://learn.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-3.1 to execute tests for my API implementation.
There is one test that verifies that cancellation of the user is returning 499 status code (https://httpstatuses.com/499). The implementation of this feature is done via middleware.
In the test the critical assertion looks like this:
var tokenSource = new CancellationTokenSource();
var token = tokenSource.Token;
var client = this.CreateHttpClient(repositoryMock);
HttpResponseMessage responseMessage = null;
await Task.WhenAll(
Task.Run(async () =>
{
await Task.Delay(500);
tokenSource.Cancel();
}),
Task.Run(async () =>
{
responseMessage = await client.GetAsync(new Uri($"http://localhost/api/values/haxi?haxiIds={haxiGuid}"), token);
}));
Assert.That(
responseMessage.StatusCode,
Is.EqualTo((HttpStatusCode)499));
In 2.2 everything is working fine. The server cancels, returns 499 and the HttpClient receives it.
In 3.1 it looks like the server cancels, returns 499, but the HttpClient always throws an exception:
Message:
System.OperationCanceledException : The operation was canceled.
Stack Trace:
HttpClient.HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
<<Cancellation_ShouldReturn499>b__3>d.MoveNext() line 57
--- End of stack trace from previous location where exception was thrown ---
ControllerTests.Cancellation_ShouldReturn499() line 49
GenericAdapter`1.BlockUntilCompleted()
NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaitable)
AsyncToSyncAdapter.Await(Func`1 invoke)
TestMethodCommand.RunTestMethod(TestExecutionContext context)
TestMethodCommand.Execute(TestExecutionContext context)
SimpleWorkItem.PerformWork()
I have setup a complete new solution to reproduce the problem: https://github.com/schrufygroovy/assert-api-cancellation.
Is there some alternative way in 3.1 to verify the response of a user-cancelled http request? Or is my setup of the API in 3.1 wrong? Is the middleware executed, but then overruled somehow by some other new 3.1 feature?
Looks like the behaviour of the HttpClient changed from 2.2 to 3.1
I was able to resolve the problem by using custom HttpMessageInvoker instead of HttpClient.
Make some function to create a HttpMessageInvoker that is using the HttpMessageHandler of the TestServer (using WebApplicationFactory with WithWebHostBuilder):
private HttpMessageInvoker CreateHttpMessageInvoker(Mock<IHaxiRepository> repositoryMock)
{
return new HttpMessageInvoker(this.factory.WithWebHostBuilder(
builder =>
{
builder.ConfigureTestServices(services =>
{
services.AddScoped(typeof(IHaxiRepository), provider => repositoryMock.Object);
});
}).Server.CreateHandler(), true);
}
Use that one instead of HttpClient:
var tokenSource = new CancellationTokenSource();
var token = tokenSource.Token;
using var httpMessageInvoker = this.CreateHttpMessageInvoker(repositoryMock);
HttpResponseMessage responseMessage = null;
await Task.WhenAll(
Task.Run(async () =>
{
await Task.Delay(500);
tokenSource.Cancel();
}),
Task.Run(async () =>
{
responseMessage = await httpMessageInvoker.SendAsync(
new HttpRequestMessage(HttpMethod.Get, new Uri($"http://localhost/api/values/haxi?haxiIds={haxiGuid}")),
token);
}));
Assert.That(
responseMessage.StatusCode,
Is.EqualTo((HttpStatusCode)499));