I've created a retry policy on my HttpClient in the Startup.ConfigureServices method. Note also that by default, asp.net core 2.1 logs 4 [Information] lines for each call made by the HttpClient which are shows in the logs at the end of my question.
services.AddHttpClient("ResilientClient")
.AddPolicyHandler(
Policy.WrapAsync(
PollyRetryPolicies.TransientErrorRetryPolicy(),
Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(60))));
The policy is defined as follows. Note that I write the retry attempt to logs, so I will know if the retry policy is invoked.
public static IAsyncPolicy < HttpResponseMessage > TransientErrorRetryPolicy() {
return HttpPolicyExtensions
.HandleTransientHttpError()
.Or < TimeoutRejectedException > ()
.WaitAndRetryAsync(sleepDurations: ExponentialBackoffPolicy.DecorrelatedJitter(3, SEED_DELAY, MAX_DELAY),
onRetry: (message, timespan, attempt, context) => {
context.GetLogger() ? .LogInformation($ "Retrying request to {message?.Result?.RequestMessage?.RequestUri} in {timespan.TotalSeconds} seconds. Retry attempt {attempt}.");
});
}
HandleTransientHttpError() is a Polly extension that states in it's comments:
The conditions configured to be handled are:
• Network failures (as System.Net.Http.HttpRequestException)
My httpclient usage is like this:
using (HttpResponseMessage response = await _httpClient.SendAsync(request))
{
response.EnsureSuccessStatusCode();
try
{
string result = await response.Content.ReadAsStringAsync();
if (result == null || result.Trim().Length == 0) {
result = "[]";
}
return JArray.Parse(result);
} catch (Exception ex) {
_logger.LogInformation($ "Failed to read response from {url}. {ex.GetType()}:{ex.Message}");
throw new ActivityException($ "Failed to read response from {url}.", ex);
}
}
The following logs are captured:
[Information] System.Net.Http.HttpClient.ResilientClient.LogicalHandler: Start processing HTTP request GET https://api.au.... obfuscated
[Information] System.Net.Http.HttpClient.ResilientClient.CustomClientHandler: Sending HTTP request GET https://api.au..... obfuscated
[Information] System.Net.Http.HttpClient.ResilientClient.CustomClientHandler: Received HTTP response after 2421.8895ms - 200
[Information] System.Net.Http.HttpClient.ResilientClient.LogicalHandler: End processing HTTP request after 2422.1636ms - OK
Unknown error responding to request: HttpRequestException:
System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: The server returned an invalid or unrecognized response.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.CopyToAsyncCore(Stream destination, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
--- End of inner exception stack trace ---
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at nd_activity_service.Controllers.ActivityController.GetND(String url) in /codebuild/output/src251819872/src/src/nd-activity-service/Controllers/ActivityController.cs:line 561
The Http call succeeds, and I can see it returns 200 - OK. But then the HttpRequestException is thrown. I assume the policy is not being invoked because the HttpClient message pipeline has already resolved, as we can see it returned 200 - OK. So how is it throwing an exception outside of this?
And how do I handle it? Wrap another policy around the method that handles HttpRequestExceptions specifically?
This error does appear to be transient. It is a scheduled job and works the next time it is called.
Your policy is defined against the HttpClient not against the HttpResponseMessage.
So, the response.EnsureSuccessStatusCode() will not trigger retry even if you receive for example 428.
The HandleTransientHttpError will trigger retry if you receive 408 or 5XX status codes from the downstream system. And when the SendAsync throws the HttpRequestException
Because your exception StackTrace looks like this:
System.Net.Http.HttpRequestException: Error while copying content to a stream.
System.IO.IOException: The server returned an invalid or
unrecognized response.
that's why my educated guess is that this exception is thrown by the HttpContent class while you try to read the response body (ReadAsStringAsync).
This will not trigger retry since you have defined your policy on the HttpClient.
If you want to retry in those cases as well when either the response.EnsureSuccessStatusCode() throws HRE or when the response.Content.ReadAsStringAsync() does then you have to wrap your whole http communication and response processing logic into a retry policy.
Let me show you how to do that.
First use a PolicyRegistry instead of AddPolicyHandler:
//services.AddHttpClient("ResilientClient")
// .AddPolicyHandler(
// Policy.WrapAsync(
// TransientErrorRetryPolicy(),
// Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(60))));
services.AddHttpClient("ResilientClient");
var registry = services.AddPolicyRegistry();
registry.Add("retry", Policy.WrapAsync(
TransientErrorRetryPolicy(),
Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(60))));
Then ask the DI for the register, for example:
private readonly IHttpClientFactory factory;
private readonly IReadOnlyPolicyRegistry<string> registry;
public TestController(IHttpClientFactory factory, IReadOnlyPolicyRegistry<string> registry)
{
this.factory = factory;
this.registry = registry;
}
Finally retrieve the combined policy and execute the http call:
var retryPolicy = registry.Get<IAsyncPolicy<HttpResponseMessage>>("retry");
await retryPolicy.ExecuteAsync(async () => await IssueRequest());
private async Task<HttpResponseMessage> IssueRequest()
{
var _httpClient = factory.CreateClient("ResilientClient");
HttpResponseMessage response = await _httpClient.GetAsync("http://httpstat.us/428");
response.EnsureSuccessStatusCode();
return response;
}
I've used the httpstat.us to simulate 428 response.
Related
I have a Azure function that is doing HTTP calls to another Azure Function endpoint, both are running in the consumption plan. The HTTP Client is setup via HttpClientFactory. In 99 of 100 times the call gets through successfully, but there are a few requests that throws the SocketException with the error message "Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request.."
Stack trace:
System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](TIOAdapter adapter)
at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
This is how the client that does the request look like:
public class ReadClient : BaseClient, IRClient
{
private HttpClient Client { get; }
public ReadClient(IConfiguration config, IHttpClientFactory httpClientFactory) : base(config)
{
Client = httpClientFactory.CreateClient(nameof(ReadClient));
}
public async Task<TResponse> Get<TResponse>(string url)
{
var authenticationToken = GetAuthenticationToken();
Client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(authenticationToken.Type, authenticationToken.Token);
var response = await Client.GetAsync(url);
return ProccessResponse<TResponse>(response);
}
}
My first thought was that perhaps there were too many requests coming in at the same time and that the function didn't have time to scale up with a new instance, but these occurs randomly at times when there aren't that much traffic. Do you have any idea on how to investigate why these errors occurs?
The problem is because you're not reusing HttpClient properly and it's leading to port exaustion.
This is not an azure function but http client issue and you can read more about it in here:
https://www.aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
To solve it when using Azure Functions, please check the following:
https://learn.microsoft.com/en-us/azure/azure-functions/manage-connections?tabs=csharp#http-requests
https://learn.microsoft.com/en-us/azure/azure-functions/errors-diagnostics/sdk-rules/azf0002
Please explain why this doesn't work with "https://localhost:5001" but works correctly with "https://localhost:44386". If you will send a link to any documentation, it will be large plus.
2 asp.net core 5.0 WebAPI applications
one application sends an http request to another
throwing an exception in the first option
public class SomeHttpClient: ISomeHttpClient
{
private readonly HttpClient _httpClient;
public ServerHealthCheckServiceHttpClient(HttpClient httpClient, IOptions<SomeOptions> options)
{
_httpClient = httpClient;
_httpClient.BaseAddress = options.Value.SomeAddress;
}
public async Task<bool> DoSomething(CancellationToken token)
{
var request = new HttpRequestMessage(HttpMethod.Get, "someMethod");
var response = await _httpClient.SendAsync(request, token);
}
}
First variant error System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Cannot determine the frame size or a corrupted frame was received.
Change port number. That will unblock you
I have a gRPC server in Golang that has mTLS enabled using the following ServerOptions:
// getServerOptions returns a list of GRPC server options.
// Current options are TLS certs and opencensus stats handler.
func (h *serviceHandler) getServerOptions() []grpc.ServerOption {
tlsCer, err := tls.LoadX509KeyPair(tlsDir+"tls.crt", tlsDir+"tls.key")
if err != nil {
logger.WithError(err).Fatal("failed to generate credentials")
}
cfg := &tls.Config{
Certificates: []tls.Certificate{tlsCer},
ClientAuth: tls.RequireAndVerifyClientCert,
GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {
h.certMutex.RLock()
defer h.certMutex.RUnlock()
return &tls.Config{
Certificates: []tls.Certificate{tlsCer},
ClientAuth: tls.RequireAndVerifyClientCert,
ClientCAs: h.caCertPool,
}, nil
},
}
// Add options for creds and OpenCensus stats handler to enable stats and tracing.
return []grpc.ServerOption{grpc.Creds(credentials.NewTLS(cfg)), grpc.StatsHandler(&ocgrpc.ServerHandler{})}
}
The server works fine for a gRPC client in Golang, but fails for the following gRPC c# client after the cert exchange handshake.
static async Task Main(string[] args)
{
string baseAddress = "x.x.x.x";
var x509Cert = new X509Certificate2("client.pfx", "123");
var client = CreateClientWithCert("https://" + baseAddress + ":443", x509Cert);
try {
var response = await client.PostAllocateAsync(new AllocationRequest {Namespace = "Default"});
Console.Write(response.State.ToString());
}
catch(RpcException e)
{
Console.WriteLine($"gRPC error: {e.Status.Detail}");
}
catch
{
Console.WriteLine($"Unexpected error calling agones-allocator");
throw;
}
}
public static AllocationService.AllocationServiceClient CreateClientWithCert(
string baseAddress,
X509Certificate2 certificate)
{
var loggerFactory = LoggerFactory.Create(logging =>
{
logging.AddConsole();
logging.SetMinimumLevel(LogLevel.Trace);
});
// Add client cert to the handler
var handler = new HttpClientHandler();
handler.ClientCertificates.Add(certificate);
handler.ServerCertificateCustomValidationCallback =
HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;
// Create the gRPC channel
var channel = GrpcChannel.ForAddress(baseAddress, new GrpcChannelOptions
{
HttpClient = new HttpClient(handler),
LoggerFactory = loggerFactory,
});
return new AllocationService.AllocationServiceClient(channel);
}
}
Here is the trace log:
dbug: Grpc.Net.Client.Internal.GrpcCall[1]
Starting gRPC call. Method type: 'Unary', URI: 'https://x.x.x.x/v1alpha1.AllocationService/PostAllocate'.
dbug: Grpc.Net.Client.Internal.GrpcCall[18]
Sending message.
trce: Grpc.Net.Client.Internal.GrpcCall[21]
Serialized 'V1Alpha1.AllocationRequest' to 9 byte message.
trce: Grpc.Net.Client.Internal.GrpcCall[19]
Message sent.
fail: Grpc.Net.Client.Internal.GrpcCall[6]
Error starting gRPC call.
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The response ended prematurely.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request)
dbug: Grpc.Net.Client.Internal.GrpcCall[8]
gRPC call canceled.
fail: Grpc.Net.Client.Internal.GrpcCall[3]
Call failed with gRPC error status. Status code: 'Internal', Message: 'Error starting gRPC call: An error occurred while sending the request.'.
dbug: Grpc.Net.Client.Internal.GrpcCall[4]
Finished gRPC call.
gRPC error: Error starting gRPC call: An error occurred while sending the request.
Can someone please help me understand what is the reason for the failure? Using SslCredentials also fails.
x.x.x.x is the IP replacement for the privacy reasons.
I wrote some sample codes to reproduce the cross platform incompatibility issue reported here.
The root cause is when using GetConfigForClient in golang server, which in this case is used to refresh the client CA certificates, the requests from C# client fails. However, when it was replaced with VerifyPeerCertificate, the issue got resolved.
I'm trying to make a fairly simple POST request to a .NET WebAPI. The request is being made from an Azure Function, in case that makes a difference.
Here's the code:
public async Task SetBackupRecordAsync(BackupRecord backupRecord)
{
var content = new StringContent(JsonConvert.SerializeObject(backupRecord),
Encoding.UTF8,
"application/json");
var uri = new Uri(_httpClient.BaseAddress, "backup");
HttpResponseMessage result = null;
try
{
result = await _httpClient.PostAsync(uri, content).ConfigureAwait(false);
if (result.StatusCode == HttpStatusCode.Accepted)
{
return;
}
throw new Exception($"Unexpected response: The API responded with {(int) result.StatusCode} ({result.StatusCode}) when setting the backup record.");
}
catch (Exception exception)
{
// ...
}
}
And this is the JSON content which is being included:
{"AddedById":14000,"AddedOnUtc":"2019-06-17T09:43:25.9821306Z","Backup":false,"DeletedById":null,"DeletedOnUtc":null,"Id":"4c3ef086-3e2a-4964-bdc1-f5e72f525fbd","LastBackupUtc":null,"Name":"something"}
I've also tried setting every property to a non-null value, which made no difference.
Unlike other questions on the same exception message, this request doesn't even get sent - as soon as _httpClient.PostAsync is called an exception is thrown:
An error occurred while sending the request.
The inner exception says
The server committed a protocol violation. Section=ResponseStatusLine
And the inner stack trace is quite short:
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
Some additional points:
I can see from Fiddler that no request is being put on the wire so there must be - as the top-level exception says - something about the request itself which is preventing it from being sent.
If I try to hit the same API method from Postman, using exactly the same JSON content, then it works.
I'm able to successfully make GET requests to the same API from the same Azure Function.
So what's wrong with this request?
I have a WEB API application, where we add some session verification logic and then call base SendAsync method after that.
When we pass request containing byte[] larger than 4MB line response = await base.SendAsync(request, cancellationToken); throws a httpException "maximum request length exceeded", which is ok and can be managed through IIS settings. What bothers me is that next step - it's still calling controller method, where I can see that this byte[] parameter set to null. Controller method throws ArgumentNullException and this what end user gets.
First of all I can't understand why controller method still gets called after HttpException. And more important how can I manage to catch this HttpException, so that end user will see actual reason, why the request was not processed.
Thank you in advance.
public class SessionIdHandler : DelegatingHandler
{
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
//some additional logic here
HttpResponseMessage response;
try
{
response = await base.SendAsync(request, cancellationToken);
}
catch (Exception e)
{
response = new HttpResponseMessage(HttpStatusCode.Forbidden)
{
Content = new StringContent(e.Message, Encoding.UTF8, "application/json")
};
}
finally
{
if (signedOnHere)
{
Signoff(sessionId);
}
}
return response;
}
}
What bothers me is that next step - it's still calling controller
method, where I can see that this byte[] parameter set to null.
When you call base.SendAsync(request, cancellationToken);, you're effectively continuing the WebAPI pipeline, which will eventually results in the controller getting invoked:
What you actually want is to check the Content-Length header before you even send the request to the controller. You do so by retrieving it for the header of the request. If the size exceeds, you can return a response indicating that, otherwise, continue the pipeline:
protected override Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
// 4MB, perhaps you want to loosen this up a bit as other request
// characteristics will effect the size.
const int maxRequestSize = 4194304;
IEnumerable<string> contentLengthHeader;
request.Headers.TryGetValues("Content-Length", out contentLengthHeader);
var contentLength = contentLengthHeader.FirstOrDefault();
if (contentLength == null)
{
//No content-length sent, decide what to do.
}
long actualContentlength;
if (!long.TryParse(contentLength, out actualContentlength))
{
// Couldn't parse, decide what to do.
}
if (actualContentlength > maxRequestSize)
{
// If reached, the content-length of the request was too big.
// Return an error response:
return Task.FromResult(request.CreateErrorResponse(HttpStatusCode.Forbidden,
string.Format("Request size exceeded {0} bytes", maxRequestSize)));
}
return base.SendAsync(request, cancellationToken);
}
If you don't want this check for all the controllers in your WebAPI application, you can pre-route each DelegatingHandler.