Update/Edit
I pull their repo build it and step through it, I find the HttpClient is simply timing out here
var result = Await client.SendAsync(request) //Times out after a while
If i change it to a blocking call, it works
var result = Await client.SendAsync(request).GetAwaiter().GetResult() //Works
The next call, gets the results Async, and it works as expected
var body = await result.Content.ReadAsStringAsync(); //Works if the Send is synchronous
Why would sending the request Async timeout, yet performing the send blocking and then reading the result async work? I don't quite understand what is happening here?
Original Post
I'm struggling to debug an issue i'm facing using a 3rd party library that handles the HTTP requests to their API. I'm targeting .Net 4.6.1 - It's a mess of a web project with both WebApi and webforms.
The package is using HttpClient() for it's requests (Postmark.Net)
PostMark Package URL
https://github.com/wildbit/postmark-dotnet/tree/71c05efafd266a4403f8e6c18a8b485252082fd3
Issue
When I run the project, the application just appears to hang. It does not timeout and return back to the caller, it just appears stuck. In my debug window it just shows the threads exiting with code 0.
Sample Code
Dim cc = New PostmarkDotNet.PostmarkClient("key")
Dim msg = cc.GetOutboundMessageDetailsAsync("some-id").GetAwaiter.GetResult()
Log.Debug(msg.Status.ToString & " - " & msg.ReceivedAt.ToString() & " - " & msg.To.FirstOrDefault.Email.ToString())
Note: Intentionally using getAwaiter to block the thread to help troubleshoot.
What I've Tried:
Created a new .net project targeting the same framework version (4.7.1) - The library and code functions as expected; so it must be related to this project
Clean / Rebuild the project
Running Fiddler to see what is happening on the wire - i see an outbound request with a response with the SSL cert from the API - then nothing. I'm thinking perhaps some issue is occurring around the SSL handshake - however, i doubt it is a TLS issue since a test project on same machine worked as expected.
Further Questions:
How can I debug or step into the library itself to see what is happening there?
The package requires .Net.Http 4.3.4, but in my web.config i'm showing a bindingRedirect to 4.2.0.0, but in my test project i threw together seeing the same thing and not having an issue.
Suggestions?
Related
I am working on a .NET Core 3.1 web api application and am having a terrible time with something closing my long running API request.
I have a POST endpoint that looks like this (slightly simplified):
[HttpPost]
[IgnoreAntiforgeryToken]
[ActionName("LoadDataIntoCache")]
public async Task<IActionResult> LoadDataIntoCache([FromQuery] string filter)
{
//long running process (15-20 mins)
var success = await _riskService.LoadDataIntoCache(filter);
if (success == false)
{
return StatusCode(StatusCodes.Status500InternalServerError);
}
return Ok();
}
This endpoint works fine when I test it locally via Postman. However, when I push this to the server (IIS), and hit the endpoint via Postman it produces an error after 5 minutes: Error: read ECONNRESE.
No more details are produced that this. Checking the logs of the application, it does not throw an exception, in fact it appears that the long running processes continues to run as if nothing is wrong. Its as if the connection itself is just being closed by something, but that the application is working fine.
I have also tried calling this endpoint via C# instead of Postman. My calling code produced the following exception message Processing of the HTTP request resulted in an exception. and additionally The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
I have checked the IIS timeout, which is set to 120s, which does not align with the 5 minute time I am seeing. I have checked a bunch of timeout settings on the .NET side, but my understanding is the .NET Core 3.1 does not need this settings because it will wait forever by default? This application is also set to run inProcess if that is significant...
I am really scratching my head on this one. Any pointers would be much appreciated.
I have a service that after my latest updates have started crashing.
It does crash on both the server and in my development environment, so it's not system specific.
It crashes on code generated by Autorest
_httpResponse = await HttpClient.SendAsync(_httpRequest, cancellationToken)
.ConfigureAwait(false);
It throws a StackOverFLowException and nothing is attached... There is no more info given.
Fiddler shows that no data is being sent, so it dosen't start sending anything.
As far as i can recall, the only change i did when it stopped working is i changed another autorest namespace out.
Actually managed to solve it myself.
I was trying to add an Accept-Header by using a delegate, this somehow created an infinite loop thus causing a StackOverflow.
The correct way of adding it is:
Client.HttpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
But i was adding it like this
httpRequest.Headers.Add("Accept", "application/json");
Easy mistake and very weird error message... ^^
I'm using RestSharp to communicate with a .Net Core Web API. Both Client and Server are written by me.
I have a suite of Services all inheriting a Base class which contains an async method to perform the request via a RestClient. Here is the method within the base class that creates the RestClient.
private async Task<ServiceResponse> RequestAsync(ServiceRequest request)
{
try
{
var result = await new RestClient(_digiCore.Config.GetApiBaseUrl()).ExecuteTaskAsync(request.Request, request.CancellationTokenSource.Token);
switch (result.StatusCode)
{
case HttpStatusCode.OK:
case HttpStatusCode.Created:
case HttpStatusCode.NoContent:
return new ServiceResponse
{
Code = ServiceResponseCode.Success,
Content = result.Content
};
// User wasn't authenticated for this one - better luck next time!
case HttpStatusCode.Unauthorized:
Logger.LogError($"Unauthorized {request.Method.ToString()}/{request.Path}");
default:
Logger.LogError($"An error occurred {request.Method.ToString()}/{request.Path}");
}
}
catch (Exception e)
{
Logger.LogError($"A Rest Client error occurred {request.Method.ToString()}/{request.Path}");
}
}
My understanding is that the RestClient (unlike HttpClient) is thread safe and OK to create a new instance each time.
When Load testing my application, one I reach a certain point, I find that I occasionally receive the following response back from the API. Refreshing the page might bring back the correct result, or I might get another error. When the Load Test has finished everything goes back to normal. The numbers of the load test are nothing crazy, so its started to get me worried about real-life performance...
Only one usage of each socket address (protocol/network address/port) is normally permitted
I believe this is down to port exhaustion. Both API and Client are running in Azure as App services.
I have tried making the RestClient part of a singleton service, and checked that it is only created once - but this did not alleviate the issue.
I have also looked into setting the DefaultConnectionLimit in the Startup function of my startup class from the default (2) to 12, but I did not see any improvements here either.
I'm really struggling to see how I can improve this...
Found out the reason for this...
It seems there is currently an issue with RestSharp whereby socket connections are not closed down immediately and instead are left in TIME_WAIT state.
This issue is currently logged with the RestSharp guys here ->
https://github.com/restsharp/RestSharp/issues/1322
I have chosen to switch over to using HttpClient - and running the exact same load test I had zero problems.
Here is an screenshot taken from Azure showing the difference between the two load tests.
Was with my app using RestSharp
Was with my app using Http Client
Hope this helps somebody. It's a shame I had to ditch RestSharp as it was a nice little library - unfortunately with an issue like this it's just too risky to run in a Production environment.
We have a self-hosted Apple Enterprise Developer based iOS app that was released around six weeks ago. We're working on the next version of the app and are seeing problems when people update to the new test version. An HttpRequestException is thrown with a "Connection reset by peer" message whenever calls over HTTPS that include a bearer token are made. The call stack shown in the MS App Center error report provides little info to go on.
Main thread
System.Net.Sockets
Socket.Receive (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.Net.Sockets.SocketFlags socketFlags)
System.Net.Sockets
NetworkStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 size)
The initial web request to sign the user in appears to be working. It is the only request that doesn't include a bearer token in the headings. All subsequent requests in a given session cause this exception to be thrown.
We have found that users can resolve the error by removing the currently released version of the app from their phone before installing the new version. The error no longer occurs in the new version once that is done. We've also received a couple of reports of users getting the error even without upgrading the version of the app. It seems to occur for those users after not having used the app in a while.
We have tried to reproduce the error by following the same steps of installing the new test version on top of the released version on our devices and it works fine.
Our project manager has the issue on her personal device. She has been testing new versions without upgrading to see if the exception is no longer thrown. We have tried the following to resolve the problem without success:
Removed all local caching of previous web request results and bearer tokens, forcing live requests at all times
Switching from the iOS NSUrlSession implementation of HttpClient back to the default Xamarin managed implementation
Verifying the HttpClient base URI and individual request URIs are what are expected
Verifying the bearer token value has been assigned to the HttpClient Authorization default headers when the calls are made
Has anyone had this happen to them, or maybe can shed some light on possible causes of this exception being thrown? Thanks a ton for any help provided. It is driving us crazy!
Here's an example of one of the request calls. The HttpClient property is a static instance that is used for the lifetime of the app. The SaveContext property is a bool that is used to enable/disable context saving to test performing the requests in background threads without saving the context. Both options have been tried and had no impact on the error occurring. The HttpClient BaseAddress property has the API uri root assigned to it. The uri passed to GetAsync is a relative uri.
HttpResponseMessage serverResponse = await HttpClient.GetAsync(uri).ConfigureAwait(SaveContext);
Here's more of the setup:
var handler = new HttpClientHandler {
AllowAutoRedirect = true, AutomaticDecompression =
DecompressionMethods.Deflate | DecompressionMethods.GZip,
MaxAutomaticRedirections = 20
};
var httpClient = new HttpClient(handler);
httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(JsonSerializerHelper.JsonEncoding));
HttpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
I have a WCF client (running on Win7) pointing to a WebSphere service.
All is good from a test harness (a little test fixture outside my web app) but when my calls to the service originate from my web project one of the calls (and only that one) is extremely slow to deserialize (it takes minutes VS seconds) and not just the first time.
I can see from fiddler that the response comes back quickly but then the WCF client hangs on the response itself for more than a minute before the next line of code is hit by the debugger, almost if the client was having trouble deserializing. This happens only if in the response I have a given pdf string (the operation generates a pdf), base64 encoded chunked. If for example the service raises a fault (thus the pdf string is not there) then the response is deserialized immediately.
Again, If I send the exact same envelope through Soap-UI or from outside the web project all is good.
I am at loss - What should I be looking for and is there some config setting that might do the trick?
Any help appreciated!
EDIT:
I coded a stub against the same service contract. Using the exact same basicHttpBinding and returning the exact same pdf string there is no delay registered. I think this rules out the string and the binding as a possible cause. What's left?
Changing transferMode="Buffered" into transferMode="Streamed" on the binding did the trick!
So the payload was apparently being chunked in small bits the size of the buffer.
I thought the same could have been achieved by increasing the buffersize (maxBufferSize="1000000") but I had that in place already and it did not help.
I have had this bite me many times. Check in your WCF client configuration that you are not trying to use the windows web proxy, that step to check on the proxy (even if there is not one configured) will eat up a lot of time during your connection.
If the tips of the other users don't help, you might want to Enable WCF Tracing and open it in the Service Trace Viewer. The information is detailed, but it has enabled me to fix a number of hard-to-identity problems in the past.
More information on WCF Tracing can be found on MSDN.
Two thing you can try:
Adjust the readerQoutas settings for your client. See http://msdn.microsoft.com/en-us/library/ms731325.aspx
Disable "Just My Code" in debugging options. Tools -> Options -> Debugging -> General "Enable Just My Code (Managed only)" and see if you can catch interal WCF exceptions.
//huusom
I had the very same issue... The problem of WCF, IMO, is in the deserialization of the base64 string returned by the service into a byte[] client side.
The easiest to solve this if you may not change your service configuration (Ex.: use a transferMode="Streamed") is to adapt your DataContract/ServiceContract client side. Replace the type "byte[]" with "string" in the Response DataContract.
Next simply decode the returned string yourself with a piece of code such as:
byte[] file = Convert.FromBase64String(pdfBase64String);
To download a PDF of 70KB, it used to required ~6 sec. With the suggested change here above, it takes now < 1 sec.
V.
PS.: Regarding the transfer mode, I did try to only change the client side (transferMode="StreamedResponse") but without improvement...
First things to check:
Is the config the same in the web project and the test project?
When you test from SOAP UI are you doing it from the same server and in the same security context as when the code is running from the web project.
-Is there any spike in the memory when the pdf comes back?
Edit
From your comments the 1 min wait, appears that it is waiting for a timeout. You also mention transactions.
I am wondering if the problem is somewhere else. The call to the WCF service goes OK, but the call is inside a transaction, and there is no complete or dispose on the transaction (I am guessing here), then the transaction / code will hang for 1 min while it waits to timeout.
Edit 2
Next things to check:
Is there any difference in the code is the test and in the web project, on how the service is being called.
Is there any differnce in the framework version, for example is one 3.0 and the other 3.5
Can it be that client side is trying to analyse what type of content is coming from server side? Try to specify mime type of the service response explicitly on the server side, e.g. Response.ContentType = "application/pdf" EDIT: By client side I mean any possible mediator like a firewall or a security suite.