We are using an HttpClient to post json to a restful web service. In one instance, we are running into something that has us baffled. Using tools like postman, fiddler etc, we can post to an endpoint and see that it is working. When we do the same with HttpClient.PostAsJsonAsync, we can verify in the software we are posting to that it received the data just fine. However, our PostAsJsonAsync will always eventually time out rather than give us a response.
We have worked with the team that created the service we are consuming, plus our additional testing on our side, and we have not yet been able to truly time out that service.
Every time we do a post with HttpClient, we then can verify that the target software we post to does indeed get the data. Any time we do a post to that target software from any other tool, we always very quickly see a response with status code of 200. Something about HttpClient is failing to accept the response from this particular service. Does anyone have an idea what we can look at from here?
Here's the code (though it is so cookie cutter that I hardly feel it is needed)
public string PostData(string resourcePath, Object o, Boolean isCompleteUrl = false, int timeoutMinutes = -1)
{
using (var client = new HttpClient())
{
if (timeoutMinutes > 0)
{
client.Timeout = new TimeSpan(0,timeoutMinutes,0);
}
var useUrl = isCompleteUrl ? resourcePath : ApiBase + resourcePath;
var response = client.PostAsJsonAsync(useUrl, o).Result;
if(response.StatusCode == System.Net.HttpStatusCode.OK)
{
return response.Content.ReadAsStringAsync().Result;
}
return "";
}
}
This:
var response = client.PostAsJsonAsync(useUrl, o).Result;
Is causing you code to deadlock. This is often the case when blocking on async API's, and that's why you're experiencing the "I don't see any response coming back" effect.
How is this causing a deadlock? The fact that you are executing this request in an environment that contains a synchronization context, perhaps one which belongs to the UI. It's executing the async request, and when the response arrives, it continues via an IO completion thread which attempts to post the continuation onto that same UI context, which is currently blocked by your .Result call.
If you want to make an HTTP request synchronously, use WebClient instead. If you want to take advantage of the async API properly, then await instead of block with .Result.
I had the same issue and this SO answer fixed it for me.
In a nutshell, you have to use the ConfigureAwait(false) extension to avoid the deadlock:
var response = await client.PostAsJsonAsync(useUrl, o).ConfigureAwait(false);
Is there a reason why you're not following the async await pattern? You're calling an async method, but not awaiting it. You didn't say if the code calling your REST service was a Windows Forms or ASP.NET application, but that .Result is probably causing you issues.
Can you restructure your method like this:
public async Task<string> PostData(string resourcePath, Object o, Boolean isCompleteUrl = false, int timeoutMinutes = -1)
{
using (var client = new HttpClient())
{
if (timeoutMinutes > 0)
{
client.Timeout = new TimeSpan(0,timeoutMinutes,0);
}
var useUrl = isCompleteUrl ? resourcePath : ApiBase + resourcePath;
var response = await client.PostAsJsonAsync(useUrl, o);
if(response.StatusCode == System.Net.HttpStatusCode.OK)
{
return await response.Content.ReadAsStringAsync();
}
return "";
}
}
This is a slight modification to #Justin Helgerson's solution. There are 2 blocking .Result calls in your method; once you go async you should fix them both.
public async Task<string> PostDataAsync(string resourcePath, Object o, Boolean isCompleteUrl = false, int timeoutMinutes = -1)
{
using (var client = new HttpClient())
{
if (timeoutMinutes > 0)
{
client.Timeout = new TimeSpan(0,timeoutMinutes,0);
}
var useUrl = isCompleteUrl ? resourcePath : ApiBase + resourcePath;
var response = await client.PostAsJsonAsync(useUrl, o);
if(response.StatusCode == System.Net.HttpStatusCode.OK)
{
return await response.Content.ReadAsStringAsync();
}
return "";
}
}
Note I've also renamed the method to PostDataAsync in accordance with the TAP pattern.
System.Net.ServicePointManager.Expect100Continue = false;
That one line of code in our case fixed the problem. A developer from a different team offered that suggestion, and it works. I have yet to google it and read up on it enough to offer an explanation of what that is addressing.
Related
Seems like simple task, but the interwebs are flooded with different ways to approach this, so what is best practice for making a simple HTTP POST from asp.net Page_Load (I'm on .net 4.7) ?
Many resources point to the fact that HttpClient.PostAsync is the most lightweight approach.
However, the example here: https://learn.microsoft.com/en-us/aspnet/web-forms/overview/performance-and-caching/using-asynchronous-methods-in-aspnet-45 - uses WebClient.
Also, both of theses approaches require setting Page Async="true" on the page - I am seeing conflicting information on whether this is actually the right approach.
Context: My page looks like it is spending a lot of time in BLOCKED_TIME during two requests that are made in Page_Load.
I suspect this is due to the way I currently make the POST:
public string makePost(string parametersToPassOn, string command)
{
HttpContent c = new StringContent(parametersToPassOn, Encoding.UTF8, "application/json");
var t = Task.Run(() => fireRESTcall(new Uri(walletProxyURL + command), c));
t.Wait();
return t.Result;
}
static async Task<String> fireRESTcall(Uri u, HttpContent c)
{
var response = string.Empty;
using (var client = new HttpClient())
{
HttpRequestMessage request = new HttpRequestMessage
{
Method = HttpMethod.Post,
RequestUri = u,
Content = c
};
HttpResponseMessage result = await client.SendAsync(request);
if (result.IsSuccessStatusCode)
{
response = await result.Content.ReadAsStringAsync();
}
}
return response;
}
Any help/thoughts would be greatly appreciated.
During the time your page makes an outgoing HTTP request, it can't do anything useful other than waiting for the response. See also Understanding BLOCKED_TIME in PerfView
You probably can't do anything about this delay, as you need the external API for some data, and you need to wait for it to complete in order to do something useful with that data.
So it's advisable to use async/await all the way, using RegisterAsyncTask():
void Page_Load()
{
RegisterAsyncTask(new PageAsyncTask(CallApiAsync));
Page.ExecuteRegisteredAsyncTasks();
}
async Task CallApiAsync() // calls
async Task<string> MakePost() // calls
async Task<String> FireRESTcall()
This frees up the thread to handle a new incoming request, until the outgoing request is finished and the incoming request continues on the same or another thread pool thread.
For this to work with WebForms, you do need Async="true" on your page.
I'm creating an application with Xamarin.Forms which consume from SOAP services using POST method; I have a bunch of services that work correctly, indeed, one of these methods is used to send information in multiple cases and my problem is related with this.
I have the following HttpClient method, but sometimes doesn't work... unfortunately I don't have access to back-end code and they are not for the labour of help me with that.
Any idea about how to improve my method or get any approach to the real error? I'm stuck here, since I send the same fields each time.
public async Task<string> InvokeAsync (string uri, string xmlSOAP) {
try {
using (var handler = new HttpClientHandler () { UseCookies = false })
using (var client = new HttpClient (new NativeMessageHandler ())) {
client.DefaultRequestHeaders.Accept.Add (new MediaTypeWithQualityHeaderValue ("application/xml"));
client.DefaultRequestHeaders.Add ("Cache-Control", "no-cache, no-store, must-revalidate");
client.DefaultRequestHeaders.Add ("Pragma", "no-cache");
client.Timeout = TimeSpan.FromSeconds (timeout);
client.DefaultRequestHeaders.CacheControl.NoCache = true;
var req = new HttpRequestMessage (HttpMethod.Post, uri)
{
Content = new StringContent (xmlSOAP, Encoding.UTF8)
};
req.Content.Headers.ContentType = MediaTypeHeaderValue.Parse ("text/xml; charset=utf-8");
if (uri.ToLowerInvariant ().Equals (jsessionUrlCheck)) {
if (jsession != null && jsession.Count > 0) {
foreach (var cookie in jsession) {
req.Headers.Add ("JSESSIONID", cookie);
}
}
jsession = null;
}
HttpResponseMessage response = await client.SendAsync (req);
string responseBodyAsText = response.IsSuccessStatusCode ? await response.Content.ReadAsStringAsync () : string.Empty;
if (!string.IsNullOrEmpty (responseBodyAsText))
{
return responseBodyAsText;
}
return null;
}
} catch (Exception e) {
Debug.WriteLine ("========= InvokeAsync Exception =========");
Debug.WriteLine ("Error: " + e.Message);
return null;
}}
Any idea about how to [...] get any approach to the real error?
It sounds like you don't really know what exactly happens when it "doesn't work". The way you approach the real error is by finding out what exactly happens in this code when it is reported not to work.
Do you have logs? Check the logs. If the exception is there, that should point you in the right direction. Exception not there? Maybe start logging the data received too. No logs? Start logging; there's no better way to handle intermittent failures that you can't reproduce on demand.
I was reading the following topic http://blog.stephencleary.com/2012/07/dont-block-on-async-code.html
and decided to write a common utility method in my library to do a GET on remote url via HTTPClient
public static async Task<T> GetAsync<T>(HttpGetObject getObject)
{
string baseUrl = getObject.BaseUrl;
string actionUrl = getObject.ActionRelativeUrl;
string acceptType = getObject.AcceptType;
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(baseUrl);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(acceptType));
AddCustomHeadersToHttpClient(client, getObject);
// HTTP GET
HttpResponseMessage httpResponseMessage = await client.GetAsync(actionUrl).ConfigureAwait(false);
if (httpResponseMessage.IsSuccessStatusCode)
{
T response = await httpResponseMessage.Content.ReadAsAsync<T>().ConfigureAwait(false);
return response;
}
else
{
string message = httpResponseMessage.Content.ReadAsStringAsync().Result;
throw new Exception(message);
}
}
return default(T);
}
I know the "await httpResponseMessage.Content.ReadAsAsync().ConfigureAwait(false)" will prevent the deadlock in the above code
First:
My query is for "string message = httpResponseMessage.Content.ReadAsStringAsync().Result" line, will .Result can cause deadlock or not in that line?
Second:
If I call that code from UI like this:
public static object DoGet()
{
// Build getObject
var task = Utility.GetAsync(getObject);
task.Wait();
var response = task.Result;
return response;
}
Will that cause a deadlock?
Please note that I know to avoid all the mess with async-await, all the methods from UI to DAL must be async-await but I am not in position at this moment to change all that structure, my goal at this moment is to call HttpClient library and do a few GET operations.
So my questions is that will the above code can cause a deadlock?
Third:
Is task.Wait(); even needed in the above code?
In the general case, you should assume that yes, calling .Result or .Wait() on anything awaitable is dangerous and can deadlock (unless you are the library issuing the task, and you understand the full context). It is possible that it will work OK in some specific cases, but you should not rely on that behaviour, even if it works today.
In my app I need to do lot of parallel http requests and I have read that it is proper to do it using async/await. In each request I need to get string content from it (often it is html of some site) and my question is: how can I do it in best way?
My current implementation:
public static async Task<string> GetStringContentAsync(HttpWebRequest webRequest)
{
try
{
using (var response = (HttpWebResponse) await webRequest.GetResponseAsync()
.ConfigureAwait(false))
{
var content = await GetStringContentFromResponseAsync(response)
.ConfigureAwait(false);
return content;
}
}
catch (Exception exception)
{
return null;
}
}
private static async Task<string> GetStringContentFromResponseAsync(HttpWebResponse response)
{
using (var responseStream = GetResponseStream(response))
{
if (responseStream == null)
return null;
using (var streamReader = new StreamReader(responseStream))
{
var content = await streamReader.ReadToEndAsync()
.ConfigureAwait(false);
return content;
}
}
}
private static Stream GetResponseStream(HttpWebResponse webResponse)
{
var responseStream = webResponse.GetResponseStream();
if (responseStream == null)
return null;
Stream stream;
switch (webResponse.ContentEncoding.ToUpperInvariant())
{
case "GZIP":
stream = new GZipStream(responseStream, CompressionMode.Decompress);
break;
case "DEFLATE":
stream = new DeflateStream(responseStream, CompressionMode.Decompress);
break;
default:
stream = responseStream;
break;
}
return stream;
}
And example of using:
var httpWebRequest = (HttpWebRequest) WebRequest.Create("http://stackoverflow.com/");
var content = await HttpHelper.GetStringContentAsync(httpWebRequest)
.ConfigureAwait(false);
Is this correct implementation, or we can improve something here? Maybe I'm doing some overhead when using async/await when reading stream?
Reason of my question is that when I'm using my code like this:
for(var i=0;i<1000;i++)
{
Task.Run(()=>{
var httpWebRequest = (HttpWebRequest) WebRequest.Create("http://google.com/");
var content = await HttpHelper.GetStringContentAsync(httpWebRequest)
.ConfigureAwait(false);
});
}
this tasks take to long to execute, but one request to google is very fast. I thought that async requests in this example must be ready almost in same time and this time must be pretty close to "one google request" time.
EDIT:
I forgot to say that I know about ServicePointManager.DefaultConnectionLimit and set it 5000 in my app. So it is not a problem.
I can't use HttpClient because my final goal is to do 100-300 requests at one time from different proxies. And if I understand right, HttpClient can work with only one proxy at one time and can't setup each request separately.
That's a tricky one. Since you know about DefaultConnectionLimit, it's already something good, but there is one more interesting and rather surprising thing:
httpRequest.ServicePoint.ConnectionLeaseTimeout
httpRequest.ServicePoint.MaxIdleTime
Information is here, your latencies might be caused by its default behavior and connections being held to ServicePoint while trying to make next request
Here's the answer answer to your issue: https://msdn.microsoft.com/en-us/library/86wf6409(v=vs.90).aspx
Using synchronous calls in asynchronous callback methods can result in severe performance penalties. Internet requests made with WebRequest and its descendants must use Stream.BeginRead to read the stream returned by the WebResponse.GetResponseStream method.
That means absolutely no synchronous code (including awaits) when reading the response stream. But even that isn't enough, as DNS lookups and TCP connection are still blocking. If you can use .NET 4.0, there's a much more easy to use System.Net.Http.HttpClient class. Otherwise, you can use System.Threading.ThreadPool, which is the workaround I ended up using on 3.5:
ThreadPool.QueueUserWorkItem((o) => {
// make a synchronous request via HttpWebRequest
});
I'm working with two web API projects that get communicated. The first web API calls the second one using an HttpClient class.
What I would like to do is to set a short timeout (500 ms) when I call the second web API, and if I don't get response in that time, just to skip the next lines that process the result in the client, but continue processing the request at server side(second API).
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Clear();
client.Timeout = this.Timeout; // (500ms)
HttpResponseMessage response = client.PostAsJsonAsync(EndPoint, PostData).Result;
if (response.IsSuccessStatusCode)
{
return response.Content.ReadAsAsync<T>().Result;
}
else
{
throw new CustomException()
}
}
It works in the first API side, however in the second API(server), I get the following exceptions:
"A task was canceled."
"The operation was cancelled."
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Threading.CancellationToken.ThrowIfCancellationRequested()
I think it is caused by the small timeout of the call, that ends when the second API is still processing the result.
How could I avoid this behaviour in the second API and continue processing the request ?
Thanks in advance.
That is the expected behavior. When you set a timeout and the call does not respond in that amount of time, the task is canceled and that exception is thrown.
And by the way, do not use .Result. That will cause blocking. Mark your method async and use await.
The whole thing should look something like this:
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Clear();
client.Timeout = this.Timeout; // (500ms)
try
{
HttpResponseMessage response = await client.PostAsJsonAsync(EndPoint, PostData);
if (response.IsSuccessStatusCode)
{
return await response.Content.ReadAsAsync<T>();
}
else
{
throw new CustomException()
}
}
catch (TaskCanceledException)
{
// request did not complete in 500ms.
return null; // or something else to indicate no data, move on
}
}