Develop Reference

System.Net.Http.HttpClient not timing out after set time

I'm using HttpClient to handle web requests. I'll perform the request like so...
var response = await httpClient.PostAsync(Uri, stringContent);
The timeout is set in the http client setup and I then call it in the class with IHttpClientFactory dependency.
// Setup HttpClient
services.AddHttpClient("ApiName", client =>
{
client.BaseAddress = new Uri("BaseUri");
client.Timeout = new TimeSpan(0, 0, 8);
});
I've inspected httpClient before it runs to see if its using the correct API and it is, the timeout is set to 8 seconds. However, the request takes around a good 1-2 minutes to timeout. I'm not sure why this is happening.
I've tried using a CancellationToken like so...
var cts = new CancellationTokenSource();
cts.CancelAfter(8000);
var response = await httpClient.PostAsync(Uri, stringContent, cts.Token);
and waiting for the exception to occur but it still takes 1-2 minutes.

Based on the HttpClient.Timeout documentation, a DNS query may take up to 15 seconds to return or time out. However that does not explain the 1-2 minute delay you are seeing.
Have you looked at "Better timeout handling with HttpClient"? I don't know if it will work, but rather than using PostAsync you might try building the POST with HttpRequestMessage and use SendAsync instead, similar to the example in the article:
var request = new HttpRequestMessage(HttpMethod.Post, "http://foo/");
request.Content = stringContent;
request.SetTimeout(TimeSpan.FromSeconds(8)); // Uses HttpRequestExtensions from article
var cts = new CancellationTokenSource(TimeSpan.FromSeconds(8));
var response = await client.SendAsync(request, cts.Token);

UWP http client delay in getting response

I found this peculiar behaviour of UWP HttpClient.
For a simple get call of WCF service is taking time almost more than 100 seconds(happens for first few calls). I observed that the same service is way too faster in iOS and rest client.
Attaching the code, Please let me know, if i doing wrong.
HttpClientHandler clientHandler = new HttpClientHandler();
clientHandler.UseCookies = true;
var client = new HttpClient(clientHandler);
client.Timeout = TimeSpan.FromSeconds(100);
client.DefaultRequestHeaders.Add("Accept", "application/json");
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get,
requestUri);
var response = await client.SendAsync(httpRequestMessage);
//Between these lines there is a delay, it doesnt fire time out
also. I tried with HttpCompletionOption. No luck
if (response.IsSuccessStatusCode)
{
var content = await response.Content.ReadAsStringAsync();
}
Any help will be really appreciated.
Thanks and Regards,
Rummy

Rapid web requests to many different websites using HttpClient C#

My team maintains a tool that is responsible for doing rapid verification of over 1000 different client websites. The tool is a Windows Service (.NET 4.5.2, C#) that reads requests from a queue, and executes a 'health check' for each request. It usually handles over 500 requests a minute, but can be responsible for more. Each request takes a second or two to execute.
A request contains a Uri and credentials needed for doing the health check. A health check is a POST against the AUTH page with the credentials (the app has custom auth, it's not header based auth), and then a GET to the home page, with a quick verification that it's the home page we expect. It then goes to a status page in the application, and does some quick checks against that. The GET requests have to use the cookies from the Set-Cookie header in the auth post.
We've been having performance problems with the tool as it scales. It currently creates a new HttpWebRequest object for each post and get in the process. There is a shared CookieContainer that is populated by the first post, so that we can get to the home page and then the status page.
I want to change this service to use the HttpClient object available in .NET 4.5. The problem is everywhere I read online says you want to avoid rapid creation and destruction of HttpClients. You'd rather keep one instance alive for the lifetime of the application. The problem I have is that HttpClient seems to work really well against one endpoint, not many.
I have looked into several options, and am not sure which is best to proceed:
Create a new HttpClient for each request, and use it for the duration of that request. That means it will live for a couple seconds, and be used for 3 calls. This would not be easy to implement, but I'm concerned about the overhead of creating and destroying hundreds of HttpClients a minute.
Figure out if it's possible to use one HttpClient instance for different endpoints by avoiding usage of a BaseAddress, and using the client to pass HttpRequestMessages using SendAsync. I haven't been able to figure out cookies with this method yet. To avoid having the HttpClient store the cookies, I set UseCookies to false in the HttpClientHandler, and tried managing cookies via headers in the HttpRequest/ResponseMessages themselves, but it looks like HttpClient simply strips cookies when UseCookies is set to false, so I was unable to pass cookies between request. edit: cookies work fine because they are stored per domain.
Store several hundred different HttpClient instances in some sort of dictionary, and pull the appropriate one for each Uri as the requests come in. I'm not sure about the memory overhead on this though. Also each unique Uri is only verified once every 5 minutes, so I'm not sure if having an HttpClient used once every 5 minutes keeps an unnecessary number of ports open.
Keep using HttpWebRequests. Maybe this older method still performs better in this situation.
If anyone has faced a similar issue, I'd love some input on where to proceed on this.
Thanks!

The problem with creating new HttpClients for each request is that HttpClientHandler will close the underlying TCP/IP connection. However, if you are using each HttpClient for the 3 requests to one host and then hitting a different host, then keeping the connection open doesn't help when you move to a new host. So, you probably will not see perf problem with one client per host. HttpClient itself is a very lightweight object. It isn't going to cost much to create one.
However, HttpClient simply delegates the real work to HttpClientHandler which uses HttpWebRequest under the covers, therefore will be unlikely to have any better performance than directly using HttpWebRequest.
If you are looking for better performance, then I suggest looking into replacing HttpClientHandler with the new WinHttpHandler which bypasses HttpWebRequest and goes directly to the Win32 API to make calls.
The full source is available for WinHttpHandler on GitHub so you can see exactly how it handles cookies and credentials.
And I would be really interested to hear if you do get much better perf with WinHttpHandler.

To start with, what part of this would you need modified to suit your needs?
var urisToCheck = new List<Uri>(); //get these somehow
//basic auth work?
var credentials = new NetworkCredential("user", "pass");
var handler = new HttpClientHandler { Credentials = credentials };
var client = new HttpClient(handler);
Parallel.ForEach(urisToCheck,
async uri =>
{
var response = await client.GetAsync(uri.AbsoluteUri);
//check for whatever you want here
}
);

here's my basic API client that uses the same HttpClient object for every request.
using System;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;
//You need to install package Newtonsoft.Json > https://www.nuget.org/packages/Newtonsoft.Json/
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
public class MyApiClient : IDisposable
{
private readonly TimeSpan _timeout;
private HttpClient _httpClient;
private HttpClientHandler _httpClientHandler;
private readonly string _baseUrl;
private const string ClientUserAgent = "my-api-client-v1";
private const string MediaTypeJson = "application/json";
public MyApiClient(string baseUrl, TimeSpan? timeout = null)
{
_baseUrl = NormalizeBaseUrl(baseUrl);
_timeout = timeout ?? TimeSpan.FromSeconds(90);
}
public async Task<string> PostAsync(string url, object input)
{
EnsureHttpClientCreated();
using (var requestContent = new StringContent(ConvertToJsonString(input), Encoding.UTF8, MediaTypeJson))
{
using (var response = await _httpClient.PostAsync(url, requestContent))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
}
public async Task<TResult> PostAsync<TResult>(string url, object input) where TResult : class, new()
{
var strResponse = await PostAsync(url, input);
return JsonConvert.DeserializeObject<TResult>(strResponse, new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
}
public async Task<TResult> GetAsync<TResult>(string url) where TResult : class, new()
{
var strResponse = await GetAsync(url);
return JsonConvert.DeserializeObject<TResult>(strResponse, new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
}
public async Task<string> GetAsync(string url)
{
EnsureHttpClientCreated();
using (var response = await _httpClient.GetAsync(url))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public async Task<string> PutAsync(string url, object input)
{
return await PutAsync(url, new StringContent(JsonConvert.SerializeObject(input), Encoding.UTF8, MediaTypeJson));
}
public async Task<string> PutAsync(string url, HttpContent content)
{
EnsureHttpClientCreated();
using (var response = await _httpClient.PutAsync(url, content))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public async Task<string> DeleteAsync(string url)
{
EnsureHttpClientCreated();
using (var response = await _httpClient.DeleteAsync(url))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public void Dispose()
{
_httpClientHandler?.Dispose();
_httpClient?.Dispose();
}
private void CreateHttpClient()
{
_httpClientHandler = new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip
};
_httpClient = new HttpClient(_httpClientHandler, false)
{
Timeout = _timeout
};
_httpClient.DefaultRequestHeaders.UserAgent.ParseAdd(ClientUserAgent);
if (!string.IsNullOrWhiteSpace(_baseUrl))
{
_httpClient.BaseAddress = new Uri(_baseUrl);
}
_httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(MediaTypeJson));
}
private void EnsureHttpClientCreated()
{
if (_httpClient == null)
{
CreateHttpClient();
}
}
private static string ConvertToJsonString(object obj)
{
if (obj == null)
{
return string.Empty;
}
return JsonConvert.SerializeObject(obj, new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
}
private static string NormalizeBaseUrl(string url)
{
return url.EndsWith("/") ? url : url + "/";
}
}
Usage
using ( var client = new MyApiClient("http://localhost:8080"))
{
var response = client.GetAsync("api/users/findByUsername?username=alper").Result;
var userResponse = client.GetAsync<MyUser>("api/users/findByUsername?username=alper").Result;
}
Register this object as singleton to your dependency injection library. It's safe to reuse because it's stateless.
Do NOT recreate HTTPClient for each request.
Reuse Httpclient as much as possible

Adding headers when using httpClient.GetAsync

I'm implementing an API made by other colleagues with Apiary.io, in a Windows Store app project.
They show this example of a method I have to implement:
var baseAddress = new Uri("https://private-a8014-xxxxxx.apiary-mock.com/");
using (var httpClient = new HttpClient{ BaseAddress = baseAddress })
{
using (var response = await httpClient.GetAsync("user/list{?organizationId}"))
{
string responseData = await response.Content.ReadAsStringAsync();
}
}
In this and some other methods, I need to have a header with a token that I get before.
Here's an image of Postman (chrome extension) with the header I'm talking about:
How do I add that Authorization header to the request?

A later answer, but because no one gave this solution...
If you do not want to set the header on the HttpClient instance by adding it to the DefaultRequestHeaders, you could set headers per request.
But you will be obliged to use the SendAsync() method.
This is the right solution if you want to reuse the HttpClient -- which is a good practice for
performance and port exhaustion problems
doing something thread-safe
not sending the same headers every time
Use it like this:
using (var requestMessage =
new HttpRequestMessage(HttpMethod.Get, "https://your.site.com"))
{
requestMessage.Headers.Authorization =
new AuthenticationHeaderValue("Bearer", your_token);
await httpClient.SendAsync(requestMessage);
}

When using GetAsync with the HttpClient you can add the authorization headers like so:
httpClient.DefaultRequestHeaders.Authorization
= new AuthenticationHeaderValue("Bearer", "Your Oauth token");
This does add the authorization header for the lifetime of the HttpClient so is useful if you are hitting one site where the authorization header doesn't change.
Here is an detailed SO answer

The accepted answer works but can get complicated when you want to try adding Accept headers. This is what I ended up with. It seems simpler to me, so I think I'll stick with it in the future:
client.DefaultRequestHeaders.Add("Accept", "application/*+xml;version=5.1");
client.DefaultRequestHeaders.Add("Authorization", "Basic " + authstring);

Sometimes, you only need this code.
httpClient.DefaultRequestHeaders.Add("token", token);

Following the greenhoorn's answer, you can use "Extensions" like this:
public static class HttpClientExtensions
{
public static HttpClient AddTokenToHeader(this HttpClient cl, string token)
{
//int timeoutSec = 90;
//cl.Timeout = new TimeSpan(0, 0, timeoutSec);
string contentType = "application/json";
cl.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(contentType));
cl.DefaultRequestHeaders.Add("Authorization", String.Format("Bearer {0}", token));
var userAgent = "d-fens HttpClient";
cl.DefaultRequestHeaders.Add("User-Agent", userAgent);
return cl;
}
}
And use:
string _tokenUpdated = "TOKEN";
HttpClient _client;
_client.AddTokenToHeader(_tokenUpdated).GetAsync("/api/values")

These days, if you are using MS Dependency Injection, it's highly recomended to plug in the IHttpClientFactory:
builder.Services.AddHttpClient("GitHub", httpClient =>
{
httpClient.BaseAddress = new Uri("https://api.github.com/");
// using Microsoft.Net.Http.Headers;
// The GitHub API requires two headers.
httpClient.DefaultRequestHeaders.Add(
HeaderNames.Accept, "application/vnd.github.v3+json");
httpClient.DefaultRequestHeaders.Add(
HeaderNames.UserAgent, "HttpRequestsSample");
});
var httpClient = _httpClientFactory.CreateClient("GitHub");
This way you avoid adding default request headers to a globally shared httpclient and moreover don't have to deal with manual creation of the HttpRequestMessage.
Source:
https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-6.0#consumption-patterns

Parallel async HttpWebRequests with different proxies. How to maximize performance?

I'm writing an app that needs to do a lot of parallel httpwebrequests to bookmaker site from different proxies. The reason why i'm using different proxies is that bookmaker can ban ip if there are many requests from it. My goal is to get freshest site content as fast as possible.
Here is my code with all settings that I have:
ServicePointManager.DefaultConnectionLimit = 1000;
ServicePointManager.Expect100Continue = false;
ServicePointManager.UseNagleAlgorithm = false;
for (var i = 0; i < proxyCollection.Count; i++)
{
var proxyLocal = proxyCollection[i];
var iLocal = i;
Task.Run(async () =>
{
var httpWebRequest = (HttpWebRequest) WebRequest.Create(String.Format("https://bookmaker.com/page{0}", iLocal));
httpWebRequest.Proxy = proxyLocal;
httpWebRequest.PreAuthenticate = true;
httpWebRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
using (var httpWebResponse = (HttpWebResponse) await httpWebRequest.GetResponseAsync())
using (var responseStream = httpWebResponse.GetResponseStream())
using (var streamReader = new StreamReader(responseStream))
{
var stringContent = await streamReader.ReadToEndAsync();
//Here i'm processing new data. It works pretty fast, so this isn't a problem
ProcessStringContent(stringContent);
}
});
}
And this code works not so fast than I expected.
First problem is that in strange reason all requests don't start at one time. As I can see in task manager loading has two or more maximums. More over I have one realy fast proxy and proxyCollection contains it. But if I use await Task.Delay(5000) after code above, in some cases to the moment when 5000ms have passed the request with my fast proxy doesn't even start!
Second problem is that total time of all task execution is slow. I expected that if one request needs 200-300ms to execute, than 100 requests in parallel and async needs a little more time. But some times this "more" is 10-20 times. I have a suspicion, that something is wrong.
Third problem is that when I'm running this code it freezes UI (not full freeze, but UI lags). I read that WebRequest.Create is processing synchronously and can get some time (dns lookup, proxy settings e.t.c) and if I'm making a lot of requests they can simply fill all my threads (UI thread too) for creating WebRequests. But I tried to create requests to direct ip address (WebRequest.Create(String.Format("https://10.10.10.1/page{0}", iLocal)) - nothing changed, and i'm setting proxy (so auto detect proxy isn't needed), so I don't understand why can creating take so much time (and is problem with creating or maybe with smth else?).
Please can someone point me what i'm doing wrong? I'm lost in all ServicePointManager settings (all what I tried didn't help). Can .Net deal with such type of task? Or maybe I need to use nodejs for example for best performance?
P.S: Collection of proxies is not such big (50-100 different proxies).

Back to (Parallel) Basics: Don't Block Your Threads, Make Async I/O Work For You
Example:
Parallel.For(0, 10, delegate(int i) {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(
new Uri("http://www.mysi.com"));
string dataToSend = "Data";
byte[] buffer = System.Text.Encoding.GetEncoding(1252).
GetBytes(dataToSend);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = buffer.Length;
request.Host = "www.mysite.com";
Stream requestStream = request.GetRequestStream();
requestStream.Write(buffer, 0, buffer.Length);
requestStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
});
Send multiple WebRequest in Parallel.For