I discovered that a single HttpClient could be shared by multiple requests. If shared, and the requests are to the same destination, multiple requests could reuse the connections. WebRequest needs to recreate the connection for each request.
I also looked up some documentation on other ways to use HttpClient in examples.
The following article summarizes the high-speed NTLM-authenticated connection sharing: HttpWebRequest.UnsafeAuthenticatedConnectionSharing
Possible implementations that I tried out are shown below
A)
private WebRequestHandler GetWebRequestHandler()
{
CredentialCache credentialCache = new CredentialCache();
credentialCache.Add(ResourceUriCanBeAnyUri, "NTLM", CredentialCache.DefaultNetworkCredentials);
WebRequestHandler handler = new WebRequestHandler
{
UnsafeAuthenticatedConnectionSharing = true,
Credentials = credentialCache
};
return handler;
}
using (HttpClient client = new HttpClient(GetWebRequestHandler(), false))
{
}
B)
using (HttpClient client = new HttpClient)
{
}
C)
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("some uri string")
I would appreciate any help in making me understand which approach I should take so as to achieve max performance, minimizing connections and making sure security is not impacted.
If you use either of them with async it should be good for the performance point of view as it will not block the resources waiting for the response and you will get good throughput.
HttpClient is preferred over HttpWebRequest due to async methods available out of the box and you would not have to worry about writing begin/end methods.
Basically when you use async call (using either of the class), it will not block the resources waiting for the response and any other request would utilise the resources to make further calls.
Another thing to keep in mind that you should not be using HttpClient in the 'using' block to allow reuse of same resources again and again for other web requests.
See following thread for more information
Do HttpClient and HttpClientHandler have to be disposed?
This is my ApiClient which creates the HttpClient for only once. Register this object as singleton to your dependency injection library. It's safe to reuse because it's stateless. Do NOT recreate HTTPClient for each request. Reuse Httpclient as much as possible
using System;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;
//You need to install package Newtonsoft.Json > https://www.nuget.org/packages/Newtonsoft.Json/
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
public class MyApiClient : IDisposable
{
private readonly TimeSpan _timeout;
private HttpClient _httpClient;
private HttpClientHandler _httpClientHandler;
private readonly string _baseUrl;
private const string ClientUserAgent = "my-api-client-v1";
private const string MediaTypeJson = "application/json";
public MyApiClient(string baseUrl, TimeSpan? timeout = null)
{
_baseUrl = NormalizeBaseUrl(baseUrl);
_timeout = timeout ?? TimeSpan.FromSeconds(90);
}
public async Task<string> PostAsync(string url, object input)
{
EnsureHttpClientCreated();
using (var requestContent = new StringContent(ConvertToJsonString(input), Encoding.UTF8, MediaTypeJson))
{
using (var response = await _httpClient.PostAsync(url, requestContent))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
}
public async Task<TResult> PostAsync<TResult>(string url, object input) where TResult : class, new()
{
var strResponse = await PostAsync(url, input);
return JsonConvert.DeserializeObject<TResult>(strResponse, new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
}
public async Task<TResult> GetAsync<TResult>(string url) where TResult : class, new()
{
var strResponse = await GetAsync(url);
return JsonConvert.DeserializeObject<TResult>(strResponse, new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
}
public async Task<string> GetAsync(string url)
{
EnsureHttpClientCreated();
using (var response = await _httpClient.GetAsync(url))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public async Task<string> PutAsync(string url, object input)
{
return await PutAsync(url, new StringContent(JsonConvert.SerializeObject(input), Encoding.UTF8, MediaTypeJson));
}
public async Task<string> PutAsync(string url, HttpContent content)
{
EnsureHttpClientCreated();
using (var response = await _httpClient.PutAsync(url, content))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public async Task<string> DeleteAsync(string url)
{
EnsureHttpClientCreated();
using (var response = await _httpClient.DeleteAsync(url))
{
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
public void Dispose()
{
_httpClientHandler?.Dispose();
_httpClient?.Dispose();
}
private void CreateHttpClient()
{
_httpClientHandler = new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip
};
_httpClient = new HttpClient(_httpClientHandler, false)
{
Timeout = _timeout
};
_httpClient.DefaultRequestHeaders.UserAgent.ParseAdd(ClientUserAgent);
if (!string.IsNullOrWhiteSpace(_baseUrl))
{
_httpClient.BaseAddress = new Uri(_baseUrl);
}
_httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(MediaTypeJson));
}
private void EnsureHttpClientCreated()
{
if (_httpClient == null)
{
CreateHttpClient();
}
}
private static string ConvertToJsonString(object obj)
{
if (obj == null)
{
return string.Empty;
}
return JsonConvert.SerializeObject(obj, new JsonSerializerSettings
{
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
}
private static string NormalizeBaseUrl(string url)
{
return url.EndsWith("/") ? url : url + "/";
}
}
The usage;
using ( var client = new MyApiClient("http://localhost:8080"))
{
var response = client.GetAsync("api/users/findByUsername?username=alper").Result;
var userResponse = client.GetAsync<MyUser>("api/users/findByUsername?username=alper").Result;
}
There is a problem in your implementation 'A'. The lifetime of the instance returned from GetWebRequestHandler() is shortlived (maybe just for the sake of the example?). If this was done on purpose, it negates the passing of false for the 2nd param of HttpClient constructor. The value of false tells the HttpClient to not dispose the underlying HttpMessageHandler (which helps with scaling since it does not close the port for the request). This is, of course, assuming that the lifetime of the HttpMessageHandler is long enough for you to take advantage of the benefit of not opening/closing ports (which is a big impact on the scalability of your server). Thus, I have a recommendation below of option 'D'.
There is also an option 'D' that you do not list above - to make the Httpclient instance static and re-used across all api calls. This is much more efficient from a memory allocation and GC perspective - as well as the opening of ports on the client. You don't have the overhead of allocating memory for and creating HttpClient instances (and all of its underlying objects) and thus avoid the cleanup via GC for them too.
Please refer to my answer provided about a similar question - What is the overhead of creating a new HttpClient per call in a WebAPI client?
Related
what exactly I want :
public static CustomDnsResolver : Dns
{
.....
}
public static void Main(string[] args)
{
var httpClient = new HttpClient();
httpClient.Dns(new CustomDnsResolver());
}
basically I just want to use my custom DNS resolver in HttpClient instead of System default, Is there any way to achieve it?
The use case you have is exactly why Microsoft build the HttpClient stack. It allow you to put your business logic in layered class with the help of HttpMessageHandler class. You can find some sample in ms docs or visualstudiomagazine
void Main()
{
var dnsHandler = new DnsHandler(new CustomDnsResolver());
var client = new HttpClient(dnsHandler);
var html = client.GetStringAsync("http://google.com").Result;
}
public class DnsHandler : HttpClientHandler
{
private readonly CustomDnsResolver _dnsResolver;
public DnsHandler(CustomDnsResolver dnsResolver)
{
_dnsResolver = dnsResolver;
}
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
var host = request.RequestUri.Host;
var ip = _dnsResolver.Resolve(host);
var builder = new UriBuilder(request.RequestUri);
builder.Host = ip;
request.RequestUri = builder.Uri;
return base.SendAsync(request, cancellationToken);
}
}
public class CustomDnsResolver
{
public string Resolve(string host)
{
return "127.0.0.1";
}
}
Not sure if this works for all cases, but it worked brilliantly for my case, even when using HTTPS. So what you do is you replace the host in the URL with the actual IP-address that your custom resolver has resolved, and then you simply add a "host" header with the host name. Like this:
var requestUri = new Uri("https://123.123.123.123/some/path");
using var request = new HttpRequestMessage(HttpMethod.Get, requestUri);
request.Headers.TryAddWithoutValidation("host", "www.host-name.com");
using var response = await httpClient.SendAsync(request);
I hope this helps as this is far by the first time I've run into this issue and I've never been able to solve it before now.
Consider using SocketsHttpHandler.ConnectCallback.
I,m testing ASP.NET Core app with TestServer, and there are controllers that require cookie auth. I've created test server instance like this:
_testServer = new TestServer(new WebHostBuilder()
.UseEnvironment(CustomEnvironments.Test)
.UseContentRoot(currentDirectory)
.UseStartup<Web.Startup>()
.UseUrls("http://localhost/"));
ApiClient = _testServer.CreateClient();
and now I have to add auth cookie, but it is ignored by server. If the client could be created directly I could pass HttpClientHandler to constractor and set UseCookies to false, and it works, but I can't access the handler when I get client from test server. Is there a way to add auth cookies to test client?
I've found the solution. TestServer has method CreateRequest(string path), it returns RequestBuilder, which allows to insert cookies to header
Using #AlexK's answer for inspiration, combined with information from a blog post (as an aside, this post goes into a lot of other useful details when dealing with other issues when sending requests to the test server), here is one way of getting cookies to work with the TestServer using CreateRequest(string path) based on what I used for my own project.
public class TestWebEnvironment : IDisposable
{
private TestServer Server { get; }
private CookieContainer CookieContainer { get; }
public TestWebEnvironment()
{
var builder = new WebHostBuilder()
.UseEnvironment("Test")
.UseStartup<TestWebStartup>();
Server = new TestServer(builder);
CookieContainer = new CookieContainer();
}
private RequestBuilder BuildRequest(string url)
{
var uri = new Uri(Server.BaseAddress, url);
var builder = Server.CreateRequest(url);
var cookieHeader = CookieContainer.GetCookieHeader(uri);
if (!string.IsNullOrWhiteSpace(cookieHeader))
{
builder.AddHeader(HeaderNames.Cookie, cookieHeader);
}
return builder;
}
private void UpdateCookies(string url, HttpResponseMessage response)
{
if (response.Headers.Contains(HeaderNames.SetCookie))
{
var uri = new Uri(Server.BaseAddress, url);
var cookies = response.Headers.GetValues(HeaderNames.SetCookie);
foreach (var cookie in cookies)
{
CookieContainer.SetCookies(uri, cookie);
}
}
}
public async Task<string> GetAsync(string url)
{
using (var response = await BuildRequest(url).GetAsync())
{
UpdateCookies(url, response);
return await response.Content.ReadAsStringAsync();
}
}
public async Task<string> PostAsync(string url, HttpContent content)
{
var builder = BuildRequest(url);
builder.And(request => request.Content = content);
using (var response = await builder.PostAsync())
{
UpdateCookies(url, response);
return await response.Content.ReadAsStringAsync();
}
}
public void Dispose()
{
Server.Dispose();
}
}
I have the following code
static void Main(string[] args)
{
string url = "http://www.google.com";
Console.WriteLine(GetUrl(url).Result); // throws TaskCanceledException
Console.WriteLine(GetUrl2(url).Result);
}
public static Task<string> GetUrl(string url)
{
using (var client = new HttpClient())
{
return client.GetStringAsync(url);
}
}
public static Task<string> GetUrl2(string url)
{
using (var client = new WebClient())
{
return client.DownloadStringTaskAsync(url);
}
}
I'm trying to get the string of an url, the problem is GetUrl method (uses HttpClient's GetStringAsync) throws an TaskCacelledException, but GetUrl2 method (uses WebClient's DownloadStringTaskAsync) runs correctly. Is this caused due to using statement? What am I missing?
Edit. In this example I'm calling Result on the task because this is a console application, I know that it is best to await the result in a event handler for example.
Is this caused due to using statement?
Yes. In both code examples, you're disposing the underlying client before the operation completes. Both code examples should be changed as such:
public static async Task<string> GetUrlAsync(string url)
{
using (var client = new HttpClient())
{
return await client.GetStringAsync(url);
}
}
public static async Task<string> GetUrl2Async(string url)
{
using (var client = new WebClient())
{
return await client.DownloadStringTaskAsync(url);
}
}
The behavior of asynchronous downloads when their underlying clients are disposed is undocumented. It's best not to dispose the clients until your code is done using them.
I have a PCl in which I want to make a async call usingg HttpClient. I coded like this
public static async Task<string> GetRequest(string url)
{
var httpClient = new HttpClient() { MaxResponseContentBufferSize = int.MaxValue };
HttpResponseMessage response = await httpClient.GetAsync(url);
return response.Content.ReadAsStringAsync().Result;
}
But await is showing error "cannot await System.net.http.httpresponsemessage" like message.
If I use code like this than everything goes well but not in async way
public static string GetRequest(string url)
{
var httpClient = new HttpClient() { MaxResponseContentBufferSize = int.MaxValue };
HttpResponseMessage response = httpClient.GetAsync(url).Result;
return response.Content.ReadAsStringAsync().Result;
}
I just want that this method executes in async way.
This is the screen shot:
Follow the TAP guidelines, don't forget to call EnsureSuccessStatusCode, dispose your resources, and replace all Results with awaits:
public static async Task<string> GetRequestAsync(string url)
{
using (var httpClient = new HttpClient() { MaxResponseContentBufferSize = int.MaxValue })
{
HttpResponseMessage response = await httpClient.GetAsync(url);
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync();
}
}
If your code doesn't need to do anything else, HttpClient has a GetStringAsync method that does this for you:
public static async Task<string> GetRequestAsync(string url)
{
using (var httpClient = new HttpClient() { MaxResponseContentBufferSize = int.MaxValue })
return await httpClient.GetStringAsync(url);
}
If you share your HttpClient instances, this can simplify to:
private static readonly HttpClient httpClient =
new HttpClient() { MaxResponseContentBufferSize = int.MaxValue };
public static Task<string> GetRequestAsync(string url)
{
return httpClient.GetStringAsync(url);
}
If you are using a PCL platform that supports .net4 then I suspect you need to install the Microsoft.bcl.Async nuget.
I'm trying to start async task (on .NET 4.5) which downloads content of web page, but somehow this task never finishes.
My PageDownloader class:
using System.Net;
using System.Text;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
using System;
namespace ParserConsole.WebClient
{
public class PageDownloader
{
private System.Net.Http.HttpClient _client;
public PageDownloader()
: this(Encoding.UTF8) { }
private Encoding _encoding;
public PageDownloader(Encoding encoding)
{
_encoding = encoding;
_client = new HttpClient() { Timeout = TimeSpan.FromSeconds(10)};
}
private HttpRequestMessage _request;
private HttpResponseMessage _response;
private string _responseString;
public string GetPageData(string link)
{
_request = new HttpRequestMessage(HttpMethod.Get, link);
_request.Headers.Add("User-Agent", "Chrome/21.0.1180.89");
_request.Headers.Add("Accept", "text/html");
GetResponse().Wait();
GetStringFromResponse().Wait();
return _responseString;
}
private async Task<HttpResponseMessage> GetResponse() {
return _response = await _client.GetAsync(_request.RequestUri);
}
private async Task<string> GetStringFromResponse() {
return _responseString = await _response.Content.ReadAsStringAsync();
}
}
}
I start downloading page by calling
new PageDownloader().GetPageData(url);
When I'm trying to debug the code, everything is fine till GetResponse().Wait(). But somehow GetResponse() task never finishes - breakpoint on the next line is never reached. I get no exceptions, application continues running. Any suggestions?
This is a standard deadlock condition you get when you start an async operation and then block on the returned task.
Here is a blog post discussion the topic.
Basically, the await call ensures that the continuation it wires up of the task will run in the context you were originally in (which is very helpful) but because you are calling Wait in that same context it's blocking, so the continuation never runs, and that continuation needs to run for the wait to end. Classic deadlock.
As for the fix; usually it means you just shouldn't be doing a blocking wait on the async operation; it's contrary to the design of the whole system. You should, "async all the way up". In this case it would mean that GetPageData should return a Task<string> rather than a string, and rather than waiting on the other operations that return a task you should await on them.
Now, having said that, there are ways of doing a blocking wait on the async operations without deadlocking. While it can be done, it honestly defeats the purpose of using async/await in the first place. The primary advantage of using that system is that the main context isn't blocked; when you block on it that entire advantage goes away, and you might as well just use blocking code all the way through. async/await is really more of an all-or-nothing paradigm.
Here is how I would structure that class:
public class PageDownloader
{
private System.Net.Http.HttpClient _client;
private Encoding _encoding;
public PageDownloader()
: this(Encoding.UTF8) { }
public PageDownloader(Encoding encoding)
{
_encoding = encoding;
_client = new HttpClient() { Timeout = TimeSpan.FromSeconds(10) };
}
public async Task<string> GetPageData(string link)
{
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, link);
request.Headers.Add("User-Agent", "Chrome/21.0.1180.89");
request.Headers.Add("Accept", "text/html");
HttpResponseMessage response = await _client.GetAsync(request.RequestUri);
return await response.Content.ReadAsStringAsync(); ;
}
}
Why not just do this if you want to have a function like that.
public string GetPageData(string link)
{
_request = new HttpRequestMessage(HttpMethod.Get, link);
_request.Headers.Add("User-Agent", "Chrome/21.0.1180.89");
_request.Headers.Add("Accept", "text/html");
var readTask = _client.GetStringAsync(link);
readTask.Wait();
return readTask.Result;
}
It would be better to return the Task all the way back and handle it with async/await in the calling code.
public Task<string> GetPageData(string link)
{
_request = new HttpRequestMessage(HttpMethod.Get, link);
_request.Headers.Add("User-Agent", "Chrome/21.0.1180.89");
_request.Headers.Add("Accept", "text/html");
return _client.GetStringAsync(link);
}