I have a class that has a private HttpClient. I have a DoRequest method that handles all my requests and takes a template object for the content/parameters. So far I have 2 calls. One is to retrieve a token and one is to get some other info (GetInfo). my GetInfo method will need the authorization header set so I do this before calling DoRequest:
this.Client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
As soon as I started doing a Parallel.For for a load test, I start getting a An item with the same key has already been added exception, sometimes. Now I understand that the headers are stored in a dictionary and that's why I am getting the exception but I am looking for ideas on how to solve this while retaining my DoRequest design. I guess one way to solve it is to set the header in my HttpContent object but that lives in my DoRequest method and I would need to pass a Token parameter to it now. I'm not saying that's the worst thing in the world but am looking for alternatives, if any exist.
The way I thought this problem through is by analyzing what I wanted. I wanted to re-use HttpClient but dispose of any Authorization headers. The best way I can think to do that is to use an HttpRequestMessage and pass the Token to my DoRequest method. In my method, I do:
HttpRequestMessage reqmsg = new HttpRequestMessage(method, uri);
reqmsg.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token ?? string.Empty);
reqmsg.Content = formContent;
You could check and see if it's already on there:
static readonly object _o = new object();
...
lock (_o){
if (!this.Client.DefaultRequestHeaders.Contains("Authorization"))
this.Client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
}
Related
Am querying multiple APIs with thousands of requests. Thus, I am looping over the end points and the requests. As it is suggested to re-use HttpClient instances, that's what I am doing. However, I need to set some parameters like timeouts, passwords etc. in the header for each API. Thus, the first API works perfectly, when trying to set the Parameters for the next API, it fails:
This instance has already started one or more requests. Properties can only be modified before sending the first request.
Generally I know that the properties need to be set before making any requests. So I considered resetting the HttpClient for each API and then just re-use it for the thousands of requests to that API. Surprisingly, I get the same error - and I have absolutely no idea why.
This is about what the code looks like:
private HttpClient ApiClient;
private List<Api> Endpoints;
[...]
foreach(Api api in this.Endpoints)
{
this.ApiClient = new HttpClient();
this.ApiClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(api.mediaType));
this.ApiClient.Timeout = TimeSpan.FromMinutes(api.timeout);
this.ApiClient.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", api.credentials);
foreach (string url in api.urls)
{
# retrieve data from APIs and do something with it
}
}
As mentioned earlier, the first loop works perfectly fine. But when it starts over with the second api, I get a System.InvalidOperationException with the error message above when I try to set the ApiClient's timeout value.
Why so? I have created a brand new instance of HttpClient. Is there a better way to just reset the HttpClient?
The preferred way for generating HttpClients seems to be httpfactory: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-3.1. Also note that reinstatiating httpclients as you are, even without your specific exception can lead to problems, as your code seems to be able to run into socket exhaustion as described in https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests.
You should have only one instance of HttpClient during the lifetime of your application.
So instead of creating a HttpClient and setting the DefaultRequestHeaders every time you loop over your endpoints use HttpRequestMessage and do the following:
this.ApiClient = new HttpClient();
foreach(Api api in this.Endpoints)
{
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, "url");
request.Headers.Accept.Clear();
request.Headers.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("header value"));
var data = await ApiClient.SendAsync(request , HttpCompletionOption.ResponseContentRead);
}
I'm trying to reuse my HttpClient instance, as that's best practice. Therefore, in one particular request, I want to set the Authorization header on the request, instead of setting it globally on the client. From everything I've read, this ought to work:
var request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
var response = await _client.SendAsync(request);
This compiles, but then throws when I attempt to call it. The exception I get is:
"Misused header name. Make sure request headers are used with
HttpRequestMessage, response headers with HttpResponseMessage, and
content headers with HttpContent objects."
I'm confused why I'm allowed to set this value on the request message if it's just going to throw, and I also haven't figured out a workaround.
I've tried directly adding the header using TryAddWithoutValidation but I still get the same exception (not even a return of false like I'd expect!)
I'm suspecting this might be a bug in the framework since multiple people have posted this exact code with no apparent problems, but would be interested in any insights/workarounds.
Turns out that due to a different bug, the value of "token" was null. I would recommend checking this first if you're having this issue: note that the error message was 100% a lie.
Given that the .net HttpClient has been designed with reuse in mind and is intended to be long lived and memory leaks have been reported in short lived instances. What guide lines are there where you want to make restful calls to a given endpoint using different bearer tokens (or any authorization header) when calling the endpoint for multiple users?
private void CallEndpoint(string resourceId, string bearerToken) {
httpClient.DefaultRequestHeaders.Authorization =
new AuthenticationHeaderValue("bearer", bearerToken);
var response = await httpClient.GetAsync($"resource/{resourceid}");
}
Given the above code could be called by any number of threads on a web application it is easily possible that the header set in the first line is not the same one that is used when calling the resource.
Without causing contention using locks and maintaining a stateless web application what is the recommended approach to creating and disposing HttpClients for a single endpoint (My current practice is to create a single client per endpoint)?
Lifecycle
Although HttpClient does indirectly implement the IDisposable
interface, the recommended usage of HttpClient is not to dispose of it
after every request. The HttpClient object is intended to live for as
long as your application needs to make HTTP requests. Having an object
exist across multiple requests enables a place for setting
DefaultRequestHeaders and prevents you from having to respecify things
like CredentialCache and CookieContainer on every request, as was
necessary with HttpWebRequest.
If your headers are usually going to be the same then you can set the DefaultRequestHeaders. But you don't need to use that property to specify headers. As you've determined, that just wouldn't work if you're going to have multiple threads using the same client. Changes to the default headers made on one thread would impact requests sent on other threads.
Although you can set default headers on the client and apply them to each request, the headers are really properties of the request. So when the headers are specific to a request, you would just add them to the request.
request.Headers.Authorization = new AuthenticationHeaderValue("bearer", bearerToken);
That means you can't use the simplified methods that don't involve creating an HttpRequest. You'll need to use
public Task<HttpResponseMessage> SendAsync(HttpRequestMessage request)
documented here.
Some have found it helpful to use extension methods to isolate the code that updates the headers from the rest of a method.
Example of GET and POST methods done through an extension method that allow you to manipulate the request header and more of the HttpRequestMessage before it is sent:
public static Task<HttpResponseMessage> GetAsync
(this HttpClient httpClient, string uri, Action<HttpRequestMessage> preAction)
{
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, uri);
preAction(httpRequestMessage);
return httpClient.SendAsync(httpRequestMessage);
}
public static Task<HttpResponseMessage> PostAsJsonAsync<T>
(this HttpClient httpClient, string uri, T value, Action<HttpRequestMessage> preAction)
{
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, uri)
{
Content = new ObjectContent<T>
(value, new JsonMediaTypeFormatter(), (MediaTypeHeaderValue)null)
};
preAction(httpRequestMessage);
return httpClient.SendAsync(httpRequestMessage);
}
These could then be used like the following:
var response = await httpClient.GetAsync("token",
x => x.Headers.Authorization = new AuthenticationHeaderValue("basic", clientSecret));
Given that the .net HttpClient has been designed with reuse in mind and is intended to be long lived and memory leaks have been reported in short lived instances. What guide lines are there where you want to make restful calls to a given endpoint using different bearer tokens (or any authorization header) when calling the endpoint for multiple users?
private void CallEndpoint(string resourceId, string bearerToken) {
httpClient.DefaultRequestHeaders.Authorization =
new AuthenticationHeaderValue("bearer", bearerToken);
var response = await httpClient.GetAsync($"resource/{resourceid}");
}
Given the above code could be called by any number of threads on a web application it is easily possible that the header set in the first line is not the same one that is used when calling the resource.
Without causing contention using locks and maintaining a stateless web application what is the recommended approach to creating and disposing HttpClients for a single endpoint (My current practice is to create a single client per endpoint)?
Lifecycle
Although HttpClient does indirectly implement the IDisposable
interface, the recommended usage of HttpClient is not to dispose of it
after every request. The HttpClient object is intended to live for as
long as your application needs to make HTTP requests. Having an object
exist across multiple requests enables a place for setting
DefaultRequestHeaders and prevents you from having to respecify things
like CredentialCache and CookieContainer on every request, as was
necessary with HttpWebRequest.
If your headers are usually going to be the same then you can set the DefaultRequestHeaders. But you don't need to use that property to specify headers. As you've determined, that just wouldn't work if you're going to have multiple threads using the same client. Changes to the default headers made on one thread would impact requests sent on other threads.
Although you can set default headers on the client and apply them to each request, the headers are really properties of the request. So when the headers are specific to a request, you would just add them to the request.
request.Headers.Authorization = new AuthenticationHeaderValue("bearer", bearerToken);
That means you can't use the simplified methods that don't involve creating an HttpRequest. You'll need to use
public Task<HttpResponseMessage> SendAsync(HttpRequestMessage request)
documented here.
Some have found it helpful to use extension methods to isolate the code that updates the headers from the rest of a method.
Example of GET and POST methods done through an extension method that allow you to manipulate the request header and more of the HttpRequestMessage before it is sent:
public static Task<HttpResponseMessage> GetAsync
(this HttpClient httpClient, string uri, Action<HttpRequestMessage> preAction)
{
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, uri);
preAction(httpRequestMessage);
return httpClient.SendAsync(httpRequestMessage);
}
public static Task<HttpResponseMessage> PostAsJsonAsync<T>
(this HttpClient httpClient, string uri, T value, Action<HttpRequestMessage> preAction)
{
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, uri)
{
Content = new ObjectContent<T>
(value, new JsonMediaTypeFormatter(), (MediaTypeHeaderValue)null)
};
preAction(httpRequestMessage);
return httpClient.SendAsync(httpRequestMessage);
}
These could then be used like the following:
var response = await httpClient.GetAsync("token",
x => x.Headers.Authorization = new AuthenticationHeaderValue("basic", clientSecret));
In Asp.Net Web Api 2 what is the difference between setting an HttpClient Accept Header using the following traditional method :
HttpClient client = HttpClientFactory.Create(handler);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
and the following method :
var headers = new Dictionary<string, string>
{
{"Accept", "application/json"}};
headers.ForEach(h => client.DefaultRequestHeaders.Add(h.Key, h.Value));
Update 1:
Based on the answer by #DarrenMiller in the following post What is the overhead of creating a new HttpClient per call in a WebAPI client? it appears that the preferred method is using DefaultRequestHeaders property because it contains properties which are intended for multiple calls. Does this mean if I set a default header using a simple dictionary my HttpClient client will not be as efficient as the one which uses DefaultRequestHeaders? In addition I really cant understand how the values inside DefaultRequestHeaders will be reused? Lets say I create 20 HttpClient client using HttpClientFactory.Create and inside every single one of them I set DefaultRequestHeaders property [Do I really need to do it because DefaultRequestHeaders was meant to be reused?!]. Where does this reuse kick-in and does setting the DefaultRequestHeaders every time I create a HttpClient client result in some kind of performance hit?
Part one of your question: Is there any difference for adding headers?
HttpClient client = HttpClientFactory.Create(handler);
Method 1:
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
Method 2:
var headers = new Dictionary<string, string>{{"Accept", "application/json"}};
headers.ForEach(h => client.DefaultRequestHeaders.Add(h.Key, h.Value));
Method 1 gives you nice strongly typed values with the ability to add multiple accept types. Method 2 has one more "magic string" that could be a place for typos and there is no way to add multiple accept types.
Part 2 of your question: Where is the performance and reuse value?
The performance hit of using a new HttpClient for every request depends on your use case. Get a bench mark and measure to see if it matters. The performance on the developer is most likely where the gains will be. Consider that every HttpClient you use you have to remember a bunch of headers to add. If you forget to add a proper header, errors happen. So, you can use DefaultRequestHeaders to set these up in a factory.
public class ApiService
{
public static HttpClient GetClient()
{
var client = new HttpClient(new Uri("https://someservice/"));
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
//add any other setup items here.
return client;
}
}
Now use it:
public async Task DoStuff()
{
using(var client = ApiService.GetClient())
{
//client will have the proper base uri and all the headers set.
var data = await client.GetAsync<dynamic>("Sales");
//client will still have the proper base uri and all the headers set.
var data2 = await client.GetAsync<dynamic>("Products");
}
}
HttpClients should be short lived and always wrapped in a using statement. The reuse occurs when multiple requests are made using the same client.
UPDATE:
As others have mentioned, Microsoft recommends keeping one HttpClient for the life of the application to avoid excess overhead. The preferred way to get an HttpClient would be having a static one declared and then reference that as needed. Be aware that anything set in the default headers will go out with every request so make sure you don't put things there such as Authorization unless you're sure of the final destination of the request.