My code looks something like this:
private readonly HttpClient _httpClient;
public SomeConstructor(HttpClient httpClient){
_httpClient = httpClient;
}
public void SomeMethod(string reqUrl, string payload){
var result = GetResponseStringAsync(reqUrl, payload).GetAwaiter().GetResult();
// do something with result
}
private async Task<string> GetResponseStringAsync(string reqUrl, string payload){
using (var req = new HttpRequestMessage("POST", reqUrl)){
using (var content = new StringContent(payload)){
// Attach content headers here (specific for each request)
req.Content = content;
// Attach request headers here (specific for each request)
// using req.Headers.TryAddWithoutValidation()
using (var response = await _httpClient.SendAsync(req))
{
return await response.Content.ReadAsStringAsync();
}
}
}
}
I need to send API requests that have different, signed headers per request, otherwise I will get back 401 (Unauthorized). That said, when I send a single request, I always got 200, indicating that the authorization headers are sent correctly. However, if I send multiple requests at once (say with concurrency level set to 10), only 1 request got 200 back, whereas the other 9 got 401s. If I click on these 9 links individually, however, I got 200s for every single one of them, as expected.
It seems to me that somehow, there's a concurrency issue that results in the proper headers not being attached to their corresponding requests, even when I create a new HttpRequestMessage for each request. HttpClient and HttpRequestMessage both are supposedly thread-safe but could someone provide an explanation as to why I'm still getting weird results when sending multiple requests at once?
Add:
I have something like this in my AppHost: Container.Register<ISomeConstructor>(x => new SomeConstructor(new HttpClient())); so I am sure I'm not accidentally modifying the
client anywhere else
Placing a lock around the HttpClient (just before the SendAsync call) makes it work and returns 200s 100% of the time, further convincing me that it's a concurrency issue
I'm deploying and running on Mono 6.8.0.105 -- could this be a Mono issue? I couldn't find any issues/bug reports on this though
Related
Am querying multiple APIs with thousands of requests. Thus, I am looping over the end points and the requests. As it is suggested to re-use HttpClient instances, that's what I am doing. However, I need to set some parameters like timeouts, passwords etc. in the header for each API. Thus, the first API works perfectly, when trying to set the Parameters for the next API, it fails:
This instance has already started one or more requests. Properties can only be modified before sending the first request.
Generally I know that the properties need to be set before making any requests. So I considered resetting the HttpClient for each API and then just re-use it for the thousands of requests to that API. Surprisingly, I get the same error - and I have absolutely no idea why.
This is about what the code looks like:
private HttpClient ApiClient;
private List<Api> Endpoints;
[...]
foreach(Api api in this.Endpoints)
{
this.ApiClient = new HttpClient();
this.ApiClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(api.mediaType));
this.ApiClient.Timeout = TimeSpan.FromMinutes(api.timeout);
this.ApiClient.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", api.credentials);
foreach (string url in api.urls)
{
# retrieve data from APIs and do something with it
}
}
As mentioned earlier, the first loop works perfectly fine. But when it starts over with the second api, I get a System.InvalidOperationException with the error message above when I try to set the ApiClient's timeout value.
Why so? I have created a brand new instance of HttpClient. Is there a better way to just reset the HttpClient?
The preferred way for generating HttpClients seems to be httpfactory: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-3.1. Also note that reinstatiating httpclients as you are, even without your specific exception can lead to problems, as your code seems to be able to run into socket exhaustion as described in https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests.
You should have only one instance of HttpClient during the lifetime of your application.
So instead of creating a HttpClient and setting the DefaultRequestHeaders every time you loop over your endpoints use HttpRequestMessage and do the following:
this.ApiClient = new HttpClient();
foreach(Api api in this.Endpoints)
{
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, "url");
request.Headers.Accept.Clear();
request.Headers.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("header value"));
var data = await ApiClient.SendAsync(request , HttpCompletionOption.ResponseContentRead);
}
I've seen a lot of question about this, and all points to me using ConfigureAwait(false), but even after doing so, it still doesn't returned any response. When I run the debugger, the code stops at the PostAsync and does not continue with my code. Am I doing something wrong here? Does it have to do with me calling an API via HTTPS?
Here's the code:
public async static Task<PaymentModel> AddAsync(Card card)
{
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", Convert.ToBase64String(ASCIIEncoding.ASCII.GetBytes(string.Format("{0}:", "hidden"))));
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
var cardJson = JsonConvert.SerializeObject(card);
var postRequest = new StringContent(cardJson, Encoding.UTF8, "application/json");
var request = await client.PostAsync(new Uri("https://sample-3rd-party-api/api/endpoint/here"), postRequest).ConfigureAwait(false);
var content = await request.Content.ReadAsStringAsync().ConfigureAwait(false);
}
EDIT:
In response to the comments below, the code is contained from a method AddAsync(Card card) called from a button click with a handler:
public async void OnExecute(object sender, EventArgs args)
{
//some code here
payment = await PaymentModel.AddAsync(card).ConfigureAwait(false);
}
EDIT 2:
I tried pinging the API, but it returns a request timed out, but when I tried it using Postman, it's doing fine (the API is just a Sandbox which is open for all, so it's okay to share this):
EDIT 3:
I think the problem lies with where I don't have an SSL certificate to access the API. I have a PHP server that connects to the same API and I have to set SSL_VERIFYPEER to false just so I can access it (don't worry, I added a cacert now so its on true again). Can the same issue be happening here? If so, what can I do to create a valid certificate for my xamarin forms app
You can use this
var json = JsonConvert.SerializeObject(card);
using (var client = new HttpClient())
{
var t = await client.PostAsJsonAsync("https://sample-3rd-party-api/api/endpoint/here", json);
Response R =JsonConvert.DeserializeObject<Response>((JsonConvert.DeserializeObject(t.Content.ReadAsStringAsync().Result.ToString())).ToString());
}
What's most likely happening here is your OnExecute method has a return type of void instead of Task which prevents the UI thread from being able to await it. Try either changing that return type to Task or creating a new UI thread to perform this work. I wouldn't worry about the ping timing out as long as Postman works. Many public web servers disable their ping response.
Does it have to do with me calling an API via HTTPS?
As you are remaining in the same network and calling the same API from POSTMAN and .NET HTTP Client and only getting success with POSTMAN.So this issue gets cancelled.
Next
tried pinging the API, but it returns a request timed out
This is answered on top of mine.
Can you Please try setting the timeout option for HTTPClient while initializing.
client.Timeout = TimeSpan.FromSeconds(10);
and if still Problem persists please setup Fiddler and compare both the req sent from POstman and .NET client
So I think the problem is resolved now since I'm able to receive content from the request, what I did was simply follow the docs here: https://learn.microsoft.com/en-us/xamarin/cross-platform/app-fundamentals/transport-layer-security?tabs=windows
It looks like my settings are outdated in platform level.
Update the HttpClient implementation and SSL/TLS implementation
options to enable TLS 1.2 security.
Update the HttpClient Implementation option to enable TSL 1.2
security. (NSUrlSession (iOS 7.0+)
I was having the same issue and below trick fixed the issue.
Change your var request = await client.PostAsync(...); as below
var task = client.PostAsync(new Uri("https://sample-3rd-party-api/api/endpoint/here"), postRequest);
var request = task.GetAwaiter().GetResult();
Given that the .net HttpClient has been designed with reuse in mind and is intended to be long lived and memory leaks have been reported in short lived instances. What guide lines are there where you want to make restful calls to a given endpoint using different bearer tokens (or any authorization header) when calling the endpoint for multiple users?
private void CallEndpoint(string resourceId, string bearerToken) {
httpClient.DefaultRequestHeaders.Authorization =
new AuthenticationHeaderValue("bearer", bearerToken);
var response = await httpClient.GetAsync($"resource/{resourceid}");
}
Given the above code could be called by any number of threads on a web application it is easily possible that the header set in the first line is not the same one that is used when calling the resource.
Without causing contention using locks and maintaining a stateless web application what is the recommended approach to creating and disposing HttpClients for a single endpoint (My current practice is to create a single client per endpoint)?
Lifecycle
Although HttpClient does indirectly implement the IDisposable
interface, the recommended usage of HttpClient is not to dispose of it
after every request. The HttpClient object is intended to live for as
long as your application needs to make HTTP requests. Having an object
exist across multiple requests enables a place for setting
DefaultRequestHeaders and prevents you from having to respecify things
like CredentialCache and CookieContainer on every request, as was
necessary with HttpWebRequest.
If your headers are usually going to be the same then you can set the DefaultRequestHeaders. But you don't need to use that property to specify headers. As you've determined, that just wouldn't work if you're going to have multiple threads using the same client. Changes to the default headers made on one thread would impact requests sent on other threads.
Although you can set default headers on the client and apply them to each request, the headers are really properties of the request. So when the headers are specific to a request, you would just add them to the request.
request.Headers.Authorization = new AuthenticationHeaderValue("bearer", bearerToken);
That means you can't use the simplified methods that don't involve creating an HttpRequest. You'll need to use
public Task<HttpResponseMessage> SendAsync(HttpRequestMessage request)
documented here.
Some have found it helpful to use extension methods to isolate the code that updates the headers from the rest of a method.
Example of GET and POST methods done through an extension method that allow you to manipulate the request header and more of the HttpRequestMessage before it is sent:
public static Task<HttpResponseMessage> GetAsync
(this HttpClient httpClient, string uri, Action<HttpRequestMessage> preAction)
{
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, uri);
preAction(httpRequestMessage);
return httpClient.SendAsync(httpRequestMessage);
}
public static Task<HttpResponseMessage> PostAsJsonAsync<T>
(this HttpClient httpClient, string uri, T value, Action<HttpRequestMessage> preAction)
{
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, uri)
{
Content = new ObjectContent<T>
(value, new JsonMediaTypeFormatter(), (MediaTypeHeaderValue)null)
};
preAction(httpRequestMessage);
return httpClient.SendAsync(httpRequestMessage);
}
These could then be used like the following:
var response = await httpClient.GetAsync("token",
x => x.Headers.Authorization = new AuthenticationHeaderValue("basic", clientSecret));
Given that the .net HttpClient has been designed with reuse in mind and is intended to be long lived and memory leaks have been reported in short lived instances. What guide lines are there where you want to make restful calls to a given endpoint using different bearer tokens (or any authorization header) when calling the endpoint for multiple users?
private void CallEndpoint(string resourceId, string bearerToken) {
httpClient.DefaultRequestHeaders.Authorization =
new AuthenticationHeaderValue("bearer", bearerToken);
var response = await httpClient.GetAsync($"resource/{resourceid}");
}
Given the above code could be called by any number of threads on a web application it is easily possible that the header set in the first line is not the same one that is used when calling the resource.
Without causing contention using locks and maintaining a stateless web application what is the recommended approach to creating and disposing HttpClients for a single endpoint (My current practice is to create a single client per endpoint)?
Lifecycle
Although HttpClient does indirectly implement the IDisposable
interface, the recommended usage of HttpClient is not to dispose of it
after every request. The HttpClient object is intended to live for as
long as your application needs to make HTTP requests. Having an object
exist across multiple requests enables a place for setting
DefaultRequestHeaders and prevents you from having to respecify things
like CredentialCache and CookieContainer on every request, as was
necessary with HttpWebRequest.
If your headers are usually going to be the same then you can set the DefaultRequestHeaders. But you don't need to use that property to specify headers. As you've determined, that just wouldn't work if you're going to have multiple threads using the same client. Changes to the default headers made on one thread would impact requests sent on other threads.
Although you can set default headers on the client and apply them to each request, the headers are really properties of the request. So when the headers are specific to a request, you would just add them to the request.
request.Headers.Authorization = new AuthenticationHeaderValue("bearer", bearerToken);
That means you can't use the simplified methods that don't involve creating an HttpRequest. You'll need to use
public Task<HttpResponseMessage> SendAsync(HttpRequestMessage request)
documented here.
Some have found it helpful to use extension methods to isolate the code that updates the headers from the rest of a method.
Example of GET and POST methods done through an extension method that allow you to manipulate the request header and more of the HttpRequestMessage before it is sent:
public static Task<HttpResponseMessage> GetAsync
(this HttpClient httpClient, string uri, Action<HttpRequestMessage> preAction)
{
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, uri);
preAction(httpRequestMessage);
return httpClient.SendAsync(httpRequestMessage);
}
public static Task<HttpResponseMessage> PostAsJsonAsync<T>
(this HttpClient httpClient, string uri, T value, Action<HttpRequestMessage> preAction)
{
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, uri)
{
Content = new ObjectContent<T>
(value, new JsonMediaTypeFormatter(), (MediaTypeHeaderValue)null)
};
preAction(httpRequestMessage);
return httpClient.SendAsync(httpRequestMessage);
}
These could then be used like the following:
var response = await httpClient.GetAsync("token",
x => x.Headers.Authorization = new AuthenticationHeaderValue("basic", clientSecret));
I have the following set up:
JS client -> Web Api -> Web Api
I need to send the auth cookie all the way down. My problem is sending it from one web api to another. Because of integration with an older system, that uses FormsAuthentication, I have to pass on the auth cookie.
For performance reasons I share a list of HttpClients (one for each web api) in the following dictionary:
private static ConcurrentDictionary<ApiIdentifier, HttpClient> _clients = new ConcurrentDictionary<ApiIdentifier, HttpClient>();
So given an identifier I can grab the corresponding HttpClient.
The following works, but I'm pretty sure this is bad code:
HttpClient client = _clients[identifier];
var callerRequest = HttpContext.Current.Items["MS_HttpRequestMessage"] as HttpRequestMessage;
string authCookieValue = GetAuthCookieValue(callerRequest);
if (authCookieValue != null)
{
client.DefaultRequestHeaders.Remove("Cookie");
client.DefaultRequestHeaders.Add("Cookie", ".ASPXAUTH=" + authCookieValue);
}
HttpResponseMessage response = await client.PutAsJsonAsync(methodName, dataToSend);
// Handle response...
Whats wrong about this is that 1) it seems wrong to manipulate DefaultRequestHeaders in a request and 2) potentially two simultanious requests may mess up the cookies, as the HttpClient is shared.
I've been searching for a while without finding a solution, as most having a matching problem instantiates the HttpClient for every request, hence being able to set the required headers, which I'm trying to avoid.
At one point I had get requests working using a HttpResponseMessage. Perhaps that can be of inspiration to a solution.
So my question is: is there a way to set cookies for a single request using a HttpClient, that will be safe from other clients using the same instance?
Instead of calling PutAsJsonAsync() you can use HttpRequestMessage and SendAsync():
Uri requestUri = ...;
HttpMethod method = HttpMethod.Get /*Put, Post, Delete, etc.*/;
var request = new HttpRequestMessage(method, requestUri);
request.Headers.TryAddWithoutValidation("Cookie", ".ASPXAUTH=" + authCookieValue);
request.Content = new StringContent(jsonDataToSend, Encoding.UTF8, "application/json");
var response = await client.SendAsync(request);
UPDATE:
To make sure that your HTTP client does not store any cookies from a response you need to do this:
var httpClient = new HttpClient(new HttpClientHandler() { UseCookies = false; });
Otherwise you might get unexpected behavior by using one client and sharing other cookies.