I've been using the HttpClient in code for a while now and have always felt its use of Uris has resulted in some brittleness in my implementation. Most of our service endpoint base addresses are in the app./web.config. As a result, they can be easily changed.
I've found that when using these endpoint strings to generate a Uri, if they don't end in a /, I get really wonky behavior. When calling GetAsync() with a non-/-terminated BaseAddress, the resulting concatenated URL that is sent the GET request often drops either the string after the final / in the BaseAddress, or it will drop the string preceeding the first / in the GetUri.
For example:
BaseAddress: http://test/serviceEndpoint
GetUri: api/customer
When HttpClient.GetAsync() is called with that GetUri, it will attempt to GET from http://test/api/customer. If I cap BaseAddress with a /, everything works as expected.
My problem is that BaseAddress is config-driven, and putting in a comment in the .config file saying "Make sure you end all Service URLs with a /!" is a really brittle solution.
So I've gotten into the habit of using the following code in all of my HttpClient construction:
var service = settings.GetValue("ServiceBaseUrl");
var serviceUri = !service.EndsWith("/")
? new Uri(service + "/")
: new Uri(service);
_client = new HttpClient
{
BaseAddress = serviceUri
};`
While this isn't brittle, it feels repetitive to have it in every HttpClient constructor. Is there something in either HttpClient or Uri that I can use to avoid this boilerplate code?
There's nothing in HttpClient or Uri to address this, which is why I addressed it in a couple ways in Flurl. Flurl's AppendPathSegment and AppendPathSegments methods will ensure one and only one "/" separator between segments. For example, these yield the identical results:
"http://mysite/".AppendPathSegment("/endpoint")
"http://mysite".AppendPathSegment("endpoint")
The static Url.Combine method also has this behavior, acting as sort of a Path.Combine for URLs.
These and other helpful URL-building bits are available in the core Flurl package, but the real fun is in Flurl.Http, which combines the fluent URL builder with a lightweight wrapper on top of HttpClient and Json.NET that lets you go from string to URL to HTTP request to deserialized result without lifting pen from paper, so to speak:
var result = await settings.GetValue("ServiceBaseUrl")
.AppendPathSegment("endpoint")
.GetJsonAsync<T>();
Related
Am querying multiple APIs with thousands of requests. Thus, I am looping over the end points and the requests. As it is suggested to re-use HttpClient instances, that's what I am doing. However, I need to set some parameters like timeouts, passwords etc. in the header for each API. Thus, the first API works perfectly, when trying to set the Parameters for the next API, it fails:
This instance has already started one or more requests. Properties can only be modified before sending the first request.
Generally I know that the properties need to be set before making any requests. So I considered resetting the HttpClient for each API and then just re-use it for the thousands of requests to that API. Surprisingly, I get the same error - and I have absolutely no idea why.
This is about what the code looks like:
private HttpClient ApiClient;
private List<Api> Endpoints;
[...]
foreach(Api api in this.Endpoints)
{
this.ApiClient = new HttpClient();
this.ApiClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(api.mediaType));
this.ApiClient.Timeout = TimeSpan.FromMinutes(api.timeout);
this.ApiClient.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", api.credentials);
foreach (string url in api.urls)
{
# retrieve data from APIs and do something with it
}
}
As mentioned earlier, the first loop works perfectly fine. But when it starts over with the second api, I get a System.InvalidOperationException with the error message above when I try to set the ApiClient's timeout value.
Why so? I have created a brand new instance of HttpClient. Is there a better way to just reset the HttpClient?
The preferred way for generating HttpClients seems to be httpfactory: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-3.1. Also note that reinstatiating httpclients as you are, even without your specific exception can lead to problems, as your code seems to be able to run into socket exhaustion as described in https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests.
You should have only one instance of HttpClient during the lifetime of your application.
So instead of creating a HttpClient and setting the DefaultRequestHeaders every time you loop over your endpoints use HttpRequestMessage and do the following:
this.ApiClient = new HttpClient();
foreach(Api api in this.Endpoints)
{
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, "url");
request.Headers.Accept.Clear();
request.Headers.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("header value"));
var data = await ApiClient.SendAsync(request , HttpCompletionOption.ResponseContentRead);
}
Integrating older ASP.NET server-side application into ADFS for authentication, which means I pretty much had to write everything from scratch. have everything working (/authorize, /token) up until the /userinfo call.
My code, in a nutshell -
HttpClient client = new HttpClient();
var req = new HttpRequestMessage {
RequestUri = new Url("https://<server_ip>/adfs/oauth2/userinfo"),
Method = HttpMethod.Get,
};
req.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token.AccessToken);
req.Headers.UserAgent.Clear();
req.Headers.UserAgent.Add(new ProductInfoHeaderValue("OldApp", "11.3.0"));
var result = await client.SendAsync(req);
The result is a HTTP error 405 - Method Not Allowed. Doing searches online, I see this as a common issue when the trailing "/" is left off the url, but I get the same result with a trailing slash.
After poking around, there are a lot of examples that use newer libraries and such that I can't use, sadly. None mention usage of the /userinfo, and I'm thinking that the issue isn't necessarily in how I'm calling the URL, but configuration of the 'Application Group' in ADFS.
Okay - I found the issue, and will document it here in case others come across the same thing..
While I am not sure why /userinfo is giving a 405 - the URL I was using is wrong, despite it being listed in the Endpoints folder. There shouldn't be any "oauth2" in the URL. The correct code (and URL) is:
var req = new HttpRequestMessage {
RequestUri = new Url("https://<server_ip>/adfs/userinfo"),
Method = HttpMethod.Get,
};
req.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token.AccessToken);
req.Headers.UserAgent.Clear();
req.Headers.UserAgent.Add(new ProductInfoHeaderValue("OldApp", "11.3.0"));
var result = await client.SendAsync(req);
Also something to keep in mind - this has been stated elsewhere, but not as clearly as here, I hope:
The /userinfo will ONLY give you the NameIdentifier ("sub") claim. (As far as I can see.) No matter what scope you pass it. You will get all your information (that should normally be in the /userinfo call) in the "id_token" parameter from you /token call, encoded as JWT.
Personally I was led to do the same thing as you, the only solution was to download the ADAL library (You will find the link below) and debug the code in order to re-produce the same HTTP stream from ADAL.
You can create a new project so that you can integrate ADAL, for debugging or else intercepting the HTTP stream
Link ADAL
In Asp.Net Web Api 2 what is the difference between setting an HttpClient Accept Header using the following traditional method :
HttpClient client = HttpClientFactory.Create(handler);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
and the following method :
var headers = new Dictionary<string, string>
{
{"Accept", "application/json"}};
headers.ForEach(h => client.DefaultRequestHeaders.Add(h.Key, h.Value));
Update 1:
Based on the answer by #DarrenMiller in the following post What is the overhead of creating a new HttpClient per call in a WebAPI client? it appears that the preferred method is using DefaultRequestHeaders property because it contains properties which are intended for multiple calls. Does this mean if I set a default header using a simple dictionary my HttpClient client will not be as efficient as the one which uses DefaultRequestHeaders? In addition I really cant understand how the values inside DefaultRequestHeaders will be reused? Lets say I create 20 HttpClient client using HttpClientFactory.Create and inside every single one of them I set DefaultRequestHeaders property [Do I really need to do it because DefaultRequestHeaders was meant to be reused?!]. Where does this reuse kick-in and does setting the DefaultRequestHeaders every time I create a HttpClient client result in some kind of performance hit?
Part one of your question: Is there any difference for adding headers?
HttpClient client = HttpClientFactory.Create(handler);
Method 1:
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
Method 2:
var headers = new Dictionary<string, string>{{"Accept", "application/json"}};
headers.ForEach(h => client.DefaultRequestHeaders.Add(h.Key, h.Value));
Method 1 gives you nice strongly typed values with the ability to add multiple accept types. Method 2 has one more "magic string" that could be a place for typos and there is no way to add multiple accept types.
Part 2 of your question: Where is the performance and reuse value?
The performance hit of using a new HttpClient for every request depends on your use case. Get a bench mark and measure to see if it matters. The performance on the developer is most likely where the gains will be. Consider that every HttpClient you use you have to remember a bunch of headers to add. If you forget to add a proper header, errors happen. So, you can use DefaultRequestHeaders to set these up in a factory.
public class ApiService
{
public static HttpClient GetClient()
{
var client = new HttpClient(new Uri("https://someservice/"));
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
//add any other setup items here.
return client;
}
}
Now use it:
public async Task DoStuff()
{
using(var client = ApiService.GetClient())
{
//client will have the proper base uri and all the headers set.
var data = await client.GetAsync<dynamic>("Sales");
//client will still have the proper base uri and all the headers set.
var data2 = await client.GetAsync<dynamic>("Products");
}
}
HttpClients should be short lived and always wrapped in a using statement. The reuse occurs when multiple requests are made using the same client.
UPDATE:
As others have mentioned, Microsoft recommends keeping one HttpClient for the life of the application to avoid excess overhead. The preferred way to get an HttpClient would be having a static one declared and then reference that as needed. Be aware that anything set in the default headers will go out with every request so make sure you don't put things there such as Authorization unless you're sure of the final destination of the request.
This is me publicly documenting my mistake so that if I or anyone does it again, they don't have to spend 3 hours tearing their hair out trying to fix such a simple thing.
Context
I was sending an HttpRequest from one C# MVC ASP.NET application to another.
The applications require an HTTPS connection, and we are using URLRewrite to redirect an HTTP request to an HTTPS url.
One application was sending a POST request with some JSON data in the body, pretty standard stuff. The other application was set up to receive this data with an MVC controller class (CollectionAction and Insert methods for GET and POST respectively).
Symptoms of the problem
The receiving application was running the GET method (CollectionAction) instead of the POST action (ItemAction). The reason for this was that the request coming in to the application was in fact a GET request, and to top it off the JSON data was missing too.
I sent the header "x-http-method" to override the request method from GET to POST (I was already setting the request httpmethod to POST but this was being ignored). This worked but still I had no data being sent.
So now I am stuck pulling my hair out, because I can see a POST request with content-length and data being sent out and I have a GET request with no data or content-length coming in (but the headers were preserved)
Turns out I was using UriBuilder to take a base URL and apply a resource path to it. For example I would have "google.com" in my web.config and then the UriBuilder would take a resource like Pages and construct the url "google.com/Pages". Unfortunately, I was not initializing the UriBuilder with the base URL, and instead was using a second UriBuilder to extract the host and add that to the path like so:
public Uri GetResourceUri(string resourceName)
{
var domain = new UriBuilder(GetBaseUrl());
var uribuilder = new UriBuilder()
{
Path = domain.Path.TrimEnd('/') + "/" + resourceName.TrimStart('/'),
Host = domain.Host
};
var resourceUri = uribuilder.Uri;
return resourceUri;
}
The problem with this code is that the scheme is ignored (HTTP:// vs HTTPS://) and it defaults to HTTP. So my client was sending out the request to an HTTP url instead of the required HTTPS url. This is the interesting part, URLRewrite was kicking in and saying that we needed to go to an HTTPS url instead so it redirected us there. But in doing so, it ignored the Http-Method and the POST data, which just got set to defaults GET and null. This is what the 2nd application could see at the receiving end.
So the function had to be rewritten to this which fixed the problem:
public Uri GetResourceUri(string resourceName)
{
var baseUrl = GetBaseUrl();
var domain = new UriBuilder(baseUrl);
var uribuilder = new UriBuilder(baseUrl)
{
Path = domain.Path.TrimEnd('/') + "/" + resourceName.TrimStart('/'),
};
var resourceUri = uribuilder.Uri;
return resourceUri;
}
I'm using HttpListener to provide a web server to an application written in another technology on localhost. The application is using a simple form submission (application/x-www-form-urlencoded) to make its requests to my software. I want to know if there is already a parser written to convert the body of the html request document into a hash table or equivalent.
I find it hard to believe I need to write this myself, given how much .NET already seems to provide.
Thanks in advance,
You mean something like HttpUtility.ParseQueryString that gives you a NameValueCollection? Here's some sample code. You need more error checking and maybe use the request content type to figure out the encoding:
string input = null;
using (StreamReader reader = new StreamReader (listenerRequest.InputStream)) {
input = reader.ReadToEnd ();
}
NameValueCollection coll = HttpUtility.ParseQueryString (input);
If you're using HTTP GET instead of POST:
string input = listenerRequest.Url.QueryString;
NameValueCollection coll = HttpUtility.ParseQueryString (input);
The magic bits that fill out HttpRequest.Form are in System.Web.HttpRequest, but they're not public (Reflector the method "FillInFormCollection" on that class to see). You have to integrate your pipeline with HttpRuntime (basically write a simple ASP.NET host) to take full advantage.
If you want to avoid the dependency on System.Web that is required to use HttpUtility.ParseQueryString, you could use the Uri extension method ParseQueryString found in System.Net.Http.
Make sure to add a reference (if you haven't already) to System.Net.Http in your project.
Note that you have to convert the response body to a valid Uri so that ParseQueryString (in System.Net.Http)works.
string body = "value1=randomvalue1&value2=randomValue2";
// "http://localhost/query?" is added to the string "body" in order to create a valid Uri.
string urlBody = "http://localhost/query?" + body;
NameValueCollection coll = new Uri(urlBody).ParseQueryString();