Develop Reference

.net HttpClient in C#: Exception when re-instantiating

Am querying multiple APIs with thousands of requests. Thus, I am looping over the end points and the requests. As it is suggested to re-use HttpClient instances, that's what I am doing. However, I need to set some parameters like timeouts, passwords etc. in the header for each API. Thus, the first API works perfectly, when trying to set the Parameters for the next API, it fails:
This instance has already started one or more requests. Properties can only be modified before sending the first request.
Generally I know that the properties need to be set before making any requests. So I considered resetting the HttpClient for each API and then just re-use it for the thousands of requests to that API. Surprisingly, I get the same error - and I have absolutely no idea why.
This is about what the code looks like:
private HttpClient ApiClient;
private List<Api> Endpoints;
[...]
foreach(Api api in this.Endpoints)
{
this.ApiClient = new HttpClient();
this.ApiClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(api.mediaType));
this.ApiClient.Timeout = TimeSpan.FromMinutes(api.timeout);
this.ApiClient.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", api.credentials);
foreach (string url in api.urls)
{
# retrieve data from APIs and do something with it
}
}
As mentioned earlier, the first loop works perfectly fine. But when it starts over with the second api, I get a System.InvalidOperationException with the error message above when I try to set the ApiClient's timeout value.
Why so? I have created a brand new instance of HttpClient. Is there a better way to just reset the HttpClient?

The preferred way for generating HttpClients seems to be httpfactory: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-3.1. Also note that reinstatiating httpclients as you are, even without your specific exception can lead to problems, as your code seems to be able to run into socket exhaustion as described in https://learn.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests.

You should have only one instance of HttpClient during the lifetime of your application.
So instead of creating a HttpClient and setting the DefaultRequestHeaders every time you loop over your endpoints use HttpRequestMessage and do the following:
this.ApiClient = new HttpClient();
foreach(Api api in this.Endpoints)
{
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, "url");
request.Headers.Accept.Clear();
request.Headers.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("header value"));
var data = await ApiClient.SendAsync(request , HttpCompletionOption.ResponseContentRead);
}

Can I change the content of an incoming HTTP request using a HTTP Module?

I'm trying to extend my REST service (built using WCF/webHttpBinding) so that clients can upload gzipped data. I'm not sure what the best way to accomplish this but I thought it would be fairly easy by adding a HTTP module which will decompress the data if Content-Encoding for the incoming request is set to gzip.
So I created an class deriving from IHttpModule with the following implementation:
private void OnBeginRequest(object sender, EventArgs e)
{
var app = (HttpApplication) sender;
var context = app.Context;
var contentEncoding = context.Request.Headers["Content-Encoding"];
if (contentEncoding == "gzip")
{
// some debug code:
var decompressedStream = new GZipStream(context.Request.InputStream, CompressionMode.Decompress);
var memoryStream = new MemoryStream();
decompressedStream.CopyTo(memoryStream);
memoryStream.Seek(0, SeekOrigin.Begin);
var streamReader = new StreamReader(memoryStream);
string msg = streamReader.ReadToEnd();
context.Request.InputStream.Seek(0, SeekOrigin.Begin);
app.Request.Filter = //new TestFilterStream(app.Request.Filter);
new System.IO.Compression.GZipStream(
app.Request.Filter, CompressionMode.Decompress);
}
}
The issue I'm seeing is that the GZipStream decompression is never actually performed. I've confirmed that the incoming data is in fact gzip'd (the msg-variable contains the proper data). I've also tried creating my own stream class (TestFilterStream) above and assign that to app.Request.Filter and I've confirmed that no members on the stream class is actually called by ASP.NET. So it seems like while it's possible to specify a filter, that filter isn't actually used.
Isn't HttpApplication.Request.Filter actually used?

I tried setting the Request Filter in two ways:
Using a HttpModule
Setting it in the start of Application_BeginRequest() (Global.asax)
Both with the same results (VS2012 web project + IISExpress):
If there is no input data (GET request or similar), the Filter.Read is not invoked
In case of a POST with actual data, the filter is executed and the web service gets the filtered data
Even if I read from the Request.InputStream before the Filter is set, I still get the filter triggered from my service code.
I have no easy way of testing with Gzippet input, so I have not tried if the actual filter works. However, I know it is getting triggered, since I get an error from GZipStream while it attempts to look for the input.
Perhaps you are having other HttpModules or Filters that disrupt your input or control flow?
This post proposes a method similar to yours, but also states the following, which may be causing some side effects (my tests were not using WCF):
"It appears that this approach trigger a problem in WCF, as WCF relies on the original Content-Length and not the value obtained after decompressing."

I've just done a couple of tests, and my Request.Filter stream is called into, as long as there is a request body and the request body gets read by the http handler. I'm guessing you use either a PUT or a POST, and certainly read the request body, so that shouldn't be a problem.
I suspect Knaģis' comment is correct. Have you tried without the debug code? If I dig into the HttpRequest source, I see a variable _rawContent being written to exactly once; at that same time the request filters are applied. After that the _rawContent value just gets cached, and is never updated (nor reset when a filter is added).
So by calling Request.InputStream in your debug code you are definitely preventing your filter from being applied later on. Reading the Request.Headers collection is no problem.

Are you sure, that application itself should bother?
Usually that's handled per configuration of host (IIS). So, basically, you only need to implement custom GZip support, when you host the service yourself.
You can take a look here

pass local path to HttpWebRequest

I need to pass local path to HttpWebRequest in c#. i have test.xml in my c drive and i need get that xml file in HttpWebRequest. but it throws exception in
HttpWebRequest rqst = (HttpWebRequest)HttpWebRequest.Create(Uri.EscapeUriString(urlServ))
line "Invalid URI: The Authority/Host could not be parsed."
my coding->
string urlServ = "file:\\c:\\test.xml";
try
{
HttpWebRequest rqst = (HttpWebRequest)HttpWebRequest.Create(Uri.EscapeUriString(urlServ));
rqst.KeepAlive = false;
}
catch{}

I believe a file: URI is supposed to be created with forward-slashes, not back slashes. So, use this:
string urlServ = "file:///c:/test.xml";
I noticed when I typed it into my browser with backslashes, FF converted it to forward slashes for me.

You should use WebRequest.Create(uri) - this will automatically create the right object based on the URI type (e.g. file, http, etc). Now you can use the same code for real web pages or local test files.
I saw this in the documentation of FileWebRequest:
Do not use the FileWebRequest constructor. Use the WebRequest.Create
method to initialize new instances of the FileWebRequest class. If the
URI scheme is file://, the Create method returns a FileWebRequest
object.

help to convert java code to C#

i was trying to get the C# version of the following java code snippet,
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestProperty("Range", "bytes=1024-");
this is what i have so far
WebRequest request = WebRequest.Create(someUri);
request.Headers.Add("Range", "bytes=1024-");
but it is not working,what is the right way for me go?

Presumably your URI is HTTP since Java's HttpURLConnection is designed for a HTTP connection. WebRequest is abstract and can handle multiple protocols. However, by specifiying a HttpWebRequest type, you can access HTTP-specific methods.
The Range header is protected and you should use AddRange to set the property instead of directly adding it to the Header collection.
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(someUri);
request.AddRange("bytes",1024);

You are setting two different things.
A request property is a value passed to the page.
A header property is a header in the HTTP request. Something like setting the HTTP REFERER (sic).

Is there a .NET ready made method to process response body of a HttpListener HttpListenerRequest body?

I'm using HttpListener to provide a web server to an application written in another technology on localhost. The application is using a simple form submission (application/x-www-form-urlencoded) to make its requests to my software. I want to know if there is already a parser written to convert the body of the html request document into a hash table or equivalent.
I find it hard to believe I need to write this myself, given how much .NET already seems to provide.
Thanks in advance,

You mean something like HttpUtility.ParseQueryString that gives you a NameValueCollection? Here's some sample code. You need more error checking and maybe use the request content type to figure out the encoding:
string input = null;
using (StreamReader reader = new StreamReader (listenerRequest.InputStream)) {
input = reader.ReadToEnd ();
}
NameValueCollection coll = HttpUtility.ParseQueryString (input);
If you're using HTTP GET instead of POST:
string input = listenerRequest.Url.QueryString;
NameValueCollection coll = HttpUtility.ParseQueryString (input);

The magic bits that fill out HttpRequest.Form are in System.Web.HttpRequest, but they're not public (Reflector the method "FillInFormCollection" on that class to see). You have to integrate your pipeline with HttpRuntime (basically write a simple ASP.NET host) to take full advantage.

If you want to avoid the dependency on System.Web that is required to use HttpUtility.ParseQueryString, you could use the Uri extension method ParseQueryString found in System.Net.Http.
Make sure to add a reference (if you haven't already) to System.Net.Http in your project.
Note that you have to convert the response body to a valid Uri so that ParseQueryString (in System.Net.Http)works.
string body = "value1=randomvalue1&value2=randomValue2";
// "http://localhost/query?" is added to the string "body" in order to create a valid Uri.
string urlBody = "http://localhost/query?" + body;
NameValueCollection coll = new Uri(urlBody).ParseQueryString();