Develop Reference

Custom proxy server headers with .NET Framework HttpWebRequest using C#

I am using the HttpWebRequest class to implement a web crawler. I would like to use a proxy server with it which usually can be implemented like this:
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("https://www.example.com/");
webRequest.Proxy = new WebProxy(proxyUrl, proxyPort);
webRequest.Proxy.Credentials = new NetworkCredential(proxyPassword, proxyPassword);
HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
using (Stream stream = webResponse.GetResponseStream())
{
StreamReader reader = new StreamReader(stream, Encoding.UTF8);
Console.WriteLine(reader.ReadToEnd());
}
However, I need to submit an additional parameter to the proxy server. Because the target site uses HTTPS, the parameters must be submitted in the initial CONNECT request that establishes the secure HTTPS tunnel. Otherwise the proxy server is unable to read them due the end-to-end encryption.
The header for the initial CONNECT request should look like this:
CONNECT example.com:443 HTTP/1.1
X-CustomProperty: 12345
So far I was unable to find a solution to achieve this with neither the HttpWebRequest nor the WebClient class from the .NET Framework. Is there any option to pass additional headers in the initial CONNECT request? I am also happy with a solution that uses an additional library.

HttpClient and WebClient give 401 error yet direct link in browser works

I'm the middleman on an integration project between a Woocommerce site and a retail POS system. The POS is trying to synchronize products with the Woocommerce store. The site has an SLL certificate so all communication is over SSL. The POS vendor tells me they are using TLS1.2. We have a URL that contains a customer key and secret so I can't paste the full URL here, but the problem is as follows...
The same URL pasted directly into the browser works perfectly well, returning the expected JSON payload. The same URL used with a .NET WebClient or HttpClient returns a 401 Not Authorized error. Here is an example of an integration test using WebClient;
[Test]
public void DownloadString_UsingWebClient_ReturnsNonEmptyResponse()
{
using (var client = new WebClient())
{
var response = client.DownloadString("https://demostore.mycompany.com.au/wp-json/wc/v2/settings/tax/woocommerce_prices_include_tax?consumer_key=ck_xxxskeyxxx&consumer_secret=cs_xxxsecretxxx");
Assert.IsNotEmpty(response);
}
}
And here is a similar test using HttpClient;
[Test]
public async Task DownloadString_UsingHttpClient_ReturnsNonEmptyResponse()
{
using (var client = new HttpClient())
{
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
var response = await client.GetStringAsync("https://demostore.mycompany.com.au/wp-json/wc/v2/settings/tax/woocommerce_prices_include_tax?consumer_key=ck_xxxskeyxxx&consumer_secret=cs_xxxsecretxxx"");
Assert.IsNotEmpty(response);
}
}
Both tests never make it to the Assert. Just to reiterate, if I take the same URL and paste it into my browser, everything works as expected. I've tried a couple of different things, including enforcing the security protocol type (as in the latter example), but it is all just a poke in the dark, really.
The website is using the latest version of Wordpress, and it is in a shared linux hosting environment.
Is there something I'm missing with WebClient and HttpClient? Do I go back to the web hosting company and find out if there is something in the configuration that would prevent a .NET client from performing the same as a browser (I have logged a support ticket to this effect, but I'm not getting any headway)?

If you're in a corporate network and the browser works and your code doesn't, it's likely you're missing the Proxy settings for the WebClient or HttpClient instance.
I typically pass a HttpClientHandler instance with Preauthenticate = true and UseDefaultCredentials = true to the HttpClient.

I found that usually when something works in the browser, but not outside of it, that suggests that the API relies on User-Agent header being present in the request.
One of the reasons for this may be a reverse-proxy/firewall in front of that API.

WebException SecureChannelFailure when invoking web service over https from Xamarin Droid app

I'm creating a Xamarin.Droid app with Visual Studio (C#).
I have a RESTful web service (WebAPI) from which I want to retrieve some data in a Xamarin Droid project, using HttpWebRequest. The fact that it's WebAPI and HttpWebRequest is not the key as I had the exact same problem (described below) with some older WSE 3.0 web services and the use of HttpClient in the generated proxies. In the case of those WSE 3.0 web services we solved the problem by extending the generated proxy class to use ModernHttpClient instead. We didn't realize this problem manifests outside of HttpClient. I now think we must be doing something wrong, as opposed to blaming HttpClient.
In the current iteration, connecting to WebAPI, I'm using HttpWebRequest. Whether using HttpWebRequest or HttpClient, everything works fine as long as I'm connecting via http. As soon as I try to connect via https the request fails. For clarification, the web server hosting the web service does have a valid certificate (it's not expired, and has a valid CA.)
I have not been able to find any solution to this problem, despite a ton of Googling. I can barely find references to other people having this problem, much less a solution, which makes me think there must be something wrong in what I'm doing.
Basically, you can this method to a Xamarin.Forms Droid project and call it. If url is using https then request.GetResponse() throws a WebException: "Error: SecureChannelFailure (The authentication or decryption has failed.)". If url is using plain http then the call executes cleanly and returns the expected data.
using System;
using System.IO;
using System.Net;
using Android.App;
using Android.Content.PM;
using Android.OS;
// ...
private string FetchData()
{
// Public test server that's handy for testing JSON code.
// http://jsonplaceholder.typicode.com
string url = "https://jsonplaceholder.typicode.com/posts/1";
// Create an HTTP web request using the URL:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Timeout = 10 * 1000; // 10 seconds
request.Accept = "application/json";
request.Method = "GET";
string result = "";
try
{
using (WebResponse response = request.GetResponse())
{
result = "Presumably, do something with response.GetResponseStream()";
}
}
catch (Exception ex)
{
result = $"{ex.GetType().Name}: {ex.Message}";
}
return result;
}
The basic pattern there comes from a Xamarin "recipe": Call a REST Web Service. I changed it to be synchronous because when I change request.GetResponse() to request.GetResponseAsync() (and async-await) then the call either times out, or hangs indefinitely, obfuscating the underlying WebException (SecureChannelFailure).
This is a simple HTTP GET, and any browser, or REST/SOAP utility (e.g. "Simple REST Client" for Chrome) can be used to build and send that exact same request, and it works fine with both http and https. Similarly, I can take that code above and paste it into a Windows Console application and it works fine with both http and https.
For the record, love it or hate it, I have also tried adding a delegate of { return true; } to the System.Net.ServicePoint.ServerCertificateAuthenticationCallback event. To me that wouldn't be a solution anyway, but it might help diagnose the problem. Too bad a breakpoint set in that delegate is never tripped, indicating my code never even gets to that point.
It seems impossible to me that invoking web services through https is not supported in a Xamarin Droid app, so I'm going on the assumption that I'm doing something wrong. What could be causing this failure in Xamarin Droid, when it works fine in a browser, and in a plain Windows Console app?

if you call a HTTP url and get this exception , be sure set redirect to false:
request.AllowAutoRedirect = false;
if you call a HTTPS url then put these lines before your request code:
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072;

Webservice Client and Compression

I have Managed to Setup II7 with Gzip Compression .
I can check via web sniffer that my asmx web service encoding is Gzip but how to i enable
gzip Compression on my C# Client , i am using the Web Service is Service reference in my application.
Actually i am trying to send large amount of data , 10k objects of array so Compression with be great effect on bw.
but how do I enable Compression on my C# Client.
i am trying to see that many people sees same problem but there nothing clear answer some says use third party tools or some says about custom headers etc etc .
is not there any proper way , built in to consume Compressed web service

As #Igby Largeman pointed out, you can use your IIS7 to enable the compression on the server, but this is not enough.
The main idea is to set the headers on the client side and server side:
Client:
Accept-Encoding = "gzip, deflate";
You can achieve this by code:
var request = HttpWebRequest.Create("http://foofoo");
request.Headers["Accept"] = "application/json";
request.Headers["Accept-Encoding"] = "gzip, deflate";
or
var request = HttpWebRequest.Create("http://foofoo");
request.AutomaticDecompression = DecompressionMethods.GZip |
DecompressionMethods.Deflate;
If you use some WCF client, and not the HttpWebRequest, you should use custom inspector and dispatcher, like in this article:
So I used a message inspector implementing IClientMessageInspector and IDispatchMessageInspector to automatically set the AcceptEncoding and ContentEncoding http headers.
This was working perfectly but I could not achieve to decompress the response on the server by first detecting the ContentEncoding header thus I used the work around to first try to decompress it and if it fails just try to process the request as normal.
I also did this in the client pipeline and this also works.
Server:
// This is the nearly same thing after all
Content-Encoding = "gzip" OR Content-Encoding = "deflate"
To do this on the Server side, you should enable httpCompression in the IIS.
I think you should check the original article to get this work

HTTP library that doesn't depend on HttpWebRequest

I need a C# HTTP library that doesn't depend on HttpWebRequest, as I can't access this from the environment I need to run my code (Unity's WebPlayer).
Ideally this would be light weight, but any suggestions are welcome!
I just need to be able to do simple HTTP GET and POST requests, but better REST support would be good.
Thanks!
Edit: As people have pointed out, HttpWebRequest isn't in System.Web, but the fact remains - I can't use it. I've updated my post above.
This post
http://forum.unity3d.com/threads/24445-NotSupportedException-System.Net.WebRequest.GetCreator shows the same error I'm getting.

Implementing your own simple HTTP client using Socket is not all that difficult.
Just use TcpClient().
For the protocol itself, drop down to a connection-per-request paradigm. A typical GET request would look as follows:
GET /url HTTP/1.1
Host: <hostname-of-server>
Connection: close
For the code itself (from memory)
TcpClient client = new TcpClient();
IPEndPoint target = ... // get an endpoint for the target using DNS class
client.Connect(target);
using(NetworkStream stream = client.GetStream())
{
// send the request.
string request = "GET /url HTTP/1.1\r\nConnection: close\r\n\r\n";
stream.Write(Encoding.ASCII.GetBytes(request));
// then drain the stream to get the server response
}
Note that you will need to wrap this code with a simple class that provides HTTPWebRequest like semantics.

Look at System.Net.HttpWebRequest.
It is in System.dll.
Documentation: http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx
HttpRequest is located in System.Web, which is probably what you were thinking of.

HttpWebRequest is in the System assembly, not in System.Web (perhaps you're confusing with HttpRequest which is used in ASP.NET). It is available in all .NET Framework versions (including Silverlight, WP7, and Client Profile)