I am using the HttpWebRequest class to implement a web crawler. I would like to use a proxy server with it which usually can be implemented like this:
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("https://www.example.com/");
webRequest.Proxy = new WebProxy(proxyUrl, proxyPort);
webRequest.Proxy.Credentials = new NetworkCredential(proxyPassword, proxyPassword);
HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
using (Stream stream = webResponse.GetResponseStream())
{
StreamReader reader = new StreamReader(stream, Encoding.UTF8);
Console.WriteLine(reader.ReadToEnd());
}
However, I need to submit an additional parameter to the proxy server. Because the target site uses HTTPS, the parameters must be submitted in the initial CONNECT request that establishes the secure HTTPS tunnel. Otherwise the proxy server is unable to read them due the end-to-end encryption.
The header for the initial CONNECT request should look like this:
CONNECT example.com:443 HTTP/1.1
X-CustomProperty: 12345
So far I was unable to find a solution to achieve this with neither the HttpWebRequest nor the WebClient class from the .NET Framework. Is there any option to pass additional headers in the initial CONNECT request? I am also happy with a solution that uses an additional library.
Related
Trying to use Oracle Warehouse Cloud REST API through ASP.net C#. API Documentation
When I make a call to the Rest Service to the Object Inquiry API, I'm getting 2 errors:
IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
SocketException: An existing connection was forcibly closed by the remote host
Oracle Support directed me to Doc ID 967964.1 in their Support Library, which states SendChunked = true; has resolved the error before, but I haven't had luck adding it, nor do I feel it's appropriate for GET REST calls.
Here was the code I started with:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(HTTPsURL);
request.Method = "GET";
request.PreAuthenticate = true;
request.Credentials = cred;
request.ContentType = "application/xml";
using (var response = (HttpWebResponse)request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (var reader = new StreamReader(stream))
{
var content = reader.ReadToEnd();
return content;
}
I have been able to get a response back from SOAP UI and Postman. In Both cases I needed to set the Header Content-Type to "application/xml", and supply the authorization preemptively.
In SOAP UI my Request looks like this:
GET {My URL is HERE} HTTP/1.1
Accept-Encoding: gzip,deflate
Authorization: Basic { BASIC KEY }
Content-Type: application/xml
Host: ta3.wms.ocs.oraclecloud.com
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
When I try watching my .Net code through Fiddler, I'm not seeing the Content-Type being passed from the .Net Application. Is there something I'm missing there? Or is it possible I do need to pass the call in chunks?
When SendChunked = true, I get the error: Content-Length or Chunked Encoding cannot be set for an operation that does not write data
When I try passing data, I get the Error: Cannot send a content-body with this verb-type
A few things I've tried:
Modifying AutomaticDecompression
Modifying the Security Protocol
Transfer Encoding gzip,deflate
Enable/Disable Auto Redirect
And several variations of: Accept, KeepAlive, UserAgent, CachePolicy, ProtocolVersion
Perhaps It's not possible with the HttpWebRequest. Is there a better method I should try employing?
My Final requirements are to get data back from this call, then kick off other .Net processes.
This sounds like a TLS 1.2 issue, especially since it works in Postman, but not in .Net. Older versions of .Net don't automatically use TLS 1.2 and try to authenticate using older protocols and promptly get rejected.
There are lots of fixes for this one, either registry on the server, or app specific, but you should understand what you are doing first so you don't shoot yourself in the foot. I suggest reading this article and trying some of the fixes.
Stack Overflow discussion
I am trying to make a REST request to a server via proxy using UnityWebRequest. I have tried to use
Network.useProxy = true;
but this fails with the error
Cannot connect to destination host
Network.useProxy has been removed in Unity 2018.2.12. I am able to connect over Postman and curl. Can anybody confirm that UnityWebRequest does not support connection via proxy?
Even if it hasn't been deprecated, it wouldn't still work because Unity's Network.useProxy is used for the legacy networking system and the UnityWebRequest API is not part of that.
There is no support for proxy with UnityWebRequest and there is no plan to add support for this on Unity roadmap. Vote for its support here.
Your only workaround is to use one of the standard C# web request API such as HttpWebRequest and WebClient. They are supported in Unity. With HttpWebRequest, you can use proxy as below:
string proxyHost = "192.168.1.3";
int proxyPort = 8080;
string url = "http://YourUrl.com/blah";
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Proxy = new WebProxy(proxyHost, proxyPort);
Since this is not UnityWebRequest, you have to do that in another Thread to prevent blocking your game main Thread.
I notice something between the v5.2 and 5.1 of the Microsoft.NETCore.UniversalWindowsPlatform
I have an APP in v5.1 and a do a "POST" opperation whith a HttpWebRequest , with parameter in the body encode in UTF8. (to simulate a login on a webSite)
If a upgrade to v5.2, my POST stop working !
I restore to v5.1 and it work again !
Have you got an idea ?
mycode
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlrest.AbsoluteUri); request.CookieContainer = new CookieContainer();
ASCIIEncoding encoding = new ASCIIEncoding();
//Encoding.UTF8.GetBytes(postString);
byte[] postData = Encoding.UTF8.GetBytes(allparaminbody); //allparamin_body contain : user=myuser&password=mypassword
request.ContentType = "POST";
request.Headers["User-Agent"] = "Apache-HttpClient/4.1.1 (java 1.5)";
Stream newStream = await request.GetRequestStreamAsync();
newStream.Write(postData, 0, postData.Length);
newStream.Dispose();
response = (HttpWebResponse)await request.GetResponseAsync();
Thanks
HttpWebRequest is not recommended to use in UWP apps. Ref Demystifying HttpClient APIs in the Universal Windows Platform:
As a Universal Windows Platform (UWP) app developer, if you are trying to communicate over HTTP with a web service or any server endpoint, you have multiple API choices. Two of the most used and recommended APIs for implementing the HTTP client role in a managed UWP app are System.Net.Http.HttpClient and Windows.Web.Http.HttpClient. These APIs should be preferred over older, discouraged APIs such as WebClient and HttpWebRequest (although a small subset of HttpWebRequest is available in UWP for backward compatibility).
Also, you can refer to my previous answer here. According to your requirement, you can choose either System.Net.Http.HttpClient or Windows.Web.Http.HttpClient. And here is a official HttpClient sample on GitHub that uses Windows.Web.Http namespace.
Besides, HttpWebRequest.ContentType should be the MIME type like x-www-form-urlencoded or text/html. POST ia not a valid value. It should be the value of HttpWebRequest.Method Property.
I've contacted Microsoft, there is a BUG in the v5.2, they're going to correct it asap.
=> https://github.com/dotnet/corefx/issues/9003
cdt
I have requirement of having proxy server where request from browser will be forwarded to my custom proxy server. Afterwards Proxy server will read request from client and talk to requested resource using some thing like this --
HttpWebRequest request =(HttpWebRequest)WebRequest.Create ("http://www.google.com");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
then once I have response back in response object write returned HTML to browser , at this point I am stuck and do not know how to write information to browser ?
Can someone please help ???
I'm using an external api which allows the request body of a PUT request to be streamed.
I've been using a TcpClient to achieve this and it works fine but now we have a client that has a proxy. So I figured I'd switch to using either the HttpClient in .net of the WebRequest api.
But I haven't been able to write directly to a request body stream that actually gets sent to the server immediately.
You can get the stream of a WebRequest like this:
var req = WebRequest.Create("url");
var stream = req.GetRequestStream();
But this will send what you've written to the stream after you call GetResponse. And I'm not really interested in the response.
Is it possible to write directly to the server using a stream?