Trying to use Oracle Warehouse Cloud REST API through ASP.net C#. API Documentation
When I make a call to the Rest Service to the Object Inquiry API, I'm getting 2 errors:
IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
SocketException: An existing connection was forcibly closed by the remote host
Oracle Support directed me to Doc ID 967964.1 in their Support Library, which states SendChunked = true; has resolved the error before, but I haven't had luck adding it, nor do I feel it's appropriate for GET REST calls.
Here was the code I started with:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(HTTPsURL);
request.Method = "GET";
request.PreAuthenticate = true;
request.Credentials = cred;
request.ContentType = "application/xml";
using (var response = (HttpWebResponse)request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (var reader = new StreamReader(stream))
{
var content = reader.ReadToEnd();
return content;
}
I have been able to get a response back from SOAP UI and Postman. In Both cases I needed to set the Header Content-Type to "application/xml", and supply the authorization preemptively.
In SOAP UI my Request looks like this:
GET {My URL is HERE} HTTP/1.1
Accept-Encoding: gzip,deflate
Authorization: Basic { BASIC KEY }
Content-Type: application/xml
Host: ta3.wms.ocs.oraclecloud.com
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
When I try watching my .Net code through Fiddler, I'm not seeing the Content-Type being passed from the .Net Application. Is there something I'm missing there? Or is it possible I do need to pass the call in chunks?
When SendChunked = true, I get the error: Content-Length or Chunked Encoding cannot be set for an operation that does not write data
When I try passing data, I get the Error: Cannot send a content-body with this verb-type
A few things I've tried:
Modifying AutomaticDecompression
Modifying the Security Protocol
Transfer Encoding gzip,deflate
Enable/Disable Auto Redirect
And several variations of: Accept, KeepAlive, UserAgent, CachePolicy, ProtocolVersion
Perhaps It's not possible with the HttpWebRequest. Is there a better method I should try employing?
My Final requirements are to get data back from this call, then kick off other .Net processes.
This sounds like a TLS 1.2 issue, especially since it works in Postman, but not in .Net. Older versions of .Net don't automatically use TLS 1.2 and try to authenticate using older protocols and promptly get rejected.
There are lots of fixes for this one, either registry on the server, or app specific, but you should understand what you are doing first so you don't shoot yourself in the foot. I suggest reading this article and trying some of the fixes.
Stack Overflow discussion
Related
I am using the HttpWebRequest class to implement a web crawler. I would like to use a proxy server with it which usually can be implemented like this:
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("https://www.example.com/");
webRequest.Proxy = new WebProxy(proxyUrl, proxyPort);
webRequest.Proxy.Credentials = new NetworkCredential(proxyPassword, proxyPassword);
HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
using (Stream stream = webResponse.GetResponseStream())
{
StreamReader reader = new StreamReader(stream, Encoding.UTF8);
Console.WriteLine(reader.ReadToEnd());
}
However, I need to submit an additional parameter to the proxy server. Because the target site uses HTTPS, the parameters must be submitted in the initial CONNECT request that establishes the secure HTTPS tunnel. Otherwise the proxy server is unable to read them due the end-to-end encryption.
The header for the initial CONNECT request should look like this:
CONNECT example.com:443 HTTP/1.1
X-CustomProperty: 12345
So far I was unable to find a solution to achieve this with neither the HttpWebRequest nor the WebClient class from the .NET Framework. Is there any option to pass additional headers in the initial CONNECT request? I am also happy with a solution that uses an additional library.
I wrote a xml grabber to receive/decode xml files from website. It works fine mostly but it always return error:
"The remote server returned an error: (403) Forbidden."
for site http://w1.weather.gov/xml/current_obs/KSRQ.xml
My code is:
CookieContainer cookies = new CookieContainer();
HttpWebRequest webRequest = (HttpWebRequest)HttpWebRequest.Create(Path);
webRequest.Method = "GET";
webRequest.CookieContainer = cookies;
using (HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse())
{
using (StreamReader streamReader = new StreamReader(webResponse.GetResponseStream()))
{
string xml = streamReader.ReadToEnd();
xmldoc.LoadXml(xml);
}
}
And the exception is throw in GetResponse method. How can I find out what happened?
It could be that your request is missing a header that is required by the server. I requested the page in a browser, recorded the exact request using Fiddler and then removed the User-Agent header and reissued the request. This resulted in a 403 response.
This is often used by servers in an attempt to prevent scripting of their sites just like you are doing ;o)
In this case, the server header in the 403 response is "AkamaiGHost" which indicates an edge node from some cloud security solution from Akamai. Maybe a WAF rule to prevent bots is triggering the 403.
It seems like adding any value to the User-Agent header will work for this site. For example I set it to "definitely-not-a-screen-scraper" and that seems to work fine.
In general, when you have this kind of problem it very often helps to look at the actual HTTP requests and responses using browser tools or a proxy like Fiddler. As Scott Hanselman says
The internet is not a black box
http://www.hanselman.com/blog/TheInternetIsNotABlackBoxLookInside.aspx
Clearly, the URL works from a browser. It just doesn't work from the code. It would appear that the server is accepting/rejecting requests based on the user agent, probably as a very basic way of trying to prevent crawlers.
To get through, just set the UserAgent property to something it will recognize, for instance:
webRequest.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36";
That does seem to work.
In my particular case, it was not the UserAgent header, but the Accept header that the server didn't like.
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
You can use the browsers network tab of dev tools to see what the correct headers should be.
Is your request going through a proxy server? If yes, add the following line before your GetResponse() call.
webRequest.Proxy.Credentials = System.Net.CredentialCache.DefaultCredentials;
I am attempting to send a PUT request to a URI defined by the documentation of an API. This documentation states you only need to send a PUT request to the URL to reactivate an account.
However, when I send a PUT request to this URL with appropriate authentication I receive a 411 length required response. After reading around I found that this can be bypassed by setting the content-length of my PUT request to 0, which works in bypassing the 411 and DOES successfully reactivate the account, however I now receive a 500 internal server error response.
The documentation (and the technical support staff) state the PUT request is to be empty, but with content-length 0. The staff have also been unable to reproduce my issue, but as they do not support C# for the API anymore I assume they are just sending cURL requests / using PHP or similar which is not an option for me.
Within C# I have tried with WebClient, HTTPWebRequest and even Raw sockets to attempt to get this working fully (I need the response code for verification within my code) but to no avail.
The API does not support C# so I'm sailing blind, does anyone know of a way to perhaps replicate an empty entity PUT cURL to a url within C# or know of any other potential workaround?
Edit: Examples of code tried:
HttpWebRequest tried, returns 500 server error.
Remove Content-length header to produce 411
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Headers.Add("Authorization", "Basic " + encodeB64);
request.Method = "PUT";
request.ContentType = "text/XML";
request.ContentLength = 0;
}
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
WebClient method (quickly typed out, might be a little wrong I've now scrapped this code so this is just an idea of what it was, it worked but also returned server error (500)
try
{
WebClient wc = new WebClient();
wc.Headers.Add("Authorization", "Basic " + encodeB64);
wc.UploadString(uri, "PUT", "");
String status = wc.ResponseHeaders.Get("Status");
}
Inspected outgoing and incoming requests with Fiddler, also attempted to compose the request within Fiddler, same issue either method, needing content length or server error, so I am beginning to think that the server I am contacted itself is the issue, not me.
Be aware these requests DID reactivate the subscription despite the 500 error code
Fiddler raw request:
PUT https://xxxxxx.recurly.com/v2/subscriptions/xxxxxxxxxxxxxxxxxxxxxxx/reactivate HTTP/1.1
Authorization: Basic Y2U0OWZlZGNjMGU3NDEXuIOSmNNYxSakYTU4MzBhZDE0YmU=
Host: xxxxx.recurly.com
Content-Length: 0
Response raw:
HTTP/1.1 500 Internal Server Error
Server: blackhole
Date: Thu, 12 Sep 2013 10:38:18 GMT
Content-Type: text/html
Content-Length: 2482
Connection: close
X-Request-Id: 69192eb570323c495bb605b432e424fb
I have Managed to Setup II7 with Gzip Compression .
I can check via web sniffer that my asmx web service encoding is Gzip but how to i enable
gzip Compression on my C# Client , i am using the Web Service is Service reference in my application.
Actually i am trying to send large amount of data , 10k objects of array so Compression with be great effect on bw.
but how do I enable Compression on my C# Client.
i am trying to see that many people sees same problem but there nothing clear answer some says use third party tools or some says about custom headers etc etc .
is not there any proper way , built in to consume Compressed web service
As #Igby Largeman pointed out, you can use your IIS7 to enable the compression on the server, but this is not enough.
The main idea is to set the headers on the client side and server side:
Client:
Accept-Encoding = "gzip, deflate";
You can achieve this by code:
var request = HttpWebRequest.Create("http://foofoo");
request.Headers["Accept"] = "application/json";
request.Headers["Accept-Encoding"] = "gzip, deflate";
or
var request = HttpWebRequest.Create("http://foofoo");
request.AutomaticDecompression = DecompressionMethods.GZip |
DecompressionMethods.Deflate;
If you use some WCF client, and not the HttpWebRequest, you should use custom inspector and dispatcher, like in this article:
So I used a message inspector implementing IClientMessageInspector and IDispatchMessageInspector to automatically set the AcceptEncoding and ContentEncoding http headers.
This was working perfectly but I could not achieve to decompress the response on the server by first detecting the ContentEncoding header thus I used the work around to first try to decompress it and if it fails just try to process the request as normal.
I also did this in the client pipeline and this also works.
Server:
// This is the nearly same thing after all
Content-Encoding = "gzip" OR Content-Encoding = "deflate"
To do this on the Server side, you should enable httpCompression in the IIS.
I think you should check the original article to get this work
I need a C# HTTP library that doesn't depend on HttpWebRequest, as I can't access this from the environment I need to run my code (Unity's WebPlayer).
Ideally this would be light weight, but any suggestions are welcome!
I just need to be able to do simple HTTP GET and POST requests, but better REST support would be good.
Thanks!
Edit: As people have pointed out, HttpWebRequest isn't in System.Web, but the fact remains - I can't use it. I've updated my post above.
This post
http://forum.unity3d.com/threads/24445-NotSupportedException-System.Net.WebRequest.GetCreator shows the same error I'm getting.
Implementing your own simple HTTP client using Socket is not all that difficult.
Just use TcpClient().
For the protocol itself, drop down to a connection-per-request paradigm. A typical GET request would look as follows:
GET /url HTTP/1.1
Host: <hostname-of-server>
Connection: close
For the code itself (from memory)
TcpClient client = new TcpClient();
IPEndPoint target = ... // get an endpoint for the target using DNS class
client.Connect(target);
using(NetworkStream stream = client.GetStream())
{
// send the request.
string request = "GET /url HTTP/1.1\r\nConnection: close\r\n\r\n";
stream.Write(Encoding.ASCII.GetBytes(request));
// then drain the stream to get the server response
}
Note that you will need to wrap this code with a simple class that provides HTTPWebRequest like semantics.
Look at System.Net.HttpWebRequest.
It is in System.dll.
Documentation: http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx
HttpRequest is located in System.Web, which is probably what you were thinking of.
HttpWebRequest is in the System assembly, not in System.Web (perhaps you're confusing with HttpRequest which is used in ASP.NET). It is available in all .NET Framework versions (including Silverlight, WP7, and Client Profile)