I wrote a xml grabber to receive/decode xml files from website. It works fine mostly but it always return error:
"The remote server returned an error: (403) Forbidden."
for site http://w1.weather.gov/xml/current_obs/KSRQ.xml
My code is:
CookieContainer cookies = new CookieContainer();
HttpWebRequest webRequest = (HttpWebRequest)HttpWebRequest.Create(Path);
webRequest.Method = "GET";
webRequest.CookieContainer = cookies;
using (HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse())
{
using (StreamReader streamReader = new StreamReader(webResponse.GetResponseStream()))
{
string xml = streamReader.ReadToEnd();
xmldoc.LoadXml(xml);
}
}
And the exception is throw in GetResponse method. How can I find out what happened?
It could be that your request is missing a header that is required by the server. I requested the page in a browser, recorded the exact request using Fiddler and then removed the User-Agent header and reissued the request. This resulted in a 403 response.
This is often used by servers in an attempt to prevent scripting of their sites just like you are doing ;o)
In this case, the server header in the 403 response is "AkamaiGHost" which indicates an edge node from some cloud security solution from Akamai. Maybe a WAF rule to prevent bots is triggering the 403.
It seems like adding any value to the User-Agent header will work for this site. For example I set it to "definitely-not-a-screen-scraper" and that seems to work fine.
In general, when you have this kind of problem it very often helps to look at the actual HTTP requests and responses using browser tools or a proxy like Fiddler. As Scott Hanselman says
The internet is not a black box
http://www.hanselman.com/blog/TheInternetIsNotABlackBoxLookInside.aspx
Clearly, the URL works from a browser. It just doesn't work from the code. It would appear that the server is accepting/rejecting requests based on the user agent, probably as a very basic way of trying to prevent crawlers.
To get through, just set the UserAgent property to something it will recognize, for instance:
webRequest.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36";
That does seem to work.
In my particular case, it was not the UserAgent header, but the Accept header that the server didn't like.
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
You can use the browsers network tab of dev tools to see what the correct headers should be.
Is your request going through a proxy server? If yes, add the following line before your GetResponse() call.
webRequest.Proxy.Credentials = System.Net.CredentialCache.DefaultCredentials;
Related
This is something I have never seen before. When executing an identical request to ASP.NET Web API 2 from two different applications I get a different response. I narrowed it down to ClaimsPrincipal.Current being the culprit.
var principal = ClaimsPrincipal.Current;
if (principal == null) return false;
if (!principal.Identity.IsAuthenticated) return false;
According to MSDN ClaimsPrincipal.Current just calls Thread.CurrentPrincipal by default but I still do not understand how this can happen.
https://learn.microsoft.com/en-us/aspnet/core/migration/claimsprincipal-current?view=aspnetcore-2.1
I then tried to use System.Web.HttpContext.Current.User instead but this gives the same result.
How can two identical HTTP Requests from the same machine generate different response in this case? The requests can be sent over and over again with the same result. I can even stop the application and IIS Express, start it again and the result is the same. What is happening here? It must be some sort of session that is stored on the server but I don't get why the values differ with an identical requests. There is nothing misspelled in the request itself, I can copy the request generated by Postman and it works with BURP, if it is sent via Postman it fails. I don't think it is Postman specific either. I used the command Copy as PowerShell from Chrome Developer Tools -> Network tab from a working request and I got the same result with Invoke-WebRequest.
IIS uses Anonymous Authentication. The application uses IAppBuilder - app.UseCookieAuthentication with AuthenticationType = DefaultAuthenticationTypes.ApplicationCookie and a custom CookieName.
With Burp the request gives principal.Identity.IsAuthenticated = true
With Postman the exact same request gives principal.Identity.IsAuthenticated = false. Postman Gui does a line break on - but the copied value works in Burp so nothing wrong there.
Update:
Postman cookie value in UTF-8:
XXX=_1gQcJZ_zwNNS6f5OO0mD5y4pPHATpzw7uRHQZnZidNfYYec9S3MkR-d9aaxx1AilQSCK_h1-9LVS1uVM_JLJDTty5Nilsx4njjOCsrefgBOvnkt9CIzt_fGu0kzgsi_VbrCSO-txXtLhrOBT61bFskQd0i2yF_xrnqdOoW6yzKmUPrdomxiABMsC-NYw5aSGD9d81ht-oreUGqJKoDQ7EJ0BzUc-Y6BDqrJv5TrIfdgwgOsk2cFN9gfrlN9DQQQpRAAEv5mgiXDmMpUpNvsP-k-CFu69sl1ZlTXOLR5ECSrq7woeIhea6-L9g1mwpslqAV_saLtv0DcbR525gR0tSrpEIuHLwj_TSqTQ1IPHqfcqSP-RzP2jGoz85y6W2glFkfFxAXJBMTjoz4U1fvjURL5qMEuC2IpQZqKGoSbp8xICFA01yY1zzHKxXnKL8MIqDNAe9urQn2W-gmwje9bzFAkft3eYYjctrCrGMRocgQ; __RequestVerificationToken=HOA5v8aiHqUhzZP3fkKMUyi336D7JydqWMSWI-VThQgMrVRZEllKglaGaLOUP0z49ZEuJsrEaYbrLaLCxMgAwxJtfSJhGvsRaB6e3tlMPjc1
BURP cookie value in UTF-8:
XXX=_1gQcJZ_zwNNS6f5OO0mD5y4pPHATpzw7uRHQZnZidNfYYec9S3MkR-d9aaxx1AilQSCK_h1-9LVS1uVM_JLJDTty5Nilsx4njjOCsrefgBOvnkt9CIzt_fGu0kzgsi_VbrCSO-txXtLhrOBT61bFskQd0i2yF_xrnqdOoW6yzKmUPrdomxiABMsC-NYw5aSGD9d81ht-oreUGqJKoDQ7EJ0BzUc-Y6BDqrJv5TrIfdgwgOsk2cFN9gfrlN9DQQQpRAAEv5mgiXDmMpUpNvsP-k-CFu69sl1ZlTXOLR5ECSrq7woeIhea6-L9g1mwpslqAV_saLtv0DcbR525gR0tSrpEIuHLwj_TSqTQ1IPHqfcqSP-RzP2jGoz85y6W2glFkfFxAXJBMTjoz4U1fvjURL5qMEuC2IpQZqKGoSbp8xICFA01yY1zzHKxXnKL8MIqDNAe9urQn2W-gmwje9bzFAkft3eYYjctrCrGMRocgQ; __RequestVerificationToken=HOA5v8aiHqUhzZP3fkKMUyi336D7JydqWMSWI-VThQgMrVRZEllKglaGaLOUP0z49ZEuJsrEaYbrLaLCxMgAwxJtfSJhGvsRaB6e3tlMPjc1
Update 2:
From the command Copy as PowerShell from Chrome Developer Tools -> Network tab.
Invoke-WebRequest -Uri "https://localhost:44349/api/crud/customer" -Headers #{"path"="/api/crud/custo
mer"; "pragma"="no-cache"; "cookie"="__RequestVerificationToken=3gvrynl8SRhi5CBG-umg5eGii3yUOrHJAQQ7jMXhN_hOk0EGS2XdIDIS
afhbBZuS3JCCJdP6V60K_crzcQF71aw2totf9CUTPheHBmTNBRM1; io=iki1JghnuzWahlUBAAAJ; XXX=SUdlUpzYNbXJbhPxj4KY6-GC31hHyyPN_IZ88
zsXHXIpqzro6t_C5-m8BC_s2xev5SINoI-0316o7ITb6dsRA5b5oYJX2MXIWD2iaMWGADqAZeLDLoeQPHo6B6a8dQ-j2YkI17I4cjQ7SQKBiUCwN3DIZckY8
HHnWqF6LGVr79nWG3R1pqI62S3UKgEXOjhFTpEA3fD3clPti4ShG88PWnxa5ypGGDjUolcqjkusylpLAWZ3Jc8K4y-K_WnA-3EX_nNyCHp3Tk8omXHq1LgvQ
J3EsqdNvELL2KcwvUCn3ni7ktSt0Vzl6G7vL3AfZhDQb41bn90l4haR9UGvLOqSkZ_cu5IiHzvsFrps6QJ3HJ8d-Dcb4A2soVjnozh7SsZxnz-HppwhV2UaW
ANvi6MsD4kwvBreJrO9nLMOBRBXhzEInoL0baqkn_nhEtxqAndZHiHcbuoPfz8xGmgV-ilTxZRAnJ8ZAwD3yHREgJsodVg"; "accept-encoding"="gzip
, deflate, br"; "accept-language"="en-US,en;q=0.9,sv-SE;q=0.8,sv;q=0.7"; "user-agent"="Mozilla/5.0 (Windows NT 10.0; Win
64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36"; "accept"="*/*"; "cache-control"="no
-cache"; "authority"="localhost:44349"; "referer"="https://localhost:44349/"; "scheme"="https"; "xsrf-token"="_GHoZagVRo
FBIAyoMmT7UEZk44wfKsGlscub-bvoeRMTpysPS_d2uccvyyvPdWDf7srVfmNqM4JN1firyN-Q35UN5DCMew0eq6OV9M_4--i_klYEJcXYSodFi_wAymDVlQ
CPLroCvDNkwuhdoZvyug2"; "method"="GET"}
Huge thanks to #CodeCaster. Sometimes you feel like a n00b all over. Looked at System.Web.HttpContext.Current.Request.Cookies and indeed these were empty. Hovered over the Cookie header in Postman and then I saw the value Restricted Header (use Postman Interceptor). What really got me here was that Invoke-WebRequest in Powershell got the same error code.
Upgraded to the Postman native app instead of using the Chrome Application and then everything worked.
Trying to use Oracle Warehouse Cloud REST API through ASP.net C#. API Documentation
When I make a call to the Rest Service to the Object Inquiry API, I'm getting 2 errors:
IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
SocketException: An existing connection was forcibly closed by the remote host
Oracle Support directed me to Doc ID 967964.1 in their Support Library, which states SendChunked = true; has resolved the error before, but I haven't had luck adding it, nor do I feel it's appropriate for GET REST calls.
Here was the code I started with:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(HTTPsURL);
request.Method = "GET";
request.PreAuthenticate = true;
request.Credentials = cred;
request.ContentType = "application/xml";
using (var response = (HttpWebResponse)request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (var reader = new StreamReader(stream))
{
var content = reader.ReadToEnd();
return content;
}
I have been able to get a response back from SOAP UI and Postman. In Both cases I needed to set the Header Content-Type to "application/xml", and supply the authorization preemptively.
In SOAP UI my Request looks like this:
GET {My URL is HERE} HTTP/1.1
Accept-Encoding: gzip,deflate
Authorization: Basic { BASIC KEY }
Content-Type: application/xml
Host: ta3.wms.ocs.oraclecloud.com
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
When I try watching my .Net code through Fiddler, I'm not seeing the Content-Type being passed from the .Net Application. Is there something I'm missing there? Or is it possible I do need to pass the call in chunks?
When SendChunked = true, I get the error: Content-Length or Chunked Encoding cannot be set for an operation that does not write data
When I try passing data, I get the Error: Cannot send a content-body with this verb-type
A few things I've tried:
Modifying AutomaticDecompression
Modifying the Security Protocol
Transfer Encoding gzip,deflate
Enable/Disable Auto Redirect
And several variations of: Accept, KeepAlive, UserAgent, CachePolicy, ProtocolVersion
Perhaps It's not possible with the HttpWebRequest. Is there a better method I should try employing?
My Final requirements are to get data back from this call, then kick off other .Net processes.
This sounds like a TLS 1.2 issue, especially since it works in Postman, but not in .Net. Older versions of .Net don't automatically use TLS 1.2 and try to authenticate using older protocols and promptly get rejected.
There are lots of fixes for this one, either registry on the server, or app specific, but you should understand what you are doing first so you don't shoot yourself in the foot. I suggest reading this article and trying some of the fixes.
Stack Overflow discussion
I started off with the simple code below in order to grab the html from webpages into a string to later process. For some sites like Digikey it works but for others like Mouser it doesn't.
I have tried putting headers and userAgents onto the WebClient along with converting the url to a Uri with no success. Does anybody have any other suggestions of what I could try? Or could anybody try to get the code to work and let me know how it goes?
String url = "http://www.mouser.com/ProductDetail/Vishay-Thin-Film/PCNM2512E1000BST5/?
qs=sGAEpiMZZMu61qfTUdNhG6MW4lgzyHBgo9k7HJ54G4u10PG6pMa7%252bA%3d%3d"
WebClient web = new WebClient();
String html = web.DownloadString(url);
MessageBox.Show(html);
EDIT : The link should lead here: link
EDIT : I tried the following chunk of code with no luck:
String url = "http://www.mouser.com/ProductDetail/Vishay-Thin-Film/PCNM2512E1000BST5/?
qs=sGAEpiMZZMu61qfTUdNhG6MW4lgzyHBgo9k7HJ54G4u10PG6pMa7%252bA%3d%3d"
WebClient web = new WebClient();
web.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
String html = web.DownloadString(url);
MessageBox.Show(html);
Need to download Fiddler it's free (was originally developed by Microsoft) and it lets you record browser sessions. So launch it open chrome or whatever your browser is and go though the steps. Once you done you can stop it and look at every request and response and the raw data sent.
Makes it easy to spot the difference between your code and the browser.
There are also many free tools that will take your request/response data and generate the C# code for you such as Request To Code. That is not the only one, I'm not at work and I can't recall the one I use there, but there are plenty to choose from.
Hope this helps
I am attempting to send a PUT request to a URI defined by the documentation of an API. This documentation states you only need to send a PUT request to the URL to reactivate an account.
However, when I send a PUT request to this URL with appropriate authentication I receive a 411 length required response. After reading around I found that this can be bypassed by setting the content-length of my PUT request to 0, which works in bypassing the 411 and DOES successfully reactivate the account, however I now receive a 500 internal server error response.
The documentation (and the technical support staff) state the PUT request is to be empty, but with content-length 0. The staff have also been unable to reproduce my issue, but as they do not support C# for the API anymore I assume they are just sending cURL requests / using PHP or similar which is not an option for me.
Within C# I have tried with WebClient, HTTPWebRequest and even Raw sockets to attempt to get this working fully (I need the response code for verification within my code) but to no avail.
The API does not support C# so I'm sailing blind, does anyone know of a way to perhaps replicate an empty entity PUT cURL to a url within C# or know of any other potential workaround?
Edit: Examples of code tried:
HttpWebRequest tried, returns 500 server error.
Remove Content-length header to produce 411
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Headers.Add("Authorization", "Basic " + encodeB64);
request.Method = "PUT";
request.ContentType = "text/XML";
request.ContentLength = 0;
}
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
WebClient method (quickly typed out, might be a little wrong I've now scrapped this code so this is just an idea of what it was, it worked but also returned server error (500)
try
{
WebClient wc = new WebClient();
wc.Headers.Add("Authorization", "Basic " + encodeB64);
wc.UploadString(uri, "PUT", "");
String status = wc.ResponseHeaders.Get("Status");
}
Inspected outgoing and incoming requests with Fiddler, also attempted to compose the request within Fiddler, same issue either method, needing content length or server error, so I am beginning to think that the server I am contacted itself is the issue, not me.
Be aware these requests DID reactivate the subscription despite the 500 error code
Fiddler raw request:
PUT https://xxxxxx.recurly.com/v2/subscriptions/xxxxxxxxxxxxxxxxxxxxxxx/reactivate HTTP/1.1
Authorization: Basic Y2U0OWZlZGNjMGU3NDEXuIOSmNNYxSakYTU4MzBhZDE0YmU=
Host: xxxxx.recurly.com
Content-Length: 0
Response raw:
HTTP/1.1 500 Internal Server Error
Server: blackhole
Date: Thu, 12 Sep 2013 10:38:18 GMT
Content-Type: text/html
Content-Length: 2482
Connection: close
X-Request-Id: 69192eb570323c495bb605b432e424fb
I need a web browser engine for using in my C# application (same as 'WebBrowser' of .Net). I need an engine which can give me back a cookie collection per request but with its full details such as path/domain etc...
I tried many browser engines but couldn't managed to find one that can give me back this trivial thing I need, actually, the 'WebBrowser' of .Net is perfect for me but I can only get the cookie name and value from it.
Does anyone know of such browser engine that can do this work?
I would appreciate any help.
As I remember the HttpWebRequest can do the job for you:
HttpWebRequest request =
(HttpWebRequest)WebRequest.Create("http://your.site");
request.Method = "GET";
request.CookieContainer = new CookieContainer();
HttpWebResponse response = loginReq.GetResponse() as HttpWebResponse;
CookieCollection cookies = response.Cookies;
The cookies will contain all the cookies from the site.