All I'm trying to do is create a program that gets a web response from Nike's upcoming shoe's page, however I keep running into an error saying this is forbidden. No other threads on this topic have been of use to me, is there anything I can do for this or am I just screwed? This is the code:
WebRequest request = WebRequest.Create("https://www.nike.com/launch/?s=upcoming");
WebResponse response = request.GetResponse();
and this is the error:
System.Net.WebException: 'The remote server returned an error: (403) Forbidden.'
Seems like a header issue, try this:
WebClient client = new WebClient();
client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
client.Headers.Add("Content-Type", "application / zip, application / octet - stream");
client.Headers.Add("Referer", "http://whatevs");
client.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
String someStuff = client.DownloadString("https://www.hassanhabib.com");
Console.WriteLine(someStuff);
Console.Read();
Removed the Accept-Encoding line, should be fine now.
Related
I currently have this code that is supposed to grab the HTML source of the website. Specifically, I am telling it to read the source of 4chan. It WILL get the source code for a board, such as /pol/ or /news/, but it will NOT get the source code for specific threads. It throws the error: [System.Net.WebException: 'The remote server returned an error: (403) Forbidden.']
Here is the code I am working with.
public string GetSource(string url)
{
WebClient client = new WebClient();
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12; //tried with & without this
client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/6.0;)");
try
{
return client.DownloadString(url);
}
catch
{
Error(2); //error code 2
}
return "";
}
It will download the source of "https://boards.4chan.org/pol" for example.
It will not download the source of "https://boards.4chan.org/pol/thread/#"
I am completely lost as to how to proceed. I have a "user-agent" tag, and it works sometimes, so I don't know what the problem is. Any help would be appreciated. Thanks.
Code which i tried :
string contents = string.Empty;
using (var wc = new System.Net.WebClient())
{
contents = wc.DownloadString("http://www.bizjournals.com/albany/blog/health-care/2015/10/what-this-local-bank-did-to-control-health-care.html");
}
but its throwing error
The remote server returned an error: (416) Requested Range Not
Satisfiable
It appears that some webservers may return a 416 if your client does not send a User-Agent header. Try adding the header like this:
wc.Headers.Add("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705");
Based on the this sourcecode I'm not able to retrieve the data from the API into XDocument.
I retrieve the error message
{"The remote server returned an error: (400) Bad Request."}
Question:
I don't know what to do?
XDocument xml = XDocument.Parse(new
WebClient().DownloadString("http://api.arbetsformedlingen.se/af/v0/platsannonser/matchning?lanid=1&kommunid=180&yrkesid=2419&1&antalrader=10000"));
You need to send HTTP headers:
using (WebClient client = new WebClient())
{
client.Headers.Add("Accept-Language", " en-US");
client.Headers.Add("Accept", "application/xml");
client.Headers.Add("User-Agent", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)");
XDocument xml = XDocument.Parse(client.DownloadString("http://api.arbetsformedlingen.se/af/v0/platsannonser/matchning?lanid=1&kommunid=180&yrkesid=2419&1&antalrader=10000"));
}
Trying To Loading Html Content Of "http://links.casemakerlegal.com/states/CA/books/Case_Law/results?search[Cite]=214 Cal.App.3d 533" but HttpWebResponse object Giving This Error "(500) Internal Server Error"
And Code Is------
request = WebRequest.Create(urlCheck); request.Timeout = 100000; response = request.GetResponse(); strmRead = new StreamReader(response.GetResponseStream(),System.Text.Encoding.UTF8); result = strmRead.ReadToEnd();
You need to use a tool like Wireshark or Ethereal, or the developer tools in your browser to investigate this further. It is likely the browser is sending some values in the HTTP Header that your code is not, and the server is returning a 500 due to these missing values. Try replicating all of the headers that the browser is using in your code to see if this resolves the problem.
It is usually browser agent. try adding a valid browser agent to your request headers along with Accept and Accept-Encoding headers,
*Edit: For example:
request.UserAgent = "Mozilla/5.0 (Windows NT 5.2; WOW64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1";
request.Headers.Add("Accept-Encoding: gzip, deflate");
request.Headers.Add("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
request.Headers.Add("Accept-Language: en;q=0.8");
I'm attempting to write a small screen-scraping tool for statistics aggregation in c#. I have attempted to use this code, (posted many times here but again for detail):
public static string GetPage(string url)
{
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)";
WebResponse response = (HttpWebResponse) request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
string result = reader.ReadToEnd();
stream.Dispose();
reader.Dispose();
return result;
}
However, some (not all) websites I attempt to connect to that use Ajax or server side includes throw NameResolutionFailure exceptions and cannot read the data.
An example of this is : pgatour stats
I am led to believe the HttpWebRequest class emulates a browser when requesting information so you get the post-generated HTML. Currently, the only way I can read the data is making an iMacro that grabs it from the page source after it runs through the browser. As said before, it works in the browser so I don't think the error is related to a DNS issue and the website does generate a response (.haveresponse is set).
Has anyone else encountered this issue and what did you use tor resolve it?
Thanks.