I want to run a URL and get the result.
I used the below code but it does not work correctly. It just returns the main website URL as a result.
this is the main page which has a blue box is called filenext. I get its link and this is what I want to get as a result.
WebClient client = new WebClient();
client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
WebClient getNitroflareLink = new WebClient();
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
Stream stream = response.GetResponseStream();
You should set Refererfrom the first URL for the second one.
var page = firstLink.LoadIt();
HttpWebRequest request = WebRequest.Create(Refererlink) as HttpWebRequest;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36";
request.Referer = firstLink;
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
Related
I guys, i need to make a POST request but i been rejected with a Remote server Error: (416) Requested Range Not Satisfiable. with a request structured like this:
req.Method = "POST";
req.Host = "launches-api.endclothing.com";
req.KeepAlive = true;
req.ContentLength = sentData.Length;
req.Accept = "application/json, text/plain, */*";
req.ContentType = "application/json;charset=UTF-8";
req.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36";
req.Headers["Origin"] = "https://launches.endclothing.com";
req.Headers["Sec-Fetch-Site"] = "same-site";
req.Headers["Sec-Fetch-Mode"] = "cors";
req.Headers["Sec-Fetch-Dest"] = "empty";
req.Headers["Access-Control-Allow-Origin"] = "https://launches.endclothing.com";
req.Referer = "https://launches.endclothing.com/";
req.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate, br";
req.Headers[HttpRequestHeader.AcceptLanguage] = "it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7";
The original POST request sniffed from Fiddler got some extra-headers which change at every request and them are mentioned in the OPTIONS package sent before making the request.
Fiddler POST request
OPTIONS package
So i don't know where to take that Headers i am missing in my request.
Can you please give me a tip?
I am trying to download the HTML from a site and parse it. I am actually interested in the OpenGraph data in the head section only. For most sites using the WebClient, HttpClient or HtmlAgilityPack works, but some domains I get 403, for example: westelm.com
I have tried setting up the Headers to be absolutely the same as they are when I use the browser, but I still get 403. Here is some code:
string url = "https://www.westelm.com/m/products/brushed-herringbone-throw-t5792/?";
var doc = new HtmlDocument();
using(WebClient client = new WebClient()) {
client.Headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36";
client.Headers["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9";
client.Headers["Accept-Encoding"] = "gzip, deflate, br";
client.Headers["Accept-Language"] = "en-US,en;q=0.9";
doc.Load(client.OpenRead(url));
}
At this point, I am getting a 403.
Am I missing something or the site administrator is protecting the site from API requests?
How can I make this work? Is there a better way to get OpenGraph data from a site?
Thanks.
I used your question to resolve the same problem. IDK if you're already fixed this but I tell you how it worked for me
A page was giving me 403 for the same reasons. The thing is: you need to emulate a "web browser" from the code, sending a lot of headers.
I used one of yours headers I wasn't using (like Accept-Language)
I didn't use WebClient though, I used HttpClient to parse the webpage
private static async Task<string> GetHtmlResponseAsync(HttpClient httpClient, string url)
{
using var request = new HttpRequestMessage(HttpMethod.Get, new Uri(url));
request.Headers.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");
request.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, br");
request.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36");
request.Headers.TryAddWithoutValidation("Accept-Charset", "UTF-8");
request.Headers.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.9");
using var response = await httpClient.SendAsync(request).ConfigureAwait(false);
if (response == null)
return string.Empty;
using var responseStream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false);
using var decompressedStream = new GZipStream(responseStream, CompressionMode.Decompress);
using var streamReader = new StreamReader(decompressedStream);
return await streamReader.ReadToEndAsync().ConfigureAwait(false);
}
If it helps you, I'm glad. If not, I will leave this answer here to help someone else in the future!
I'm trying to login on a xbox live page, and got some problems with that, have no idea why, I think I set everything properly... here is my code
CookieCollection cookies = new CookieCollection();
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("https://account.xbox.com/en-US/PaymentAndBilling/RedeemCode");
Request.CookieContainer = new CookieContainer();
Request.CookieContainer.Add(cookies);
//Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko";
HttpWebResponse Response = (HttpWebResponse)Request.GetResponse();
Response.Cookies.Add(cookies);
Response.Close();
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create("https://login.live.com/");
getRequest.Method = "POST";
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko";
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies);
string postData = String.Format("login=/*dd*/&passwd=/*pass*/");
byte[] byteArray = Encoding.ASCII.GetBytes(postData);
getRequest.ContentLength = byteArray.Length;
Stream newStream = getRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close();
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
getResponse.Cookies = cookies;
StreamReader sr1 = new StreamReader(getResponse.GetResponseStream());
string sourceCode = sr1.ReadToEnd();
richTextBox1.Text = sourceCode;
sr1.Close();
I would really apprieciate any help, or any info where can I find some explain of cookiecontainers, http protocols in c# etc.... as it's my first program working with WebRequests, really thank u for help in advance.
Okay, you're going to hate me, but if I had any choice I would not use HttpwebResponse, I would use http://www.seleniumhq.org/projects/webdriver/
It's so easy because it uses the full blown browser instead of needing to maintain cookies. And if you need to run it interactively, or without the gui you can use SimpleDriver()
First of all sorry for my bad english.
Task is to authenticate in instagram with API. When I try to get a token after the user clicked "OK" I get 403 error from HttpWebResponse and I can't get my token:(. I don't know what to do. Help me please. And when I enter this link in me browser - it's ok
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("https://instagram.com" + url);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36";
request.Method = "POST";
request.CookieContainer = cookies;
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
request.Headers.Add("Accept-Encoding: gzip, deflate");
request.Headers.Add("Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4");
request.Headers.Add("Cache-Control:max-age=0");
request.ContentType = "application/x-www-form-urlencoded";
request.Host = "instagram.com";
request.Headers.Add("Origin: https://instagram.com");
StreamWriter writer = new StreamWriter(request.GetRequestStream());
writer.Write("csrfmiddlewaretoken=" + par + "&username=LOGIN&password=PASSWORD");
writer.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse(); //403 ERORR
You cannot get an access token by simply sending a username/password pair to the API! You have to direct a real user to the Instagram's authorization url, and after the user logged in successfully Instagram will send your application a code to be used for getting the access token.
More information: https://instagram.com/developer/authentication/
I want to get some information from the web with HtmlAgilityPack, the application was normal before I use the application to get the data from this page, the number of the error is 403, And my code is as follows:
string wikipageurl = geturl.Text;
WebClient wc1 = new WebClient();
Stream stream1 = wc1.OpenRead(wikipageurl);
StreamReader sr1 = new StreamReader(stream1, Encoding.UTF8);
showhtml.Text = sr1.ReadToEnd();
I use showhtml textbox to show me the information the application got.
This is how you can do it using HtmlAgilityPack:
HtmlDocutment doc;
HtmlWeb web = new HtmlWeb();
web.OverrideEncoding = Encoding.UTF8;
web.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0";
doc = web.Load("http://zh.wikipedia.org/wiki/%E6%97%A5%E6%9C%AC%E5%85%83%E5%B8%85%E5%88%97%E8%A1%A8");
showhtml.Text = doc.DocumentNode.OuterHtml;
If you want to do it using WebClient check Oscar Mederos answer
Just try to simulate you're accessing to it through a web browser. For that, you use the User-Agent header:
...
WebClient wc1 = new WebClient();
wc1.Headers.Add(
"User-Agent",
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:18.0) Gecko/20100101 Firefox/18.0"
);
Stream stream1 = wc1.OpenRead(wikipageurl);
...