Develop Reference

Set the same cookies webrequest to web browser client in c#?

I use webrequest in asp.net page (run on server) to get google drive direct download link. After that, I send this link to client ( they use browser ex: chrome, firefox) for download this file. But issuie is cookie, webrequest cookie and client cookie not the same same. Ofcourse they can't download.
So, how I can set webrequest cookie to client?
Or how make webrequest cookie and client cookie to the same same? Thank and hope some idea to solve this problem. This my code:
HttpWebRequest objWebRequest = (HttpWebRequest)WebRequest.Create(url);
objWebRequest.CookieContainer = cookies;
objWebRequest.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse objWebResponse = (HttpWebResponse)objWebRequest.GetResponse();
Stream receiveStream = objWebResponse.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream, System.Text.Encoding.UTF8);
HtmlAgilityPack.HtmlDocument doc = new HtmlDocument();
doc.Load(readStream);
string link = ""; //=> this link i need to send client
foreach (HtmlNode row in doc.DocumentNode.SelectNodes("//a[#id='uc-download-link']"))
{ link += row.Attributes["href"].Value; }

HttpWebRequest an Angular website

I have the following code for getting a website and it works fine. The problem come up when I try to get a web page developed in Angular.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; rv:2.2) Gecko/20110201";
request.Method = "GET";
request.Timeout = 30000;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream flujo = response.GetResponseStream();
Encoding encode = Encoding.GetEncoding("utf-8");
StreamReader readStream = new StreamReader(flujo, encode);
String html;
try
{
html = readStream.ReadToEnd();
} catch(System.IO.IOException)
{
return;
}
response.Close();
readStream.Close();
HtmlAgilityPack.HtmlDocument DOM = new HtmlAgilityPack.HtmlDocument();
DOM.LoadHtml(html);
I know Angular first supply the skeleton of the page and in client side, fecth for info and display it.
When I try to get some info using HtmlAgilityPack, I get nothing.
My question is if it's possible to setup HttpWebRequest or HttpWebResponse or any other class to indicate to wait for javascript is done before getting the content or something similar.
Also, I tried to get content using WebBrowser and used the loadCompleted event and the same problem.
Any help?
Thanks.

Passing cookie to login-page

I am trying to scrape data from a webpage that runs on Asp.Net Webforms. I had a look at this page for doing this. It looks like what i wan't to do, but i'm not gonna get it to work out of the box, because the login page requires a cookie to be presented. The cookie is set on a page that sets the cookie, and then redirects the user to the login page. How should i modify the code on the link so that it first browses the redirect-page, saves the cookie and then passes the cookie to the login page?

Worked great =)
CookieContainer cookieJar = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://mydomain.com/Start.aspx?g=4");
request.CookieContainer = cookieJar;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
CookieContainer cookies = new CookieContainer();
HttpWebRequest request1 = (HttpWebRequest)WebRequest.Create("http://mydomain.com/Login.aspx");
request1.CookieContainer = cookieJar;
HttpWebResponse response1 = (HttpWebResponse)request1.GetResponse();
StreamReader reader = new StreamReader(response1.GetResponseStream());
string loginPage = reader.ReadToEnd();
reader.Close();

webBrowser not navigating, even when DocumentStream is set?

I'm trying to load a page in WebBrowser thorugh responseStream sent by the server. I tried setting, its DocumentStream property but the browser is not navigating. This is my code:
private void btnFbConnect_Click(object sender, EventArgs e)
{
WebRequest request = WebRequest.Create("http://www.facebook.com");
request.Method = "GET";
WebResponse response = request.GetResponse();
StreamReader rdr = new StreamReader(response.GetResponseStream());
webBrowser1.DocumentStream = rdr.BaseStream;// after this line, I'm expecting the browser to display the facebook login page.
MessageBox.Show(rdr.ReadToEnd());
rdr.Close();
response.Close();
}
Am I missing an intermediate step or something??? If not, what might be causing the browser not to cause navigation???

Don't close the stream before the WebBrowser accesses it.
WebRequest request = WebRequest.Create("http://www.facebook.com");
request.Method = "GET";
WebResponse response = request.GetResponse();
StreamReader rdr = new StreamReader(response.GetResponseStream());
webBrowser1.DocumentStream = rdr.BaseStream;// after this line, I'm expecting the browser to display the facebook login page.
MessageBox.Show(rdr.ReadToEnd());
//rdr.Close();
//response.Close();
Additionally, you might encounter a error because the relative path resources on the server cannot be resolved. So, I recommend you use:
webBrowser1.Navigate("http://www.facebook.com");

How do you login to a webpage and retrieve its content in C#?

How do you login to a webpage and retrieve its content in C#?

That depends on what's required to log in. You could use a webclient to send the login credentials to the server's login page (via whatever method is required, GET or POST), but that wouldn't persist a cookie. There is a way to get a webclient to handle cookies, so you could just POST the login info to the server, then request the page you want with the same webclient, then do whatever you want with the page.

Look at System.Net.WebClient, or for more advanced requirements System.Net.HttpWebRequest/System.Net.HttpWebResponse.
As for actually applying these: you'll have to study the html source of each page you want to scrape in order to learn exactly what Http requests it's expecting.

How do you mean "login"?
If the subfolder is protected on the OS level, and the browser pops of a login dialog when you go there, you will need to set the Credentials property on the HttpWebRequest.
If the website has it's own cookie-based membership/login system, you will have to use HttpWebRequest to first response to the login form.

string postData = "userid=ducon";
postData += "&username=camarche" ;
byte[] data = Encoding.ASCII.GetBytes(postData);
WebRequest req = WebRequest.Create(
URL);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = data.Length;
Stream newStream = req.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
StreamReader reader = new StreamReader(req.GetResponse().GetResponseStream(), System.Text.Encoding.GetEncoding("iso-8859-1"));
string coco = reader.ReadToEnd();

Use the WebClient class.
Dim Html As String
Using Client As New System.Net.WebClient()
Html = Client.DownloadString("http://www.google.com")
End Using

You can use the build in WebClient Object instead of crating the request yourself.
WebClient wc = new WebClient();
wc.Credentials = new NetworkCredential("username", "password");
string url = "http://foo.com";
try
{
using (Stream stream = wc.OpenRead(new Uri(url)))
{
using (StreamReader reader = new StreamReader(stream))
{
return reader.ReadToEnd();
}
}
}
catch (WebException e)
{
//Error handeling
}

Try this:
public string GetContent(string url)
{
using (System.Net.WebClient client =new System.Net.WebClient())
{
return client.DownloadString(url);
}
}