I am trying to read a page from a website that logs me in automatically. The problem is the site when launched first gives a page saying "Logging you in" with a loading icon. That happens for about 5 seconds or so and than the actual page loads.
When I run my code it only gets the logon page with icon code and not the actual page after the logon.
Anyway I can get both?
Thanks
WebRequest request = WebRequest.Create("https://server1/test");
request.Credentials = CredentialCache.DefaultCredentials;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream dataStream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(dataStream))
{
Console.WriteLine(reader.ReadToEnd());
}
}
}
I have the following code for getting a website and it works fine. The problem come up when I try to get a web page developed in Angular.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; rv:2.2) Gecko/20110201";
request.Method = "GET";
request.Timeout = 30000;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream flujo = response.GetResponseStream();
Encoding encode = Encoding.GetEncoding("utf-8");
StreamReader readStream = new StreamReader(flujo, encode);
String html;
try
{
html = readStream.ReadToEnd();
} catch(System.IO.IOException)
{
return;
}
response.Close();
readStream.Close();
HtmlAgilityPack.HtmlDocument DOM = new HtmlAgilityPack.HtmlDocument();
DOM.LoadHtml(html);
I know Angular first supply the skeleton of the page and in client side, fecth for info and display it.
When I try to get some info using HtmlAgilityPack, I get nothing.
My question is if it's possible to setup HttpWebRequest or HttpWebResponse or any other class to indicate to wait for javascript is done before getting the content or something similar.
Also, I tried to get content using WebBrowser and used the loadCompleted event and the same problem.
Any help?
Thanks.
I have a Request which I make to a page and works fine. I can also view that page the response page with Fiddler.
But how do I open this response in my browser?
Currently what I have:
Cookie cookie = new Cookie("test","this");
cookie.Domain = "foobar";
HttpWebRequest request = (HttpWebRequest) HttpWebRequest.Create("http://foobar/ReportServer/");
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookie);
WebResponse response = request.GetResponse();
Stream sr = response.GetResponseStream();
StreamReader sre = new StreamReader(sr);
string s = sre.ReadToEnd();
Response.Write(s);
Save it to an HTML file and open the browser with the path to that file.
Because you have addressibility to the request "Stream" you can use this method: NavigateToStream :
http://msdn.microsoft.com/en-us/library/system.windows.controls.webbrowser.navigatetostream(v=vs.100).aspx
this.webBrowser.NavigateToStream(sr);
You can use System.Windows.Forms.WebBrowser control.
I have to use HttpWebRequest in C#, to retrieve an access token.
My base url redirects me to another url that contains the access token.
Is there a way to use this to get the access token?
This is my code after applying suggestions from the answer:
// Create a request for the URL.
string url = "https://stackexchange.com/oauth/dialog?client_id=2532&scope=no_expiry&redirect_uri=https://stackexchange.com/oauth/login_success";
WebRequest request = WebRequest.Create(
url);
request.Method = "GET";
// If required by the server, set the credentials.
request.Credentials = CredentialCache.DefaultCredentials;
// Get the response.
WebResponse response = request.GetResponse();
// Display the status.
Console.WriteLine(((HttpWebResponse)response).StatusDescription);
// Get the stream containing content returned by the server.
Stream dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content.
string responseFromServer = reader.ReadToEnd();
// Display the content.
Console.WriteLine(responseFromServer);
// Clean up the streams and the response.
reader.Close();
response.Close();
Debug breaks at line: WebResponse response = request.GetResponse();
VS tells me that the server has the a generic error: 500.
If I try with https://www.google.com, no errors.
You can get the resulting url from the ResponseUri property.
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
NaveValueCollection urlParameters = HttpUtility.ParseQueryString(response.ResponseUri.Query);
// extract the access token from the url.
string accessToken = urlParameters["access_token"];
}
How do you login to a webpage and retrieve its content in C#?
That depends on what's required to log in. You could use a webclient to send the login credentials to the server's login page (via whatever method is required, GET or POST), but that wouldn't persist a cookie. There is a way to get a webclient to handle cookies, so you could just POST the login info to the server, then request the page you want with the same webclient, then do whatever you want with the page.
Look at System.Net.WebClient, or for more advanced requirements System.Net.HttpWebRequest/System.Net.HttpWebResponse.
As for actually applying these: you'll have to study the html source of each page you want to scrape in order to learn exactly what Http requests it's expecting.
How do you mean "login"?
If the subfolder is protected on the OS level, and the browser pops of a login dialog when you go there, you will need to set the Credentials property on the HttpWebRequest.
If the website has it's own cookie-based membership/login system, you will have to use HttpWebRequest to first response to the login form.
string postData = "userid=ducon";
postData += "&username=camarche" ;
byte[] data = Encoding.ASCII.GetBytes(postData);
WebRequest req = WebRequest.Create(
URL);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = data.Length;
Stream newStream = req.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
StreamReader reader = new StreamReader(req.GetResponse().GetResponseStream(), System.Text.Encoding.GetEncoding("iso-8859-1"));
string coco = reader.ReadToEnd();
Use the WebClient class.
Dim Html As String
Using Client As New System.Net.WebClient()
Html = Client.DownloadString("http://www.google.com")
End Using
You can use the build in WebClient Object instead of crating the request yourself.
WebClient wc = new WebClient();
wc.Credentials = new NetworkCredential("username", "password");
string url = "http://foo.com";
try
{
using (Stream stream = wc.OpenRead(new Uri(url)))
{
using (StreamReader reader = new StreamReader(stream))
{
return reader.ReadToEnd();
}
}
}
catch (WebException e)
{
//Error handeling
}
Try this:
public string GetContent(string url)
{
using (System.Net.WebClient client =new System.Net.WebClient())
{
return client.DownloadString(url);
}
}