Develop Reference

Can you make a server side httprequest that mimicks a html form post?

When you perform a HTML POST to a page, it does this:
validates based on your form fields
redirects the user to a different url and responds with a authentication cookie and another cookie.
If I create a simple html page with a form on it, it works fine and I get redirected.
Would it be possible to do this on the server side only? If the endpoint reads the form fields, and then redirects with cookies then I guess not right since the server side won't have any notion of the cookies?

This should work for you to post server side:
WebRequest request = default(WebRequest);
request = WebRequest.Create(your_url);
request.Method = "POST";
request.ContentType = "application/x-www-form-encoded";
StreamWriter sw = new StreamWriter(request.GetRequestStream);
//'// Read the Response
WebResponse wr = request.GetResponse;
StreamReader sr = new StreamReader(wr.GetResponseStream);
var ReturnValue = sr.ReadToEnd.Trim;
You can set cookies on the post like this:
request.Headers(HttpRequestHeader.Cookie) = "MyCookie=value;";

C# : OAuth 2.0 authentication doesn't redirect for user consent

I'm working on a Web App that gets my users' gmails. For OAuth Authentication, I'm Making a Ajax call from UI to HTTPHandler and from HttpHandler i'm making HttpWebRequest GET call to google server. I'm getting the response of complete HTML page (Google User login) that i'm rendering in my application's UI. And after logging in and clicking on the login, It does nothing. As per the document it actually have to redirect to the google page and after user's consent i should get the respponse to the Redirect_uri provided. Nothing of that sort is happening. Below is code block on my hanlder process request method.
public void ProcessRequest(HttpContext context)
{
String url = "https://accounts.google.com/o/oauth2/auth?scope=https%3A%2F%2Fmail.google.com%2Fmail%2Ffeed%2Fatom&response_type=code&access_type=offline&redirect_uri=http%3A%2F%2Flocalhost%3A50639%2FresponseHandler.ashx&approval_prompt=force&client_id=XXXXXXXXXXXX.apps.googleusercontent.com";
HttpWebRequest webrequest = (HttpWebRequest)WebRequest.Create(url);
webrequest.Method = "GET";
HttpWebResponse response = (HttpWebResponse)webrequest.GetResponse();
Stream picasaResponse = response.GetResponseStream();
HttpContext.Current.Response.Redirect(response.ResponseUri.ToString());
StreamReader picasaReader = new StreamReader(picasaResponse);
string picasaAPIResponse = picasaReader.ReadToEnd();
Response.InnerHtml = picasaAPIResponse;
context.Response.ContentType = "text/plain";
context.Response.Write(response);
}
Thanks in Advance.

You should redirect your users to the auth url (either using full page redirect or in a popup) instead of fetching the url.

How to interact with a website without a browser? [closed]

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.
Closed 10 years ago.
Say I am building a c# application.
The purpose of application to :
get username & password from user.
and show some information present on the website.
in the background, after taking username and password, it should :
log in to a website with those credentials.
and click on the anchor link that appears after logging in.
find out the span that hold the info.
get the info.
that was an example. I am actually building an app to show bandwidth usage information.
The server does not expose any API for that.
Is there any tutorial/info/article available for similar purpose ? I just don't what to search for ?

Basic Introduction To HttpWebRequests
Firstly, you're going to need the right tools for the job. Go and download the Live HTTP Headers plugin for Firefox. This will allow you to view HTTP headers in real time so you can view the POST data that is sent when you interact with the website. Once you know the data that is sent to the website you can emulate the process by creating your own HTTP web requests programmatically. Tool > Live HTTP Headers
Load Live HTTP Headers by navigating to Tools > Live HTTP Headers. Once you've loaded the GUI navigate to the website you wish to login to, I will use Facebook for demonstration purposes. Type in your credentials ready to login, but before you do Clear the GUI text window and ensure that the check box labeled Capture is checked. Once you hit login you will see the text window flood with various information about the requests including the POST data which you need.
I find it best to click Save All... and then search for your username in the text document so that you can identify the POST data easily. For my request the POST data looked like this:
lsd=AVp-UAbD&display=&legacy_return=1&return_session=0&trynum=1&charset_test=%E2%82%AC%2C%C2%B4%2C%E2%82%AC%2C%C2%B4%2C%E6%B0%B4%2C%D0%94%2C%D0%84&timezone=0&lgnrnd=214119_mDgc&lgnjs=1356154880&email=%myfacebookemail40outlook.com&pass=myfacebookpassword&default_persistent=0
Which can then be defined in C# like so:
StringBuilder postData = new StringBuilder();
postData.Append("lsd=AVqRGVie&display=");
postData.Append("&legacy_return=1");
postData.Append("&return_session=0");
postData.Append("&trynum=1");
postData.Append("&charset_test=%E2%82%AC%2C%C2%B4%2C%E2%82%AC%2C%C2%B4%2C%E6%B0%B4%2C%D0%94%2C%D0%84");
postData.Append("&timezone=0");
postData.Append("&lgnrnd=153743_eO6D");
postData.Append("&lgnjs=1355614667");
postData.Append(String.Format("&email={0}", "CUSTOM_EMAIL"));
postData.Append(String.Format("&pass={0}", "CUSTOM_PASSWORD"));
postData.Append("&default_persistent=0");
I'm aiming to show you the relation between the POST data that we can send 'manually' via the web browser and how we can use said data to emulate the request in C#. Understand that sending POST data is far from deterministic. Different websites work in different ways and can throw all kinds of things your way. Below is a function I put together to validate that Facebook credentials are correct. I can't and shouldn't go into extraordinary depth here as the classes and their members are well self-documented. You can find better information than I can offer about the methods used at MSDN for example, WebRequest.Method Property
private bool ValidateFacebookCredentials(string email, string password)
{
CookieContainer cookies = new CookieContainer();
HttpWebRequest request = null;
HttpWebResponse response = null;
string returnData = string.Empty;
//Need to retrieve cookies first
request = (HttpWebRequest)WebRequest.Create(new Uri("https://www.facebook.com/login.php?login_attempt=1"));
request.Method = "GET";
request.CookieContainer = cookies;
response = (HttpWebResponse)request.GetResponse();
//Set up the request
request = (HttpWebRequest)WebRequest.Create(new Uri("https://www.facebook.com/login.php?login_attempt=1"));
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13";
request.Referer = "https://www.facebook.com/login.php?login_attempt=1";
request.AllowAutoRedirect = true;
request.KeepAlive = true;
request.CookieContainer = cookies;
//Format the POST data
StringBuilder postData = new StringBuilder();
postData.Append("lsd=AVqRGVie&display=");
postData.Append("&legacy_return=1");
postData.Append("&return_session=0");
postData.Append("&trynum=1");
postData.Append("&charset_test=%E2%82%AC%2C%C2%B4%2C%E2%82%AC%2C%C2%B4%2C%E6%B0%B4%2C%D0%94%2C%D0%84");
postData.Append("&timezone=0");
postData.Append("&lgnrnd=153743_eO6D");
postData.Append("&lgnjs=1355614667");
postData.Append(String.Format("&email={0}", email));
postData.Append(String.Format("&pass={0}", password));
postData.Append("&default_persistent=0");
//write the POST data to the stream
using(StreamWriter writer = new StreamWriter(request.GetRequestStream()))
writer.Write(postData.ToString());
response = (HttpWebResponse)request.GetResponse();
//Read the web page (HTML) that we retrieve after sending the request
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
returnData = reader.ReadToEnd();
return !returnData.Contains("Please re-enter your password");
}

Sample Code on Grabbing Contents (Screen Scraping)
Uri uri = new Uri("http://www.microsoft.com/default.aspx");
if(uri.Scheme = Uri.UriSchemeHttp)
{
HttpWebRequest request = HttpWebRequest.Create(uri);
request.Method = WebRequestMethods.Http.Get;
HttpWebResponse response = request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string tmp = reader.ReadToEnd();
response.Close();
Response.Write(tmp);
}
Sample Code on how to Post Data to remote Web Page using HttpWebRequest
Uri uri = new Uri("http://www.amazon.com/exec/obidos/search-handle-form/102-5194535-6807312");
string data = "field-keywords=ASP.NET 2.0";
if (uri.Scheme == Uri.UriSchemeHttp)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
request.Method = WebRequestMethods.Http.Post;
request.ContentLength = data.Length;
request.ContentType = "application/x-www-form-urlencoded";
StreamWriter writer = new StreamWriter(request.GetRequestStream());
writer.Write(data);
writer.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string tmp = reader.ReadToEnd();
response.Close();
Response.Write(tmp);
}
Source

Any HTTP client implementation, there are tons of open-source libraries for that. look at curl for example. Some dude made a .NET wrapper for it.

You can continue using WebClient to POST (instead of GET, which is the HTTP verb you're currently using with DownloadString), but I think you'll find it easier to work with the (slightly) lower-level classes WebRequest and WebResponse.
There are two parts to this - the first is to post the login form, the second is recovering the "Set-cookie" header and sending that back to the server as "Cookie" along with your GET request. The server will use this cookie to identify you from now on (assuming it's using cookie-based authentication which I'm fairly confident it is as that page returns a Set-cookie header which includes "PHPSESSID").
Click Here to Check in Detail

Retrieve DOM data from site

Is there any chance to retrieve DOM results when I click older posts from the site:
http://www.facebook.com/FamilyGuy
using C# or Java? I heard that it is possible to execute a script with onclick and get results. How I can execute this script:
onclick="(JSCC.get('j4eb9ad57ab8a19f468880561') && JSCC.get('j4eb9ad57ab8a19f468880561').getHandler())(); return false;"

I think older posts link sends an Ajax request and appends the response to the page. (I'm not sure. You should check the page source).
You can emulate this behavior in C#, Java, and JavaScript (you already have the code for javascript).
Edit:
It seems that Facebook uses some sort of internal APIs (JSCC) to load the content and it's undocumented.
I don't know about Facebook Developers' APIs (you may want to check that first) but if you want to emulate exactly what happens in your browser then you can use TamperData to intercept GET requests when you click on more posts link and find the request URL and it's parameters.
After you get this information you have to Login to your account in your application and get the authentication cookie.
C# sample code as you requested:
private CookieContainer GetCookieContainer(string loginURL, string userName, string password)
{
var webRequest = WebRequest.Create(loginURL) as HttpWebRequest;
var responseReader = new StreamReader(webRequest.GetResponse().GetResponseStream());
string responseData = responseReader.ReadToEnd();
responseReader.Close();
// Now you may need to extract some values from the login form and build the POST data with your username and password.
// I don't know what exactly you need to POST but again a TamperData observation will help you to find out.
string postData =String.Format("UserName={0}&Password={1}", userName, password); // I emphasize that this is just an example.
// cookie container
var cookies = new CookieContainer();
// post the login form
webRequest = WebRequest.Create(loginURL) as HttpWebRequest;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.CookieContainer = cookies;
// write the form values into the request message
var requestWriter = new StreamWriter(webRequest.GetRequestStream());
requestWriter.Write(postData);
requestWriter.Close();
webRequest.GetResponse().Close();
return cookies;
}
Then you can perform GET requests with the cookie you have, on the URL you've got from analyzing that JSCC.get().getHandler() requests using TamperData, and eventually you'll get what you want as a response stream:
var webRequest = WebRequest.Create(url) as HttpWebRequest;
webRequest.CookieContainer = GetCookieContainer(url, userName, password);
var responseStream = webRequest.GetResponse().GetResponseStream();
You can also use Selenium for browser automation. It also has C# and Java APIs (I have no experience using Selenium).

Facebook loads it's content dynamically with AJAX. You can use a tool like Firebug to examine what kind of request is made, and then replicate it.
Or you can use a browser render engine like webkit to process the JavaScript for you and expose the resulting HTML:
http://webscraping.com/blog/Scraping-JavaScript-webpages-with-webkit/

C# get or set cookies, to download content from the web using cookies

I need help with cookies. I'm planing use cookies to download web content. To get the content I need to log into a website because only authorized users can download web content or files. I'm using
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
and then I'm scraping source code, and I need to get link to file, but I can't get because I'm not authorized, so I need to set cookies. I have not worked with cookies before.
How do I to do this?

If you are talking about creating a asp.net web app that can create cookies, then doing this should work:
Not sure if this would work in asp.net mvc though, this works in asp.net web forms
Response.Cookies["nameOfCookie"].Value = "someValue";
Response.Cookies["nameOfCookies].Expires = DateTime.Now.AddDays(1);
then on a post back on say same page or different page, can get cookie by
string value = string.Empty;
if (Request.Cookies["nameOfCookie"] != null)
value = Request.Cookies["nameOfCookie"].Value;

I've created a quick little application that helps with generating web requests for me
public class HttpRequestHandler {
private CookieContainer cookies;
public HttpRequestHandler() {
cookies = new CookieContainer();
}
public HttpWebRequest GenerateWebRequest(string url) {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new System.Uri(url));
request.CookieContainer = cookies;
request.AllowAutoRedirect = true;
request.KeepAlive = true;
request.Referer = HttpUtility.UrlEncode(referer);
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.8) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)";
request.Headers.Add("Pragma", "no-cache");
request.Timeout = 40000;
return request;
}
}
Your problem is probably related to lack of a CookieContainer. If you create a cookie container you can save/use cookies in your web requests.

You should set the CookieContainer property of the HTTPWebRequest class to an instance of a CookieContainer class. From MSDN, it's stated that:
CookieContainer is null by default. You must assign a CookieContainer object to the property to have cookies returned in the Cookies property of the HttpWebResponse returned by the GetResponse method.
In other words, after you have set the CookieContainer property of the HTTPWebRequest object in your code, you can get the corresponding Cookies in the HTTPWebResponse object in your code. The sample code in the MSDN link shared above should get you started.

Could you be more speciffic about the project? is it a Desktop Application, ASP.NET, ASP.NET MVC?
in ASP.NET MVC I used SetPersistenCookie method every time when user is logging in, and for methods I use Authorize Attribute, every time when a user wants to acces somenthig and he is not logged in it redirects him to LogIn page...