Develop Reference

How can i login to linkedin using HTTP POST Request with C#

I am working on a simple Windows Forms program that take a username and password from a "Textbox" then it show my linked-in name in a "Messagebox".
I want to accomplish the code with the using of "HttpWebRequest" or using any method to send my POST request to Linked-in then i can get the response and find my name to shown it in a "Messagebox".
I am familiar with creating a "GET" Request and also i made some "POST" Requests but in this case i didn't know how can i send my "txt_UserName.Text" and "txt_Password" with the POST Request and how can i receives the Response.
I tried to using Fiddler to capture POST request (=POST) from linkedin when i try to login but it captures more than 4 requests when i see the header of them it seem like a GET Request this is an example of one:
GET /nhome/?trk= HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
and all of them have a multiple cookies values.
This is my POST request code:
public void SubmitData()
{
try
{
string postData = "This is a test that posts this string to a Web server.";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
// Create a request using a URL that can receive a post.
WebRequest request = WebRequest.Create("http://www.linkedin.com");
// Set the Method property of the request to POST.
request.Method = "POST";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
//Content Length
request.ContentLength = byteArray.Length;
// Get the request stream.
Stream dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
// Get the response.
WebResponse response = request.GetResponse();
dataStream = response.GetResponseStream();
StreamReader sr = new StreamReader(dataStream);
MessageBox.Show(sr.ReadToEnd());
sr.Close();
dataStream.Close();
}
catch (Exception ex)
{
MessageBox.Show("Error: " + ex.Message);
}
}
Now the only thing i wish to know, how can i send my username and password as a values to login to linked-in?
Edit:
Below is my second try, it's ok, i can now send the User and Password in postData and i can store the Cookies and retrive it. but there are two issues:
1- how can i make sure that the login is accomplished and not failed
2- if the login is accomplished i want to know what is the second step to get my name from profile, is it made another request or what ?
private void button1_Click(object sender, EventArgs e)
{
PostMessage();
}
private void PostMessage()
{
try {
// POST Data and the POST uri
string postData = "isJsEnabled=true&source_app=&session_key=" + textBox1.Text + "&session_password=" + textBox2.Text + "&signin=Sign+In&session_redirect=";
string uri = "https://www.linkedin.com/uas/login-submit";
// Encoding the POST Data
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
// Create the POST Request
HttpWebRequest WebReq = (HttpWebRequest)WebRequest.Create(uri);
//POST Parameters (Method and etc.)
WebReq.Method = "POST";
WebReq.ContentType = "application/x-www-form-urlencoded";
WebReq.ContentLength = byteArray.Length;
// Set the POST Request Cookies
var cookieContainer = new CookieContainer();
WebReq.CookieContainer = cookieContainer;
// Get the request stream.
Stream dataStream = WebReq.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
// Get the response.
HttpWebResponse response = (HttpWebResponse)WebReq.GetResponse();
dataStream = response.GetResponseStream();
StreamReader sr = new StreamReader(dataStream);
//MessageBox.Show(sr.ReadToEnd());
sr.Close();
dataStream.Close();
if (response.StatusCode != HttpStatusCode.OK)
{
MessageBox.Show(" Error: " + response.StatusDescription);
response.Close();
}
foreach (Cookie cook in response.Cookies)
{
MessageBox.Show(cook.Name + " " + cook.Value);
}
}
catch (Exception ex)
{
MessageBox.Show("POST Message Error: " + ex.Message);
}
}

I used Fiddler while I was logging in and found a request to https://www.linkedin.com/uas/login-submit containing the username and password. Found it? Now, if you want to look at it completely from an HTTP Request perspective, you will have to figure out how to generate the other data in the post data/cookie header using the other requests and responses that your browser sent and received to and from the site before this particular request (the information should be there). I think this will lead you to what you need to do, but there's some work to be done!

You're going to need more than you think to log in there is good documentation on how to do this just here, you are going to need an auth token etc, This is because like other services, for example google, they are using oauth2 to secure applications etc.
oauth works by issuing tokens and refreshing tokens and there's a bit of a learning curve but its not especially difficult.
Essentially the following happens
You register your application with linked in and they give you a
client secret.
You pass this code to linked in in your application
and they will generate an auth screen saying that the application is
requesting permission.
you then approve this and it will give you an access token
You then log in with the access token (access tokens on linkedin are valid for 60 days, you must refresh them by this time).
On the plus side the linked in api is pretty straight forward and once authorised you will be able to get stuff pretty easily. All of this is detailed in the link provided in nice step by step stages.
By the way there is also a nuget package that gives you access to profile information.
Try Install-Package LinkedIn
I should point out that the nuget package above gives you a login provider to help authenticate if you don't want to roll your own.
Added after your comments below.
If all you want to do is know how to send a post request here's a generic bit of code that does just that:
string url = "www.foo.bar.com";
using (var webClient= new WebClient())
{
var data = new NameValueCollection();
data["username"] = "<yourusername>";
data["password"] = "<yourPassword>";
var response = webClient.UploadValues(url, "POST", data);
}
note: because its a web uri you should use POST in the method argument here.

The problem you up against is that Linkedin uses a redirect within the login session and is differcult to catch.
So somehow within the login session you need to code that it redirects to the https://www.linkedin.com/nhome/?trk= this provides the user page access.
I am also testing with this, however did not manage to figure this part out, normally
httpWebRequest.AllowAutoRedirect = true;
should do the trick but not in this case it does not work.
So if you find the solution let me know, if I find will post it also.

How to login a website using httpwebrequest via my web app or generic handler and access the content?

Basically I am making a chat app for my university students only and for that I have to make sure they are genuine by checking there details on UMS(university management system) and get their basic detail so they chat genuinely. I am nearly done with my chat app only the login is left.
So I want to login to my UMS page via my website from a generic handler.
and then navigate to another page in it to access there basic info keeping the session alive.
I did research on httpwebrequest and failed to login with my credentials.
https://ums.lpu.in/lpuums
(made in asp.net)
I did tried codes from other posts for login.
I am novice to this part so bear with me.. any help will be appreciated.

Without the actual handshake with UMS via a defined API, you would end up scraping UMS html, which is bad for various reasons.
I would suggest you read up on Single Sign On (SSO).
A few articles on SSO and ASP.NET -
1. Codeproject
2. MSDN
3. asp.net forum
Edit 1
Although, I think this is a bad idea, since you say you are out of options, here is a link that shows how Html Agility Pack can help in scraping the web pages.
Beware of the drawbacks of screen scraping, changes from UMS will not be communicated to you, and you will see your application not working all of a sudden.

public string Scrap(string Username, string Password)
{
string Url1 = "https://www.example.com";//first url
string Url2 = "https://www.example.com/login.aspx";//secret url to post request to
//first request
CookieContainer jar = new CookieContainer();
HttpWebRequest request1 = (HttpWebRequest)WebRequest.Create(Url1);
request1.CookieContainer = jar;
//Get the response from the server and save the cookies from the first request..
HttpWebResponse response1 = (HttpWebResponse)request1.GetResponse();
//second request
string postData = "***viewstate here***";//VIEWSTATE
HttpWebRequest request2 = (HttpWebRequest)WebRequest.Create(Url2);
request2.CookieContainer = jar;
request2.KeepAlive = true;
request2.Referer = Url2;
request2.Method = WebRequestMethods.Http.Post;
request2.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request2.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
request2.ContentType = "application/x-www-form-urlencoded";
request2.AllowWriteStreamBuffering = true;
request2.ProtocolVersion = HttpVersion.Version11;
request2.AllowAutoRedirect = true;
byte[] byteArray = Encoding.ASCII.GetBytes(postData);
request2.ContentLength = byteArray.Length;
Stream newStream = request2.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close();
HttpWebResponse response2 = (HttpWebResponse)request2.GetResponse();
using (StreamReader sr = new StreamReader(response2.GetResponseStream()))
{
responseData = sr.ReadToEnd();
}
return responseData;
}
This is the code which works for me any one can add there links and viewstate for asp.net websites to scrap and you need to take care of cookie too.
and for other websites(non asp.net) they don't require viewstate.
Use fiddler to find things needed to add in header and viewstate or cookie.
Hope this helps if some one having the problem. :)

Problems authenticating to website from code

I am trying to write code that will authenticate to the website wallbase.cc. I've looked at what it does using Firfebug/Chrome Developer tools and it seems fairly easy:
Post "usrname=$USER&pass=$PASS&nopass_email=Type+in+your+e-mail+and+press+enter&nopass=0" to the webpage "http://wallbase.cc/user/login", store the returned cookies and use them on all future requests.
Here is my code:
private CookieContainer _cookies = new CookieContainer();
//......
HttpPost("http://wallbase.cc/user/login", string.Format("usrname={0}&pass={1}&nopass_email=Type+in+your+e-mail+and+press+enter&nopass=0", Username, assword));
//......
private string HttpPost(string url, string parameters)
{
try
{
System.Net.WebRequest req = System.Net.WebRequest.Create(url);
//Add these, as we're doing a POST
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
((HttpWebRequest)req).Referer = "http://wallbase.cc/home/";
((HttpWebRequest)req).CookieContainer = _cookies;
//We need to count how many bytes we're sending. Post'ed Faked Forms should be name=value&
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(parameters);
req.ContentLength = bytes.Length;
System.IO.Stream os = req.GetRequestStream();
os.Write(bytes, 0, bytes.Length); //Push it out there
os.Close();
//get response
using (System.Net.WebResponse resp = req.GetResponse())
{
if (resp == null) return null;
using (Stream st = resp.GetResponseStream())
{
System.IO.StreamReader sr = new System.IO.StreamReader(st);
return sr.ReadToEnd().Trim();
}
}
}
catch (Exception)
{
return null;
}
}
After calling HttpPost with my login parameters I would expect all future calls using this same method to be authenticated (assuming a valid username/password). I do get a session cookie in my cookie collection but for some reason I'm not authenticated. I get a session cookie in my cookie collection regardless of which page I visit so I tried loading the home page first to get the initial session cookie and then logging in but there was no change.
To my knowledge this Python version works: https://github.com/sevensins/Wallbase-Downloader/blob/master/wallbase.sh (line 336)
Any ideas on how to get authentication working?
Update #1
When using a correct user/password pair the response automatically redirects to the referrer but when an incorrect user/pass pair is received it does not redirect and returns a bad user/pass pair. Based on this it seems as though authentication is happening, but maybe not all the key pieces of information are being saved??
Update #2
I am using .NET 3.5. When I tried the above code in .NET 4, with the added line of System.Net.ServicePointManager.Expect100Continue = false (which was in my code, just not shown here) it works, no changes necessary. The problem seems to stem directly from some pre-.Net 4 issue.

This is based on code from one of my projects, as well as code found from various answers here on stackoverflow.
First we need to set up a Cookie aware WebClient that is going to use HTML 1.0.
public class CookieAwareWebClient : WebClient
{
private CookieContainer cookie = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);
request.ProtocolVersion = HttpVersion.Version10;
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = cookie;
}
return request;
}
}
Next we set up the code that handles the Authentication and then finally loads the response.
var client = new CookieAwareWebClient();
client.UseDefaultCredentials = true;
client.BaseAddress = #"http://wallbase.cc";
var loginData = new NameValueCollection();
loginData.Add("usrname", "test");
loginData.Add("pass", "123");
loginData.Add("nopass_email", "Type in your e-mail and press enter");
loginData.Add("nopass", "0");
var result = client.UploadValues(#"http://wallbase.cc/user/login", "POST", loginData);
string response = System.Text.Encoding.UTF8.GetString(result);
We can try this out using the HTML Visualizer inbuilt into Visual Studio while staying in debug mode and use that to confirm that we were able to authenticate and load the Home page while staying authenticated.
The key here is to set up a CookieContainer and use HTTP 1.0, instead of 1.1. I am not entirely sure why forcing it to use 1.0 allows you to authenticate and load the page successfully, but part of the solution is based on this answer.
https://stackoverflow.com/a/10916014/408182
I used Fiddler to make sure that the response sent by the C# Client was the same as with my web browser Chrome. It also allows me to confirm if the C# client is being redirect correctly. In this case we can see that with HTML 1.0 we are getting the HTTP/1.0 302 Found and then redirects us to the home page as intended. If we switch back to HTML 1.1 we will get an HTTP/1.1 417 Expectation Failed message instead.
There is some information on this error message available in this stackoverflow thread.
HTTP POST Returns Error: 417 "Expectation Failed."
Edit: Hack/Fix for .NET 3.5
I have spent a lot of time trying to figure out the difference between 3.5 and 4.0, but I seriously have no clue. It looks like 3.5 is creating a new cookie after the authentication and the only way I found around this was to authenticate the user twice.
I also had to make some changes on the WebClient based on information from this post.
http://dot-net-expertise.blogspot.fr/2009/10/cookiecontainer-domain-handling-bug-fix.html
public class CookieAwareWebClient : WebClient
{
public CookieContainer cookies = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
var request = base.GetWebRequest(address);
var httpRequest = request as HttpWebRequest;
if (httpRequest != null)
{
httpRequest.ProtocolVersion = HttpVersion.Version10;
httpRequest.CookieContainer = cookies;
var table = (Hashtable)cookies.GetType().InvokeMember("m_domainTable", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.GetField | System.Reflection.BindingFlags.Instance, null, cookies, new object[] { });
var keys = new ArrayList(table.Keys);
foreach (var key in keys)
{
var newKey = (key as string).Substring(1);
table[newKey] = table[key];
}
}
return request;
}
}
var client = new CookieAwareWebClient();
var loginData = new NameValueCollection();
loginData.Add("usrname", "test");
loginData.Add("pass", "123");
loginData.Add("nopass_email", "Type in your e-mail and press enter");
loginData.Add("nopass", "0");
// Hack: Authenticate the user twice!
client.UploadValues(#"http://wallbase.cc/user/login", "POST", loginData);
var result = client.UploadValues(#"http://wallbase.cc/user/login", "POST", loginData);
string response = System.Text.Encoding.UTF8.GetString(result);

You may need to add the following:
//get response
using (System.Net.WebResponse resp = req.GetResponse())
{
foreach (Cookie c in resp.Cookies)
_cookies.Add(c);
// Do other stuff with response....
}
Another thing that you might have to do is, if the server responds with a 302 (redirect) the .Net web request will automatically follow it and in the process you might lose the cookie you're after. You can turn off this behavior with the following code:
req.AllowAutoRedirect = false;

The Python you reference uses a different referrer (http://wallbase.cc/start/). It is also followed by another post to (http://wallbase.cc/user/adult_confirm/1). Try the other referrer and followup with this POST.
I think you are authenticating correctly, but that the site needs more info/assertions from you before proceeding.

FormsAuthenticate through WCF (How do I make the Session apply to browser?!)

Users are authenticating to a REST WCF Service (my own). The credentials are sent through AJAX with Javascript and JSON format. The service reply with a OK and little info (redirect url) to the client, when authenticated.
Now, There are a new method provided for external authentication, and I have to create a compact code snippet that are easy to paste & run inside a asp.net code file method.
A typical wcf request could end up like this,
http://testuri.org/WebService/AuthenticationService.svc/ExtLogin?cId=197&aId=someName&password=!!pwd
My code snippet so far,
protected void bn_Click(object sender, EventArgs e)
{
WebHttpBinding webHttpBinding = new WebHttpBinding();
EndpointAddress endpointAddress = new EndpointAddress(url);
ContractDescription cd =
ContractDescription.GetContract(typeof(IAuthenticationService));
ServiceEndpoint sep = new ServiceEndpoint(cd);
sep.Behaviors.Add(new WebHttpBehavior());
sep.Address = endpointAddress;
sep.Binding = webHttpBinding;
var resp = new ChannelFactory<IAuthenticationService>(sepREST).CreateChannel();
LoginResult result = resp.ExtLogin(cId, aId, hashPwd);
Response.Redirect(result.RedirectUri);
// I.e. http://testuri.org/Profile.aspx (Require authenticated to visit)
}
I recieve correct authenticated reply in the resp/result objects. So, the communication are fine. When redirecting to the actual website, I'm not authenticated. I can't locate the problem? If I take the URI above (with valid credentials) and paste into my Webbrowser URL, and then manually type the uri, i'm authenticated.
I've spent a day searched the net for this, without success.
There are a LOT of info but none seem to apply.
What am I missing?
I also tried another approach but the same problem persist.
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uriWithParameters);
CookieContainer cookieContainer = new CookieContainer();
request.CookieContainer = cookieContainer;
request.ContentType = "application/json";
request.Accept = "application/json";
request.Method = "GET";
string result;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream, Encoding.UTF8))
result = reader.ReadToEnd();
JavaScriptSerializer jsonDeserializer = new JavaScriptSerializer();
LoginResult contact = jsonDeserializer.Deserialize<LoginResult>(result);
Response.Redirect(result.RedirectUri);

I'm not sure about this answer, but will offer it anyway as nobody else has posted:
I think it's because the request that has been authenticated is the request sent via code.
When you redirect it's a totally different request - so is still not authenticated.
All authentication techniques require some way of maintaining the authenticated state across 'stateless' requests = session cookies or some kind of authentication token.
Whatever token you get back from the call to the authentication service needs to be available to your website requests as well - dumping the token from the request into a cookie might be an option.
Can you see (in something like Fiddler) an auth token being sent as part of the request to 'RedirectUrl'?

How to make my web scraper log in to this website via C#

I have an application that reads parts of the source code on a website. That all works; but the problem is that the page in question requires the user to be logged in to access this source code. What my program needs a way to initially log the user into the website- after that is done, I'll be able to access and read the source code.
The website that needs to be logged into is:
mmoinn.com/index.do?PageModule=UsersLogin

You can continue using WebClient to POST (instead of GET, which is the HTTP verb you're currently using with DownloadString), but I think you'll find it easier to work with the (slightly) lower-level classes WebRequest and WebResponse.
There are two parts to this - the first is to post the login form, the second is recovering the "Set-cookie" header and sending that back to the server as "Cookie" along with your GET request. The server will use this cookie to identify you from now on (assuming it's using cookie-based authentication which I'm fairly confident it is as that page returns a Set-cookie header which includes "PHPSESSID").
POSTing to the login form
Form posts are easy to simulate, it's just a case of formatting your post data as follows:
field1=value1&field2=value2
Using WebRequest and code I adapted from Scott Hanselman, here's how you'd POST form data to your login form:
string formUrl = "http://www.mmoinn.com/index.do?PageModule=UsersAction&Action=UsersLogin"; // NOTE: This is the URL the form POSTs to, not the URL of the form (you can find this in the "action" attribute of the HTML's form tag
string formParams = string.Format("email_address={0}&password={1}", "your email", "your password");
string cookieHeader;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
Here's an example of what you should see in the Set-cookie header for your login form:
PHPSESSID=c4812cffcf2c45e0357a5a93c137642e; path=/; domain=.mmoinn.com,wowmine_referer=directenter; path=/; domain=.mmoinn.com,lang=en; path=/;domain=.mmoinn.com,adt_usertype=other,adt_host=-
GETting the page behind the login form
Now you can perform your GET request to a page that you need to be logged in for.
string pageSource;
string getUrl = "the url of the page behind the login";
WebRequest getRequest = WebRequest.Create(getUrl);
getRequest.Headers.Add("Cookie", cookieHeader);
WebResponse getResponse = getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
EDIT:
If you need to view the results of the first POST, you can recover the HTML it returned with:
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
Place this directly below cookieHeader = resp.Headers["Set-cookie"]; and then inspect the string held in pageSource.

You can simplify things quite a bit by creating a class that derives from WebClient, overriding its GetWebRequest method and setting a CookieContainer object on it. If you always set the same CookieContainer instance, then cookie management will be handled automatically for you.
But the only way to get at the HttpWebRequest before it is sent is to inherit from WebClient and override that method.
public class CookieAwareWebClient : WebClient
{
private CookieContainer cookie = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = cookie;
}
return request;
}
}
var client = new CookieAwareWebClient();
client.BaseAddress = #"https://www.site.com/any/base/url/";
var loginData = new NameValueCollection();
loginData.Add("login", "YourLogin");
loginData.Add("password", "YourPassword");
client.UploadValues("login.php", "POST", loginData);
//Now you are logged in and can request pages
string htmlSource = client.DownloadString("index.php");

Matthew Brindley, your code worked very good for some website I needed (with login), but I needed to change to HttpWebRequest and HttpWebResponse otherwise I get a 404 Bad Request from the remote server. Also I would like to share my workaround using your code, and is that I tried it to login to a website based on moodle, but it didn't work at your step "GETting the page behind the login form" because when successfully POSTing the login, the Header 'Set-Cookie' didn't return anything despite other websites does.
So I think this where we need to store cookies for next Requests, so I added this.
To the "POSTing to the login form" code block :
var cookies = new CookieContainer();
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(formUrl);
req.CookieContainer = cookies;
And To the "GETting the page behind the login form" :
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(resp.Cookies);
getRequest.Headers.Add("Cookie", cookieHeader);
Doing this, lets me Log me in and get the source code of the "page behind login" (website based moodle) I know this is a vague use of the CookieContainer and HTTPCookies because we may ask first is there a previously set of cookies saved before sending the request to the server. This works without problem anyway, but here's a good info to read about WebRequest and WebResponse with sample projects and tutorial:
Retrieving HTTP content in .NET
How to use HttpWebRequest and HttpWebResponse in .NET

Sometimes, it may help switching off AllowAutoRedirect and setting both login POST and page GET requests the same user agent.
request.UserAgent = userAgent;
request.AllowAutoRedirect = false;