I am using webbrowser control , and i get list of all the url of all the profiles from the search result.
After this is there any way that i use httpwebrequest to get the data from the urls?
I wanted to use the Linked in search profile api but that is very confusing.
Also i tried using httpwebrequest but it takes me to the linkedin login page.
I was thinking of any way that as i signed in to linkedin using the webbrowser control maybe using that information of webbrowser and adding with my request to pretend to be logged in .
Any ideas? Please help
The HttpWebRequest sent you to the login page, because there isn´t the cookie with the validation.
So, you'll can connect using WebBrowser control and get the cookie, then put the cookie in the webrequest
webBrowser.Navigate(someUrl);
...
CookieContainer cookies = new CookieContainer();
foreach (string cookie in webBrowser.Document.Cookie.Split(';'))
{
string name = cookie.Split('=')[0];
string value = cookie.Substring(name.Length + 1);
string path = "/";
string domain = "yourdomain.com";
cookies.Add(new Cookie(name.Trim(), value.Trim(), path, domain));
}
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.CookieContainer = cookies;
...
Related
I am creating an application for data retrieval from the web page. The page is password protected and when the user logs in the cookie is created.
In order to retrieve the data the application first has to log in: make web request with username and password and store the cookie. Then when the cookie is stored it has to be added into the headers of all requests.
Here is the method which makes requests and retrieves the content:
public void getAsyncDailyPDPContextActivationDeactivation()
{
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(dailyPDPContextActivationDeactivation);
IAsyncResult asyncResult = httpWebRequest.BeginGetResponse(null, null);
asyncResult.AsyncWaitHandle.WaitOne();
using (HttpWebResponse httpWebResponse = (HttpWebResponse)httpWebRequest.EndGetResponse(asyncResult))
using (StreamReader responseStreamReader = new StreamReader(httpWebResponse.GetResponseStream()))
{
string responseText = responseStreamReader.ReadToEnd();
}
}
Does anyone know how to modify this method in order to add a cookie into the header?
I would be also thankful if anyone suggested a way to store cookie from the response (when the application makes a request http:xxx.xxx.xxx/login?username=xxx&password=xxx the cookie is created and has to be stored for future requests).
CookieContainer cookieContainer = new CookieContainer();
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(...);
httpWebRequest.CookieContainer = cookieContainer;
Then you reuse this CookieContainer in subsequent requests:
HttpWebRequest httpWebRequest2 = (HttpWebRequest)WebRequest.Create(...);
httpWebRequest2.CookieContainer = cookieContainer;
Use the CookieContainer or you could use a CookieAwareWebClient
I'm trying to get access token from LinkedIn.
I'm follwing this URL https://developer.linkedin.com/documents/authentication
I am able to get an authorization code.
But when I'm passing the authorization code to this URL
https://www.linkedin.com/uas/oauth2/accessToken?grant_type=authorization_code &code=AUTHORIZATION_CODE &redirect_uri=YOUR_REDIRECT_URI &client_id=YOUR_API_KEY &client_secret=YOUR_SECRET_KEY
I get an error in the below format
{"error":"invalid_request","error_description":"missing required parameters, includes an invalid parameter value, parameter more then once. : Unable to retrieve access token : appId or redirect uri does not match authorization code or authorization code expired"}
Any ideas? Thanks in advance.
This is because authorization code expires in 20 seconds. So you have to get the Access Token within that time frame.
I got the same error as you. I also met the following conditions:
My request was a POST request.
My redirect_uri's were the same in /authorization and /accessToken calls.
The /accessToken call was executed immediately after receiving the authorization code, so
it wouldn't expire.
What finally did the trick for me was revoking the access token generated on the application details page on https://www.linkedin.com/secure/developer.
This is an access token for oAuth 1.a and is not compatible with oAuth 2.0 on which the linkedIn api is currently running.
After revoking this access token I was able to get a new one with the /authorization and /accessToken calls.
I see this is an older thread, however if it will help anyone, here is my working solution, working on MVC core 2.0 as of december 2018:
first, redirect to LinkedIn like this
var url = "https://" + Request.Host + "/Login/LoginLinkedIn";
url = WebUtility.UrlEncode(url);
var redirectLinkedIn = "https://www.linkedin.com/oauth/v2/authorization?response_type=code&client_id=*ClientId*&client_secret=*ClientSecret*&redirect_uri=" + url + "&state=*random required nummeric value*";
return Redirect(redirectLinkedIn);
after that, you will receive the answer in your Login/LoginLinkedIn action (don't forget to specify this path in your app settings Authorized Redirect URLs).
There you will use this private method to get a dynamic object filled with user data
private dynamic GetLinkedInUser(string code)
{
dynamic jresult;
NameValueCollection parameters = new NameValueCollection {
{"client_id", *ClientId*},
{"client_secret", *ClientSecret*},
{"grant_type", "authorization_code"},
{"redirect_uri", "https://" + Request.Host + "/Login/LoginLinkedIn"},
{"code", code}
};
WebClient client = new WebClient();
byte[] result = client.UploadValues("https://www.linkedin.com/oauth/v2/accessToken", "POST", parameters);
string response = System.Text.Encoding.Default.GetString(result);
string accessToken = JsonConvert.DeserializeObject<dynamic>(response).access_token;
WebRequest webReq = WebRequest.Create("https://api.linkedin.com/v1/people/~:(id,email-address,first-name,last-name)?format=json");
webReq.Method = "GET";
webReq.Headers.Add("Authorization","Bearer "+accessToken);
HttpWebResponse webResponse = (HttpWebResponse)webReq.GetResponse();
using (StreamReader reader = new StreamReader(webResponse.GetResponseStream())) {
string objText = reader.ReadToEnd();
jresult = JsonConvert.DeserializeObject<dynamic>(objText);
}
return jresult;
}
hope it helps someone :)
I'm using the below code to reuse the sessions.
Will I get the same session id on the grabUrl response cookies?
How do I know if both requests are using the same session/cookie?
Thanks.
CookieContainer cookCon = new CookieContainer();
Httpwebrequest loginReq = (httpwebrequest) loginReq.create(loginurl);
loginReq.cookiecontainer = cookCon;
... All the response stuffs
Httpwebrequest grabReq = (httpwebrequest) grabReq.create(grabUrl);
grabUrl.cookiecontainer = cookCon
When I add in below code to see the container contents, it shows that there is a session id:
foreach (Cookie cook in cookieContainer.GetCookies(new Uri(loginurl)))
{
Console.WriteLine("Cookie_Get_Container:");
Console.WriteLine("====================================================");
Console.WriteLine("String: {0}", cook.ToString());
Console.WriteLine("====================================================");
}
So I added cookies to the login response:
cookieContainer.Add(new Uri(domainUrl), loginRes.Cookies);
I am not able to get the session id when I try to get from the cookieContainer.grabRes.Cookies.
Any advise?
Since you are not showing code that grabs session cookie from first response I assume that you are not and the answer is "no, new session is created for second requset".
You want to get session ID (and other) from first response and set them on new requests (i.e. by adding them to the container).
Side note: depending on authenitcation method the information to authenticate the next requests (different from session ID) may not come in cookies.
I know how to get the source code but the website that I have been working on uses PHP session, this mean that you will need to login(which I have) and use the session ID that the server send back. How do I do this?
Assuming you meant Session, you'll need CookieContainer:
CookieContainer cookies = new CookieContainer();
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(someSite);
getRequest.CookieContainer = cookies;
getRequest.Method = "GET";
HttpWebResponse form = (HttpWebResponse)getRequest.GetResponse();
using (StreamReader response =
new StreamReader(form.GetResponseStream(), Encoding.UTF8))
{
formPage = response.ReadToEnd();
}
You first make a GET request to server and it'll return your SessionID in a cookie. If you need to make new requests to same server, you must pass it through and server will identify you as a returning user.
I have an application that reads parts of the source code on a website. That all works; but the problem is that the page in question requires the user to be logged in to access this source code. What my program needs a way to initially log the user into the website- after that is done, I'll be able to access and read the source code.
The website that needs to be logged into is:
mmoinn.com/index.do?PageModule=UsersLogin
You can continue using WebClient to POST (instead of GET, which is the HTTP verb you're currently using with DownloadString), but I think you'll find it easier to work with the (slightly) lower-level classes WebRequest and WebResponse.
There are two parts to this - the first is to post the login form, the second is recovering the "Set-cookie" header and sending that back to the server as "Cookie" along with your GET request. The server will use this cookie to identify you from now on (assuming it's using cookie-based authentication which I'm fairly confident it is as that page returns a Set-cookie header which includes "PHPSESSID").
POSTing to the login form
Form posts are easy to simulate, it's just a case of formatting your post data as follows:
field1=value1&field2=value2
Using WebRequest and code I adapted from Scott Hanselman, here's how you'd POST form data to your login form:
string formUrl = "http://www.mmoinn.com/index.do?PageModule=UsersAction&Action=UsersLogin"; // NOTE: This is the URL the form POSTs to, not the URL of the form (you can find this in the "action" attribute of the HTML's form tag
string formParams = string.Format("email_address={0}&password={1}", "your email", "your password");
string cookieHeader;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
Here's an example of what you should see in the Set-cookie header for your login form:
PHPSESSID=c4812cffcf2c45e0357a5a93c137642e; path=/; domain=.mmoinn.com,wowmine_referer=directenter; path=/; domain=.mmoinn.com,lang=en; path=/;domain=.mmoinn.com,adt_usertype=other,adt_host=-
GETting the page behind the login form
Now you can perform your GET request to a page that you need to be logged in for.
string pageSource;
string getUrl = "the url of the page behind the login";
WebRequest getRequest = WebRequest.Create(getUrl);
getRequest.Headers.Add("Cookie", cookieHeader);
WebResponse getResponse = getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
EDIT:
If you need to view the results of the first POST, you can recover the HTML it returned with:
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
Place this directly below cookieHeader = resp.Headers["Set-cookie"]; and then inspect the string held in pageSource.
You can simplify things quite a bit by creating a class that derives from WebClient, overriding its GetWebRequest method and setting a CookieContainer object on it. If you always set the same CookieContainer instance, then cookie management will be handled automatically for you.
But the only way to get at the HttpWebRequest before it is sent is to inherit from WebClient and override that method.
public class CookieAwareWebClient : WebClient
{
private CookieContainer cookie = new CookieContainer();
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest request = base.GetWebRequest(address);
if (request is HttpWebRequest)
{
(request as HttpWebRequest).CookieContainer = cookie;
}
return request;
}
}
var client = new CookieAwareWebClient();
client.BaseAddress = #"https://www.site.com/any/base/url/";
var loginData = new NameValueCollection();
loginData.Add("login", "YourLogin");
loginData.Add("password", "YourPassword");
client.UploadValues("login.php", "POST", loginData);
//Now you are logged in and can request pages
string htmlSource = client.DownloadString("index.php");
Matthew Brindley, your code worked very good for some website I needed (with login), but I needed to change to HttpWebRequest and HttpWebResponse otherwise I get a 404 Bad Request from the remote server. Also I would like to share my workaround using your code, and is that I tried it to login to a website based on moodle, but it didn't work at your step "GETting the page behind the login form" because when successfully POSTing the login, the Header 'Set-Cookie' didn't return anything despite other websites does.
So I think this where we need to store cookies for next Requests, so I added this.
To the "POSTing to the login form" code block :
var cookies = new CookieContainer();
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(formUrl);
req.CookieContainer = cookies;
And To the "GETting the page behind the login form" :
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(resp.Cookies);
getRequest.Headers.Add("Cookie", cookieHeader);
Doing this, lets me Log me in and get the source code of the "page behind login" (website based moodle) I know this is a vague use of the CookieContainer and HTTPCookies because we may ask first is there a previously set of cookies saved before sending the request to the server. This works without problem anyway, but here's a good info to read about WebRequest and WebResponse with sample projects and tutorial:
Retrieving HTTP content in .NET
How to use HttpWebRequest and HttpWebResponse in .NET
Sometimes, it may help switching off AllowAutoRedirect and setting both login POST and page GET requests the same user agent.
request.UserAgent = userAgent;
request.AllowAutoRedirect = false;