I'm using the HttpClient in theSystem.Net.Http but I'm having problems dealing with cookies. I have come through other posts explaining how to setup up cookies with HttpClient. I create a CookieContainer for the HttpClientHandler.CookieHandler and use this container for the cookies. When initialized, its value is obviously 0 and after making the PostAsync, the CookieContainer contains one cookie. I run through a foreach loop to print the values and I get the value of the one cookie. This all seems like it's working as it should.
The problem is that even though it seems as I'm accepting cookies the website redirects me to a page informing me that I do not accept cookies. I have checked with a browser(making it not accept cookies) and it's the exact result i get with my HttpClient.
The overall function of my code is giving values for a login and try to post these. The login seems to be working as expected though it redirects me telling me I don't accept cookies - even though my CookieContainer does contain a cookie. I'm not sure where this goes wrong but it seems like the website can't see that I'm actually accepting cookies. I have tried setting httpClientHandler.UseCookies = true; but without any difference. The website I'm trying to access is my university's system for reading grades and I do not have access to any coding of it.
My code is shown below:
{
var httpClientHandler = new HttpClientHandler();
//Creating cookiecontainer and printing Count for debug purpose
var cookieContainer = new CookieContainer();
Debug.WriteLine("Cookies " + cookieContainer.Count);
httpClientHandler.CookieContainer = cookieContainer;
var httpClient = new HttpClient(httpClientHandler);
HttpResponseMessage response = await httpClient.PostAsync(url, new FormUrlEncodedContent(values));
Debug.WriteLine("Response code" + response.StatusCode);
Debug.WriteLine("Response: " + response.IsSuccessStatusCode);
response.EnsureSuccessStatusCode();
string content = await response.Content.ReadAsStringAsync();
//Print amount of cookies after PostAsync - Should contain cookie now
Debug.WriteLine("Cookies " + cookieContainer.Count);
//Printing the values of the added cookies
foreach (Cookie cookie in cookieContainer.GetCookies(new Uri(url)))
{
Debug.WriteLine("Cookie contains: " + cookie.Value.ToString());
}
}
This is my first Windows Phone App and it is my first time using the HttpClient but I have used a few days now reading about it so hope the answer is not to obvious. Thank you very much - your help is very much appreciated.
I solved the problem by tracking the the website with chrome. I realized that the login page was giving me 2 cookies when i first went to the page and 1 cookie when i logged in. The scenario can be described like this
Get loginPage - 2 cookies
Post LoginPage - 1 cookie
The problem was that I started out by doing a httpClient.PostAsync for the login. When doing the login i was missing the first 2 cookies.
I solved it by doing a httpClient.GetAsync before the post, I then got the totalt 3 cookies that was needed.
HttpResponseMessage responseGet = await httpClient.GetAsync(User.School.Url);
HttpResponseMessage responsePost = await httpClient.PostAsync(User.School.Url, new FormUrlEncodedContent(values));
The problem was probably caused by my lack of knowledge about cookies. If the website response is saying your are missing cookies then you probably are, be sure you tracked ALL the traffic correctly. Hope this can be helpful to someone else :)
Related
I want to scrape the HTML of a website. When I access this website with my browser (no matter if it is Chrome or FireFox), I have no problem accessing the website + HTML.
When I try to parse the HTML with C# using methods like HttpWebRequest and HtmlAgilityPack, the website redirects me to another website and thus I parse the HTML of the redirected website.
Any idea how to solve this problem?
I thought the site recognises my program as a program and redirects immediately, so I tried using Selenium and a GoogleDriver and FireFoxDriver but also no luck, I get redirected immediately.
The Website: https://www.jodel.city/7700#!home
private void bt_load_Click(object sender, EventArgs e)
{
var url = #"https://www.jodel.city/7700#!home";
var req = (HttpWebRequest)WebRequest.Create(url);
req.AllowAutoRedirect = false;
// req.Referer = "http://www.muenchen.de/";
var resp = req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
String returnedContent = sr.ReadToEnd();
Console.WriteLine(returnedContent);
return;
}
And of course, cookies are to blame again, because cookies are great and amazing.
So, let's look at what happens in Chrome the first time you visit the site:
(I went to https://www.jodel.city/7700#!home):
Yes, I got a 302 redirect, but I also got told by the server to set a __cfduid cookie (twice actually).
When you visit the site again, you are correctly let into the site:
Notice how this time a __cfduid cookie was sent along? That's the key here.
Your C# code needs to:
Go to the site once, get redirected, but obtain the cookie value from the response header.
Go BACK to the site with the correct cookie value in the request header.
You can go to the first link in this post to see an example of how to set cookie values for requests.
I have already read many articles about the topic but I can't find solution.
So please don't mark this question as duplicate because other solutions won't work and are out to date.
I have a web application with a page containing a GridView (one button per row).
The button will create a HttpWebRequest (or WebClient, it's the same) and get its html.
I tried using one cookie or all the cookies but I have no success.
This is the code:
String path = Request.Url.GetLeftPart(UriPartial.Authority) + VirtualPathUtility.ToAbsolute("~/");
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(path + "MyPage.aspx");
CookieContainer cookieContainer = new CookieContainer();
HttpCookie httpCookie = HttpContext.Current.Request.Cookies.Get("ASP.NET_SessionId");
if (httpCookie != null)
{
Cookie myCookie = new Cookie();
// Convert between the System.Net.Cookie to a System.Web.HttpCookie...
myCookie.Domain = webRequest.RequestUri.Host;
myCookie.Expires = httpCookie.Expires;
myCookie.Name = httpCookie.Name;
myCookie.Path = httpCookie.Path;
myCookie.Secure = httpCookie.Secure;
myCookie.Value = httpCookie.Value;
cookieContainer.Add(myCookie);
}
webRequest.CookieContainer = cookieContainer;
string responseHTML = string.Empty;
using (HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
using (StreamReader responseReader = new StreamReader(responseStream))
{
responseHTML = responseReader.ReadToEnd();
}
}
}
webRequest.GetResponse will get timeout.
I think the problem is the domain (localhost), i know it's not possible but i have not any domain and i won't create a fake one in web.config. Moreover i have tried using a fake domain without success.
Without the following line
webRequest.CookieContainer = cookieContainer;
it works nicely without sharing session.
I would remember domain must be set otherwise i will received the relative error.
Session access must be serialized. When you use ASP.NET session, it is necessary to "serialize" HTTP requests to avoid threading issues. If two or more requests were processed in parallel, that would mean two threads could change or read session variables at the same time, which could cause a variety of issues.
The good news: ASP.NET will serialize the requests for you, automatically. If you send a second request with the same ASP.NET_SessionId, it will wait until the first one has completed.
The bad news: That means that a mechanism like the one you are attempting will not work. Your web request runs in the context of one HTTP request that is already in progress; it will block any additional HTTP requests until it is completed, including the request that you are sending via WebRequest.
More good news: If your page reads session data and does not write it, it can specify a hint that will allow two threads to run concurrently. Try adding this to both pages (the page your code is behind and the page that your code is attempting to access):
<% #Page EnableSessionState="ReadOnly" %>
If ASP.NET recognizes that the session needs are read-only, it'll allow two read-only threads to run at the same time with the same session ID.
If you need read/write access in either page, you are out of luck.
An alternative would be to use HttpServerUtility.Transfer instead. The role of the first page would change. Instead of serving as a proxy to the second page, it hands off control to the second page. By putting the pages in series, you avoid any issues with parallelism.
Example:
Server.Transfer("MyPage.aspx");
UPDATE: Figured this out. I DID need to add the authorization header as answered below but I also believe the issue for my particular use case is that the access token (which I verified through Postman) required more scopes to authenticate me fully, which makes sense since this API contains surveys that I am trying to access, which are also linked to a Google account. Once I added the extra scopes needed to access the surveys to the token request along with the authorization header code below I was able to connect successfully.
More info on adding scopes to C# code can be found here: http://www.oauthforaspnet.com/providers/google/
Hope this helps anyone running into similar issues. Thanks all!
I am trying to make a GET call to a Google API but it keeps responding with "Unauthorized" while I am logged in to Gmail. I've already implemented Google+ Sign-In in StartUp.Auth.cs and even saved the access token for later use.
So how do I get the HttpClient to authorize me?
I have an access token available but I do not know how to pass it in properly. I've seen examples with usernames and passwords, but I should not need to pass those parameters if I already have an access token? If anything, I should be able to have the user redirected to a login page instead if needed when I log out before running the solution.
What I am expecting when the project is run, is the result of the GET call to come back in the form of json but it always says I'm "Unauthorized" and I am probably missing 1 line of code somewhere...
Here is the code I am using:
using (HttpClient client = new HttpClient())
{
string _url = "https://www.googleapis.com/consumersurveys/v2/surveys?key={MY_API_KEY}";
client.BaseAddress = new Uri(_url);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
using (HttpResponseMessage response = client.GetAsync(_url).Result)
{
if (response.IsSuccessStatusCode)
{
using (HttpContent content = response.Content)
{
var Content = content.ReadAsStringAsync();
ViewBag.GoogleResponse = Content.ToString();
}
}
else
{
// THIS IS ALWAYS UNAUTHORIZED!
ViewBag.GoogleResponse = response.StatusCode + " - " + response.ReasonPhrase;
}
}
}
Please help with ideas or suggestions. Thanks!
You need to pass the auth token in an Authorization Header:
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
Have you ever gotten an response for this api/survey? If you were unable to get a response from the API by hitting it with Postman, you may have issues in the way you are targeting the API. The error being returned there seems like you weren't including the token in your request header. Did you click the Authorization tab below the request URL to add the OAuth token to your header? (Keep in mind that the {} characters need to be URL encoded)
Also, when you are referencing MY_API_KEY, is that analagous to your surveyId?
I don't have a lot of experience here, but I have a couple of suggestions :
1) I agree with Pedro, you definitely need to include the Authorization Header in your request.
2) If your MY_API_KEY is in fact the survey ID, you may be providing an incorrect URL (GoogleAPIs documentation indicates that it should be < http://www.googleapis.com/consumer-surveys/v2/surveys/surveyId >
Recommendation (after moving your API key to a string var named MY_API_KEY) :
string _url = "https://www.googleapis.com/consumersurveys/v2/surveys/";
client.BaseAddress = new Uri(_url);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", ViewBag.GoogleAccessToken);
using (HttpResponseMessage response = client.GetAsync(MY_API_KEY).Result)
{
if (response.IsSuccessStatusCode)
{
using (HttpContent content = response.Content)
{
var Content = content.ReadAsStringAsync();
ViewBag.GoogleResponse = Content.ToString();
}
}
Reference:
https://developers.google.com/consumer-surveys/v2/reference/
http://www.asp.net/web-api/overview/advanced/calling-a-web-api-from-a-net-client
im trying to build a program that can "login" to site as user to get the html code,
then ill fix the code to make more options to the user:)
i googled it and as i understand i need to send a cookie to identify myself as the user,
i use firefox edit cookies to see which cookies saves at my cmputer and saw only one 'phpsessid' that saves a string represent the session ,
i use wireshark to see how its real going and as i saw when im getting the response page im getting also this line
PHPSESSID=xxxxxxxxxxxxxxxxxxxxxxx; path=/
how i can read it from the response string to be able surf another pages as the 'user'?
tyvm for your help:)
edit:
i got it:
Req.GetResponse().Headers.Get("Set-Cookie");
Before you do the login, create a CookieContainer and assign it to the request:
var request = (HttpWebRequest)WebRequest.Create(loginUrl);
var cookies = new CookieContainer();
request.CookieContainer = cookies;
After you make the request, cookies will contain the cookies that make you logged-in.
When you want to make another request, now as a logged-in user, use cookies again:
var request = (HttpWebRequest)WebRequest.Create(anotherUrl);
request.CookieContainer = cookies;
I would need to make a simple program that logs with given credentials to certain website and then navigate to some element (link).
It is even possible (I mean this Authlogin thing)?
EDIT: SORRY - I am on my company machine and I cannot click on "Vote" or "Add comment" - the page says "Done, but with errors on page" (IE..). I do appreciate your answers and comments, you have helped me a lot!
Main things to do are:
Start using Fiddler to see what needs to be sent and in what way
Assuming we're talking a normal web form you'll probably need to use a CookieContainer with your WebRequests in order to accept the cookies that come from the login request and then re-supply them when sending subsequent requests (such context is not automagically maintained by HttpWebRequest) :-
CookieContainer _cookieContainer = new CookieContainer();
((HttpWebRequest)request).CookieContainer = _cookieContainer;
yes. it is possible.
see following code:
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.AuthenticationLevel = System.Net.Security.AuthenticationLevel.MutualAuthRequested;
req.Credentials = new NetworkCredential("admin", "admin");
req.PreAuthenticate = true;
It will partly depend on how the login process is managed. Is this actually done via a web form? If so, you'll need to post the form, just as a normal browser would.
If it's done over HTTP authentication, you should be able to set the credentials in the web request, tell it to pre-authenticate, and all should be well.