So i am just getting to learn HttpWebRequests and it's functions.
I've gotten to the point where I want to learn how to capture cookies in a CookieContainer and parse through them.
The issue is that some websites return a 503 error and I am not sure.
One of the websites will be used in this example.
From what Iäve read online a 503 error is this.
The HyperText Transfer Protocol (HTTP) 503 Service Unavailable server
error response code indicates that the server is not ready to handle
the request.
Common causes are a server that is down for maintenance or that is
overloaded. This response should be used for temporary conditions and
the Retry-After HTTP header should, if possible, contain the estimated
time for the recovery of the service.
Which doesnt seem to fit in at all since the website is up and running.
Why is my request returning a 503 status code and what should I do to resolve this issue in a propper manner?
static void Main(string[] args)
{
//1. Create a HTTP REQUEST
//Build the request
Uri site = new Uri("https://ucp.nordvpn.com/login/");
//Inizializing a new instance of the HttpWebRequest and casting it as a WebRequest
//And calling the Create function and using our site as a paramter which the Create function takes.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(site);
//Inizialize a new instance of the CookieContainer
CookieContainer cookies = new CookieContainer();
//The request has a CookieContainer, which is null by default, so we are just assinging the newly inizialized instance
//of our CookieContainer to our requests CookieContainer
request.CookieContainer = cookies;
//Print out the number of cookies before the response (of course it will be blank)
Console.WriteLine(cookies.GetCookieHeader(site));
//Get the response and print out the cookies again
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine(cookies.GetCookieHeader(site));
}
Console.ReadKey();
}
The URL that you are trying to get to appears to be protected by CloudFlare. You can't use the basic HttpWebRequest for that type of request without some additional work. While I haven't tried this, it may be an option for you to get around that protection:
CloudFlareUtilities
The url you are trying to access is using cloud hosting which use many security measurement including which browser are accessing the site
for that to work you need to change the userAgent property of HttpWebRequest
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0";
Related
The title explains it mostly. I have declared my HttpClient, HttpClientHandler, and CookieContainer as class variables.
private HttpClient client;
private HttpClientHandler handler;
private CookieContainer cookies;
Then in the form creation I initialize the variables like so
public FrmMain()
{
InitializeComponent();
handler = new HttpClientHandler();
cookies = new CookieContainer();
handler.AllowAutoRedirect = true;
handler.UseCookies = true;
handler.CookieContainer = cookies;
client = new HttpClient(handler);
client.DefaultRequestHeaders.Connection.Clear();
client.DefaultRequestHeaders.ConnectionClose = false;
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36");
}
Later on in the program, when I call the requests, I am able to log in to the url (in this case a device on my local net) just fine. As part of the troubleshooting for this, I started printing the cookie data to the console each time a request is made. When I initially log on, it gives me single cookie, a sessionID. Any subsequent requests that I make using the same client gives me a new sessionID. This causes my requests to get a return code of badRequest, most likely because it is trying to route me back to the login page. I know that I am successfully logging in with the first request because printing the response content gives me the HTML of the index page that I am redirected to upon a successful login. I've tested all the data I'm sending via Postman, where I'm able to do a log in request, then do whatever other requests I need without issue. The only difference between Postman and my program is that in my program I am getting a new sessionID for every request instead of it persisting. Anyone know why my cookies are not persisting despite the client handler, client, and cookie container all being declared in the class scope?
It turns out my issue was not a cookie issue. The cookie was supposed to change to a new sessionID after logging in. The sessionID never changes after that. HttpClient was saving the cookie persistently. The issue was with a hidden CSRFToken that I was submitting with the formdata. That changes with each request and while I was doing the steps to get the new one before each POST, I was not actually assigning it.
I'd like to thank Jonathan. If I hadn't jumped back in to try some tweaks to make sure the program was only loading the initializations once, I probably wouldn't be looking in the area where I was neglecting to assign the new CSRFToken.
I'm trying to get ASP.NET_SessionId from a HttpWebResponse but it seems that no such data comes on the response.
Basically I'm trying to simulate some steps over some pages, where authentication is required. The problem is not in the authentication, my problem is to get ASP.NET_SessionId after the authentication so I can use it in my future requests/steps.
From Chrome on developer tools > network, I can see the ASP.NET_SessionId on the headers, but it does not come in my HttpWebResponse. Any ideia why this is happening?
There is my code:
var httpWebRequest = (HttpWebRequest) WebRequest.Create(url);
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36"; httpWebRequest.Method = "POST";
httpWebRequest.ContentType = "application/x-www-form-urlencoded";
httpWebRequest.ContentLength = 0;
var httpWebResponse = (HttpWebResponse) httpWebRequest.GetResponse();
After my request I should see a ASP.NET_SessionId header Set-Cookie, but no luck. Any ideia?
I've seen some people say that
httpWebResponse.Headers["ASP.NET_SessionId"]
or
httpWebResponse.Headers["SESSION_ID"]
should work but no, no session id header is set nor any Cookie.
After many research, the answer was here
Basically we have to keep the same CookieContainer object reference across all requests. I was extracting some Set-Cookie from the responses and adding them into my requests, but now I don't need to do anything, CookieContainer manages all of it transparently.
Set-Cookie from responses are set on CookieContainer of your request. It's the way they found to resolve possible security issues, so don't lose more time and just keep a reference to your CookieContainer because you will not be able to access session id (and you don't need it).
There's the example of my code now.
var cookieContainer = new CookieContainer();
var httpWebRequest1 = (HttpWebRequest) WebRequest.Create(url);
httpWebRequest1.CookieContainer = cookieContainer;
// do the request and some logic
var httpWebRequest2 = (HttpWebRequest) WebRequest.Create(anotherUrl);
httpWebRequest2.CookieContainer = cookieContainer; // same cookieContainer reference
Everything is working great now, hope it helps someone.
I'm working on a project of a mobile version of an archaic Online Learning System in my campus. I've been trying for weeks to scrape something in this website, but I need to login first in order to get it. I have search anything including HttpWebRequest, CookiesAwareWebClient, etc
My method until now is:
Find the "action" URL in the login form of the site
Sent POST request to that URL
Receive response containing cookies in the Headers["Set-Cookie"]
Create new HttpWebRequest with the URL to the content(that need to be logged in first).
Copy the headers of set-cookie into that request.
Run it (but fails)
I also have tried using CookieCollection in CookieAwareWebClient but it didn't work too.
How to do it properly? Is the location of a Cookie in HttpWebRequest is only in Headers, or in HTTP Packets, where is the location of CookieCollection? Does CookieCollection included in the next request?
Thanks
You need to use a CookieContainer. That will process and hold the cookies for you between HttpWebRequest objects:
var cookieJar = new CookieContainer();
var loginWebRequest = WebRequest.Create(loginUrl) as HttpWebRequest;
loginWebRequest.CookieContainer = cookieJar;
// Execute the Web Request
var authRequiredWebRequest = WebRequest.Create(protectedUrl) as HttpWebRequest;
authRequiredWebRequest.CookieContainer = cookieJar;
// Execute the next request
// It will have the auth cookie set appropriately
I would like to grab some content from a website that is made with Drupal.
The challenge here is that i need to login on this site before i can access the page i want to scrape. Is there a way to automate this login process in my C# code, so i can grab the secure content?
To access the secured content, you'll need to store and send cookies with every request to your server, starting with the request that sends your log in info and then saving the session cookie that the server gives you (which is your proof that you are who you say you are).
You can use the System.Windows.Forms.WebBrowser for a less control but out-of-the-box solution that will handle cookies.
My preferred method is to use System.Net.HttpWebRequest to send and receive all web data and then use the HtmlAgilityPack to parse the returned data into a Document Object Model (DOM) which can be easily read from.
The trick to getting System.Net.HttpWebRequest to work is that you must create a long-lived System.Net.CookieContainer that will keep track of your log in info (and other things the server expects you to keep track of). The good news is that the HttpWebRequest will take care of all of this for you if you provide the container.
You need a new HttpWebRequest for each call you make, so you must sets their .CookieContainer to the same object every time. Here is an example:
UNTESTED
using System.Net;
public void TestConnect()
{
CookieContainer cookieJar = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/login.htm");
request.CookieContainer = cookieJar;
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
// do page parsing and request setting here
request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/submit_login.htm");
// add specific page parameters here
request.CookeContainer = cookieJar;
response = (HttpWebResponse) request.GetResponse();
request = (HttpWebRequest)WebRequest.Create("http://www.mysite.com/secured_page.htm");
request.CookeContainer = cookieJar;
// this will now work since you have saved your authentication cookies in 'cookieJar'
response = (HttpWebResponse) request.GetResponse();
}
http://msdn.microsoft.com/en-us/library/system.windows.forms.webbrowser.aspx
HttpWebRequest Class
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.cookiecontainer.aspx
You'll have to use the Services module to do that. Also check out this link for a bit of explanation.
I need help with cookies. I'm planing use cookies to download web content. To get the content I need to log into a website because only authorized users can download web content or files. I'm using
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
and then I'm scraping source code, and I need to get link to file, but I can't get because I'm not authorized, so I need to set cookies. I have not worked with cookies before.
How do I to do this?
If you are talking about creating a asp.net web app that can create cookies, then doing this should work:
Not sure if this would work in asp.net mvc though, this works in asp.net web forms
Response.Cookies["nameOfCookie"].Value = "someValue";
Response.Cookies["nameOfCookies].Expires = DateTime.Now.AddDays(1);
then on a post back on say same page or different page, can get cookie by
string value = string.Empty;
if (Request.Cookies["nameOfCookie"] != null)
value = Request.Cookies["nameOfCookie"].Value;
I've created a quick little application that helps with generating web requests for me
public class HttpRequestHandler {
private CookieContainer cookies;
public HttpRequestHandler() {
cookies = new CookieContainer();
}
public HttpWebRequest GenerateWebRequest(string url) {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new System.Uri(url));
request.CookieContainer = cookies;
request.AllowAutoRedirect = true;
request.KeepAlive = true;
request.Referer = HttpUtility.UrlEncode(referer);
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.8) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)";
request.Headers.Add("Pragma", "no-cache");
request.Timeout = 40000;
return request;
}
}
Your problem is probably related to lack of a CookieContainer. If you create a cookie container you can save/use cookies in your web requests.
You should set the CookieContainer property of the HTTPWebRequest class to an instance of a CookieContainer class. From MSDN, it's stated that:
CookieContainer is null by default. You must assign a CookieContainer object to the property to have cookies returned in the Cookies property of the HttpWebResponse returned by the GetResponse method.
In other words, after you have set the CookieContainer property of the HTTPWebRequest object in your code, you can get the corresponding Cookies in the HTTPWebResponse object in your code. The sample code in the MSDN link shared above should get you started.
Could you be more speciffic about the project? is it a Desktop Application, ASP.NET, ASP.NET MVC?
in ASP.NET MVC I used SetPersistenCookie method every time when user is logging in, and for methods I use Authorize Attribute, every time when a user wants to acces somenthig and he is not logged in it redirects him to LogIn page...