I send a POST request by this way:
HttpWebResponse res = null;
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Method = "POST";
req.CookieContainer = cookieContainer;
req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3";
req.ContentType = "application/x-www-form-urlencoded";
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] loginDataBytes = encoding.GetBytes(requestCommand);
req.ContentLength = loginDataBytes.Length;
Stream stream = req.GetRequestStream();
stream.Write(loginDataBytes, 0, loginDataBytes.Length);
stream.Close();
//line below is my way to send request, but it return a response.
res = (HttpWebResponse)req.GetResponse();
I just want to send request to WebServer and I don't want to receive reponse. It wastes time and bandwidth. Are there anyway to do that?
Thank you very much!
No, there's no way. The HTTP protocol is two way. It is a request/response protocol. You need to call the GetResponse method if you want to send the request that you prepared earlier.
Related
So I am currently trying to log into my account on a website using WebRequest.
I have been reading about it to the point where I feel like I wanted to use an example to learn by trial and error.
This is the example I am using
Login to website, via C#
So when I try to execute my code it returns an unhandled exception and its this one
System.Net.WebException: 'The remote server returned an error: (404)
Not Found.'
I tried stepping through the code and I THINK it might be that it's trying to POST somewhere where it can't.
I wanted to fix this before moving onto getting a confirmation that it successfully logged in.
I changed the username and password to dummy text for the sake of this question.
What did I do wrong here and whats the most logical way of fixing this issue?
Thanks in advance.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
string formUrl = "https://secure.runescape.com/m=weblogin/login.ws"; // NOTE: This is the URL the form POSTs to, not the URL of the form (you can find this in the "action" attribute of the HTML's form tag
string formParams = string.Format("login-username={0}&login-password={1}", "myUsername", "password");
string cookieHeader;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
When you scrape a website, you have to make sure you mimic everything that happens. That includes any client-side state (Cookies) that is sent earlier before a form is POST-ed. As most sites don't like to be scraped or steered by bots they are often rather picky about what is the payload. Same is true for the site you're trying to control.
Three important things you have missed:
You didn't start with an initial GET so you have the required cookies in a CookieContainer.
on the post you missed an header (Referrer) and three hidden fields in the form.
The form fields are named username and password (as can be seen in the name attribute of the input tags). You have used the id's.
Fixing those omissions will result in the following code:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
string useragent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36";
// capture cookies, this is important!
var cookies = new CookieContainer();
// do a GET first, so you have the initial cookies neeeded
string loginUrl = "https://secure.runescape.com/m=weblogin/loginform.ws?mod=www&ssl=0&dest=community";
// HttpWebRequest
var reqLogin = (HttpWebRequest) WebRequest.Create(loginUrl);
// minimal needed settings
reqLogin.UserAgent = useragent;
reqLogin.CookieContainer = cookies;
reqLogin.Method = "GET";
var loginResp = reqLogin.GetResponse();
//loginResp.Dump(); // LinqPad testing
string formUrl = "https://secure.runescape.com/m=weblogin/login.ws"; // NOTE: This is the URL the form POSTs to, not the URL of the form (you can find this in the "action" attribute of the HTML's form tag
// in ther html the form has 3 more hidden fields, those are needed as well
string formParams = string.Format("username={0}&password={1}&mod=www&ssl=0&dest=community", "myUsername", "password");
string cookieHeader;
// notice the cast to HttpWebRequest
var req = (HttpWebRequest) WebRequest.Create(formUrl);
// put the earlier cookies back on the request
req.CookieContainer = cookies;
// the Referrer is mandatory, without it a timeout is raised
req.Headers["Referrer"] = "https://secure.runescape.com/m=weblogin/loginform.ws?mod=www&ssl=0&dest=community";
req.UserAgent = useragent;
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
This returns for me success. It is up to you parse the resulting HTML to plan your next steps.
I'm trying to login on a xbox live page, and got some problems with that, have no idea why, I think I set everything properly... here is my code
CookieCollection cookies = new CookieCollection();
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("https://account.xbox.com/en-US/PaymentAndBilling/RedeemCode");
Request.CookieContainer = new CookieContainer();
Request.CookieContainer.Add(cookies);
//Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko";
HttpWebResponse Response = (HttpWebResponse)Request.GetResponse();
Response.Cookies.Add(cookies);
Response.Close();
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create("https://login.live.com/");
getRequest.Method = "POST";
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko";
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies);
string postData = String.Format("login=/*dd*/&passwd=/*pass*/");
byte[] byteArray = Encoding.ASCII.GetBytes(postData);
getRequest.ContentLength = byteArray.Length;
Stream newStream = getRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close();
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
getResponse.Cookies = cookies;
StreamReader sr1 = new StreamReader(getResponse.GetResponseStream());
string sourceCode = sr1.ReadToEnd();
richTextBox1.Text = sourceCode;
sr1.Close();
I would really apprieciate any help, or any info where can I find some explain of cookiecontainers, http protocols in c# etc.... as it's my first program working with WebRequests, really thank u for help in advance.
Okay, you're going to hate me, but if I had any choice I would not use HttpwebResponse, I would use http://www.seleniumhq.org/projects/webdriver/
It's so easy because it uses the full blown browser instead of needing to maintain cookies. And if you need to run it interactively, or without the gui you can use SimpleDriver()
So I have a list of URLs that I'm processing through and in a couple of cases I run into an argument exception due to gzip encoding issues, so I drew up this code to resolve the gzip encoding problems.
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate";
req.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
req.Method = "GET";
req.UserAgent = "Mozilla/5.0 (Windows; U; MSIE 9.0; WIndows NT 9.0; en-US))";
string source;
using (WebResponse webResponse = req.GetResponse())
// On the second iteration we never get beyond this line
{
HttpWebResponse httpWebResponse = webResponse as HttpWebResponse;
using (StreamReader reader = new StreamReader(httpWebResponse.GetResponseStream()))
{
source = reader.ReadToEnd();
}
httpWebResponse.Close();
}
req.Abort();
This works for the first URL that needs this processing. however the second URL that needs processing times out. I'm not sure what it is that I'm missing to get this to work consistently.
Now I do have the URLs being sent to the above method inside a foreach loop.
foreach (string uri in _UriAddresses)
{
ProcessListItem(uri);
}
Let me know if there's anything that's not shown that would shed light on this issue.
Following on from the post How to create a simple proxy in C#? I have been playing around with implementing a basic proxy.
Where I am getting stuck and confused is trying to issue a WebRequest with the information provided in the original request.
Using the following code.
WebRequest webRequest = WebRequest.Create("http://www.google.com");
(webRequest as HttpWebRequest).UserAgent = "MOZILLA/5.0 (WINDOWS NT 6.1; WOW64) APPLEWEBKIT/537.1 (KHTML, LIKE GECKO) CHROME/21.0.1180.75 SAFARI/537.1";
webRequest.Method = "GET";
WebResponse webResponse = webRequest.GetResponse();
Stream responseStream = webResponse.GetResponseStream();
byte[] responseBytes = responseStream.ReadFully();
I can successfully issue a request and return the page content.
However when I put it inside a Proxy request (IE: TcpListener) like such.
TcpListener _listener = new TcpListener(IPAddress.Any, 1234);
this._listener.Start();
byte[] bytes = new byte[1024];
while (true)
{
TcpClient client = this._listener.AcceptTcpClient();
NetworkStream networkStream = client.GetStream();
int i = networkStream.Read(bytes, 0, bytes.Length);
while (i != 0)
{
string data = System.Text.Encoding.ASCII.GetString(bytes, 0, i);
RequestHeader header = new RequestHeader(data.ToUpper());
WebRequest webRequest = WebRequest.Create(header.URL);
(webRequest as HttpWebRequest).UserAgent = header.UserAgent;
webRequest.Method = "GET";
WebResponse webResponse = webRequest.GetResponse(); //It gets here and never returns
Stream responseStream = webResponse.GetResponseStream();
byte[] responseBytes = responseStream.ReadFully();
networkStream.Write(responseBytes, 0, responseBytes.Length);
i = networkStream.Read(bytes, 0, bytes.Length);
}
client.Close();
}
It blocks at the line WebResponse webResponse = webRequest.GetResponse(); and never returns.
This has definitely got nothing to do with the data provided by the RequestHeader class I created as I've also tried hardcoding the values.
I'm assuming I'm missing something fundamental about the way sockets work in such a scenario and the approach required. Hopefully someone can clarify for me.
Yeah, you are assuming you have read all the header.
Instead of this, some kind of state machine should be implemented to parse the incoming HTTP-request. The state machine must collect the information about the request and, of course, detect the end of request, then you process the request (proxy stuff) and send the response. Just Google C# http state machine for examples.
So it turned out to be a proxy issue.
Basically for testing I needed to set the machine proxy to 127.0.0.1:1234 or similar.
This in turn was being used in the default settings when initializing a WebRequest.
So all I needed to do in the end was the following to bypass the proxy.
(webRequest as HttpWebRequest).UserAgent = header.UserAgent;
webRequest.Method = "GET";
webRequest.Proxy = null; //Adding this line cleared the proxy.
I have tried many ways to login to an https website programmatically, but I am having issues. Every time I get an error stating that my login and password are incorrect. I am sure they are correct because I can login to the site via the browser using the same credentials.
Failing Code
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("https://www.majesticseo.com/account/login?EmailAddress=myemail&Password=mypass&RememberMe=1");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,**;q=0.8";
request.UnsafeAuthenticatedConnectionSharing = true;
request.Method = "POST";
request.KeepAlive = true;
request.ContentType = "application/x-www-form-urlencoded";
request.AllowAutoRedirect = true;
request.CookieContainer = container;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
//String tmp;
foreach(Cookie cookie1 in response.Cookies)
{
container.Add(cookie1);
}
Stream stream = response.GetResponseStream();
string html = new StreamReader(stream).ReadToEnd();
Console.WriteLine("" + html);
That site uses HTTP POST for login, and does not send the username and password in the URL.
The correct login URL is https://www.majesticseo.com/account/login
You need to create a string of data to post, convert it to a byte array, set the content length and then do your request. It is very important that the content-length is sent. Without it the post will not work.
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("https://www.majesticseo.com/account/login?EmailAddress=myemail&Password=mypass&RememberMe=1");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0";
request.Referer = "https://www.majesticseo.com/account/login";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,**;q=0.8";
request.UnsafeAuthenticatedConnectionSharing = true;
request.Method = "POST";
request.KeepAlive = true;
request.ContentType = "application/x-www-form-urlencoded";
request.AllowAutoRedirect = true;
// the post string for login form
string postData = "redirect=&EmailAddress=EMAIL&Password=PASS";
byte[] postBytes = System.Text.Encoding.ASCII.GetBytes(postData);
request.ContentLength = postBytes.Length;
System.IO.Stream str = request.GetRequestStream();
str.Write(postBytes, 0, postBytes.Length);
str.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
System.IO.Stream stream = response.GetResponseStream();
string html = new System.IO.StreamReader(stream).ReadToEnd();
Console.WriteLine("" + html);
You are trying to post something (I don't see, what, from your code) but not credentials. I guess that your web page shows you a web form where you enter username (email address?) and password. Then the browsers posts this form. Consequently you need to replicate browser behavior - encode form contents and send them in your post request. Use some webmaster developer tools for popular browsers to see what exactly the client browser sends to the server and how it encodes form data. Next, it's very likely that your request requires special cookies which you can collect by visiting another page (eg. login page). Sending preset cookies (like you do in commented code) won't work for most sites.
In other words, proper mechanism is:
GET the login web page
collect cookies
POST form data and pass collected cookies in the request.
collect other cookies, which could have been sent after login.