I have this code in C#:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.ContentType = "application/x-www-form-urlencoded";
request.Timeout = 30000;
request.Method = "POST";
request.KeepAlive = true;
request.AllowAutoRedirect = false;
Stream newStream = request.GetRequestStream();
newStream.Write(bPostData, 0, bPostData.Length);
byte[] buf = new byte[1025]; int read = 0; string sResp = "";
HttpWebResponse wResp = (HttpWebResponse)request.GetResponse();
Stream resp = wResp.GetResponseStream();
The line HttpWebResponse wResp =... just hangs (as in no response from the URL). I'm not sure where exactly its crashing (cause i dont even get an exception error). I tested the URL in IE and it works fine. I also checked the bPostData and that one has data in it.
Where is it going wrong?
Try closing the request stream in variable newStream. Maybe the API waits for it to be done.
You have to increase the limit:
ServicePointManager.DefaultConnectionLimit = 10; // Max number of requests
Try simplifying your code and faking a user agent. Maybe the site is blocking/throttling scrapers/bots. Also ensure your application/x-www-form-urlencoded HTTP POST values are properly encoded. For this I would recommend you WebClient:
using (var client = new WebClient())
{
client.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0";
var values = new NameValueCollection
{
{ "param1", "value1" },
{ "param2", "value2" },
};
byte[] result = client.UploadValues(url, values);
}
When I commented earlier, I had run your code at my office (heavily firewalled) I got the same result you did. Came home, tried again (less firewalled) it worked fine... I'm guessing you have a barrier there. I believe you are facing a firewall issue.
Use a content-length=0
Example:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(URL);
request.Method = "POST";
request.ContentLength = 0;
var requestStream = request.GetRequestStream();
HttpWebResponse res = (HttpWebResponse)request.GetResponse();
res.Close();
Related
I made a Function for a program, which does work when the Request Type is GET, if it is POST, it always produces a Timeout Exception(and the timeout of 50s wasnt reached) on the Line HttpWebResponse response = (HttpWebResponse)request.GetResponse();
I tried many things, but I doesnt found out why, may someone here know it.
Edit: Got it to work, if someone is interested: https://gist.github.com/4347248
Any help will be greatly appreciated.
My Code is:
public ResRequest request(string URL, RequestType typ, CookieCollection cookies, string postdata ="", int timeout= 50000)
{
byte[] data;
Stream req;
Stream resp;
HttpWebRequest request = WebRequest.Create(URL) as HttpWebRequest;
request.Timeout = timeout;
request.ContinueTimeout = timeout;
request.ReadWriteTimeout = timeout;
request.Proxy = new WebProxy("127.0.0.1", 8118);
request.Headers.Add(HttpRequestHeader.AcceptLanguage, "de");
request.Headers.Add("UA-CPU", "x86");
request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate");
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; WOW64; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618) ";
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
if (typ == RequestType.POST)
{
data = System.Text.Encoding.Default.GetBytes(postdata);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
req = request.GetRequestStream();//after a few tries this produced a Timeout error
req.Write(data, 0, data.Length);
req.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();//This line produces a Timeout Exception
resp = response.GetResponseStream();
if ((response.ContentEncoding.ToLower().Contains("gzip")))
{
resp = new System.IO.Compression.GZipStream(resp, System.IO.Compression.CompressionMode.Decompress);
} else if ((response.ContentEncoding.ToLower().Contains("deflate"))) {
resp = new System.IO.Compression.DeflateStream(resp, System.IO.Compression.CompressionMode.Decompress);
}
return new ResRequest() { result = new System.IO.StreamReader(resp, System.Text.Encoding.UTF8).ReadToEnd(), cookies = response.Cookies, cstring = cookiestring(response.Cookies) };
}
else
{
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
resp = response.GetResponseStream();
if ((response.ContentEncoding.ToLower().Contains("gzip")))
{
resp = new System.IO.Compression.GZipStream(resp, System.IO.Compression.CompressionMode.Decompress);
}
else if ((response.ContentEncoding.ToLower().Contains("deflate")))
{
resp = new System.IO.Compression.DeflateStream(resp, System.IO.Compression.CompressionMode.Decompress);
}
return new ResRequest() { result = new System.IO.StreamReader(resp, System.Text.Encoding.UTF8).ReadToEnd(), cookies = response.Cookies, cstring = cookiestring(response.Cookies) };
}
}
So does it hang on req.GetRequestStream() every time, or does it work "a few tries" and then hang?
If it works a few times and then hangs, it's possible that you're not closing the requests properly, which is causing you to run out of connections. Make sure to Close() and/or Dispose() the HttpWebResponse objects and all of the Streams and Readers that you're creating.
You have to use
response.Dispose();
end of the method
I'm having issues with handling cookies when using HttpWebRequest.
I'm making a program to manage my account on a small community website.
I am able to make get and post request (successfully logged in, etc), but I can't maintain a session cookie to stay logged in.
My code looks like this :
this.cookies = new CookieCollection();
request = (HttpWebRequest)WebRequest.Create(requestURL);
request.CookieContainer = new CookieContainer();
...
request.CookieContainer.Add(cookies);
ASCIIEncoding encodage = new System.Text.ASCIIEncoding();
byte[] data = encodage.GetBytes(Post);
request.AllowAutoRedirect = true;
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "whatever";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Method = "POST";
request.Headers.Add("Accept-Encoding", "gzip,deflate,sdch");
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.AllowWriteStreamBuffering = true;
request.ContentLength = data.Length;
newStream = request.GetRequestStream();
request.ProtocolVersion = HttpVersion.Version11;
newStream.Write(data, 0, data.Length);
...
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
this.cookies = response.Cookies;
...
response.Cookies is always empty (length: 0) and it shouldn't.
Could anyone tell what I'm doing wrong? Why are there no cookies associated with the response?
thanks in advance
Just read it from Request.Cookies collection. Only new cookies added on the server side are available in Response.Cookies. Request.Cookies contains all (Request+Response) Cookies.
Considering above it seems like there are no additional cookies added by the server which is why you getting no cookies in the response. Does that make sense ?
I'm trying to write a bit of code to login to a website. But it's not working. Please can you give me some advice. This is my a bit of code:
static void Main(string[] args)
{
CookieContainer container = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://pagehype.com/login.php");
request.Method = "POST";
request.Timeout = 10000;
request.ReadWriteTimeout = 30000;
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729) (Prevx 3.0.5)";
request.CookieContainer = container;
ASCIIEncoding encoding = new ASCIIEncoding();
string postData = "username=user&password=password&processlogin=1&return=";
byte[] data = encoding.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
Stream newStream = request.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string htmldoc = reader.ReadToEnd();
response.Close();
Console.Write(htmldoc);
}
Many thanks,
Use http://www.fiddler2.com/fiddler2/ to view the http request sent went you login in using a browser and ensure that the request you build in code is the same.
PHP logins use PHPSESSID cookie. You'll need to capture this and pass it back in the CookieContainer. This is how the server will recognise you as an authenticated user.
The cookie is set in the Set-Cookie header in the initial response. You'll need to parse it to recreate the cookie in your container (don't forget the path (and domain?)
var setCookie = response.GetResponseHeader("Set-Cookie");
response.Close();
container = new CookieContainer();
foreach (var cookie in setCookie.Split(','))
{
var split = cookie.Split(';');
string name = split[0].Split('=')[0];
string value = split[0].Split('=')[1];
var c = new Cookie(name, value);
if (cookie.Contains(" Domain="))
c.Domain = split.Where(x => x.StartsWith(" Domain")).First().Split('=')[1];
else
{
c.Domain = ".pagehype.com";
}
if (cookie.Contains(" Path="))
c.Path = split.Where(x => x.StartsWith(" Path")).First().Split('=')[1];
container.Add(c);
}
Then add this container to your request.
I've tried to login to my google app engine application from ASP.NET for a few days, but no luck. I've read the following articles and got the basic ideas. But nothing works for me.
http://code.activestate.com/recipes/577217-routines-for-programmatically-authenticating-with-/
http://dalelane.co.uk/blog/?p=303
http://dalelane.co.uk/blog/?p=894
http://krasserm.blogspot.com/2010/01/accessing-security-enabled-google-app.html
http://blog.notdot.net/2010/05/Authenticating-against-App-Engine-from-an-Android-app
I know what to do. 1) Get an auth token from ClientLogin. 2) Get a cookie from Google App Engine. 3) Post data to my app with the cookie (Yes, I want to post data, not redirect after the second part). But the third part doesn't work for me at all. It give me 403 error. Here is my code:
void PostToGAE()
{
var auth = GetAuth(); // I can get the authtoken
var cookies = GetCookies(auth); // I can get the ACSID cookie
var url = string.Format("http://test.appspot.com/do/something/");
var content = "testvalue=test";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.KeepAlive = false;
request.CookieContainer = cookies;
byte[] byteArray = Encoding.UTF8.GetBytes(content);
request.ContentLength = byteArray.Length;
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse(); // This gives me 403
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
string result = reader.ReadToEnd();
reader.Close();
}
CookieContainer GetCookies(string auth)
{
CookieContainer cookies = new CookieContainer();
var url = string.Format("http://test.appspot.com/_ah/login?auth={0}",
System.Web.HttpUtility.UrlEncode(auth));
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AllowAutoRedirect = false;
request.CookieContainer = cookies;
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
string result = reader.ReadToEnd();
reader.Close();
return cookies;
}
string GetAuth()
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.google.com/accounts/ClientLogin");
var content = "Email=test#gmail.com&Passwd=testpass&service=ah&accountType=HOSTED_OR_GOOGLE";
byte[] byteArray = Encoding.UTF8.GetBytes(content);
request.ContentLength = byteArray.Length;
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
string loginStuff = reader.ReadToEnd();
reader.Close();
var auth = loginStuff.Substring(loginStuff.IndexOf("Auth")).Replace("Auth=", "").TrimEnd('\n');
return auth;
}
My app.yaml looks like this:
- url: /do/something/
script: something.py
login: admin
If I change the method POST to GET, that works. Could anyone tell me how I can post data?
Thanks.
EDITED:
Still no luck. I've tried several ways such as changing to [login: required] in app.yaml, adding [secure: always] to app.yaml and changing the request protocol to https, appending continue parameter to /_ah/login, but all of them don't work :(
I totally have no idea why POST doesn't work at all but GET. Any ideas?
I made it. I was on the wrong track. That was not the problem of app engine but Django. I am using Django-nonrel on google app engine, and I totally forgot to put #csrf_exempt decorator to my handler. I had the same problem before, but again. Anyway, the code above has been apparently working correctly since at the beginning. What a smart boy :)
I am trying to login to this website https://www.virginmobile.com.au programatically (on the right there is a Member Login form).
That form works. But when I do a POST request to the form action (https://www.virginmobile.com.au/selfcare/MyAccount/LogoutLoginPre.jsp) it failed.
It returns a 302, then following up to the new location, it returns 405.
This is my code test1.aspx.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Net;
using System.Text;
using System.IO;
using System.Security.Cryptography.X509Certificates;
using System.Net;
public partial class test1 : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
string uri = "https://www.virginmobile.com.au/selfcare/MyAccount/LogoutLoginPre.jsp";
string parameters = "username=0411222333&password=123";
System.Net.ServicePointManager.CertificatePolicy = new MyPolicy();
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
//req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 ( .NET CLR 3.0.4506.2152)";
//req.Referer = "http://www.virginmobile.com.au/";
//req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
req.AllowAutoRedirect = false;
// Send the Post
byte[] paramBytes = Encoding.ASCII.GetBytes(parameters);
req.ContentLength = paramBytes.Length;
Stream reqStream = req.GetRequestStream();
reqStream.Write(paramBytes, 0, paramBytes.Length); //Send it
reqStream.Close();
// Get the response
HttpWebResponse response = (HttpWebResponse)req.GetResponse();
if (response == null) throw new Exception("Response is null");
if (!string.IsNullOrEmpty(response.Headers["Location"]))
{
string newLocation = response.Headers["Location"];
// Request the new location
req = (HttpWebRequest)WebRequest.Create(newLocation);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
//req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 ( .NET CLR 3.0.4506.2152)";
//req.Referer = "http://www.virginmobile.com.au/";
//req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
req.AllowAutoRedirect = false;
req.CookieContainer = new CookieContainer();
req.CookieContainer.Add(response.Cookies);
// Send the Post
paramBytes = Encoding.ASCII.GetBytes(parameters);
req.ContentLength = paramBytes.Length;
reqStream = req.GetRequestStream();
reqStream.Write(paramBytes, 0, paramBytes.Length); //Send it
reqStream.Close();
// Get the response
response = (HttpWebResponse)req.GetResponse(); //**** 405 Method Not Allowed here
}
StreamReader sr = new StreamReader(response.GetResponseStream());
string responseHtml = sr.ReadToEnd().Trim();
Response.Write(responseHtml);
}
}
public class MyPolicy : ICertificatePolicy
{
public bool CheckValidationResult(ServicePoint srvPoint, X509Certificate certificate, WebRequest request, int certificateProblem)
{
return true; // Return true to force the certificate to be accepted.
}
}
Could anyone help me? Thanks in advance!
The 302 response is trying to redirect you to another page, so the problem might be that your POST data isn't being sent to the redirected page.
Maybe try setting HttpWebRequest.AllowAutoRedirect = false and catch the exception that you get
back. Then create another request to the redirected URL (specified in the Location response header) and then issue the request again with the same POST data.
You are sending pretty few headers with your request. It is very possible that they wrote their script so that it expects certain headers to be present. Headers that I can think of off the top of my head are:
User-Agent (identifies your browser and version; you can pretend to be Firefox, for example)
Referer (identifies the URL you came from; put the homepage URL in here)
Accept-Charset, Accept-Encoding, Accept-Language
but there may be others. You can probably use the Fiddler tool you mentioned to find out what headers Firefox (or whatever browser you’re using) sends with normal (non-HTTPS) requests and then add some of them to your request and see whether that makes it work. (Personally, I use TamperData for this purpose, which is a Firefox plugin.)
I'm getting a 404 error - The remote server returned an error: (404) Not Found. Below is the code I'm getting the error at the same line of code as you were getting the 405 error. If I replace the code with your previous version no 404 is returned but the 405 error is returned.
Thanks
string uri = "https://www.virginmobile.com.au/selfcare/MyAccount/LogoutLoginPre.jsp?username=0466651800&password=160392";
string parameters = "username=0411223344&password=123456";
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri);
req.Method = "GET";
req.ContentType = "application/x-www-form-urlencoded";
//req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 ( .NET CLR 3.0.4506.2152)";
//req.Referer = "http://www.virginmobile.com.au/";
//req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
req.AllowAutoRedirect = false;
// Send the Post
byte[] paramBytes = Encoding.ASCII.GetBytes(parameters);
//req.ContentLength = paramBytes.Length
//Dim reqStream As Stream = req.GetRequestStream()
//reqStream.Write(paramBytes, 0, paramBytes.Length)
//Send it
//reqStream.Close()
// Get the response
HttpWebResponse response__1 = (HttpWebResponse)req.GetResponse();
if (response__1 == null) {
throw new Exception("Response is null");
}
if (!string.IsNullOrEmpty(response__1.Headers("Location"))) {
string newLocation = response__1.Headers("Location");
// Request the new location
req = (HttpWebRequest)WebRequest.Create(newLocation + "?" + parameters);
req.Method = "GET";
req.ContentType = "application/x-www-form-urlencoded";
req.AllowAutoRedirect = false;
req.CookieContainer = new CookieContainer();
req.CookieContainer.Add(response__1.Cookies);
// Send the Post
//paramBytes = Encoding.ASCII.GetBytes(parameters)
//req.ContentLength = paramBytes.Length
//Dim reqStream As Stream = req.GetRequestStream()
//reqStream.Write(paramBytes, 0, paramBytes.Length)
//Send it
//reqStream.Close()
// Get the response
//**** The remote server returned an error: (404) Not Found.
response__1 = (HttpWebResponse)req.GetResponse();
}
StreamReader sr = new StreamReader(response__1.GetResponseStream());
string responseHtml = sr.ReadToEnd().Trim();
Resolved: 405 was because I was sending a POST instead of a GET