I am trying post some data via HttpWebRequest.
Here is the data:
string data = string.Format("username={0}&password={1}", username, password);
byte[] bytes = Encoding.UTF8.GetBytes(data);
There isn't any difference bewtwen UTF8 and ASCII in this case. The string is pure ASCII chars.
The code below will throw an exception:
using (Stream stream = request.GetRequestStream())
{
writer = new StreamWriter(stream, Encoding.UTF8);
string a = data.Substring(0, 1);
string b = data.Replace(a, string.Empty);
writer.Write(a);
writer.Flush();
writer.Write(b);
writer.Flush();
}//---->Last line with no code but right curly braces. Here's EXACTLY where the ex.StackTrace suggests.
This works perfectly:
using (Stream stream = request.GetRequestStream())
{
writer = new StreamWriter(stream, Encoding.ACSII); //---> from UTF8 to ACSII
// ... the rest is same as before
}
This also works perfectly:
using (Stream stream = request.GetRequestStream())
{
stream.Write(bytes, 0, bytes.Length);
}
The exception is this:
The request was aborted: The request was canceled.
StackTrace:
at System.Net.ConnectStream.CloseInternal(Boolean internalCall, Boolean aborting)
at System.Net.ConnectStream.System.Net.ICloseEx.CloseEx(CloseExState closeState)
at System.Net.ConnectStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at my function at the forementioned line.
The request:
request = (HttpWebRequest)HttpWebRequest.Create("https://-.-/takelogin.php");
request.Method = "POST";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Accept-Language", "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2");
request.Headers.Add("Accept-Encoding", "gzip, deflate, br");
request.Referer = "https://-.-/login.php";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = Encoding.UTF8.GetByteCount(data);
request.KeepAlive = true;
request.Headers.Add("Upgrade-Insecure-Requests", "1");
I want to know the internal reason in this situation...Any help will be appreciated. Thank you.
Encoding.UTF8 passes a “byte order mark” when used with StreamWriter, often called a BOM.
If you use new UTF8Encoding(false), that will not send the BOM, and things should work.
The Encoding.UTF8 is equivalent to new UTF8Encoding(true), where true is “write the BOM”.
Related
I haven't posted here in a long time so please forgive me if I am not formatting this question properly. I am trying to login to a website(omitted) via the .NET objects HttpWebRequest and HttpWebResponse. Using Wireshark, I can see that my POST request is identical to Chrome's POST request when I login to this website through my application. I am having issues getting the full response back though.
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
This response object has all the appropriate response headers, but there is still data I would like to see that is being sent through HTTP chunked responses. I can verify this in Wireshark as well. My understanding is that I need to instantiate a StreamReader object to read this remaining data. My code is blowing up at this line:
using (StreamReader sr = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
The stack trace is showing this error:
System.ArgumentException: Stream was not readable.
How can I use this StreamReader object to read the entire response after my POST request is sent? Below is my code that sends the POST request for logging into the website. Please let me know if you have any questions and I will be happy to clarify any confusion.
public bool Login()
{
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(#"http://WEBSITE-REMOVED/CheckAccess");
request.CookieContainer = CookieContainer;
//Set POST data.
string postData = "institution=AAA";
postData += "&ismobile=false";
postData += "&id=BBB";
postData += "&password=CCC";
byte[] data = Encoding.ASCII.GetBytes(postData);
//Configure HTTP POST request.
request.Headers.Clear();
request.Method = "POST";
request.Accept = #"text/html, application/xhtml+xml, image/jxr, */*";
request.Referer = #"http://WEBSITE-REMOVED/entry.html";
request.Headers.Add("Accept-Language", "en-US");
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko";
request.ContentType = #"application/x-www-form-urlencoded";
request.Headers.Add("Accept-Encoding", "gzip, deflate");
request.Host = "op.responsive.net";
request.ContentLength = data.Length;
request.KeepAlive = true;
request.Headers.Add("Cache-Control", "no-cache");
using (Stream stream = request.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}//using
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (StreamReader sr = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
{
//TO-DO
}
}//try
catch(Exception ex)
{
File.AppendAllText(ErrorLogDirectory + "Errors.txt", "Login() Exception\r\n" + System.DateTime.Now + "\r\n" + ex.ToString() + Environment.NewLine);
}//catch
return false;
}//Login
I am building one auto login application for one of my website. I have developed the auto login tool in C#. I have used httpwebrequest to make post call on the login page. I'm facing issue with the compression of the webpage. In some of the computers I'm not getting compressed response while in others I'm getting the response. Can anyone suggest how I can guaranteed get compressed response. The following is the code I have written for my httpwebrequest:
httpRequest.Connection = "keepalive";
httpRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate, br");
httpRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
httpRequest.UserAgent = "Mozilla/5.0 (Windows T 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36";
httpRequest.UnsafeAuthenticatedConnectionSharing = true;
httpRequest.Headers.Set("Cache-Control", "no-cache");
httpRequest.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(postdata);
httpRequest.ContentLength = bytes.Length;
requestStream = httpRequest.GetRequestStream();
requestStream.Write(bytes, 0, bytes.Length);
requestStream.Flush();
requestStream.Close();
response = (HttpWebResponse)httpRequest.GetResponse();
HttpWebResponse response2 = response;
switch (response2.StatusCode)
{
case HttpStatusCode.OK:
responseStream = response2.GetResponseStream();
if (response2.ContentEncoding.ToLower().Contains("gzip"))
{
str = new StreamReader(new GZipStream(responseStream, CompressionMode.Decompress), Encoding.UTF8).ReadToEnd();
}
else if (response2.ContentEncoding.ToLower().Contains("deflate"))
{
str = new StreamReader(new DeflateStream(responseStream, CompressionMode.Decompress), Encoding.UTF8).ReadToEnd();
}
else
{
str = new StreamReader(responseStream, Encoding.UTF8).ReadToEnd();
}
responseStream.Close();
responseStream = null;
break;
}
i using HttpWebRequest/HttpWebResponse to get html document, the code follow was running but i can not encode received stream to html string:
string uri = "https://myfavoritesite.come";
HttpWebRequest webrequest = (HttpWebRequest)WebRequest.Create(uri);
webrequest.KeepAlive = true;
webrequest.Method = "GET";
webrequest.ContentType = "text/html";
webrequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
//webrequest.Connection = "keep-alive";
webrequest.Host = "cat.sabresonicweb.com";
webrequest.Headers.Add("Accept-Encoding", "gzip, deflate");
webrequest.Headers.Add("Accept-Language", "en-US,en;q=0.5");
webrequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; rv:18.0) Gecko/20100101 Firefox/18.0";
HttpWebResponse webresponse = (HttpWebResponse)webrequest.GetResponse();
Console.Write(webresponse.StatusCode);
Stream receiveStream = webresponse.GetResponseStream();
Encoding enc = System.Text.Encoding.GetEncoding(1252);//1252
StreamReader loResponseStream = new
StreamReader(receiveStream, enc);
string Response = loResponseStream.ReadToEnd();
loResponseStream.Close();
webresponse.Close();
Console.Write(Response);
So, i use below code line to test is there successful request.
Console.Write(webresponse.StatusCode);
The result on the screen was OK, it's mean the request was sent but the Response string expose on screen was not html format, it's something so strange like this: #32u%&$&(#*#Eeeuw
By using webrequest.Headers.Add("Accept-Encoding", "gzip, deflate"); you are telling the server that you understand compressed responses. Remove that header and use a normal UTF8 encoding instead of 1252 that you are using. You should then get the proper string. You can just use System.Text.Encoding.UTF8.
What could be causing an out of memory exception in the code below? My program was running for a few hours and then died. The code only sends/receives a very small amount of data each time, so there are no huge files or strings going over the wire or coming back. The code sends and receives from the server every 3 seconds or so.
private void Read()
{
string postData = "Name=John"
using (HttpWebResponse response = SendRequest(new Uri(#"someWebSitehere"), postData))
{
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
responseFromServer = reader.ReadToEnd(); IT THROWS OUT OF MEMORY HERE
stream.Close();
｝
}
private HttpWebResponse SendRequest(Uri uri, string postData)
{
lock (SendRequestLock)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri);
req.Method = "POST";
req.CookieContainer = new CookieContainer();
req.Proxy = null;
UTF8Encoding encoding = new UTF8Encoding();
byte[] byte1 = encoding.GetBytes(postData);
// Set the content type of the data being posted.
req.ContentType = "application/x-www-form-urlencoded";
// Set the content length of the string being posted.
req.ContentLength = byte1.Length;
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705;)";
req.Method = "POST";
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
req.Headers.Add("Accept-Language: en-us,en;q=0.5");
req.Headers.Add("Accept-Encoding: gzip,deflate");
req.Headers.Add("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7");
req.KeepAlive = true;
req.Headers.Add("Keep-Alive: 300");
using (Stream stream = req.GetRequestStream())
{
stream.Write(byte1, 0, byte1.Length);
}
return (HttpWebResponse)req.GetResponse();
}
}
You'll want to dispose of the IDisposable classes Stream and StreamReader:
using (Stream stream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream))
{
responseFromServer = reader.ReadToEnd(); //IT THROWS OUT OF MEMORY HERE
}
}
Classes that implement IDisposable generally have external resources that they will hang onto unless you call Dispose() (or, same thing, put it inside a using block). It's likely that those classes are leaking memory each time your block of code runs, hence the "out of memory" exception after some time.
It's worthwhile reading MSDN's notes on IDisposable.
Have you checked Content-Length of the response. Maybe it is very huge. In this case you should read response stream part by part
I have problems with characters encoding received from http web response, I receive ? instead é.
I set the encoding to according Content-Type of web page that's text/javascript; charset=ISO-8859;
My code is:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(..);
request.Method = "GET";
request.AllowAutoRedirect = false;
request.Referer = "Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1";
request.Headers.Add("DNT", "1");
request.Accept = "text/html,application/xhtml+xml,application/xml";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader sr = new StreamReader(stream, Encoding.GetEncoding("iso-8859-1"));
char[] buf = new char[256];
int count;
StringBuilder buffer = new StringBuilder();
while ((count = sr.Read(buf, 0, 256)) > 0)
{
buffer.Append(buf, 0, count);
}
string responseStr = buffer.ToString();
Console.WriteLine(responseStr);
response.Close();
stream.Close();
sr.Close();
Can you tell me what is wrong with it?
Try adding the following before you make your request:
request.Headers.Add(HttpRequestHeader.AcceptCharset, "ISO-8859-1");
Btw, you should keep your StreamReader with ISO-8859-1 (instead of UTF8) if you want to try my proposed solution. Good luck!
Have you tried setting it at UTF-8?
Further more you send a referrer which I think you tried to set the UserAgent. The code below is the same as yours, but then does not go over the byte array and sets the useragent and utf8 encoding.
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
request.AllowAutoRedirect = false;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1";
request.Headers.Add("DNT", "1");
request.Accept = "text/html,application/xhtml+xml,application/xml";
using(var response = (HttpWebResponse)request.GetResponse())
using(var stream = response.GetResponseStream())
using (var sr = new StreamReader(stream, Encoding.UTF8))
{
string responseStr = sr.ReadToEnd();
Console.WriteLine(responseStr);
response.Close();
if (stream != null)
stream.Close();
sr.Close();
}