What could be causing an out of memory exception in the code below? My program was running for a few hours and then died. The code only sends/receives a very small amount of data each time, so there are no huge files or strings going over the wire or coming back. The code sends and receives from the server every 3 seconds or so.
private void Read()
{
string postData = "Name=John"
using (HttpWebResponse response = SendRequest(new Uri(#"someWebSitehere"), postData))
{
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
responseFromServer = reader.ReadToEnd(); IT THROWS OUT OF MEMORY HERE
stream.Close();
｝
}
private HttpWebResponse SendRequest(Uri uri, string postData)
{
lock (SendRequestLock)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri);
req.Method = "POST";
req.CookieContainer = new CookieContainer();
req.Proxy = null;
UTF8Encoding encoding = new UTF8Encoding();
byte[] byte1 = encoding.GetBytes(postData);
// Set the content type of the data being posted.
req.ContentType = "application/x-www-form-urlencoded";
// Set the content length of the string being posted.
req.ContentLength = byte1.Length;
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705;)";
req.Method = "POST";
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
req.Headers.Add("Accept-Language: en-us,en;q=0.5");
req.Headers.Add("Accept-Encoding: gzip,deflate");
req.Headers.Add("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7");
req.KeepAlive = true;
req.Headers.Add("Keep-Alive: 300");
using (Stream stream = req.GetRequestStream())
{
stream.Write(byte1, 0, byte1.Length);
}
return (HttpWebResponse)req.GetResponse();
}
}
You'll want to dispose of the IDisposable classes Stream and StreamReader:
using (Stream stream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream))
{
responseFromServer = reader.ReadToEnd(); //IT THROWS OUT OF MEMORY HERE
}
}
Classes that implement IDisposable generally have external resources that they will hang onto unless you call Dispose() (or, same thing, put it inside a using block). It's likely that those classes are leaking memory each time your block of code runs, hence the "out of memory" exception after some time.
It's worthwhile reading MSDN's notes on IDisposable.
Have you checked Content-Length of the response. Maybe it is very huge. In this case you should read response stream part by part
Related
I am trying post some data via HttpWebRequest.
Here is the data:
string data = string.Format("username={0}&password={1}", username, password);
byte[] bytes = Encoding.UTF8.GetBytes(data);
There isn't any difference bewtwen UTF8 and ASCII in this case. The string is pure ASCII chars.
The code below will throw an exception:
using (Stream stream = request.GetRequestStream())
{
writer = new StreamWriter(stream, Encoding.UTF8);
string a = data.Substring(0, 1);
string b = data.Replace(a, string.Empty);
writer.Write(a);
writer.Flush();
writer.Write(b);
writer.Flush();
}//---->Last line with no code but right curly braces. Here's EXACTLY where the ex.StackTrace suggests.
This works perfectly:
using (Stream stream = request.GetRequestStream())
{
writer = new StreamWriter(stream, Encoding.ACSII); //---> from UTF8 to ACSII
// ... the rest is same as before
}
This also works perfectly:
using (Stream stream = request.GetRequestStream())
{
stream.Write(bytes, 0, bytes.Length);
}
The exception is this:
The request was aborted: The request was canceled.
StackTrace:
at System.Net.ConnectStream.CloseInternal(Boolean internalCall, Boolean aborting)
at System.Net.ConnectStream.System.Net.ICloseEx.CloseEx(CloseExState closeState)
at System.Net.ConnectStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at my function at the forementioned line.
The request:
request = (HttpWebRequest)HttpWebRequest.Create("https://-.-/takelogin.php");
request.Method = "POST";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Accept-Language", "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2");
request.Headers.Add("Accept-Encoding", "gzip, deflate, br");
request.Referer = "https://-.-/login.php";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = Encoding.UTF8.GetByteCount(data);
request.KeepAlive = true;
request.Headers.Add("Upgrade-Insecure-Requests", "1");
I want to know the internal reason in this situation...Any help will be appreciated. Thank you.
Encoding.UTF8 passes a “byte order mark” when used with StreamWriter, often called a BOM.
If you use new UTF8Encoding(false), that will not send the BOM, and things should work.
The Encoding.UTF8 is equivalent to new UTF8Encoding(true), where true is “write the BOM”.
i have some servers with rtorrent. I want to get status information about these servers and delete torrents for that i want to use the XML rpc interface of rtorrent.
scgi_port = localhost:5000
https://github.com/rakshasa/rtorrent/wiki/RPC-Setup-XMLRPC
Now i rly need some help to get the information from the interface back to my programm.
i already have some code, but i always get an error when executing.
Additional information: The Connection with the remote server could not be established.
using System;
using System.Text;
using System.Net;
using System.IO;
namespace SimpleXmlRpcClient
{
class Program
{
static void Main(string[] args)
{
WebRequest request = WebRequest.Create("http://ip/RPC2");
request.Method = "POST";
string postData = #"<?xml version=""1.0""?>
<methodCall>
<methodName>system.listMethods
</methodCall>";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = byteArray.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
WebResponse response = request.GetResponse();
Console.WriteLine(((HttpWebResponse)response).StatusDescription);
dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
Console.WriteLine(responseFromServer);
reader.Close();
dataStream.Close();
response.Close();
Console.WriteLine("Press any key to continue ...");
Console.ReadKey();
}
}
}
i finally found the solution.
you still have parse the xml encoded response but now i finally get an response.
System.Net.ServicePointManager.Expect100Continue = false;
byte[] requestData = Encoding.ASCII.GetBytes("<?xml version=\"1.0\"?><methodCall><methodName>system.listMethods</methodName></methodCall>");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://ip/RPC2");
request.Method = "POST";
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729;)";
request.ContentType = "text/xml";
request.ContentLength = requestData.Length;
using (Stream requestStream = request.GetRequestStream())
requestStream.Write(requestData, 0, requestData.Length);
string result = null;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream, Encoding.ASCII))
result = reader.ReadToEnd();
}
}
Good ! ,the solution of Olias is correct.
I have an application that starts multiple threads, and each thread sends requests to a remote server. After profiling, I noticed that the most time is taken in sending & processing the request. This is the function:
private string GetRequest(string url, string postFields)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Proxy = proxy;
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
request.Host = Constants.URLDomain;
request.KeepAlive = true;
request.Accept = Constants.Accept;
request.ContentType = "application/json";
request.Headers.Add("MyHeader1", "true");
request.Headers.Add("MyHeader2", "header2value");
request.Headers.Add("MyHeader3", "header3value");
request.Method = "POST";
byte[] postBytes = Encoding.UTF8.GetBytes(postFields);
request.ContentLength = postBytes.Length;
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(postBytes, 0, postBytes.Length);
}
string text = string.Empty;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream answer = response.GetResponseStream())
using (StreamReader reader = new StreamReader(answer, System.Text.Encoding.UTF8))
{
text = reader.ReadToEnd();
}
}
return text;
}
I was thinking of changing this code to use sockets so I could improve performance. Will there be actually an improvement if I change to sockets, and more importantly, how do will a sockets implementation of the above code look like?
I have problems with characters encoding received from http web response, I receive ? instead é.
I set the encoding to according Content-Type of web page that's text/javascript; charset=ISO-8859;
My code is:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(..);
request.Method = "GET";
request.AllowAutoRedirect = false;
request.Referer = "Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1";
request.Headers.Add("DNT", "1");
request.Accept = "text/html,application/xhtml+xml,application/xml";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader sr = new StreamReader(stream, Encoding.GetEncoding("iso-8859-1"));
char[] buf = new char[256];
int count;
StringBuilder buffer = new StringBuilder();
while ((count = sr.Read(buf, 0, 256)) > 0)
{
buffer.Append(buf, 0, count);
}
string responseStr = buffer.ToString();
Console.WriteLine(responseStr);
response.Close();
stream.Close();
sr.Close();
Can you tell me what is wrong with it?
Try adding the following before you make your request:
request.Headers.Add(HttpRequestHeader.AcceptCharset, "ISO-8859-1");
Btw, you should keep your StreamReader with ISO-8859-1 (instead of UTF8) if you want to try my proposed solution. Good luck!
Have you tried setting it at UTF-8?
Further more you send a referrer which I think you tried to set the UserAgent. The code below is the same as yours, but then does not go over the byte array and sets the useragent and utf8 encoding.
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
request.AllowAutoRedirect = false;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1";
request.Headers.Add("DNT", "1");
request.Accept = "text/html,application/xhtml+xml,application/xml";
using(var response = (HttpWebResponse)request.GetResponse())
using(var stream = response.GetResponseStream())
using (var sr = new StreamReader(stream, Encoding.UTF8))
{
string responseStr = sr.ReadToEnd();
Console.WriteLine(responseStr);
response.Close();
if (stream != null)
stream.Close();
sr.Close();
}
I have this code in C#:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.ContentType = "application/x-www-form-urlencoded";
request.Timeout = 30000;
request.Method = "POST";
request.KeepAlive = true;
request.AllowAutoRedirect = false;
Stream newStream = request.GetRequestStream();
newStream.Write(bPostData, 0, bPostData.Length);
byte[] buf = new byte[1025]; int read = 0; string sResp = "";
HttpWebResponse wResp = (HttpWebResponse)request.GetResponse();
Stream resp = wResp.GetResponseStream();
The line HttpWebResponse wResp =... just hangs (as in no response from the URL). I'm not sure where exactly its crashing (cause i dont even get an exception error). I tested the URL in IE and it works fine. I also checked the bPostData and that one has data in it.
Where is it going wrong?
Try closing the request stream in variable newStream. Maybe the API waits for it to be done.
You have to increase the limit:
ServicePointManager.DefaultConnectionLimit = 10; // Max number of requests
Try simplifying your code and faking a user agent. Maybe the site is blocking/throttling scrapers/bots. Also ensure your application/x-www-form-urlencoded HTTP POST values are properly encoded. For this I would recommend you WebClient:
using (var client = new WebClient())
{
client.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0";
var values = new NameValueCollection
{
{ "param1", "value1" },
{ "param2", "value2" },
};
byte[] result = client.UploadValues(url, values);
}
When I commented earlier, I had run your code at my office (heavily firewalled) I got the same result you did. Came home, tried again (less firewalled) it worked fine... I'm guessing you have a barrier there. I believe you are facing a firewall issue.
Use a content-length=0
Example:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(URL);
request.Method = "POST";
request.ContentLength = 0;
var requestStream = request.GetRequestStream();
HttpWebResponse res = (HttpWebResponse)request.GetResponse();
res.Close();