Develop Reference

GetRequestStream method and hanging thread

Assume I have the following code:
private string PostData(string functionName, string parsedContent)
{
string url = // some url;
var http = (HttpWebRequest)WebRequest.Create(new Uri(url));
http.Accept = "application/json";
http.ContentType = "application/json";
http.Method = "POST";
http.Timeout = 15000; // 15 seconds
Byte[] bytes = Encoding.UTF8.GetBytes(parsedContent);
using (Stream newStream = http.GetRequestStream())
{
newStream.Write(bytes, 0, bytes.Length);
}
using (WebResponse response = http.GetResponse())
{
using (var stream = response.GetResponseStream())
{
var sr = new StreamReader(stream);
var content = sr.ReadToEnd();
return content;
}
}
}
I set up a breakpoint over this line of code:
using (Stream newStream = http.GetRequestStream())
before http.GetRequestStream() gets executed. Here is a screenshot of my active threads:
This whole method is running in background thread with ThreadId = 3 as you can see.
After pressing F10 we get http.GetRequestStream() method executed. And here is an updated screenshot of active threads:
As you can see, now we have one extra active thread that is in state of waiting. Probably the method http.GetRequestStream() spawns it. Everything is fine, but.. this thread keeps hanging like that for the whole app lifecycle, which seems not to be the intended behaviour.
Am I misusing GetRequestStream somehow?

If I use ilspy it looks like the request is send asynchronously. That would explain the extra thread.
Looking a little bit deeper the HttpWebRequest creates a static TimerQueue with one thread and a never ending loop, that has a Monitor.WaitAny in it. Every webrequest in the appdomain will register a timer callback for timeout handling and all those callbacks are handled by that thread. Due to it being static that instance will never get garbage collected and therefore it will keep hold of the thread.
It did register for the AppDomain.Unload event so if that fires it will clean up it's resources including any threads.
Do notice that these are all internal classes and those implementation details might change at any time.

Multithread HttpWebRequest hangs randomly on responseStream

I'm coding a multithreaded web-crawler that performs a lot of concurrent httpwebrequests every second using hundreds of threads, the application works great but sometimes(randomly) one of the webrequests hangs on the getResponseStream() completely ignoring the timeout(this happen when I perform hundreds of requests concurrently) making the crawling process never end, the strange thing is that with fiddler this never happen and the application never hang, it is really hard to debug because it happens randomly.
I've tried to set
Keep-Alive = false
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
but I still get the strange behavior, any ideas?
Thanks
HttpWebRequest code:
public static string RequestHttp(string url, string referer, ref CookieContainer cookieContainer_0, IWebProxy proxy)
{
string str = string.Empty;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
request.UserAgent = randomuseragent();
request.ContentType = "application/x-www-form-urlencoded";
request.Accept = "*/*";
request.CookieContainer = cookieContainer_0;
request.Proxy = proxy;
request.Timeout = 15000;
request.Referer = referer;
//request.ServicePoint.MaxIdleTime = 15000;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
List<byte> list = new List<byte>();
byte[] buffer = new byte[0x400];
int count = responseStream.Read(buffer, 0, buffer.Length);
while (count != 0)
{
list.AddRange(buffer.ToList<byte>().GetRange(0, count));
if (list.Count >= 0x100000)
{
break;
}
count = 0;
try
{
HERE IT HANGS SOMETIMES ---> count = responseStream.Read(buffer, 0, buffer.Length);
continue;
}
catch
{
continue;
}
}
//responseStream.Close();
int num2 = 0x200 * 0x400;
if (list.Count >= num2)
{
list.RemoveRange((num2 * 3) / 10, list.Count - num2);
}
byte[] bytes = list.ToArray();
str = Encoding.Default.GetString(bytes);
Encoding encoding = Encoding.Default;
if (str.ToLower().IndexOf("charset=") > 0)
{
encoding = GetEncoding(str);
}
else
{
try
{
encoding = Encoding.GetEncoding(response.CharacterSet);
}
catch
{
}
}
str = encoding.GetString(bytes);
// response.Close();
}
}
return str.Trim();
}

The Timeout property "Gets or sets the time-out value in milliseconds for the GetResponse and GetRequestStream methods." The default value is 100,000 milliseonds (100 seconds).
The ReadWriteTimeout property, "Gets or sets a time-out in milliseconds when writing to or reading from a stream." The default is 300,000 milliseconds (5 minutes).
You're setting Timeout, but leaving ReadWriteTimeout at the default, so your reads can take up to five minutes before timing out. You probably want to set ReadWriteTimeout to a lower value. You might also consider limiting the size of data that you download. With my crawler, I'd sometimes stumble upon an unending stream that would eventually result in an out of memory exception.
Something else I noticed when crawling is that sometimes closing the response stream will hang. I found that I had to call request.Abort to reliably terminate a request if I wanted to quit before reading the entire stream.

There is nothing apparent in the code you provided.
Why did you comment response.Close() out?
Documentation hints that connections may run out if not explicitly closed. The response getting disposed may close the connection but just releasing all the resources is not optimal I think. Closing the response will also close the stream so that is covered.
The system hanging without timeout can be just a network issue making the response object a dead duck or the problem is due the high number of threads resulting in memory fragmentation.
Looking at anything that may produce a pattern may help find the source:
How many threads are typically running (can you bundle request sets in less threads)
How is the network performance at the time the thread stopped
Is there a specific count or range when it happens
What data was processed last when it happened (are there any specific control characters or sequences of data that can upset the stream)
Want to ask more questions but not enough reputation so can only reply.
Good luck!

Below is some code that does something similar, it's also used to access multiple web sites, each call is in a different task. The difference is that I only read the stream once and then parse the results. That might be a way to get around the stream reader locking up randomly or at least make it easier to debug.
try
{
_webResponse = (HttpWebResponse)_request.GetResponse();
if(_request.HaveResponse)
{
if (_webResponse.StatusCode == HttpStatusCode.OK)
{
var _stream = _webResponse.GetResponseStream();
using (var _streamReader = new StreamReader(_stream))
{
string str = _streamReader.ReadToEnd();

TcpListener intermittently not working

Attempting to write a basic proxy I have implemented code along the following lines as a start.
private Thread _proxyThread;
public void Start()
{
this._proxyThread = new Thread(StartListener);
this._proxyThread.Start();
}
protected void StartListener(object data)
{
this._listener = new TcpListener(IPAddress.Any, 1234);
this._listener.Start();
while (true)
{
TcpClient client = this._listener.AcceptTcpClient();
while (client.Connected)
{
NetworkStream stream = client.GetStream();
StringBuilder request = new StringBuilder();
byte[] bytes = new byte[1024];
while (stream.DataAvailable && stream.CanRead)
{
int i = stream.Read(bytes, 0, bytes.Length);
request.Append(System.Text.Encoding.ASCII.GetString(bytes, 0, i));
}
if (stream.CanWrite)
{
byte[] response = System.Text.Encoding.Default.GetBytes("HTTP/1.1 200 OK" + Environment.NewLine + "Content-length: 4\r\n\r\n" + "Test");
stream.Write(response, 0, response.Length);
}
client.Close();
}
}
}
I then set my LAN proxy like this
My current goal (as you can hopefully deduce from my code) is to return just a text value with no headers or anything.
The problem that I am running into is that any of my attempts to run the code as is results in an intermittent behaviour where Chrome and IE9 will usually just return a ERR_CONNECTION_ABORTED or Internet Explorer cannot display the webpage respectively and very occasionally display the expected output "TEST".
I originally tried using a asynchronous approach (Using BeginAcceptTcpClient()) but have reverted back to this simpler technique because I thought this problem might stem from me doing something incorrectly in the asynchronous implementation but it seems that the cause is something else.
Could anyone please provide any guidance?

After your first iteration of the loop while (client.Connected) you close the connection. Only close it when you are done transferring all data.

It's not whether it's synchronous or asynchronous, you're just not responding with a proper HTTP response. Take a tool like Fiddler and look at the data being received when you request a regular webpage.
If you return this, it should work:
HTTP/1.1 200 OK
Content-length: 4
TEST

C# and running of HttpListener in background

I created simple HttpListener that listens to port 9090 and depending on a request's URL writes some info to the console.
But I'm stuck :( I thought of multithreading, event based system, but I dind't manage to do anything with it.
Here is the code of my listener that I launched as a separate console app:
string urlTemplate = String.Format("/prefix/{0}/suffix", id);
string prefix = String.Format("http://localhost:9090/");
HttpListener listener = new HttpListener();
listener.Prefixes.Add(prefix);
listener.Start();
Console.WriteLine("Listening to {0}...", prefix);
while (true)
{
HttpListenerContext context = listener.GetContext();
HttpListenerRequest request = context.Request;
//Response object
HttpListenerResponse response = context.Response;
//Construct response
if (request.RawUrl.Contains(urlTemplate) && request.HttpMethod == "POST")
{
string requestBody;
Stream iStream = request.InputStream;
Encoding encoding = request.ContentEncoding;
StreamReader reader = new StreamReader(iStream, encoding);
requestBody = reader.ReadToEnd();
Console.WriteLine("POST request on {0} with body = [{1}]", request.RawUrl, requestBody);
response.StatusCode = (int)HttpStatusCode.OK;
//Return a response
using (Stream stream = response.OutputStream) { }
}
else
{
response.StatusCode = (int)HttpStatusCode.BadRequest;
Console.WriteLine("Invalid HTTP request: [{0}] {1}", request.HttpMethod, request.Url);
using (Stream stream = response.OutputStream) { }
}
}
I decided to use it as an utility for unit tests (maybe somewhere else). So when test starts I need to configure the listener, run it, then make some requests and receive the info (which listener wrote earlier to Console), and at the end of the test stop the listener.
My main idea was to incapsulate this listener to separate class MyHttpListener which has methods: StartListener(), StopListener().
But when I call StartListener() my test freezes because of infinite while loop. I tried to create separate background thread or event based system, but my lack of experience with them, prevents me from doing it. I've already spent a lot of time trying to find the solution, but all for nothing.
Hope you can help me finding the solution for such trivial task.
Thanks in advance.

One of the responder's variant (it seems he deleted his post) looked good, but it didn't work for me. I tried to fix things in order it started working, but at the end that variant gave me an idea how to solve the problem - with multithreading and events :)
Here's what I have now and it works:
public delegate void HttpListenerRequestHandler(object sender, HttpListenerEventArgs e);
public event HttpListenerRequestHandler OnCorrectRequest;
...
if(OnCorrectRequest != null)
OnCorrectRequest(this, new HttpListenerEventArgs(response));
lock (threadLock)
{
Console.WriteLine("POST request on {0} with body = [{1}]", request.RawUrl, requestBody);
}
...
public class HttpListenerEventArgs : EventArgs
{
public readonly HttpListenerResponse response;
public HttpListenerEventArgs(HttpListenerResponse httpResponse)
{
response = httpResponse;
}
}
In order to receive detailed responses from HttpListener in main thread, I use the following:
private HttpListenerResponse response;
public void HttpCallbackRequestCorrect(object sender, HttpListenerEventArgs e)
{
response = e.response;
Console.WriteLine("{0} sent: {1}", sender, e.response.StatusCode);
}
Unfortunately I have the following exception which I don't know how to handle:
System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request
at System.Net.HttpListener.GetContext()

How To Make Parallel HTTP Requests

I have a console app that makes a single request to a web page and returns its server status eg. 200, 404, etc..
I'd like to apply following changes:
List of User Inputs:
Url to request
How many parallel connections to use(concurrent users)
How long(seconds) to submit as many requests as it can
List of Outputs:
Show Total Fetches
Show Fetches per Second
Show Average Response Time (ms)
I imagine the best way to do it is to run multiple http fetches in parallel and run in a single process, so it doesn't bog down the client machine.
I really like C# but I'm still new to it. I've researched other articles about this but I don't fully understand them so any help would be greatly appreciated.
My Code:
static void Main(string[] args)
{
try
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("http://10.10.1.6/64k.html");
webRequest.AllowAutoRedirect = false;
HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse();
//Returns "MovedPermanently", not 301 which is what I want.
int i_goodResponse = (int)response.StatusCode;
string s_goodResponse = response.StatusCode.ToString();
Console.WriteLine("Normal Response: " + i_goodResponse + " " + s_goodResponse);
Console.ReadLine();
}
catch (WebException we)
{
int i_badResponse = (int)((HttpWebResponse)we.Response).StatusCode;
string s_badResponse = ((HttpWebResponse)we.Response).StatusCode.ToString();
Console.WriteLine("Error Response: " + i_badResponse + " " + s_badResponse);
Console.ReadLine();
}
}
Some possible code that I found:
void StartWebRequest()
{
HttpWebRequest webRequest = ...;
webRequest.BeginGetResponse(new AsyncCallback(FinishWebRequest), webRequest);
}
void FinishWebRequest(IAsyncResult result)
{
HttpWebResponse response = (result.AsyncState as HttpWebRequest).EndGetResponse(result) as HttpWebResponse;
}

This is actually a good place to make use of the Task Parallel Library in .NET 4.0. I have wrapped your code in a Parallel.For block which will execute a number of sets of requests in parallel, collate the total times in each parallel branch, and then calculate the overall result afterwards.
int n = 16;
int reqs = 10;
var totalTimes = new long[n];
Parallel.For(0, n, i =>
{
for (int req = 0; req < reqs; req++)
{
Stopwatch w = new Stopwatch();
try
{
w.Start();
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create("http://localhost:42838/Default.aspx");
webRequest.AllowAutoRedirect = false;
HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse();
w.Stop();
totalTimes[i] += w.ElapsedMilliseconds;
//Returns "MovedPermanently", not 301 which is what I want.
int i_goodResponse = (int)response.StatusCode;
string s_goodResponse = response.StatusCode.ToString();
Console.WriteLine("Normal Response: " + i_goodResponse + " " + s_goodResponse);
}
catch (WebException we)
{
w.Stop();
totalTimes[i] += w.ElapsedMilliseconds;
int i_badResponse = (int)((HttpWebResponse)we.Response).StatusCode;
string s_badResponse = ((HttpWebResponse)we.Response).StatusCode.ToString();
Console.WriteLine("Error Response: " + i_badResponse + " " + s_badResponse);
}
}
});
var grandTotalTime = totalTimes.Sum();
var reqsPerSec = (double)(n * reqs * 1000) / (double)grandTotalTime;
Console.WriteLine("Requests per second: {0}", reqsPerSec);
The TPL is very useful here, as it abstracts away the detail of creating multiple threads of exececution within your process, and running each parallel branch on these threads.
Note that you still have to be careful here - we cannot share state which is updated during the tasks between threads, hence the array for totalTimes which collates the totals for each parallel branch, and only summed up at the very end, once the parallel execution is complete. If we didn't do this, we are open to the possibility of a race condition - where two seperate threads attempt to update the total count simultaneously, potentially corrupting the result.
I hope this makes sense and is useful as a start for you (I only calculate requests per second here, the other stats should be relatively easy to add). Add comments if you need further clarifications.

You have already answered your own question, you can use BeginGetResponse to start async request.
Another, and more convenient method might be using WebClient class, if you are more familiar with events then with AsyncResult.
DownloadDataCompletedEventHandlerd