I am currently facing a strange problem. I'm testing website response times, but when the method is looped on the third time (of timing the connection) it hangs:
internal class Program
{
Console.ReadLine();
loop();
}
}
The output before it hangs is: "HTTP Response Timer", so I assume it's something to do with the running instance.
You have to close the response. Otherwise you reach the maximum of open connections.
Use a using statement, this is the most simple way to close and dispose the response:
using(HttpWebResponse response = (HttpWebResponse)request.GetResponse()){
// .. here some action if any
}
You're not disposing of the response, which means it's hanging onto the connection from the previous requests. The new request is then waiting to get that connection from the pool, and blocking because the old response still "owns" the connection Just change your code to:
// You weren't really using the HttpWebResponse class anyway, so why cast?
using (var response = request.GetResponse())
{
}
Did you really mean to recurse though? Why aren't you looping using:
while(true)
{
GetResponse();
}
or something like that?
Related
About:
I have this Windows Form application which every 60 seconds it captures information from two common web pages, do some simple string treatment with the result and do something (or not) based in the result.
One of those sites doesn't have any protection, so I can easily get it's HTML code using HttpWebRequest and it's HttpWebResponse.GetResponseStream().
The other one has some code protection and I can't use the same approach. The solution was use the WebBrowser class to select all text of the site and copy to the clipboard, as Jake Drew posted here (method 1).
Extra information:
When the timer reachs 1 min, each method is asynchronously execuded using Task. At the end of each Task the main thread will search some information in those texts and take or not some decisions based in the result. After this process, not even the captured text will relevant anymore. Basically everything can be wipe out from memory, since I'll get everything new and process it in about 1 minute.
Problem:
Everything is working fine but the problem is that it's gradually increasing the memory usage (about 20mb for each ticking), which are unecessary as I said before I don't need to maintain data in running in memory more than I had in the begin of app's execution:
and after comparing two snapshots I've found these 3 objects. Apparently they're responsible for that excess of memory usage:
So, even after I put the main execution in Tasks and do everything I could to help the Garbage Collector, I still have this issue.
What else could I do to avoid this issue or dump the trash from memory??
Edit:
Here's the code that is capturing the HTML of the page using HttpWebRequest:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse()) {
if (response.StatusCode == HttpStatusCode.OK) {
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = null;
if (response.CharacterSet == null) {
readStream = new StreamReader(receiveStream);
} else {
readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
}
PB_value = readStream.ReadToEnd();
readStream.Close(); //Ensure
}
response.Close(); //Ensure
}
Solved:
After some research I've found a solution. I actually feel kind of ashamed because it is a quite simple solution that I haven't tried before, still, it's important to share.
The first thing I did was create an Event to identify when my two Tasks were finished then I assigned two functions to this event. The first function forced the Garbage Collector (GC.Collect()). The second function disposed the two Tasks, since all the main processes were done inside them (T.Dispose()). Then I've got the result I wanted:
I'm developing a class that can send fttp requests, it has a utility method that can execute different types of ftp methods:
private FtpWebResponse DoFttpRequest(Uri uri, NetworkCredential credentials, string method, string file = null)
{
var request = (FtpWebRequest)WebRequest.Create(uri);
request.Credentials = credentials;
request.Method = method;
if (!string.IsNullOrEmpty(file))
{
using (var stream = request.GetRequestStream())
using (var writer = new StreamWriter(stream))
{
writer.Write(file);
}
}
return (FtpWebResponse)request.GetResponse();
}
As you can see, this methods executes ftp method and returns response stream to the caller. Here is the client method that uses this method to write string contents to a file through ftp:
public void WriteToFile(string path, string contents)
{
var uri = new Uri(path);
using (var ftpResponse = DoFttpRequest(uri, _credentials, Ftp.UploadFile, contents)) { }
}
As you can see, here I'm using empty using statement using (var ftpResponse = DoFttpRequest(uri, _credentials, Ftp.UploadFile, contents)) { } to dispose of the received stream.
Is this a good approach to dispose object like that? Is it even necessary to dispose this stream, since it will probably be disposed by the garbage collector anyway?
Is it even necessary to dispose this stream, since it will probably be
disposed by the garbage collector anyway
You can use this simple code to see how not disposing response stream might completely break application. I use http request instead of ftp for simlicity of testing, but that applies equally to ftp requests.
public class Program {
static void Main(string[] args) {
// this value is *already* 2 by default, set for visibility
ServicePointManager.DefaultConnectionLimit = 2;
// replace example.com with real site
DoFttpRequest("http://example.com");
DoFttpRequest("http://example.com");
DoFttpRequest("http://example.com");
Console.ReadLine();
}
private static HttpWebResponse DoFttpRequest(string uri) {
var request = (HttpWebRequest) WebRequest.Create(uri);
var response = (HttpWebResponse) request.GetResponse();
Console.WriteLine("got response");
return response;
}
}
Note that you are not disposing HttpWebResponse. What will happen is you will see 2 "got response" messages in console and then application will hang trying to get response 3rd time. That's because concurrent connections limit per endpoint (per host) is 2, so while 2 connections to the host (example.com here) are "in progress" - next connection to the same host will have to wait for them to complete. Because you don't dispose response - those connections will not be "completed" until GC collects them. Until then - your application hangs and then fails by timeout (if request.Timeout is set to some reasonable time). All subsequent requests also hang then fail by timeout. If you dispose responses - application will work as expected.
So always dispose things that are disposable. Using block is not necessary, you can just do DoFtpRequest(..).Dispose(). But if you prefer empty using - at least don't declare unnecessary variable, just do using (DoFttpRequest(..)) {}. One thing to note when choosing between empty using and Dispose is the possibility of null being returned by DoFtpRequest, because if it will return null - explicit Dispose will throw NullReferenceException while empty using will just ignore it (you can do DoFttpRequest(...)?.Dispose(); if you expect nulls but don't want to use using).
What using statement does is actually execute some sort of code and then simply calls the Dispose method.
That's why u can use it only types inherits from IDisposible interface(in most case)
So you don't really have to use using statement. Just simply call
DoFttpRequest(uri, _credentials, Ftp.UploadFile, contents)).Dispose()
If you dont Dispose and object by yourself the Garbage Collector automatically disposes it after the scope completed.
You don't have to think much about memory when you use high level languages like c#, java ... They are called Memory Managed Languages. They handle thoose kind of staff for you.
I've been asked to write a small program which checks that all the pages we have online are not erroring.
To do this, I use the following code (where pathsToCheck is List, each string is a URL like http://www.domain.com/webpage)
foreach (string path in pathsToCheck)
{
HttpWebResponse response = null;
try
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(path);
webRequest.AllowAutoRedirect = true;
response = (HttpWebResponse)webRequest.GetResponse();
System.Diagnostics.Debug.Assert(response.StatusDescription == "OK", "Look into this, it doesn't like the response code");
System.Threading.Thread.Sleep(1000);
}
catch (Exception ex)
{
Console.WriteLine("Failed : " + path);
}
finally
{
Write(--totalPathsToCheck);
}
}
The problem I am having is it always fails (timesout) from the third item in the list (everything fails from the third). Naturally, I guessed there must be a fault with the third item, but there isn't.
Since the first item doesn't time out, I created a new list, with 5 items, all of the same URL (one I know doesn't time out). The same issue occurs, on the third iteration, it times out and continues to time out for the remainder of the list.
I then decided to test a different URL (on a different domain) and the same issue persists.
I added the sleep to the code, and increased it in-case there were too many requests within a given period but that made no differences.
What should I be doing?
You need to close your connections. Add
response.Close();
From http://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.close.aspx:
The Close method closes the response stream and releases the connection to the resource for reuse by other requests.
You must call either the Stream.Close or the HttpWebResponse.Close method to close the stream and release the connection for reuse. It is not necessary to call both Stream.Close and HttpWebResponse.Close, but doing so does not cause an error. Failure to close the stream can cause your application to run out of connections.
Given an async controller:
public class MyController : AsyncController
{
[NoAsyncTimeout]
public void MyActionAsync() { ... }
public void MyActionCompleted() { ... }
}
Assume MyActionAsync kicks off a process that takes several minutes. If the user now goes to the MyAction action, the browser will wait with the connection open. If the user closes his browser, the connection is closed. Is it possible to detect when that happens on the server (preferably inside the controller)? If so, how? I've tried overriding OnException but that never fires in this scenario.
Note: I do appreciate the helpful answers below, but the key aspect of this question is that I'm using an AsyncController. This means that the HTTP requests are still open (they are long-lived like COMET or BOSH) which means it's a live socket connection. Why can't the server be notified when this live connection is terminated (i.e. "connection reset by peer", the TCP RST packet)?
I realise this question is old, but it turned up frequently in my search for the same answer.
The details below only apply to .Net 4.5
HttpContext.Response.ClientDisconnectedToken is what you want. That will give you a CancellationToken you can pass to your async/await calls.
public async Task<ActionResult> Index()
{
//The Connected Client 'manages' this token.
//HttpContext.Response.ClientDisconnectedToken.IsCancellationRequested will be set to true if the client disconnects
try
{
using (var client = new System.Net.Http.HttpClient())
{
var url = "http://google.com";
var html = await client.GetAsync(url, HttpContext.Response.ClientDisconnectedToken);
}
}
catch (TaskCanceledException e)
{
//The Client has gone
//you can handle this and the request will keep on being processed, but no one is there to see the resonse
}
return View();
}
You can test the snippet above by putting a breakpoint at the start of the function then closing your browser window.
And another snippet, not directly related to your question but useful all the same...
You can also put a hard limit on the amount of time an action can execute for by using the AsyncTimeout attribute. To use this use add an additional parameter of type CancellationToken. This token will allow ASP.Net to time-out the request if execution takes too long.
[AsyncTimeout(500)] //500ms
public async Task<ActionResult> Index(CancellationToken cancel)
{
//ASP.Net manages the cancel token.
//cancel.IsCancellationRequested will be set to true after 500ms
try
{
using (var client = new System.Net.Http.HttpClient())
{
var url = "http://google.com";
var html = await client.GetAsync(url, cancel);
}
}
catch (TaskCanceledException e)
{
//ASP.Net has killed the request
//Yellow Screen Of Death with System.TimeoutException
//the return View() below wont render
}
return View();
}
You can test this one by putting a breakpoint at the start of the function (thus making the request take more than 500ms when the breakpoint is hit) then letting it run out.
Does not Response.IsClientConnected work fairly well for this? I have just now tried out to in my case cancel large file uploads. By that I mean if a client abort their (in my case Ajax) requests I can see that in my Action. I am not saying it is 100% accurate but my small scale testing shows that the client browser aborts the request, and that the Action gets the correct response from IsClientConnected.
It's just as #Darin says. HTTP is a stateless protocol which means that there are no way (by using HTTP) to detect if the client is still there or not. HTTP 1.0 closes the socket after each request, while HTTP/1.1 can keep it open for a while (a keep alive timeout can be set as a header). That a HTTP/1.1 client closes the socket (or the server for that matter) doesn't mean that the client has gone away, just that the socket hasn't been used for a while.
There are something called COMET servers which are used to let client/server continue to "chat" over HTTP. Search for comet here at SO or on the net, there are several implementations available.
For obvious reasons the server cannot be notified that the client has closed his browser. Or that he went to the toilet :-) What you could do is have the client continuously poll the server with AJAX requests at regular interval (window.setInterval) and if the server detects that it is no longer polled it means the client is no longer there.
As part of a suite of integration tests I am writing, I want to assert that my server behaves correctly when a client HTTP request terminates early, before all the response data has been sent.
Is it possible to create an HTTP request and terminate it after receiving just a few bytes of data in C#?
You don't have to read all bytes out fo the response. Just read as many bytes as you want and then return from your test.
You can do so more or less like this:
Stream myStream = resp.GetResponseStream();
myStream.Read(bufferArray, 0, 1); //read 1 byte into bufferArray
return;
You may find the documentation on WebReponse useful.
Just start the call asynchronously using, say a background worker, and then close the thread/channel.
I found a solution which works for me. I just close the response after getting it. This seems to leave me with the response headers, but closes the connection before the server is finished sending.
var response = request.getResponse();
response.Close();
// Assert that server has dealt with closed response correctly