I am creating HTTP WebRequests in c# to navigate a website. I create the request, get a response and read it into a StreamReader. I then use the same request variable to create a new request and get a new response...
Below is my code:
HttpWebRequest request;
WebResponse response;
Stream responseStream;
StreamReader reader;
string responseFromServer;
request = (HttpWebRequest)WebRequest.Create("https://www.firstRequest.com");
//set cookies and headers here...
response = request.GetResponse();
responseStream = response.GetResponseStream();
reader = new StreamReader(responseStream);
responseFromServer = reader.ReadToEnd();
//second request using the same variables
request = (HttpWebRequest)WebRequest.Create("https://www.secondRequest.com");
response = request.GetResponse();
responseStream = response.GetResponseStream();
reader = new StreamReader(responseStream);
responseFromServer = reader.ReadToEnd();
I know I have to close the response, stream and streamreader to free up resources.
Do I have to do it in between each request or just at the end of everything?
Yes, you need to clean up after yourself every time. Just make use of the IDisposable features that you're ignoring:
So, instead, you could
using(response = request.GetResponse())
{
using(responseStream = response.GetResponseStream())
{
using(reader = new StreamReader(responseStream))
{
responseFromServer = reader.ReadToEnd();
}
}
}
or more concisely:
using(response = request.GetResponse())
using(responseStream = response.GetResponseStream())
using(reader = new StreamReader(responseStream))
{
responseFromServer = reader.ReadToEnd();
}
...now you don't need to worry about calling Close. The IDisposable implementation takes care of it for you (when the using block ends).
Alltough #spender has a point when he's saying you should be using using directives, I don't think that his answer answers your question or is particularly helpful.
When thinking about problems like this, think about the fundamentals of C# and objects that implement IDisposable. Classes that do so make use of unmanaged resources that need to be freed when the class is no longer used. This happens either at the end of the using-block or when Dispose is called.
After a call to Dispose however, the object should not be used anymore. This is a fundamental difference to Close. Using Close the response frees resources and the same object can be reused by another request.
MSDN for Close
Now let's consider what would happen if you simply overrode the response variable. classes are a reference type in C#, the variable would now reference a new object and the old response object would vanish in nirvana, waiting to be eventually collected by the GC without Dispose being invoked. Certainly not what you want.
So if you want to reuse variables, make sure you use Close
Related
I have code below that read ftp response stream and write data to two different files (test1.html & test2.html). The 2nd StreamReader throw a stream was not readable error. The response stream should be readable because it's not out of scope yet and the dispose shouldn't be called. Can someone explain why?
static void Main(string[] args)
{
// Make sure it is ftp
if (Properties.Settings.Default.FtpEndpoint.Split(':')[0] != Uri.UriSchemeFtp) return;
// Intitalize object to used to communicuate to the ftp server
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(Properties.Settings.Default.FtpEndpoint + "/test.html");
// Credentials
request.Credentials = new NetworkCredential(Properties.Settings.Default.FtpUser, Properties.Settings.Default.FtpPassword);
// Set command method to download
request.Method = WebRequestMethods.Ftp.DownloadFile;
// Get response
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
using (Stream output = File.OpenWrite(#"C:\Sandbox\vs_projects\FTP\FTP_Download\test1.html"))
using (Stream responseStream = response.GetResponseStream())
{
responseStream.CopyTo(output);
Console.WriteLine("Successfully wrote stream to test.html");
try
{
using (StreamReader reader = new StreamReader(responseStream))
{
string file = reader.ReadToEnd();
File.WriteAllText(#"C:\Sandbox\vs_projects\FTP\FTP_Download\test2.html", file);
Console.WriteLine("Successfully wrote stream to test2.html");
}
}
catch (Exception ex)
{
Console.WriteLine($"Exception: {ex}");
}
}
}
You can't read from the stream twice. After this call:
responseStream.CopyTo(output);
... you've already read all the data in the stream. There's nothing left to read, and you can't "rewind" the stream (e.g. seeking to the beginning) because it's a network stream. Admittedly I'd expect it to just be empty rather than throwing an error, but the details don't really matter much as it's not a useful thing to try to do.
If you want to make two copies of the same data, the best option is to copy it to disk as you're already doing, then read the file that you've just written.
(Alternatively, you could just read it into memory by copying to a MemoryStream, then you can rewind that stream and read from it repeatedly. But if you're already going to save it to disk, you might as well do that first.)
I have written two codes:
code block 1:
Stream dataStream;
using (var response = (HttpWebResponse)req.GetResponse())
{
dataStream = response.GetResponseStream();
}
//Open the stream using a StreamReader for easy access
using (var reader = new StreamReader(dataStream))
{
data = reader.ReadToEnd();
}
code block 2:
Stream dataStream;
using (var response = (HttpWebResponse)req.GetResponse())
{
dataStream = response.GetResponseStream();
//Open the stream using a StreamReader for easy access
using (var reader = new StreamReader(dataStream))
{
data = reader.ReadToEnd();
}
}
Code block 1 is throwing error: stream is not reachable. While progmatically i think both code will work same. I added using block in to whole statment in code block 2 and it's working. But I'm confused why is it throwing error in code block 1
Based upon the behavior you have specified, it would seem that when the HttpWebResponse is disposed, it disposes of the response stream.
Rohit, the first code block is throwing error because, once you close the braces for the using clause, the object is considered as disposed. i.e. the object is left for the GC to collect. The scope of the stream in code block 1 had expired before the second using statement.
The right way to access the stream is the second code block.
the using block provides you easy and correct use of objects that are disposable. therefor, anything that is initialized in the using statement will be constidered as disposed after the using block, even if you declare the variable outside of the using statement.
from msdn:
As a rule, when you use an IDisposable object, you should declare and
instantiate it in a using statement. The using statement calls the
Dispose method on the object in the correct way, and (when you use it
as shown earlier) it also causes the object itself to go out of scope
as soon as Dispose is called. Within the using block, the object is
read-only and cannot be modified or reassigned.
A Stream is not a block of data, but a way to read that data from "somewhere".
When you dispose of that response (by exiting the using-block) you break the connection to that "somewhere". This means you can't read from that stream anymore.
Your second example keeps that connection alive until you are done reading it.
The using statement ensures that the scoped object has the Dispose function called on it when the execution reaches the final bracket } which means the HttpWebResponse object is out of scope in the second using block.
Although you have saved your stream in the datastream variable the stream itself has not been read and that is done when you call ReadToEnd(). And as the HttpWebResponse is out of scope or Disposed then you are getting the exception you received.
Because post compilation (in IL), your code gets converted to the following
Code Block: 1
HttpWebResponse response=null;
Stream dataStream;
try
{
response = (HttpWebResponse) req.GetResponse();
dataStream = response.GetResponseStream();
}
finally
{
if(response!=null)
((IDisposable)response).Dispose();
}
StreamReader reader = null;
try
{
//DataStream is accessed AFTER response object is disposed
reader = new StreamReader(dataStream);
data = reader.ReadToEnd();
}
finally
{
if(reader!=null)
reader.Dispose();
}
Code Block: 2
HttpWebResponse response=null;
Stream dataStream;
try
{
response = (HttpWebResponse) req.GetResponse();
dataStream = response.GetResponseStream();
StreamReader reader = null;
try
{
//DataStream is accessed while response object is alive, and connected (not disposed)
reader = new StreamReader(dataStream);
data = reader.ReadToEnd();
}
finally
{
if (reader != null)
reader.Dispose();
}
}
finally
{
if(response!=null)
((IDisposable)response).Dispose();
}
As you can see, in CodeBlock1, you are trying to access stream which is trying to read from a Disposed (and dis-connected) HttWebResponse object.
I have been noticing what seems to be memory leaks in some of my services, and I am trying to make sure that I am handling resources properly. I have the following code used to send an external Http request. I am not sure if I am missing anything, or if there are too many using statements.
string str = string.Empty;
WebRequest request = WebRequest.Create(paramBuilder.ToString());
request.ContentLength = 0;
request.Method = "GET";
using (WebResponse response = request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream))
{
str = reader.ReadToEnd();
reader.Close();
return str;
}
}
}
}
You cannot have "too many" using statements.
Using using simply creates a scope for an IDisposable resource such that when it leaves that scope, IDisposable.Dispose() is invoked. A good implementation of IDisposable.Dispose() should tolerate the object already having been disposed, so extra Dispose() calls will not affect proper implementations of the pattern.
Your call to reader.Close() is not strictly necessary since the reader will be closed when it is disposed.
I do not see any obvious sources of memory leaks in the posted code. Try using a memory profiler such as the one included in Visual Studio to isolate the actual cause.
You only need curly braces after the last using statement.
using (WebResponse response = request.GetResponse())
using (Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream))
{
str = reader.ReadToEnd();
return str;
}
Otherwise your code looks okay.
I have the following code:
System.Net.WebRequest req = System.Net.WebRequest.Create(url);
req.Credentials = new NetworkCredential("admin", "password");
System.Net.WebResponse resp = req.GetResponse();
System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream());
var result = sr.ReadToEnd().Trim();
When I run the code the result is just an empty string. However when I step through the code the result is a string with data in it, as I was expecting, when I put a breakpoint on this line:
System.Net.WebResponse resp = req.GetResponse();
So I think the problem lies with this or the subsequent line. Not sure how to proceed, help would be appreciated.
I came across a similar issue whilst using CopyToAsync() on a WebResponse, it turned out that the Stream's pointer was ending up at the end of the Stream (it's pointer position was equal to it's length).
If this is the case, you can reset the pointer before reading the contents of the string with the following...
var responseStream = resp.GetResponseStream();
responseStream.Seek(0, SeekOrigin.Begin);
var sr = new StreamReader(responseStream);
var result = sr.ReadToEnd().Trim();
Although, since you're reading the stream directly, and not copying it into a new MemoryStream, this may not apply to your case.
May be "req.GetResponse();" taking more time..... When your putting the break point its getting time to complete the task.
You need to check
resp.StatusDescription
before
System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream());
I've have some code similar to this:
HttpWebRequest req;
HttpWebResponse response;
Stream receiveStream = null;
StreamReader readStream = null;
try
{
req = (HttpWebRequest)WebRequest.Create("someUrl"));
req.Credentials = CredentialCache.DefaultCredentials;
req.Method = "GET";
response = (HttpWebResponse)req.GetResponse();
receiveStream = response.GetResponseStream();
readStream = new StreamReader(receiveStream, Encoding.Default);
return readStream.ReadToEnd();
}
catch
{
return "Error";
}
finally
{
readStream = null;
receiveStream = null;
response = null;
req = null;
}
Should this code have readStream.Dispose() and responseStream.Dispose() instead of setting both to null?
It's almost always a mistake to set local variables to null, unless you want to actually use that value later on. It doesn't force garbage collection any earlier - if you're not going to read from the variable later, the garbage collector can ignore the reference (when not in debug mode).
However, it's almost always correct to close streams - ideally in a using statement for simplicity.
It's also almost always wrong to have a bare "catch" block like that. Do you really want to handle anything going wrong, including things like OutOfMemoryException?
I would rewrite your code as:
HttpWebRequest req = (HttpWebRequest) WebRequest.Create("someUrl"));
req.Credentials = CredentialCache.DefaultCredentials;
req.Method = "GET";
using (WebResponse response = req.GetResponse())
{
using (StreamReader reader = new StreamReader(response.GetResponseStream(),
Encoding.Default))
{
return reader.ReadToEnd();
}
}
Now if something goes wrong, the exception will be propagated to the caller. You might want to catch a few specific exceptions, but it's generally not a good idea to represent errors using a value which could have been a valid "normal" response.
Finally, are you really sure you want Encoding.Default? That's the default encoding of the local machine - you normally want the encoding indicated by the response itself.
It should have using [which calls Dispose()].
Yes, Dispose() them.
Even better to do something like
using (HttpWebResponse response = (HttpWebResponse)req.GetResponse() )
using (Stream receiveStream = response.GetResponseStream() )
using (readStream = new StreamReader(receiveStream, Encoding.Default) )
{
return readStream.ReadToEnd();
}
A using(x) {} block will be rewritten (by the compiler)
as a try {} finally {x.Dispose();}
Note that the WebRequest is not IDisposable.
Also note that the following lines accomplish the same thing as all of your code:
using (var client = new System.Net.WebClient())
{
client.Encoding = ...;
client.Credentials = ...;
return client.DownloadString("SomeUrl");
}
Yes. Pretty much anything that implements a Dispose() method must have its Dispose() method called. You can implicitly call it in a with the 'using' statement:
using(StreamReader stream = GetStream())
{
stream.DoStuff();
}
Yes.
When you set to null, it only nulls the reference. It doesn't run any cleanup code the creator of the Stream class wrote.
You may also want to consider the using(){ } statement which handles this for you on IDisposable types.
Example:
using (MyDisposableObject mdo = new MyDisposableObject)
{
// Do some stuff with mdo
// mdo will automatically get disposed by the using clause
}
No, you should call Dispose or Close
Safest method:
try {
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("someUrl");
request.Credentials = CredentialCache.DefaultCredentials;
request.Method = "GET";
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse()) {
using (StreamReader reader = new StreamReader(response.GetResponseStream(), Encoding.Default)) {
return reader.ReadToEnd();
}
}
} catch {
return "Error";
}
There's no need to dispose of the response.GetResponseStream() stream explicitly because the attached StreamReader will dispose it for you.
EDIT: I agree with the other answers - catching exceptions like that is very bad practice. I just left it in for comparison purposes. :-)
Yes - you should explicitly call Dispose() on classes that implement IDisposable after you have used them - this ensures all their resources get cleaned up in a timely fashion. Wrapping the variable in a using() does the same thing (which adds wrapping code that calls Dispose for you):
using (StreamReader reader = new StreamReader()) {
// do stuff with reader
}
There are a few gotchas in the .net libraries. Stream is one, and the other is much of the Imaging API. These entities that use certain system resources don't garbage collect the attached system resources.
If anything uses the IDisposable API, the best thing to do is wrap it in a using block, as people have pointed out above.
Read up on "using", and keep it in mind whenever you're dealing with file handles or images.
Really, the question has been answered but I do want to elaborate on one thing.
Any time an object implements the IDisposable interface, you should dispose it with the Dispose method or (even better) use the using statement.
If you are ever faced with this question in the future, just find out which interfaces it implements. Once you see IDisposable you know to dispose it.
If you need to clear the stream use null; Otherwise, use the Dispose(); method if your application no longer requires the use of the stream.