I have code below that read ftp response stream and write data to two different files (test1.html & test2.html). The 2nd StreamReader throw a stream was not readable error. The response stream should be readable because it's not out of scope yet and the dispose shouldn't be called. Can someone explain why?
static void Main(string[] args)
{
// Make sure it is ftp
if (Properties.Settings.Default.FtpEndpoint.Split(':')[0] != Uri.UriSchemeFtp) return;
// Intitalize object to used to communicuate to the ftp server
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(Properties.Settings.Default.FtpEndpoint + "/test.html");
// Credentials
request.Credentials = new NetworkCredential(Properties.Settings.Default.FtpUser, Properties.Settings.Default.FtpPassword);
// Set command method to download
request.Method = WebRequestMethods.Ftp.DownloadFile;
// Get response
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
using (Stream output = File.OpenWrite(#"C:\Sandbox\vs_projects\FTP\FTP_Download\test1.html"))
using (Stream responseStream = response.GetResponseStream())
{
responseStream.CopyTo(output);
Console.WriteLine("Successfully wrote stream to test.html");
try
{
using (StreamReader reader = new StreamReader(responseStream))
{
string file = reader.ReadToEnd();
File.WriteAllText(#"C:\Sandbox\vs_projects\FTP\FTP_Download\test2.html", file);
Console.WriteLine("Successfully wrote stream to test2.html");
}
}
catch (Exception ex)
{
Console.WriteLine($"Exception: {ex}");
}
}
}
You can't read from the stream twice. After this call:
responseStream.CopyTo(output);
... you've already read all the data in the stream. There's nothing left to read, and you can't "rewind" the stream (e.g. seeking to the beginning) because it's a network stream. Admittedly I'd expect it to just be empty rather than throwing an error, but the details don't really matter much as it's not a useful thing to try to do.
If you want to make two copies of the same data, the best option is to copy it to disk as you're already doing, then read the file that you've just written.
(Alternatively, you could just read it into memory by copying to a MemoryStream, then you can rewind that stream and read from it repeatedly. But if you're already going to save it to disk, you might as well do that first.)
Related
I have written two codes:
code block 1:
Stream dataStream;
using (var response = (HttpWebResponse)req.GetResponse())
{
dataStream = response.GetResponseStream();
}
//Open the stream using a StreamReader for easy access
using (var reader = new StreamReader(dataStream))
{
data = reader.ReadToEnd();
}
code block 2:
Stream dataStream;
using (var response = (HttpWebResponse)req.GetResponse())
{
dataStream = response.GetResponseStream();
//Open the stream using a StreamReader for easy access
using (var reader = new StreamReader(dataStream))
{
data = reader.ReadToEnd();
}
}
Code block 1 is throwing error: stream is not reachable. While progmatically i think both code will work same. I added using block in to whole statment in code block 2 and it's working. But I'm confused why is it throwing error in code block 1
Based upon the behavior you have specified, it would seem that when the HttpWebResponse is disposed, it disposes of the response stream.
Rohit, the first code block is throwing error because, once you close the braces for the using clause, the object is considered as disposed. i.e. the object is left for the GC to collect. The scope of the stream in code block 1 had expired before the second using statement.
The right way to access the stream is the second code block.
the using block provides you easy and correct use of objects that are disposable. therefor, anything that is initialized in the using statement will be constidered as disposed after the using block, even if you declare the variable outside of the using statement.
from msdn:
As a rule, when you use an IDisposable object, you should declare and
instantiate it in a using statement. The using statement calls the
Dispose method on the object in the correct way, and (when you use it
as shown earlier) it also causes the object itself to go out of scope
as soon as Dispose is called. Within the using block, the object is
read-only and cannot be modified or reassigned.
A Stream is not a block of data, but a way to read that data from "somewhere".
When you dispose of that response (by exiting the using-block) you break the connection to that "somewhere". This means you can't read from that stream anymore.
Your second example keeps that connection alive until you are done reading it.
The using statement ensures that the scoped object has the Dispose function called on it when the execution reaches the final bracket } which means the HttpWebResponse object is out of scope in the second using block.
Although you have saved your stream in the datastream variable the stream itself has not been read and that is done when you call ReadToEnd(). And as the HttpWebResponse is out of scope or Disposed then you are getting the exception you received.
Because post compilation (in IL), your code gets converted to the following
Code Block: 1
HttpWebResponse response=null;
Stream dataStream;
try
{
response = (HttpWebResponse) req.GetResponse();
dataStream = response.GetResponseStream();
}
finally
{
if(response!=null)
((IDisposable)response).Dispose();
}
StreamReader reader = null;
try
{
//DataStream is accessed AFTER response object is disposed
reader = new StreamReader(dataStream);
data = reader.ReadToEnd();
}
finally
{
if(reader!=null)
reader.Dispose();
}
Code Block: 2
HttpWebResponse response=null;
Stream dataStream;
try
{
response = (HttpWebResponse) req.GetResponse();
dataStream = response.GetResponseStream();
StreamReader reader = null;
try
{
//DataStream is accessed while response object is alive, and connected (not disposed)
reader = new StreamReader(dataStream);
data = reader.ReadToEnd();
}
finally
{
if (reader != null)
reader.Dispose();
}
}
finally
{
if(response!=null)
((IDisposable)response).Dispose();
}
As you can see, in CodeBlock1, you are trying to access stream which is trying to read from a Disposed (and dis-connected) HttWebResponse object.
I'm programing an app for Microsoft PixelSense, and I'm using WPF for developing the user interface.
The app needs to download some content from the internet. It should be able to cope with a situation when the internet connection suddendly breaks while downloading. So when I need an internet connection, I got a try catch and catch every WebException or IOException due to the internet break.
Here's a snippet of my code :
System.Drawing.Image tmpimg = null;
Stream stream = null;
HttpWebResponse httpWebReponse = null;
try
{
// dowloading image
HttpWebRequest httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(urlPicturesBig +urlresource);
httpWebReponse = (HttpWebResponse)httpWebRequest.GetResponse();
stream = httpWebReponse.GetResponseStream();
tmpimg = System.Drawing.Image.FromStream(stream);
// saving
tmpimg.Save(#appDirectory + "\\resources\\" + urlresource);
}
catch (WebException)
{
Debug.WriteLine("WebException");
return -1;
}
catch (IOException)
{
Debug.WriteLine("IOException");
return -1;
}
The problem is that my GUI breaks down when the IOException is handled (a list of buttons becomes grey colored). So I've tried to do things this way :
try
{
// dowloading image
HttpWebRequest httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(urlPicturesBig +urlresource);
httpWebReponse = (HttpWebResponse)httpWebRequest.GetResponse();
}
catch (WebException)
{
Debug.WriteLine("WebException");
return -1;
}
stream = httpWebReponse.GetResponseStream();
tmpimg = System.Drawing.Image.FromStream(stream);
// saving
tmpimg.Save(#appDirectory + "\\resources\\" + urlresource);
But even if there's an internet break, the IOException is handled, and the programm doesn't read catch (WebException) instructions.
If I remove try catch blocks, WebException is handled most of the time, sometimes it's the IOException .
The problem as you state is that a failed network fetch can end up corrupting the backup resources you want to use.
As discussed in the comments below, this is due to the exception being thrown as the image stream is fetched from the network during the write to disk. Using this kind of code should protect you against this. Needless to say, you should perform some sanity checking on the length of the stream that is being returned.
var httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(urlPicturesBig + urlresource);
MemoryStream memory= new MemoryStream();
using (var httpWebReponse = (HttpWebResponse)httpWebRequest.GetResponse())
{
var stream = httpWebReponse.GetResponseStream();
//read the entire stream into memory to ensure any network issues
//are exposed
stream.CopyTo(memory);
}
var tmpimg = System.Drawing.Image.FromStream(memory); {
tmpimg.Save(#appDirectory + "\\resources\\" + urlresource);
I have a PDF which is hosted in say http://test.com/mypdf.pdf.
How can I convert the PDF to Stream and then using this Stream convert it back to PDF.
I tried the following but got an exception(see image):
private static Stream ConvertToStream(string fileUrl)
{
HttpWebResponse aResponse = null;
try
{
HttpWebRequest aRequest = (HttpWebRequest)WebRequest.Create(fileUrl);
aResponse = (HttpWebResponse)aRequest.GetResponse();
}
catch (Exception ex)
{
}
return aResponse.GetResponseStream();
}
This will work:
private static Stream ConvertToStream(string fileUrl)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(fileUrl);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
try {
MemoryStream mem = new MemoryStream();
Stream stream = response.GetResponseStream();
stream.CopyTo(mem,4096);
return mem;
} finally {
response.Close();
}
}
However you are entirely responsible for the lifetime of the returned memory stream.
A better approach is:
private static void ConvertToStream(string fileUrl, Stream stream)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(fileUrl);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
try {
Stream response_stream = response.GetResponseStream();
response_stream.CopyTo(stream,4096);
} finally {
response.Close();
}
}
You can then do something like:
using (MemoryStream mem = new MemoryStream()) {
ConvertToStream('http://www.example.com/',mem);
mem.Seek(0,SeekOrigin.Begin);
... Do something else ...
}
You may also be able to return the response stream directly but you'd have to check on the lifetime of that, releasing the response may release the stream, hence the mem copy.
You may want to take a look at WebClient.DownloadFile.
You give it a URL and local file name and it saves the file straight to disk. Might save you a step or two.
You could also try WebClient.DownloadData which saves the file to an in-memory byte[].
EDIT
You did not specify the protocol of the web-service you are posting the file to. The simplest form (RESTful) would be just to POST the file to data to another URL. Here is how you would do that.
using (WebClient wc = new WebClient())
{
// copy data to byte[]
byte[] data = wc.DownloadData("http://somesite.com/your.pdf");
// POST data to another URL
wc.Headers.Add("Content-Type","application/pdf");
wc.UploadData("http://anothersite.com/your.pdf", data);
}
If you are using SOAP, you would have to convert the file to a Base64 string, but hopefully you are using a generated client which takes care of that for you. If you could elaborate on the type of web-service you are sending the file to, I could probably provide some more information..
I am trying to retrieve HTML code from a webpage using HttpWebRequest and HttpWebResponse.
response = (HttpWebResponse)request.GetResponse();
...
Stream stream = response.GetResponseStream();
The response object has a ContentLength value of 106142. When I look at the stream object, it has a length of 65536. When reading the stream with a StreamReader using ReadToEnd(), only the first 65536 characters are returned.
How can I get the whole code?
Edit:
Using the following code segment:
catch (WebException ex)
{
errorMessage = errorMessage + ex.Message;
if (ex.Response != null) {
if (ex.Response.ContentLength > 0)
{
using (Stream stream = ex.Response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream))
{
string pageOutput = reader.ReadToEnd().Trim();
ex.Response.ContentLength = 106142
ex.Response.GetResponseStream().Length = 65536
stream.Length = 65536
pageOutput.Length = 65534 (because of the trim)
And yes, the code is actually truncated.
You can find an answer in this topic in System.Net.HttpWebResponse.GetResponseStream() returns truncated body in WebException
You have to manage the HttpWebRequest object and change DefaultMaximumErrorResponseLength property.
For example :
HttpWebRequest.DefaultMaximumErrorResponseLength = 1048576;
ReadToEnd does specifically just that, it reads to the end of the stream. I would check to make sure that you were actually being sent the entire expected response.
There seems to be a problem when calling the GetResponseStream() method on the HttpWebResponse returned by the exception. Everything works as expected when there is no exception.
I wanted to get the HTML code from the error returned by the server.
I guess I'll have to hope the error doesn't exceed 65536 characters...
I need to download a large file (2 GB) over HTTP in a C# console application. Problem is, after about 1.2 GB, the application runs out of memory.
Here's the code I'm using:
WebClient request = new WebClient();
request.Credentials = new NetworkCredential(username, password);
byte[] fileData = request.DownloadData(baseURL + fName);
As you can see... I'm reading the file directly into memory. I'm pretty sure I could solve this if I were to read the data back from HTTP in chunks and write it to a file on disk.
How could I do this?
If you use WebClient.DownloadFile you could save it directly into a file.
The WebClient class is the one for simplified scenarios. Once you get past simple scenarios (and you have), you'll have to fall back a bit and use WebRequest.
With WebRequest, you'll have access to the response stream, and you'll be able to loop over it, reading a bit and writing a bit, until you're done.
From the Microsoft documentation:
We don't recommend that you use WebRequest or its derived classes for
new development. Instead, use the System.Net.Http.HttpClient class.
Source: learn.microsoft.com/WebRequest
Example:
public void MyDownloadFile(Uri url, string outputFilePath)
{
const int BUFFER_SIZE = 16 * 1024;
using (var outputFileStream = File.Create(outputFilePath, BUFFER_SIZE))
{
var req = WebRequest.Create(url);
using (var response = req.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
var buffer = new byte[BUFFER_SIZE];
int bytesRead;
do
{
bytesRead = responseStream.Read(buffer, 0, BUFFER_SIZE);
outputFileStream.Write(buffer, 0, bytesRead);
} while (bytesRead > 0);
}
}
}
}
Note that if WebClient.DownloadFile works, then I'd call it the best solution. I wrote the above before the "DownloadFile" answer was posted. I also wrote it way too early in the morning, so a grain of salt (and testing) may be required.
You need to get the response stream and then read in blocks, writing each block to a file to allow memory to be reused.
As you have written it, the whole response, all 2GB, needs to be in memory. Even on a 64bit system that will hit the 2GB limit for a single .NET object.
Update: easier option. Get WebClient to do the work for you: with its DownloadFile method which will put the data directly into a file.
WebClient.OpenRead returns a Stream, just use Read to loop over the contents, so the data is not buffered in memory but can be written in blocks to a file.
i would use something like this
The connection can be interrupted, so it is better to download the file in small chunks.
Akka streams can help download file in small chunks from a System.IO.Stream using multithreading. https://getakka.net/articles/intro/what-is-akka.html
The Download method will append the bytes to the file starting with long fileStart. If the file does not exist, fileStart value must be 0.
using Akka.Actor;
using Akka.IO;
using Akka.Streams;
using Akka.Streams.Dsl;
using Akka.Streams.IO;
private static Sink<ByteString, Task<IOResult>> FileSink(string filename)
{
return Flow.Create<ByteString>()
.ToMaterialized(FileIO.ToFile(new FileInfo(filename), FileMode.Append), Keep.Right);
}
private async Task Download(string path, Uri uri, long fileStart)
{
using (var system = ActorSystem.Create("system"))
using (var materializer = system.Materializer())
{
HttpWebRequest request = WebRequest.Create(uri) as HttpWebRequest;
request.AddRange(fileStart);
using (WebResponse response = request.GetResponse())
{
Stream stream = response.GetResponseStream();
await StreamConverters.FromInputStream(() => stream, chunkSize: 1024)
.RunWith(FileSink(path), materializer);
}
}
}