Develop Reference

StreamReader is too greedy

I'm trying to process part of a text file, and write the remainder of the text file to a cloud blob using UploadFromStream. The problem is that the StreamReader appears to be grabbing too much content from the underlying stream, and so the subsequent write does nothing.
Text file:
3
Col1,String
Col2,Integer
Col3,Boolean
abc,123,True
def,3456,False
ghijkl,532,True
mnop,1211,False
Code:
using (var stream = File.OpenRead("c:\\test\\testinput.txt"))
using (var reader = new StreamReader(stream))
{
var numColumns = int.Parse(reader.ReadLine());
while (numColumns-- > 0)
{
var colDescription = reader.ReadLine();
// do stuff
}
// Write remaining contents to another file, for testing
using (var destination = File.OpenWrite("c:\\test\\testoutput.txt"))
{
stream.CopyTo(destination);
destination.Flush();
}
// Actual intended usage:
// CloudBlockBlob blob = ...;
// blob.UploadFromStream(stream);
}
When debugging, I observe that stream.Position jumps to the end of the file on the first call to reader.ReadLine(), which I don't expect. I expected the stream to be advanced only as many positions as the reader needed to read some content.
I imagine that the stream reader is doing some buffering for performance reasons, but there doesn't seem to be a way to ask the reader where in the underlying stream it "really" is. (If there was, I could manually Seek the stream to that position before CopyingTo).
I know that I could keep taking lines using the same reader and sequentially append them to the text file I'm writing, but I'm wondering if there's a cleaner way?
EDIT:
I found a StreamReader constructor which leaves the underlying stream open when it is disposed, so I tried this, hoping that the reader would set the stream's position as it's being disposed:
using (var stream = File.OpenRead("c:\\test\\testinput.txt"))
{
using (var reader = new StreamReader(stream, Encoding.UTF8,
detectEncodingFromByteOrderMarks: true,
bufferSize: 1 << 12,
leaveOpen: true))
{
var numColumns = int.Parse(reader.ReadLine());
while (numColumns-- > 0)
{
var colDescription = reader.ReadLine();
// do stuff
}
}
// Write remaining contents to another file
using (var destination = File.OpenWrite("c:\\test\\testoutput.txt"))
{
stream.CopyTo(destination);
destination.Flush();
}
}
But it doesn't. Why would this constructor be exposed if it doesn't leave the stream in an intuitive state/position?

Sure, there's a cleaner way. Use ReadToEnd to read the remaining data, and then write it to a new file. For example:
using (var reader = new StreamReader("c:\\test\\testinput.txt"))
{
var numColumns = int.Parse(reader.ReadLine());
while (numColumns-- > 0)
{
var colDescription = reader.ReadLine();
// do stuff
}
// write everything else to another file.
File.WriteAllText("c:\\test\\testoutput.txt", reader.ReadToEnd());
}
Edit after comment
If you want to read the text and upload it to a stream, you could replace the File.WriteAllText with code that reads the remaining text, writes it to a StreamWriter backed by a MemoryStream, and then sends the contents of that MemoryStream. Something like:
using (var memStream = new MemoryStream())
{
using (var writer = new StreamWriter(memStream))
{
writer.Write(reader.ReadToEnd());
writer.Flush();
memStream.Position = 0;
blob.UploadFromStream(memStream);
}
}

You should never access the underlying stream of a StreamReader. Trying to use both is going to have an undefined behavior.
What's going on here is that the reader is buffering the data from the underlying stream. It doesn't read each byte exactly when you request it, because that's often going to be very inefficient. Instead it will grab chunks, put them in a buffer, and then provide you with data from that buffer, grabbing a new chunk when it needs to.
You should continue to use the StreamReader throughout the remainder of that block, instead of using stream. To minimize the memory footprint of the program, the most effective way of doing this would be to read the next line from the reader in a loop until it his the end of the file, writing each line to the output stream as you go.
Also note that you don't need to be disposing of both the stream reader and the underlying stream. The stream reader will dispose of the underlying stream itself, so you can simply adjust your header to:
using (var reader = new StreamReader(
File.OpenRead("c:\\test\\testinput.txt")))

How to get contents of System.Net.Mail.Attachment

I have a System.Net.Mail.Attachment object with some .csv data in it. I need to save the contents of the attachment in a file. I tried this:
var sb = new StringBuilder();
sb.AppendLine("Accounts,JOB,Usage Count");
sb.AppendLine("One,Two,Three");
sb.AppendLine("One,Two,Three");
sb.AppendLine("One,Two,Three");
var stream = new MemoryStream(Encoding.ASCII.GetBytes(sb.ToString()));
//Add a new attachment to the E-mail message, using the correct MIME type
var attachment = new Attachment(stream, new ContentType("text/csv"))
{
Name = "theAttachment.csv"
};
var sr = new StreamWriter(#"C:\Blah\Look.csv");
sr.WriteLine(attachment.ContentStream.ToString());
sr.Close();
But the file has only the following: "System.IO.MemoryStream".
Could you please tell me how I can get the real data there?
Thanks.

You can't call ToString on an arbitrary stream. Instead you should use CopyTo:
using (var fs = new FileStream(#"C:\temp\Look.csv", FileMode.Create))
{
attachment.ContentStream.CopyTo(fs);
}
Use this to replace the last three lines of your example. By default, ToString just returns that name of the type unless the class overrides ToString. ContentStream is just the abstract Stream (at runtime it is a MemoryStream), so there is just the default implementation.
CopyTo is new in .NET Framework 4. If you aren't using the .NET Framework 4, you can mimic it with an extension method:
public static void CopyTo(this Stream fromStream, Stream toStream)
{
if (fromStream == null)
throw new ArgumentNullException("fromStream");
if (toStream == null)
throw new ArgumentNullException("toStream");
var bytes = new byte[8092];
int dataRead;
while ((dataRead = fromStream.Read(bytes, 0, bytes.Length)) > 0)
toStream.Write(bytes, 0, dataRead);
}
Credit to Gunnar Peipman for the extension method on his blog.

Assuming your stream isn't too big you can just write it all to the file like so:
StreamWriter writer = new StreamWriter(#"C:\Blah\Look.csv");
StreamReader reader = new StreamReader(attachment.ContentStream);
writer.WriteLine(reader.ReadToEnd());
writer.Close();
If it is bigger you probably want to chunk the reads up into a loop as to not demolish your RAM (and risk out of memory exceptions).

how to close network stream connection properly?

I have an application that is connecting to some network server (tcp) and is connected using a network stream:
if (!SSL)
{
_networkStream = new System.Net.Sockets.TcpClient(Server, Port).GetStream();
_StreamWriter = new System.IO.StreamWriter(_networkStream);
_StreamReader = new System.IO.StreamReader(_networkStream, Encoding.UTF8);
}
if (SSL)
{
System.Net.Sockets.TcpClient client = new System.Net.Sockets.TcpClient(Server, Port);
_networkSsl = new System.Net.Security.SslStream(client.GetStream(), false,
new System.Net.Security.RemoteCertificateValidationCallback(Protocol.ValidateServerCertificate), null);
_networkSsl.AuthenticateAsClient(Server);
_StreamWriter = new System.IO.StreamWriter(_networkSsl);
_StreamReader = new System.IO.StreamReader(_networkSsl, Encoding.UTF8);
}
///// since now I working with both reader and writer globally in multiple threads
I am using reader and writer asynchronously (in 2 threads) and I need to have both of these streams globally available (they are defined as public references on class level), so I don't know if using statement can be used here.
How should I properly close this connection? All objects (NetworkStream, stream writer and reader) have the Close() method. Should I call it for all of them? Or only 1 of them? Is there any need to release some resources?

Put them in a using block. It will close and dispose of each stream appropriately after the operation has been completed.
using (StreamReader reader = new StreamReader(_networkStream, Encoding.UTF8))
{
}

Use using() {} blocks. This way all Connections/streams are properly closed after usage.
using(_networkStream = new System.Net.Sockets.TcpClient(Server, Port).GetStream())
{
using(_StreamWriter = new System.IO.StreamWriter(_networkStream))
{
//do stuff
}
using(_StreamReader = new System.IO.StreamReader(_networkStream, Encoding.UTF8))
{
//do stuff
}
}

When they are disposed of, StreamReader and StreamWriter each dispose of the underlying NetworkStream automatically by default. Because of this, if you are using both a reader and a writer, you might get an "object already disposed" type of exception when you try to close the second one. I prefer to dispose of the stream itself but not the reader/writer since it's pointless in that case.
using stream = tcpClient.GetStream();
var reader = new StreamReader(stream);
var writer = new StreamWriter(stream);
// <use reader and writer>
// <the using statement ensures the stream is disposed of>
If you insist on disposing of the reader and writer, you can disable their default auto-close-the-stream behavior via the leaveOpen parameter:
new StreamReader(stream, leaveOpen: true)
new StreamWriter(stream, leaveOpen: true)
Making use of using is ideal if possible, but the other way to accomplish the same is to manually call the stream's Close() method or, equivalently, its Dispose() method.

Reading from memory stream to string

I am trying to write an object to an Xml string and take that string and save it to a DB. But first I need to get the string...
private static readonly Encoding LocalEncoding = Encoding.UTF8;
public static string SaveToString<T> (T settings)
{
Stream stream = null;
TextWriter writer = null;
string settingsString = null;
try
{
stream = new MemoryStream();
var serializer = new XmlSerializer(typeof(T));
writer = new StreamWriter(stream, LocalEncoding);
serializer.Serialize(writer, settings);
var buffer = new byte[stream.Length];
stream.Read(buffer, 0, (int)stream.Length);
settingsString = LocalEncoding.GetString(buffer);
}
catch (Exception ex)
{
// If the action cancels we don't want to throw, just return null.
}
finally
{
if (stream != null)
stream.Close();
if (writer != null)
writer.Close();
}
return settingsString;
}
This seems to work, the stream gets filled with bytes. But when I come to read it back into the buffer and then into the string... the buffer is filled with '0'! Not sure what I doing wrong here guys.

If you'd checked the results of stream.Read, you'd have seen that it hadn't read anything - because you haven't rewound the stream. (You could do this with stream.Position = 0;.) However, it's easier to just call ToArray:
settingsString = LocalEncoding.GetString(stream.ToArray());
(You'll need to change the type of stream from Stream to MemoryStream, but that's okay as it's in the same method where you create it.)
Alternatively - and even more simply - just use StringWriter instead of StreamWriter. You'll need to create a subclass if you want to use UTF-8 instead of UTF-16, but that's pretty easy. See this answer for an example.
I'm concerned by the way you're just catching Exception and assuming that it means something harmless, by the way - without even logging anything. Note that using statements are generally cleaner than writing explicit finally blocks.

string result = System.Text.Encoding.UTF8.GetString(fs.ToArray());

string result = Encoding.UTF8.GetString((stream as MemoryStream).ToArray());

In case of a very large stream length there is the hazard of memory leak due to Large Object Heap. i.e. The byte buffer created by stream.ToArray creates a copy of memory stream in Heap memory leading to duplication of reserved memory. I would suggest to use a StreamReader, a TextWriter and read the stream in chunks of char buffers.
In netstandard2.0 System.IO.StreamReader has a method ReadBlock
you can use this method in order to read the instance of a Stream (a MemoryStream instance as well since Stream is the super of MemoryStream):
private static string ReadStreamInChunks(Stream stream, int chunkLength)
{
stream.Seek(0, SeekOrigin.Begin);
string result;
using(var textWriter = new StringWriter())
using (var reader = new StreamReader(stream))
{
var readChunk = new char[chunkLength];
int readChunkLength;
//do while: is useful for the last iteration in case readChunkLength < chunkLength
do
{
readChunkLength = reader.ReadBlock(readChunk, 0, chunkLength);
textWriter.Write(readChunk,0,readChunkLength);
} while (readChunkLength > 0);
result = textWriter.ToString();
}
return result;
}
NB. The hazard of memory leak is not fully eradicated, due to the usage of MemoryStream, that can lead to memory leak for large memory stream instance (memoryStreamInstance.Size >85000 bytes). You can use Recyclable Memory stream, in order to avoid LOH. This is the relevant library

"Object can be disposed of more than once" error

When I run code analysis on the following chunk of code I get this message:
Object 'stream' can be disposed more than once in method 'upload.Page_Load(object, EventArgs)'. To avoid generating a System.ObjectDisposedException you should not call Dispose more than one time on an object.
using(var stream = File.Open(newFilename, FileMode.CreateNew))
using(var reader = new BinaryReader(file.InputStream))
using(var writer = new BinaryWriter(stream))
{
var chunk = new byte[ChunkSize];
Int32 count;
while((count = reader.Read(chunk, 0, ChunkSize)) > 0)
{
writer.Write(chunk, 0, count);
}
}
I don't understand why it might be called twice, and how to fix it to eliminate the error. Any help?

I struggled with this problem and found the example here to be very helpful. I'll post the code for a quick view:
using (Stream stream = new FileStream("file.txt", FileMode.OpenOrCreate))
{
using (StreamWriter writer = new StreamWriter(stream))
{
// Use the writer object...
}
}
Replace the outer using statement with a try/finally making sure to BOTH null the stream after using it in StreamWriter AND check to make sure it is not null in the finally before disposing.
Stream stream = null;
try
{
stream = new FileStream("file.txt", FileMode.OpenOrCreate);
using (StreamWriter writer = new StreamWriter(stream))
{
stream = null;
// Use the writer object...
}
}
finally
{
if(stream != null)
stream.Dispose();
}
Doing this cleared up my errors.

To illustrate, let's edit your code
using(var stream = File.Open(newFilename, FileMode.CreateNew))
{
using(var reader = new BinaryReader(file.InputStream))
{
using(var writer = new BinaryWriter(stream))
{
var chunk = new byte[ChunkSize];
Int32 count;
while((count = reader.Read(chunk, 0, ChunkSize)) > 0)
{
writer.Write(chunk, 0, count);
}
} // here we dispose of writer, which disposes of stream
} // here we dispose of reader
} // here we dispose a stream, which was already disposed of by writer
To avoid this, just create the writer directly
using(var reader = new BinaryReader(file.InputStream))
{
using(var writer = new BinaryWriter( File.Open(newFilename, FileMode.CreateNew)))
{
var chunk = new byte[ChunkSize];
Int32 count;
while((count = reader.Read(chunk, 0, ChunkSize)) > 0)
{
writer.Write(chunk, 0, count);
}
} // here we dispose of writer, which disposes of its inner stream
} // here we dispose of reader
edit: to take into account what Eric Lippert is saying, there could indeed be a moment when the stream is only released by the finalizer if BinaryWriter throws an exception. According to the BinaryWriter code, that could occur in three cases
If (output Is Nothing) Then
Throw New ArgumentNullException("output")
End If
If (encoding Is Nothing) Then
Throw New ArgumentNullException("encoding")
End If
If Not output.CanWrite Then
Throw New ArgumentException(Environment.GetResourceString("Argument_StreamNotWritable"))
End If
if you didn't specify an output, ie if stream is null. That shouldn't be a problem since a null stream means no resources to dispose of :)
if you didn't specify an encoding. since we don't use the constructor form where the encoding is specified, there should be no problem here either (i didn't look into the encoding contructor too much, but an invalid codepage can throw)
if you don't pass a writable stream. That should be caught quite quickly during development...
Anyway, good point, hence the edit :)

The BinaryReader/BinaryWriter will dispose the underlying stream for you when it disposes. You don't need to do it explicitly.
To fix it you can remove the using around the Stream itself.

A proper implementation of Dispose is explicitly required not to care if it's been called more than once on the same object. While multiple calls to Dispose are sometimes indicative of logic problems or code which could be better written, the only way I would improve the original posted code would be to convince Microsoft to add an option to BinaryReader and BinaryWriter instructing them not to dispose their passed-in stream (and then use that option). Otherwise, the code required to ensure the file gets closed even if the reader or writer throws in its constructor would be sufficiently ugly that simply letting the file get disposed more than once would seem cleaner.

Your writer will dispose your stream, always.

Suppress CA2202 whenever you are sure that the object in question handles multiple Dispose calls correctly and that your control flow is impeccably readable. BCL objects generally implement Dispose correctly. Streams are famous for that.
But don't necessarily trust third party or your own streams if you don't have unit tests probing that scenario yet. An API which returns a Stream may be returning a fragile subclass.