Develop Reference

Trying to create a bin-file using stream

I onced managed to create the bin-file in my project. I changed the primary key from int to Guid and moved code from Main to my class Quote. At the moment I can only add new entries in said file. If I remove it a new file(0 bytes) is created and the stream gets ArgumentException when I try to feed the file dummy-data. I am trying to use an if-loop to handle stream.Lenght == 0.
public static List<Quote> readBinaryToList() //Crashes if binfile is 0 bytes long
{
IFormatter formatter = new BinaryFormatter();
Stream stream = new FileStream(#"C:\Users\xxxxxx\Desktop\quotes.bin", FileMode.OpenOrCreate, FileAccess.Read, FileShare.Read);
if (stream.Length == 0)
{
Quote q = new Quote(Guid.NewGuid(), "Quote dummy", false);
List<Quote> quoteList = new List<Quote>();
quoteList.Add(q);
var bformatter = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
bformatter.Serialize(stream, quoteList);
bformatter = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
List<Quote> quoteListTmp = (List<Quote>)bformatter.Deserialize(stream);
return quoteList;
}
else
{
List<Quote> quoteList = (List<Quote>)formatter.Deserialize(stream);
stream.Close();
return quoteList;
}
}

As pointed out in previous answers, you must give your file stream write permissions which can be done in its constructor, then you should also set the position of the stream back to 0, you can achieve this by using the stream's Position property.
You are creating a lot of unnecessary objects that don't actually contribute to purpose of the method I have omitted these below. In doing so, setting the streams Position property to 0 is redundant but I've left it in a comment to show how its done.
Some other things to consider: Declare the file stream inside a using statement so that it is disposed when the method comes to an end this means you can omit the manual close in the else statement. Some of your code can be written more tersely, this is just a personal preference but I think it would be best to inline some of your code to remove as much noise as possible. It is also convention in C# to use PascalCase for Methods.
public static List<Quote> ReadBinaryToList(){
using(Stream stream = new FileStream(#"quotes.bin", FileMode.OpenOrCreate, FileAccess.ReadWrite)) {
IFormatter formatter = new BinaryFormatter();
if (stream.Length == 0) {
List<Quote> quoteList = new List<Quote> {new Quote(Guid.NewGuid(), "Quote dummy", false)};
formatter.Serialize(stream, quoteList);
//stream.Position = 0;
return quoteList;
}
else return (List<Quote>)formatter.Deserialize(stream);
}
}

The file is being opened as readonly, serializing to the file will require write permissions.
Stream stream = new FileStream(#"C:\temp\quotes.bin", FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.Read);
The stream should also be returned to the beginning before making any attempts to deserialize from it.
stream.Seek(0, SeekOrigin.Begin);
FileStreams have a single "head" where all read and write operations take places. As a new stream is being written, the head is always at the end and any attempt to read from the end will fail. Some streams (e.g. NetworkStream) behave differently and do not allow seeking at all.
Also, the initial position of the FileStream depends on how the file is opened (based on the specified FileMode). The FileMode specified in the question will result in the stream position starting at the beginning of the file, so this is not required in the else block.
And make sure that the Quote class is marked [Serializable]

OutOfMemory Exception while converting MemoryStream to Array

I use the following code to return a byte array in HttpResponseMessage:
using (WebResponse response = (HttpWebResponse)request.GetResponse())
{
byte[] bytes = ReadFully(response.GetResponseStream());
......
}
public static byte[] ReadFully(Stream input)
{
byte[] buffer = new byte[16*1024];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
return ms.ToArray(); // This line throws OutOfMemory exception
}
}
An OutOfMemory exception is thrown in the last return ms.ToArray() statement.
I need to set the resulting byte[] as HttpResponseMessage.Content.

You should return the stream directly instead of reading it into memory first.
public HttpResponseMessage CreateMessage(Stream input)
{
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
result.Content = new StreamContent(input);
return result;
}
Do not forget to set the appropriate headers etc.
Edit
... i need to write the byte array from the HttpResponseMessage into a file
Based on your last comment you changed your question and want to go the other way. Here is an example of writing to a file from a web response.
public void writetoFile(HttpWebResponse response)
{
var inStream = response.GetResponseStream();
using (var file = System.IO.File.OpenWrite("your file path here"))
{
inStream.CopyTo(file);
}
}

Igor posted the solution, and the correct way to deal with stream content. Use one of the MVC helper functions like File(stream,contentype) or classes like StreamContent to send the stream contents directly to the client, eg:
return File(myStream,myExcelContentTypeString);
or
return File(myStream,myExcelContentTypeString,"ReallyBigFile.xlsx");
The reason for the error, is that OOM can occur because memory is too fragmented to allocate a new object. A MemoryStream stores data in a buffer. When it exceeds the buffer limits, it allocates a new one with double the capacity and copies the old data. Copying 250MB of data like this is going to cause a lot of reallocations and thus a lot of memory fragmentation.
This can be avoided by specifying the desired capacity in the stream's constructor. This will allocate a large enough buffer immediatelly.
It's even better though to avoid caching this content though, by sending it to the browser directly.

Why do I need to flush and set the stream position to 0 in this serialization snippet?

Taken from here:
private static string SerializeToString<T>(T value)
{
using (var stream = new MemoryStream()) {
var formatter = new BinaryFormatter();
formatter.Serialize(stream, value);
stream.Flush();
stream.Position = 0;
return Convert.ToBase64String(stream.ToArray());
}
}
private static T DeserializeFromString<T>(string data)
{
byte[] b = Convert.FromBase64String(data);
using (var stream = new MemoryStream(b)) {
var formatter = new BinaryFormatter();
stream.Seek(0, SeekOrigin.Begin);
return (T)formatter.Deserialize(stream);
}
}
Why do I need to flush and set the position to 0 in the serialize method, and seek in the deserialize method?
I removed them, they didn't affect anything.
I know that flushing means write whatever that's in the stream immediately.
But I don't know if it's necessary here... also not sure about the position and seek.

These samples contain unecessary code. The documentation for MemoryStream.ToArray (here) explicitly states that:
Writes the stream contents to a byte array, regardless of the Position
property.
Thus, we clearly don't need to set position. The flush is more debatable. It's very, very unlikely that memory stream would buffer under the hood, since it's just writing to a memory buffer anyway. However, I'm not sure that it's documented anywhere that memory stream won't buffer, so Flush() might be reasonable since we're calling ToArray() before disposing the stream. Another approach would be to call ToArray() outside the using block (we'd have to move the declaration of the variable out as well). This will work because ToArray() states that:
This method works when the MemoryStream is closed.
On the read side, you are creating a new stream, which starts at position 0 by default. Thus, there's no need for the Seek call.

Disposing a StreamReader that read a stream that is defined out of scope?

In an utility method, which accepts a Stream parameter, I rely on some StreamReader to analyse data.
I don't want to close the incoming stream in my method. I want to let the caller method to take the decision to dispose the stream.
Is it safe to not dispose the opened StreamReader? I mean, will it eventually be automatically disposed? Will it lead to memory leaks?
Here is my utility method. Its goal is to read a Stream, and return its content as a string, regardless of how the data is encoded:
public static string GetStringAutoDetectEncoding(Stream data, out Encoding actualEncoding)
{
// 1. Is there a Bye Order Mask ?
var candidateEncoding = DetectEncodingWithByteOrderMask(data);
// 2a. No BOM, the data is either UTF8 no BOM or ANSI
if (candidateEncoding == Encoding.Default)
{
var utf8NoBomEncoding = Encoding.GetEncoding("utf-8",new EncoderExceptionFallback(), new DecoderExceptionFallback());
var positionBackup = data.Position;
var sr = new StreamReader(data, utf8NoBomEncoding);
try
{
// 3. Try as UTF8 With no BOM
var result = sr.ReadToEnd(); // will throw error if not UTF8
actualEncoding = utf8NoBomEncoding; // Probably an UTF8 no bom string
return result;
}
catch (DecoderFallbackException)
{
// 4. Rewind the stream and fallback to ASNI
data.Position = positionBackup;
var srFallback = new StreamReader(data, candidateEncoding);
actualEncoding = candidateEncoding;
return srFallback.ReadToEnd(); ;
}
}
// 2b. There is a BOM. Use the detected encoding
else
{
var sr = new StreamReader(data, candidateEncoding);
actualEncoding = candidateEncoding;
return sr.ReadToEnd(); ;
}
}
Then, I can have some methods in the like this:
void Foo(){
using(var stream = File.OpenRead(#"c:\somefile")) {
Encoding detected;
var fileContent = MyUtilityClass.GetStringAutoDetectEncoding(stream, detected);
Console.WriteLine("Detected encoding: {0}", encoding);
Console.WriteLine("File content: {0}", fileContent);
}
}

You could invert control using a closure. That is, create a method like so:
// This method will open the stream, execute the streamClosure, and then close the stream.
public static String StreamWork(Func<Stream, String> streamClosure) {
// Set up the stream here.
using (Stream stream = new MemoryStream()) { // Pretend the MemoryStream is your actual stream.
// Execute the closure. Return it's results.
return streamClosure(stream);
}
}
which is responsible for opening / closing the stream within the method.
Then you simply wrap up all the code that needs the stream into a Func<Stream, String> closure, and pass it in. The StreamWork method will open the stream, execute your code, then close the stream.
public static void Main()
{
// Wrap all of the work that needs to be done in a closure.
// This represents all the work that needs to be done while the stream is open.
Func<Stream, String> streamClosure = delegate(Stream stream) {
using (StreamReader streamReader = new StreamReader(stream)) {
return streamReader.ReadToEnd();
}
};
// Call StreamWork. This method handles creating/closing the stream.
String result = StreamWork(streamClosure);
Console.WriteLine(result);
Console.ReadLine();
}
UPDATE
Of course, this method of inversion is a matter of preference as mentioned in the comments below. The key point is to ensure that the stream is closed rather than allowing it to float around until the GC cleans it up (since the whole point of having stuff implement IDisposable is to avoid that sort of situation to begin with). Since this is a library function that accepts a Stream as input, the assumption is that the method-consumer will be creating the stream, and therefore as you point out, has the responsibility of ultimately closing the stream as well. But for sensitive resources where you are concerned about ensuring clean up occurs absolutely, inversion is sometimes a useful technique.

StreamReader close/dispose their underlying streams only when you call Dispose on them. They don't dispose of the stream if the reader/writer is just garbage collected.

"Object can be disposed of more than once" error

When I run code analysis on the following chunk of code I get this message:
Object 'stream' can be disposed more than once in method 'upload.Page_Load(object, EventArgs)'. To avoid generating a System.ObjectDisposedException you should not call Dispose more than one time on an object.
using(var stream = File.Open(newFilename, FileMode.CreateNew))
using(var reader = new BinaryReader(file.InputStream))
using(var writer = new BinaryWriter(stream))
{
var chunk = new byte[ChunkSize];
Int32 count;
while((count = reader.Read(chunk, 0, ChunkSize)) > 0)
{
writer.Write(chunk, 0, count);
}
}
I don't understand why it might be called twice, and how to fix it to eliminate the error. Any help?

I struggled with this problem and found the example here to be very helpful. I'll post the code for a quick view:
using (Stream stream = new FileStream("file.txt", FileMode.OpenOrCreate))
{
using (StreamWriter writer = new StreamWriter(stream))
{
// Use the writer object...
}
}
Replace the outer using statement with a try/finally making sure to BOTH null the stream after using it in StreamWriter AND check to make sure it is not null in the finally before disposing.
Stream stream = null;
try
{
stream = new FileStream("file.txt", FileMode.OpenOrCreate);
using (StreamWriter writer = new StreamWriter(stream))
{
stream = null;
// Use the writer object...
}
}
finally
{
if(stream != null)
stream.Dispose();
}
Doing this cleared up my errors.

To illustrate, let's edit your code
using(var stream = File.Open(newFilename, FileMode.CreateNew))
{
using(var reader = new BinaryReader(file.InputStream))
{
using(var writer = new BinaryWriter(stream))
{
var chunk = new byte[ChunkSize];
Int32 count;
while((count = reader.Read(chunk, 0, ChunkSize)) > 0)
{
writer.Write(chunk, 0, count);
}
} // here we dispose of writer, which disposes of stream
} // here we dispose of reader
} // here we dispose a stream, which was already disposed of by writer
To avoid this, just create the writer directly
using(var reader = new BinaryReader(file.InputStream))
{
using(var writer = new BinaryWriter( File.Open(newFilename, FileMode.CreateNew)))
{
var chunk = new byte[ChunkSize];
Int32 count;
while((count = reader.Read(chunk, 0, ChunkSize)) > 0)
{
writer.Write(chunk, 0, count);
}
} // here we dispose of writer, which disposes of its inner stream
} // here we dispose of reader
edit: to take into account what Eric Lippert is saying, there could indeed be a moment when the stream is only released by the finalizer if BinaryWriter throws an exception. According to the BinaryWriter code, that could occur in three cases
If (output Is Nothing) Then
Throw New ArgumentNullException("output")
End If
If (encoding Is Nothing) Then
Throw New ArgumentNullException("encoding")
End If
If Not output.CanWrite Then
Throw New ArgumentException(Environment.GetResourceString("Argument_StreamNotWritable"))
End If
if you didn't specify an output, ie if stream is null. That shouldn't be a problem since a null stream means no resources to dispose of :)
if you didn't specify an encoding. since we don't use the constructor form where the encoding is specified, there should be no problem here either (i didn't look into the encoding contructor too much, but an invalid codepage can throw)
if you don't pass a writable stream. That should be caught quite quickly during development...
Anyway, good point, hence the edit :)

The BinaryReader/BinaryWriter will dispose the underlying stream for you when it disposes. You don't need to do it explicitly.
To fix it you can remove the using around the Stream itself.

A proper implementation of Dispose is explicitly required not to care if it's been called more than once on the same object. While multiple calls to Dispose are sometimes indicative of logic problems or code which could be better written, the only way I would improve the original posted code would be to convince Microsoft to add an option to BinaryReader and BinaryWriter instructing them not to dispose their passed-in stream (and then use that option). Otherwise, the code required to ensure the file gets closed even if the reader or writer throws in its constructor would be sufficiently ugly that simply letting the file get disposed more than once would seem cleaner.

Your writer will dispose your stream, always.

Suppress CA2202 whenever you are sure that the object in question handles multiple Dispose calls correctly and that your control flow is impeccably readable. BCL objects generally implement Dispose correctly. Streams are famous for that.
But don't necessarily trust third party or your own streams if you don't have unit tests probing that scenario yet. An API which returns a Stream may be returning a fragile subclass.