Develop Reference

Stream chaining in computing a checksum: avoiding memory issues

I have a FileStream connected to a xml file that I would like to read directly into a SHA512 object in order to compute a hash for the purposes of a checksum (not a security use).
The issue is twofold:
I want to omit some of the nodes in the xml,
the file is quite large, and I would rather not load the whole thing into into memory
I can read the whole file into a xml structure, delete the node, then write it to a stream that would then be plugged into SHA512.ComputeHash, but that will cause a performance loss. I would prefer to be able to somehow do the deletion of the nodes as an operation on a stream and then chain the streams together somehow into a single stream that can be passed into SHA512.ComputeHash(Stream).
How can I accomplish this?

using (var hash = new SHA512Cng())
using (var stream = new CryptoStream(Stream.Null, hash, CryptoStreamMode.Write))
using (var writer = XmlWriter.Create(stream))
using (var reader = XmlReader.Create("input.xml"))
{
while (reader.Read())
{
// ... write node to writer ...
}
writer.Flush();
stream.FlushFinalBlock();
var result = hash.Hash;
}

converting MemoryStream array to fileStream C#

I have a function that is returning MemoryStream array, i want to convert this memory stream array to a FileStream object.
Is it possible if yes can you please provide a way to do that...
Thanks
A.S

You cannot "convert" the stream, because a MemoryStream and a FileStream are very different things. However, you can write the entire contents of the MemoryStream to a file. There is a CopyTo method that you can use for that:
// memStream is the MemoryStream
using (var output = File.Create(filename)) {
memStream.CopyTo(output);
}

A file stream object represents an open file (from disk) as a stream. A memory stream represents an area of memory (byte array) as a stream. So you can't really convert a memory stream into a file stream directly - at least not trivially.
There are two approaches you could take:
OFFLINE: fully consume the contents of the memory stream and write it all out to a file on disk; then open that file as a file stream
ONLINE: extent the FileStream class creating an adapter that will wrap a MemoryStream object and expose it as a FileStream (essentially acting as a converter)
The reason one is marked [OFFLINE] is because you need to have to full contents of the memory stream before you output it to the file (and once you do, modifications to the file stream will not affect the memory stream; nor changes to the memory stream, such as new data, be available to the file stream)
The second one is marked as [ONLINE] because once you create the adapter and you initialize the FileStream object from the MemoryStream you could process any new data in the MemoryStream using the FileStream adapter object. You would essentially be able to read/write and seek into the memory stream using the file stream as a layer on top of the memory stream. Presumably, that's what you'd want to do..
Of course, it depends on what you need to do, but I'm leaning towards the second [ONLINE] version as the better in the general sense.

Memory stream is empty

I need to generate a huge xml file from different sources (functions). I decide to use XmlTextWriter since it uses less memory than XmlDocument.
First, initiate an XmlWriter with underlying MemoryStream
MemoryStream ms = new MemoryStream();
XmlTextWriter xmlWriter = new XmlTextWriter(ms, new UTF8Encoding(false, false));
xmlWriter.Formatting = Formatting.Indented;
Then I pass the XmlWriter (note xml writer is kept open until the very end) to a function to generate the beginning of the XML file:
xmlWriter.WriteStartDocument();
xmlWriter.WriteStartElement();
// xmlWriter.WriteEndElement(); // Do not write the end of root element in first function, to add more xml elements in following functions
xmlWriter.WriteEndDocument();
xmlWriter.Flush();
But I found that underlying memory stream is empty (by converting byte array to string and output string). Any ideas why?
Also, I have a general question about how to generate a huge xml file from different sources (functions). What I do now is keeping the XmlWriter open (I assume the underlying memory stream should open as well) to each function and write. In the first function, I do not write the end of root element. After the last function, I manually add the end of root element by:
string endRoot = "</Root>";
byte[] byteEndRoot = Encoding.ASCII.GetBytes(endRoot);
ms.Write(byteEndRoot, 0, byteEndRoot.Length);
Not sure if this works or not.
Thanks a lot!

Technically you should only ask one question per question, so I'm only going to answer the first one because this is just a quick visit to SO for me at the moment.
You need to call Flush before attempting to read from the Stream I think.
Edit
Just bubbling up my second hunch from the comments below to justify the accepted answer here.
In addition to the call to Flush, if reading from the Stream is done using the Read method and its brethren, then the position in the stream must first be reset back to the start. Otherwise no bytes will be read.
ms.Position = 0; /*reset Position to start*/
StreamReader reader = new StreamReader(ms);
string text = reader.ReadToEnd();
Console.WriteLine(text);

Perhaps you need to call Flush() on the xml stream before checking the memory streazm.

Make sure you call Flush on the XmlTextWriter before checking the memory stream.

Odd behaviour with XmlReader.Read and Stream.Read

I've encountering an unusual problem with .Net Framework 3.5 and the System.Xml.XmlReader class.
Before my application calls the XmlReader.Read method it first reads the content of the stream for logging purposes using the Stream.Read method. It then seeks back to the beginning of the stream before calling Stream.Read. When I do this I am getting the following error:
Unhandled Exception: System.Xml.XmlException: Unexpected end of file while parsing Name has occurred. Line 1, position 4097.
If however I call XmlReader.Read, seek to the beginning of the stream and then call the Stream.Read method it all works fine. This only appears to be happening on large streams however. I've just seen one go through the system at about 2000 characters and it works fine?
I've included a code sample below to give an idea of what I'm doing.
XmlReaderSettings readerSettings = new XmlReaderSettings();
readerSettings.Schemas.Add(null, args[1]);
readerSettings.ValidationType = ValidationType.Schema;
readerSettings.ValidationEventHandler += new ValidationEventHandler(XmlValidatingReaderValidationEventHandler);
XmlReader reader = XmlReader.Create(fileReader, readerSettings);
byte[] buffer = new byte[fs.Length];
fs.Read(buffer, 0, buffer.Length);
string content = System.Text.UTF8Encoding.UTF8.GetString(buffer, 0, buffer.Length);
fs.Seek(0, SeekOrigin.Begin);
while(reader.Read());
Console.WriteLine("Done");
Thanks

Messing around with the stream which is backing something like XmlReader is generally a bad idea. If you want to do two different things with the same file, I suggest you open two different streams. That way they won´t interfere with each other.
Note that using File.ReadAllText is a simpler way of loading the contents of a text file into a string.

That's because XmlReader buffers the data from the Stream. If you mess with the current position of the Stream, you mess with the XmlReader too...

What is ADODB.Stream?

What exactly is it, or was it, as is a interop, used for?
Here, this is the method where I use it:
public void SaveAttachmentMime(String fileName, CDO.Message message)
{
ADODB.Stream stream = message.BodyPart.GetStream();
String messageString = stream.ReadText(stream.Size);
StreamWriter outputStream = new StreamWriter(fileName);
outputStream.Write(messageString);
outputStream.Flush();
outputStream.Close();
}

The ADODB.Stream object was used to read files and other streams. What it does is part of what the StreamReader, StreamWriter, FileStream and Stream does in the .NET framework.
For what the code in that method uses it for, in .NET you would use a StreamReader to read from a Stream.
Note that the code in the method only works properly if the stream contains non-Unicode data, as it uses the size in bytes to determine how many characters to read. With a Unicode encoding some characters may be encoded as several bytes, so the stream would run into the end of the stream before it could read the number of characters specified.

It is a COM object, which is used to represent a stream of data or text. The data can be binary. If I recall correctly, it implements the IStream interface, which stores data in a structured storage object. You can find the interop representation of the interface in System.Runtime.InteropServices.ComTypes.IStream.