I am trying to use the WCF DataContractSerializer to serialize a DataContract object into a memoryStream.
Then I use the memoryStream.ToArray to get the serialized content.
Finally, I persist the memoryStream into a file using anther fileStream.
My initial implement is like this. I am missing bytes at the end of the persisted File.
public virtual string SerializeTransient(DataObject data, string targetPath)
{
string securityCode;
using (var memoryStream = new MemoryStream())
{
using (var xmlWriter = XmlWriter.Create(memoryStream, new XmlWriterSettings {Indent = true}))
{
_serializer.WriteObject(xmlWriter, data);
using (var fileStream = new FileStream(targetPath, FileMode.Create))
{
securityCode = CalculateSecurityCode(memoryStream.ToArray());
memoryStream.WriteTo(fileStream);
}
}
}
return securityCode;
}
If I move the persist logic out of the inner using{} block (see below), the output is correct. It almost feels like the WriteObject function didnt finish what it is doing. Could someone please explain to me what is happening there? Thanks.
public virtual string SerializeTransient(DataObject data, string targetPath)
{
string securityCode;
using (var memoryStream = new MemoryStream())
{
using (var xmlWriter = XmlWriter.Create(memoryStream, new XmlWriterSettings {Indent = true}))
{
_serializer.WriteObject(xmlWriter, data);
}
using (var fileStream = new FileStream(targetPath, FileMode.Create))
{
securityCode = CalculateSecurityCode(memoryStream.ToArray());
memoryStream.WriteTo(fileStream);
}
}
return securityCode;
}
XmlWriter has an internal buffer. You should either Close/Dispose XmlWriter or call the XmlWriter.Flush() to force all content to be written to underlying stream (memoryStream).
If memoryStream.ToArray() is called before writer.Flush() then some bytes will possibly remain in internal writer buffer.
Related
How can I create a .csv file implicitly/automatically by using the correct method, add text to that file existing in memory and then convert to in memory data to a byte array?
string path = #"C:\test.txt";
File.WriteAllLines(path, GetLines());
byte[] bytes = System.IO.File.ReadAllBytes(path);
With that approach I create a file always (good), write into it (good) then close it (bad) then open the file again from a path and read it from the hard disc (bad)
How can I improve that?
UPDATE
One nearly good approach would be:
using (var fs = new FileStream(#"C:\test.csv", FileMode.Create, FileAccess.ReadWrite))
{
using (var memoryStream = new MemoryStream())
{
fs.CopyTo(memoryStream );
return memoryStream .ToArray();
}
}
but I am not able to write text into that filestream... just bytes...
UPDATE 2
using (var fs = File.Create(#"C:\temp\test.csv"))
{
using (var sw = new StreamWriter(fs, Encoding.Default))
{
using (var ms = new MemoryStream())
{
String message = "Message is the correct ääüö Pi(\u03a0), and Sigma (\u03a3).";
sw.Write(message);
sw.Flush();
fs.CopyTo(ms);
return ms.ToArray();
}
}
}
The string message is not persisted to the test.csv file. Anyone knows why?
Write text into Memory Stream.
byte[] bytes = null;
using (var ms = new MemoryStream())
{
using(TextWriter tw = new StreamWriter(ms)){
tw.Write("blabla");
tw.Flush();
ms.Position = 0;
bytes = ms.ToArray();
}
}
UPDATE
Use file stream Directly and write to File
using (var fs = new FileStream(#"C:\ed\test.csv", FileMode.Create, FileAccess.ReadWrite))
{
using (TextWriter tw = new StreamWriter(fs))
{
tw.Write("blabla");
tw.Flush();
}
}
You can get a byte array from a string using encoding:
Encoding.ASCII.GetBytes(aString);
Or
Encoding.UTF8.GetBytes(aString);
But I don't know why you would want a csv as bytes. You could load the entire file to a string, add to it and then save it:
string content;
using (var reader = new StreamReader(filename))
{
content = reader.ReadToEnd();
}
content += "x,y,z";
using (var writer = new StreamWriter(filename))
{
writer.Write(content);
}
Update: Create a csv in memory and pass back as bytes:
var stringBuilder = new StringBuilder();
foreach(var line in GetLines())
{
stringBuilder.AppendLine(line);
}
return Encoding.ASCII.GetBytes(stringBuilder.ToString());
In my program, I am basically reading in a file, doing some processing to it, and then passing it back to the main program as a memorystream, which will be handled by a streamreader. This will all be handled by a class beside my main.
The problem is, when I return the memory stream from my method in another class, the "canread" variable is set to false, and thus causes the streamreader initialization to fail.
Below is an example of the problem happening (though in here I'm writing to the memorystream in the other class, but it still causes the same error when i pass it back.
In the class named "Otherclass":
public static MemoryStream ImportantStreamManipulator()
{
MemoryStream MemStream = new MemoryStream();
StreamWriter writer = new StreamWriter(MemStream);
using (writer)
{
//Code that writes stuff to the memorystream via streamwriter
return MemStream;
}
}
The function calls in the main program:
MemoryStream MStream = Otherclass.ImportantStreamManipulator();
StreamReader reader = new StreamReader(MStream);
When I put a breakpoint on the "return MemStream", the "CanRead" property is still set to true. Once I step such that it gets back to my main function, and writes the returned value to MStream, the "CanRead" property is set to false. This then causes an exception in StreamReader saying that MStream could not be read (as the property indicated). The data is in the streams buffer as it should be, but I just can't get it out.
How do I set it so that "CanRead" will report true once it is returned to my main? Or am I misunderstanding how MemoryStream works and how would I accomplish what I want to do?
This is the problem:
using (writer)
{
//Code that writes stuff to the memorystream via streamwriter
return MemStream;
}
You're closing the writer, which closes the MemoryStream. In this case you don't want to do that... although you do need to flush the writer, and rewind the MemoryStream. Just change your code to:
public static MemoryStream ImportantStreamManipulator()
{
// Probably add a comment here stating that the lack of using statements
// is deliberate.
MemoryStream stream = new MemoryStream();
StreamWriter writer = new StreamWriter(stream);
// Code that writes stuff to the memorystream via streamwriter
writer.Flush();
stream.Position = 0;
return stream;
}
The StreamWriter takes ownership of the memory stream and when the using statement ends, the MemoryStream is also closed.
See Is there any way to close a StreamWriter without closing its BaseStream?.
As others have stated, the problem is that the Stream is closed when the StreamWriter is closed. One possible way to deal with this is to return a byte array rather than a MemoryStream. This avoids having potentially long running objects that must be disposed by the garbage collector.
public static void Main()
{
OutputData(GetData());
}
public static byte[] GetData()
{
byte[] binaryData = null;
using (MemoryStream ms = new MemoryStream())
using (StreamWriter sw = new StreamWriter(ms))
{
string data = "My test data is really great!";
sw.Write(data);
sw.Flush();
binaryData = ms.ToArray();
}
return binaryData;
}
public static void OutputData(byte[] binaryData)
{
using (MemoryStream ms = new MemoryStream(binaryData))
using (StreamReader sr = new StreamReader(ms))
{
Console.WriteLine(sr.ReadToEnd());
}
}
Another method is to copy the Stream to another stream prior to returning. However, this still has the problem that subsequent access to it with a StreamReader will close that stream.
public static void RunSnippet()
{
OutputData(GetData());
}
public static MemoryStream GetData()
{
MemoryStream outputStream = new MemoryStream();
using (MemoryStream ms = new MemoryStream())
using (StreamWriter sw = new StreamWriter(ms))
{
string data = "My test data is really great!";
sw.Write(data);
sw.Flush();
ms.WriteTo(outputStream);
outputStream.Seek(0, SeekOrigin.Begin);
}
return outputStream;
}
public static void OutputData(MemoryStream inputStream)
{
using (StreamReader sr = new StreamReader(inputStream))
{
Console.WriteLine(sr.ReadToEnd());
}
}
I'm using GZipStream to compress a string, and I've modified two different examples to see what works. The first code snippet, which is a heavily modified version of the example in the documentation, simply returns an empty string.
public static String CompressStringGzip(String uncompressed)
{
String compressedString;
// Convert the uncompressed source string to a stream stored in memory
// and create the MemoryStream that will hold the compressed string
using (MemoryStream inStream = new MemoryStream(Encoding.Unicode.GetBytes(uncompressed)),
outStream = new MemoryStream())
{
using (GZipStream compress = new GZipStream(outStream, CompressionMode.Compress))
{
inStream.CopyTo(compress);
StreamReader reader = new StreamReader(outStream);
compressedString = reader.ReadToEnd();
}
}
return compressedString;
and when I debug it, all I can tell is nothing is read from reader, which is compressedString is empty. However, the second method I wrote, modified from a CodeProject snippet is successful.
public static String CompressStringGzip3(String uncompressed)
{
//Transform string to byte array
String compressedString;
byte[] uncompressedByteArray = Encoding.Unicode.GetBytes(uncompressed);
using (MemoryStream outStream = new MemoryStream())
{
using (GZipStream compress = new GZipStream(outStream, CompressionMode.Compress))
{
compress.Write(uncompressedByteArray, 0, uncompressedByteArray.Length);
compress.Close();
}
byte[] compressedByteArray = outStream.ToArray();
StringBuilder compressedStringBuilder = new StringBuilder(compressedByteArray.Length);
foreach (byte b in compressedByteArray)
compressedStringBuilder.Append((char)b);
compressedString = compressedStringBuilder.ToString();
}
return compressedString;
}
Why is the first code snippet not successful while the other one is? Even though they're slightly different, I don't know why the minor changes in the second snippet allow it to work. The sample string I'm using is SELECT * FROM foods f WHERE f.name = 'chicken';
I ended up using the following code for compression and decompression:
public static String Compress(String decompressed)
{
byte[] data = Encoding.UTF8.GetBytes(decompressed);
using (var input = new MemoryStream(data))
using (var output = new MemoryStream())
{
using (var gzip = new GZipStream(output, CompressionMode.Compress, true))
{
input.CopyTo(gzip);
}
return Convert.ToBase64String(output.ToArray());
}
}
public static String Decompress(String compressed)
{
byte[] data = Convert.FromBase64String(compressed);
using (MemoryStream input = new MemoryStream(data))
using (GZipStream gzip = new GZipStream(input, CompressionMode.Decompress))
using (MemoryStream output = new MemoryStream())
{
gzip.CopyTo(output);
StringBuilder sb = new StringBuilder();
return Encoding.UTF8.GetString(output.ToArray());
}
}
The explanation for a part of the problem comes from this question. Although I fixed the problem by changing the code to what I included in this answer, these lines (in my original code):
foreach (byte b in compressedByteArray)
compressedStringBuilder.Append((char)b);
are problematic, because as dlev aptly phrased it:
You are interpreting each byte as its own character, when in fact that is not the case. Instead, you need the line:
string decoded = Encoding.Unicode.GetString(compressedByteArray);
The basic problem is that you are converting to a byte array based on an encoding, but then ignoring that encoding when you retrieve the bytes.
Therefore, the problem is solved, and the new code I'm using is much more succinct than my original code.
You need to move the code below outside the second using statement:
using (GZipStream compress = new GZipStream(outStream, CompressionMode.Compress))
{
inStream.CopyTo(compress);
outStream.Position = 0;
StreamReader reader = new StreamReader(outStream);
compressedString = reader.ReadToEnd();
}
CopyTo() is not flushing the results to the underlying MemoryStream.
Update
Seems that GZipStream closes and disposes it's underlying stream when it is disposed (not the way I would have designed the class). I've updated the sample above and tested it.
I am working on a list program that contains a 2-column listbox. Both columns contain strings that I wish to save as a file. The file format doesn't really matter as long as it can be read and populate the listbox again at startup.
Any help will be extremely appreciated!
You could populate your list box from a custom object and serialize that object to disk:
using System;
using System.IO;
using System.Xml;
using System.Xml.Serialization;
using System.Collections.Generic;
using System.Windows.Forms;
//Serialize an object to disk (properties must be public):
public string Serialize(Object Input, string OutFile)
{
System.IO.MemoryStream memoryStream = new System.IO.MemoryStream();
XmlSerializer XML = new XmlSerializer(Input.GetType());
XML.Serialize(memoryStream, Input);
memoryStream.Flush();
memoryStream.Position = 0;
if (!string.IsNullOrEmpty(OutFile))
{
using (FileStream fileStream = new FileStream(OutFile, FileMode.Create))
{
byte[] data = memoryStream.ToArray();
fileStream.Write(data, 0, data.Length);
fileStream.Close();
}
}
return new System.IO.StreamReader(memoryStream).ReadToEnd();
}
//Deserialize from a serialized file:
public object DeserializeFile(Type ObjectType, string FileName)
{
Type type = typeof(object);
if (ObjectType != null)
{ type = ObjectType; }
using (FileStream fileStream = new FileStream(FileName, FileMode.Open, FileAccess.Read))
{
StreamReader sr = new StreamReader(fileStream);
string XML = sr.ReadToEnd();
XmlSerializer xmlSerializer = new System.Xml.Serialization.XmlSerializer(type);
fileStream.Close();
return xmlSerializer.Deserialize(new StringReader(XML));
}
}
CustomObject co = DeserializeFile(typeof(CustomObject), fileName.xml) as CustomObject;
I use this method to save the contents of a CheckedListBoxControl to disk so users are able to define it with their own items.
First off, the display of your data and what gui widgets you use is completely irrelevant (i.e. that you are using a "2 column listbox" doesn't matter). What is relevant (in this case of a listbox) is the collection of objects that is bound to your listbox.
The collection class is really the important element here for the purposes of loading & saving. Its contents are what you will be saving and loading from your file. The listbox that displays the data doesn't care; it simply displays the data of the collection it is bound to using whatever template you've setup.
So how do you load & save you ask? Here's a simple example that uses binary formatting. There are a gazillion other walkthrus on the Intertubes; just google "C# Serialization to file"
public void Serialize(string filename, ObjectToSerialize objectToSerialize)
{
Stream stream = File.Open(filename, FileMode.Create);
BinaryFormatter bFormatter = new BinaryFormatter();
bFormatter.Serialize(stream, objectToSerialize);
stream.Close();
}
public ObjectToSerialize DeSerialize(string filename)
{
ObjectToSerialize objectToSerialize;
Stream stream = File.Open(filename, FileMode.Open);
BinaryFormatter bFormatter = new BinaryFormatter();
objectToSerialize = (ObjectToSerialize)bFormatter.Deserialize(stream);
stream.Close();
return objectToSerialize;
}
Look into the StreamReader and StreamWriter classes.
For saving the listbox contents:
using (StreamWriter tWrite = new StreamWriter(#"c:\temp\test.txt"))
{
foreach (string tItem in listBox1.Items)
{
tWrite.WriteLine(tItem);
}
}
For reading file contents:
listBox1.Items.AddRange(File.ReadAllLines(#"c:\temp\test.txt"));
I'm trying to do custom serialization/deserialization of an object as well as compressing/decompressing the serialized data with DeflateStreams. I originally did this for more complex objects but cut it down to try and figure out the problem, however it just became more puzzling as it is still there. Here is the class to be serialized/deserialized:
[Serializable]
public class RandomObject : ISerializable
{
public String Name { get; set; }
public String SavePath { get; set; }
public RandomObject()
{
}
public RandomObject(String name, String savepath)
{
Name = name;
SavePath = savepath;
}
public RandomObject(SerializationInfo info, StreamingContext context)
: this(info.GetString("name"), info.GetString("savepath"))
{
}
[SecurityPermissionAttribute(SecurityAction.Demand, SerializationFormatter = true)]
public void GetObjectData(SerializationInfo info, StreamingContext context)
{
info.AddValue("name", Name);
info.AddValue("savepath", SavePath);
}
}
And here is the code that is supposed to serialize it(which seems to work):
BinaryFormatter bf = new BinaryFormatter();
using (MemoryStream ms = new MemoryStream())
{
bf.Serialize(ms, profile);
using (DeflateStream ds = new DeflateStream(ms, CompressionMode.Compress))
{
try
{
using (FileStream fs = File.Create(path))
{
ds.Flush();
Miscellaneous.CopyStream(ds.BaseStream, fs);
fs.Flush();
fs.Close();
}
}
catch (IOException e)
{
MessageBox.Show(e.Message);
success = false;
}
ds.Close();
}
ms.Close();
}
And here is the deserialization:
RandomObject profile = null;
using (FileStream fs = File.OpenRead(path))
{
using (DeflateStream ds = new DeflateStream(fs, CompressionMode.Decompress))
{
BinaryFormatter bf = new BinaryFormatter();
ds.Flush();
using (MemoryStream ms = new MemoryStream())
{
Miscellaneous.CopyStream(ds.BaseStream, ms);
profile = bf.Deserialize(ms) as RandomObject;
profile.SavePath = path;
ms.Close();
}
ds.Close();
}
fs.Close();
}
Now, to the problem. Deserialization throws a SerializationException with the message {"No map for object '201326592'."} I have no idea how to troubleshoot or figure out what exactly is causing the problem. Very basic serialization works when I just run BinaryFormatter's Serialize and Deserialize methods on the same MemoryStream.
I tried removing the DeflateStream stuff from both methods, but it's still the same problem. When I look at the examples at MSDN and other places it looks like I'm doing it just right, and googling for the exception message doesn't give any meaningful results(or perhaps I'm just bad at searching).
PS. As you can see I use Miscellaneous.CopyStream(src, dest) which is a basic stream copier, as I can't get src.CopyTo(dest) to work at all, so any hints on that is welcome as well.
Below is a link to the whole VS2010 project if you would like to look at it more closely:
http://www.diredumplings.com/SerializationTesting.zip
UPDATE:
The_Smallest: I tried using the Compress method you posted on my serialization:
BinaryFormatter bf = new BinaryFormatter();
using (MemoryStream stream = new MemoryStream())
{
bf.Serialize(stream, profile);
byte[] array = Compress(stream);
using (MemoryStream ms = new MemoryStream(array))
{
using (FileStream fs = File.Create(path))
{
ms.WriteTo(fs);
fs.Flush();
}
}
}
However, it seems to give me the same problems that I had with srcStream.CopyTo(destStream) earlier, which is that it doesn't seem to get written to the stream. The result is a 0 kb file when I try to save it to disk. Any ideas?
Pieter: I removed the MemoryStream from the deserialization method and it seems have the same functionality as before. However I'm not sure how to implement the serialization the way you suggested. Is this what you had in mind?
BinaryFormatter bf = new BinaryFormatter();
using (FileStream fs = File.Create(path))
{
using (DeflateStream ds = new DeflateStream(fs, CompressionMode.Compress))
{
bf.Serialize(ds, profile);
fs.Flush();
ds.Close();
}
fs.Close();
}
Thanks to both of you!
I dowloaded you example and digged a little in there. See changes for your project below:
Replace LoadFromFile in Loader.cs
private static RandomObject LoadFromFile(string path)
{
try
{
var bf = new BinaryFormatter();
using (var fileStream = File.OpenRead(path))
using (var decompressed = new MemoryStream())
{
using (var deflateStream = new DeflateStream(fileStream, CompressionMode.Decompress))
deflateStream.CopyTo(decompressed);
decompressed.Seek(0, SeekOrigin.Begin);
var profile = (RandomObject)bf.Deserialize(decompressed);
profile.SavePath = path;
return profile;
}
}
catch (IOException e)
{
MessageBox.Show(e.Message);
return null;
}
}
Replace Save in Saver.cs as follows:
public static bool Save(RandomObject profile, String path)
{
try
{
var bf = new BinaryFormatter();
using (var uncompressed = new MemoryStream())
using (var fileStream = File.Create(path))
{
bf.Serialize(uncompressed, profile);
uncompressed.Seek(0, SeekOrigin.Begin);
using (var deflateStream = new DeflateStream(fileStream, CompressionMode.Compress))
uncompressed.CopyTo(deflateStream);
}
return true;
}
catch (IOException e)
{
MessageBox.Show(e.Message);
return false;
}
}
You should serialize into the DeflateStream, not the base (MemoryStream) stream.
For serializing: begin with the File.Create. Then around that stream, create the DeflateStream. Then to the DefaulteStream, serialize your objects.
For deserializing: do not create the MemoryStream and deserialize directly from the DeflateStream.
I believe there is no need for the added MemoryStream. If however you do have problems writing directly to/reading directly from the file streams, just change the serialize routine to write to the DeflateStream instead of the MemoryStream.
That should solve your issues.
There is error in streams logics, while compressing you should write to CompressStream, which writes to MemoryStream, after this you will have result in MemoryStream (not in CompressStream)
Here is example how to compress and decompress bytes
private static byte[] Compress(Stream stream)
{
using (var resultStream = new MemoryStream())
{
using (var gzipStream = new DeflateStream(resultStream, CompressionMode.Compress))
stream.CopyTo(gzipStream);
return resultStream.ToArray();
}
}
private static byte[] Decompress(byte[] bytes)
{
using (var readStream = new MemoryStream(bytes))
using (var resultStream = new MemoryStream())
{
using (var gzipStream = new DeflateStream(readStream, CompressionMode.Decompress))
gzipStream.CopyTo(resultStream);
return resultStream.ToArray();
}
}