I have, in my code, a ConcurrentBag<Point3DCollection>.
I'm trying to figure out how to serialize them. Of course I could iterate through or package it with a provider model class, but I wonder if it's already been done.
The Point3DCollections themselves are potentially quite large and could stand to be compressed to speed up reading and writing to and from the disk, but the response times I need for this are largely in the user interface scale. In other words, I prefer a binary formatting over a XAML-text formatting, for performance reasons. (There is a nice XAML-text serializer which is part of the Helix 3D CodeProject, but it's slower than I'd like.)
Is this a use case where I'm left rolling out my own serializer, or is there something out there that's already packaged for this kind of data?
Here are some extensions methods that handle string and binary serialization of Point3DCollection bags. As I said in my comment, I don't think there is a best way of doing this in all cases, so you might want to try both. Also note they're using Stream parameter as input so you can chain these with calls to GZipStream of DeflateStream.
public static class Point3DExtensions
{
public static void StringSerialize(this ConcurrentBag<Point3DCollection> bag, Stream stream)
{
if (bag == null)
throw new ArgumentNullException("bag");
if (stream == null)
throw new ArgumentNullException("stream");
StreamWriter writer = new StreamWriter(stream);
Point3DCollectionConverter converter = new Point3DCollectionConverter();
foreach (Point3DCollection coll in bag)
{
// we need to use the english locale as the converter needs that for parsing...
string line = (string)converter.ConvertTo(null, CultureInfo.GetCultureInfo("en-US"), coll, typeof(string));
writer.WriteLine(line);
}
writer.Flush();
}
public static void StringDeserialize(this ConcurrentBag<Point3DCollection> bag, Stream stream)
{
if (bag == null)
throw new ArgumentNullException("bag");
if (stream == null)
throw new ArgumentNullException("stream");
StreamReader reader = new StreamReader(stream);
Point3DCollectionConverter converter = new Point3DCollectionConverter();
do
{
string line = reader.ReadLine();
if (line == null)
break;
bag.Add((Point3DCollection)converter.ConvertFrom(line));
// NOTE: could also use this:
//bag.Add(Point3DCollection.Parse(line));
}
while (true);
}
public static void BinarySerialize(this ConcurrentBag<Point3DCollection> bag, Stream stream)
{
if (bag == null)
throw new ArgumentNullException("bag");
if (stream == null)
throw new ArgumentNullException("stream");
BinaryWriter writer = new BinaryWriter(stream);
writer.Write(bag.Count);
foreach (Point3DCollection coll in bag)
{
writer.Write(coll.Count);
foreach (Point3D point in coll)
{
writer.Write(point.X);
writer.Write(point.Y);
writer.Write(point.Z);
}
}
writer.Flush();
}
public static void BinaryDeserialize(this ConcurrentBag<Point3DCollection> bag, Stream stream)
{
if (bag == null)
throw new ArgumentNullException("bag");
if (stream == null)
throw new ArgumentNullException("stream");
BinaryReader reader = new BinaryReader(stream);
int count = reader.ReadInt32();
for (int i = 0; i < count; i++)
{
int pointCount = reader.ReadInt32();
Point3DCollection coll = new Point3DCollection(pointCount);
for (int j = 0; j < pointCount; j++)
{
coll.Add(new Point3D(reader.ReadDouble(), reader.ReadDouble(), reader.ReadDouble()));
}
bag.Add(coll);
}
}
}
And a little console app test program to play with:
static void Main(string[] args)
{
Random rand = new Random(Environment.TickCount);
ConcurrentBag<Point3DCollection> bag = new ConcurrentBag<Point3DCollection>();
for (int i = 0; i < 100; i++)
{
Point3DCollection coll = new Point3DCollection();
bag.Add(coll);
for (int j = rand.Next(10); j < rand.Next(100); j++)
{
Point3D point = new Point3D(rand.NextDouble(), rand.NextDouble(), rand.NextDouble());
coll.Add(point);
}
}
using (FileStream stream = new FileStream("test.bin", FileMode.Create))
{
bag.StringSerialize(stream); // or Binary
}
ConcurrentBag<Point3DCollection> newbag = new ConcurrentBag<Point3DCollection>();
using (FileStream stream = new FileStream("test.bin", FileMode.Open))
{
newbag.StringDeserialize(stream); // or Binary
foreach (Point3DCollection coll in newbag)
{
foreach (Point3D point in coll)
{
Console.WriteLine(point);
}
Console.WriteLine();
}
}
}
}
Compression could potentially take advantage of repeated coordinates. Serializers will often use references for repeat objects as well, although I'm not sure there are many set up to work with structs (like Point3D). Anyhow, here are some examples of how to serialize this. To use the standard formatters, you need to convert the data type to something most of them support: list/array. The code below uses Nuget packages NUnit and Json.NET.
using Newtonsoft.Json;
using Newtonsoft.Json.Bson;
using NUnit.Framework;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;
using System.Windows.Media.Media3D;
namespace DemoPoint3DSerialize
{
[TestFixture]
class Tests
{
[Test]
public void DemoBinary()
{
// this shows how to convert them all to strings
var collection = CreateCollection();
var data = collection.Select(c => c.ToArray()).ToList(); // switch to serializable types
var formatter = new BinaryFormatter();
using (var ms = new MemoryStream())
{
formatter.Serialize(ms, data);
Trace.WriteLine("Binary of Array Size: " + ms.Position);
ms.Position = 0;
var dupe = (List<Point3D[]>)formatter.Deserialize(ms);
var result = new ConcurrentBag<Point3DCollection>(dupe.Select(r => new Point3DCollection(r)));
VerifyEquality(collection, result);
}
}
[Test]
public void DemoString()
{
// this shows how to convert them all to strings
var collection = CreateCollection();
IEnumerable<IList<Point3D>> tmp = collection;
var strings = collection.Select(c => c.ToString()).ToList();
Trace.WriteLine("String Size: " + strings.Sum(s => s.Length)); // eh, 2x for Unicode
var result = new ConcurrentBag<Point3DCollection>(strings.Select(r => Point3DCollection.Parse(r)));
VerifyEquality(collection, result);
}
[Test]
public void DemoDeflateString()
{
// this shows how to convert them all to strings
var collection = CreateCollection();
var formatter = new BinaryFormatter(); // not really helping much: could
var strings = collection.Select(c => c.ToString()).ToList();
using (var ms = new MemoryStream())
{
using (var def = new DeflateStream(ms, CompressionLevel.Optimal, true))
{
formatter.Serialize(def, strings);
}
Trace.WriteLine("Deflate Size: " + ms.Position);
ms.Position = 0;
using (var def = new DeflateStream(ms, CompressionMode.Decompress))
{
var stringsDupe = (IList<string>)formatter.Deserialize(def);
var result = new ConcurrentBag<Point3DCollection>(stringsDupe.Select(r => Point3DCollection.Parse(r)));
VerifyEquality(collection, result);
}
}
}
[Test]
public void DemoStraightJson()
{
// this uses Json.NET
var collection = CreateCollection();
var formatter = new JsonSerializer();
using (var ms = new MemoryStream())
{
using (var stream = new StreamWriter(ms, new UTF8Encoding(true), 2048, true))
using (var writer = new JsonTextWriter(stream))
{
formatter.Serialize(writer, collection);
}
Trace.WriteLine("JSON Size: " + ms.Position);
ms.Position = 0;
using (var stream = new StreamReader(ms))
using (var reader = new JsonTextReader(stream))
{
var result = formatter.Deserialize<List<Point3DCollection>>(reader);
VerifyEquality(collection, new ConcurrentBag<Point3DCollection>(result));
}
}
}
[Test]
public void DemoBsonOfArray()
{
// this uses Json.NET
var collection = CreateCollection();
var formatter = new JsonSerializer();
using (var ms = new MemoryStream())
{
using (var stream = new BinaryWriter(ms, new UTF8Encoding(true), true))
using (var writer = new BsonWriter(stream))
{
formatter.Serialize(writer, collection);
}
Trace.WriteLine("BSON Size: " + ms.Position);
ms.Position = 0;
using (var stream = new BinaryReader(ms))
using (var reader = new BsonReader(stream, true, DateTimeKind.Unspecified))
{
var result = formatter.Deserialize<List<Point3DCollection>>(reader); // doesn't seem to read out that concurrentBag
VerifyEquality(collection, new ConcurrentBag<Point3DCollection>(result));
}
}
}
private ConcurrentBag<Point3DCollection> CreateCollection()
{
var rand = new Random(42);
var bag = new ConcurrentBag<Point3DCollection>();
for (int i = 0; i < 10; i++)
{
var collection = new Point3DCollection();
for (int j = 0; j < i + 10; j++)
{
var point = new Point3D(rand.NextDouble(), rand.NextDouble(), rand.NextDouble());
collection.Add(point);
}
bag.Add(collection);
}
return bag;
}
private class CollectionComparer : IEqualityComparer<Point3DCollection>
{
public bool Equals(Point3DCollection x, Point3DCollection y)
{
return x.SequenceEqual(y);
}
public int GetHashCode(Point3DCollection obj)
{
return obj.GetHashCode();
}
}
private void VerifyEquality(ConcurrentBag<Point3DCollection> collection, ConcurrentBag<Point3DCollection> result)
{
var first = collection.OrderBy(c => c.Count);
var second = collection.OrderBy(c => c.Count);
first.SequenceEqual(second, new CollectionComparer());
}
}
}
Use Google's protobuf-net. protobuf-net is an open source .net implementation of Google's protocol buffer binary serialization format which can be used as a replacement for the BinaryFormatter serializer. It is probably going to be the fastest solution and easiest to implement.
Here is a link to the the main google wiki for protobuf-net. On the left, you'll find the downloads for all of the most updated binaries.
https://code.google.com/p/protobuf-net/
Here is a great article that you might want to look at first to get a feel for how it works.
http://wallaceturner.com/serialization-with-protobuf-net
Here is a link to a discussion on google's wiki about your specific problem. The answer is at the bottom of the page. That's where I got the code below and substituted with details from your post.
https://code.google.com/p/protobuf-net/issues/detail?id=354
I haven't used it myself but it looks like a very good solution to your stated needs. From what I gather, your code would end up some variation of this.
[ProtoContract]
public class MyClass {
public ConcurrentQueue<Point3DCollection> Points {get;set;}
[ProtoMember(1)]
private Point3DCollection[] Items
{
get { return Points.ToArray(); }
set { Items = new ConcurrentBag<Point3DCollection>(value); }
}
}
I wish you the best of luck. Take care.
For a large amount of data, why don't you consider Sqlite or any other small database system etc, which can store structured data in the file.
I have seen many 3d programs using database to store structure along with relations, which allow them to partially insert/update/delete data.
Benefit of Sqlite/database will be multithreaded serialization to improve speed, however you need to do little bit of work on sqlite to enable multi threaded sqlite connection, or else you can use LocalDB of SQL Express or even Sql Compact.
Also some of workload of loading data can be done through queries, which will be indexed by database nicely. And most of things can be done on background worker without interfering with User Interface.
Sqlite has limited multi-thread support, which can be explored here http://www.sqlite.org/threadsafe.html
Sql Compact is thread safe and requires installation that can be installed without admin priviledges. And you can use Entity framework as well.
Related
What does this error mean? Failed to create compatible device context
We have a aspx page which calls a webservice. The webservice in turns called 3rd API with Hyland. The error happens in the third party API
I have contacted their API support. They are slow to respond and since this happens in production I am wondering if there is a fix or somehow lighten the impact? Is it memory related.
I got a dump and opened it in ants memory profile dump. This is not a snapshot. I am finding memory leaks. What does this mean? Is red bad. Does it really have 589 instances. I clicked on the system.Byte[] 2.31MB and see this instance list.
Why would it have so many 4120 in the instance list? Is it really creating all these instances.
Could this be the problem? I have a routine like this. This is a asp.net webpage which calls a webservice. The webservice calls a class Provider.GetDocumentData().
public static byte[] GetDocumentData(Application obApp, ParametersList parameters, ref PropertiesList Properties, ref KeywordPropertiesList Keywords)
{
var type = "";
var BUFFER_SIZE = 65536;
var byteData = new byte[BUFFER_SIZE];
long documentId = 0;
PageRangeSet pageRangeSet = null;
var docQuery = obApp.Core.CreateDocumentQuery();
var pageRanges = "";
//get the content type andd DocumentId from paramters
foreach (var paramter in parameters)
{
if (paramter.Key == GetDocumentDataRequest.DocumentID)
{
documentId = Convert.ToInt64(paramter.Value);
}
if (paramter.Key == GetDocumentDataRequest.PageRange)
{
pageRanges = paramter.Value;
}
if (paramter.Key.Contains(GetDocumentDataRequest.ContentType))
{
type = paramter.Value;
}
}
var document = obApp.Core.GetDocumentByID(documentId);
if (document != null)
{
if (String.IsNullOrEmpty(pageRanges))
{
pageRanges = "1-10000";
}
if (type == "text/plain")
{
var textProvider = obApp.Core.Retrieval.Text;
var documentdata = document.DefaultRenditionOfLatestRevision;
pageRangeSet = textProvider.CreatePageRangeSet(pageRanges);
var newpagecharacter = System.Text.Encoding.UTF8.GetBytes(new[]
{
'\f'
});
var newPageStream = new MemoryStream(newpagecharacter);
using (var ms = new MemoryStream())
{
using (var pageDataList = textProvider.GetPages(documentdata, pageRangeSet))
{
for (var i = 0; i < pageDataList.Count; i++)
{
using (var newStream = new MemoryStream())
{
var pageData = pageDataList[i];
pageData.Stream.CopyTo(newStream);
newStream.Seek(0, System.IO.SeekOrigin.Begin);
newPageStream.Seek(0, System.IO.SeekOrigin.Begin);
newStream.CopyTo(ms);
newPageStream.CopyTo(ms);
}
}
byteData = ms.ToArray();
}
}
}
else if (type == "application/pdf")
{
var pdfProvider = obApp.Core.Retrieval.PDF;
var documentdata = document.DefaultRenditionOfLatestRevision;
var pageData = pdfProvider.GetDocument(documentdata);
using (var ms = new MemoryStream())
{
pageData.Stream.CopyTo(ms);
byteData = ms.ToArray();
}
}
else if (type == "image/tiff")
{
var imageProvider = obApp.Core.Retrieval.Image;
var documentdata = document.DefaultRenditionOfLatestRevision;
var pageData = imageProvider.GetDocument(documentdata);
using (var ms = new MemoryStream())
{
pageData.Stream.CopyTo(ms);
byteData = ms.ToArray();
}
}
else
{
var defaultProvider = obApp.Core.Retrieval.Default;
var documentdata = document.DefaultRenditionOfLatestRevision;
pageRangeSet = defaultProvider.CreatePageRangeSet(pageRanges);
var newpagecharacter = System.Text.Encoding.UTF8.GetBytes(new[]
{
'\f'
});
var newPageStream = new MemoryStream(newpagecharacter);
using (var ms = new MemoryStream())
{
using (var pageDataList = defaultProvider.GetPages(documentdata, pageRangeSet))
{
for (var i = 0; i < pageDataList.Count; i++)
{
using (var newStream = new MemoryStream())
{
var pageData = pageDataList[i];
pageData.Stream.CopyTo(newStream);
newStream.Seek(0, System.IO.SeekOrigin.Begin);
newPageStream.Seek(0, System.IO.SeekOrigin.Begin);
newStream.CopyTo(ms);
newPageStream.CopyTo(ms);
}
}
byteData = ms.ToArray();
}
}
}
}
return byteData;
}
I wrote a program that reads a PDF file and then shows some information about it. It works fine.
class Program
{
static void Main(string[] args)
{
//some variables
string plabels = null;
string text2Search = "IX";
PdfReader reader = new PdfReader("file.pdf");
string[] labels = PdfPageLabels.GetPageLabels(reader);
for (int l = 0; l < labels.Length; l++)
{
plabels += labels[l] + "\n";
}
Console.WriteLine(plabels.IndexOf(text2Search, StringComparison.CurrentCultureIgnoreCase));
Console.ReadLine();
}
}
I also wrote another program that reads a PDF file and makes a copy of it. It also works fine.
class Program
{
static void Main(string[] args)
{
byte[] bytes;
using (var ms = new MemoryStream())
{
using (var reader = new PdfReader("file.pdf"))
{
using (var stamper = new PdfStamper(reader, ms))
{
}
}
//grab the bytes before closing things out
bytes = ms.ToArray();
}
File.WriteAllBytes("output.pdf", bytes);
}
}
What I cannot do is to combine these two programs into one.
EDIT
Thanks to your comments I spotted an error. Now my code compiles fine, but when run it gives "Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object." The error point to line for(int l = 0; l < labels.Length; l++).
class Program
{
static void Main(string[] args)
{
byte[] bytes;
using (var ms = new MemoryStream())
{
using (var reader = new PdfReader("file.pdf"))
{
//some variables
string plabels = null;
string text2Search = "IX";
string[] labels = PdfPageLabels.GetPageLabels(reader);
for(int l = 0; l < labels.Length; l++)
{
plabels += labels[l] + "\n";
}
Console.WriteLine(plabels.IndexOf(text2Search, StringComparison.CurrentCultureIgnoreCase));
Console.ReadLine();
using (var stamper = new PdfStamper(reader, ms))
{
}
}
//grab the bytes before closing things out
bytes = ms.ToArray();
}
File.WriteAllBytes("output.pdf", bytes);
}
}
reader is already defined in the using statement. You cannot re-use that variable name inside the using block. Change the name of one of the reader variables or get rid of it.
using (PdfReader reader = new PdfReader("file.pdf"))
{
PdfReader reader = new PdfReader("file.pdf"); //is redundant.
}
labels is null. Your PdfPageLabels.GetPageLabels(reader);
is not returning a value so look in there. No code is posted so I can't give you a clearer answer.
I have a Game application(WP8), where we are saving the scores of multiple attampts and showing it to user.
I have a Object with fields noOfStonesPicked and noOfFruitsPicked.
Here is my code:
MyTopic topicObj = new MyTopic ();
for (int i = 0; i <= 2; i++)
{
Test mt = new Test();
mt.noOfStonesPicked = 12;
mt.noOfFruitsPicked= 20;
topicObj.Stats.Add(mt);
}
WritetestTopicState(topicObj);
Now 3 attempts with each one having noOfStonesPicked -12 and noOfFruitsPicked - 20
Now i have saving this like :
public static void WritetestTopicState(MyTopic topic)
{
try
{
using (var store = IsolatedStorageFile.GetUserStoreForApplication())
{
using (StreamWriter sw = new StreamWriter(store.OpenFile("12.xml", FileMode.Append, FileAccess.Write)))
{
XmlSerializer serializer = new XmlSerializer(typeof(MyTopic));
serializer.Serialize(sw, topic);
serializer = null;
}
}
}
catch (Exception)
{
throw;
}
}
Now how can i retrive these values and display ?
EDIT
This is what i have tried:
public static MyTopic ReadMockTestTopicState()
{
MyTopic topic = null;
try
{
using (IsolatedStorageFile isoStore = IsolatedStorageFile.GetUserStoreForApplication())
{
// Read application settings.
if (isoStore.FileExists("11.xml"))
{
using (var store = IsolatedStorageFile.GetUserStoreForApplication())
{
using (StreamReader SR = new StreamReader(store.OpenFile("12.xml", FileMode.Open, FileAccess.Read)))
{
XmlSerializer serializer = new XmlSerializer(typeof(MyTopic));
topic = (MyTopic)serializer.Deserialize(SR);
serializer = null;
}
}
}
else
{
// If setting does not exists return default setting.
topic = new MyTopic();
}
}
}
catch (Exception)
{
throw;
}
return topic;
}
XmlSerializer serializer = new XmlSerializer(typeof(MyTopic));
StreamReader reader = new StreamReader(path);
_myTopic = (MyTopic)serializer.Deserialize(reader);
reader.Close();
This should be enough for deserializing, If your MyTopic object is properly serializable, I mean if properties of the MyTopic object are properly attributed for xml serialization.
I am not sure what I am doing wrong, have seen a lot of examples, but can't seem to get this working.
public static Stream Foo()
{
var memStream = new MemoryStream();
var streamWriter = new StreamWriter(memStream);
for (int i = 0; i < 6; i++)
streamWriter.WriteLine("TEST");
memStream.Seek(0, SeekOrigin.Begin);
return memStream;
}
I am doing a simple test on this method to try and get it to pass, but no matter what, my collection count is 0.
[Test]
public void TestStreamRowCount()
{
var stream = Foo();
using (var reader = new StreamReader(stream))
{
var collection = new List<string>();
string input;
while ((input = reader.ReadLine()) != null)
collection.Add(input);
Assert.AreEqual(6, collection.Count);
}
}
Note: I changed some syntax above without compiling in the Test method. What is more important is the first method which seems to be returning an empty stream (my reader.ReadLine() always reads once). Not sure what I am doing wrong. Thank you.
You are forgetting to flush your StreamWriter instance.
public static Stream Foo()
{
var memStream = new MemoryStream();
var streamWriter = new StreamWriter(memStream);
for (int i = 0; i < 6; i++)
streamWriter.WriteLine("TEST");
streamWriter.Flush(); <-- need this
memStream.Seek(0, SeekOrigin.Begin);
return memStream;
}
Also note that StreamWriter is supposed to be disposed of, since it implements IDisposable, but that in turn generates another problem, it will close the underlying MemoryStream as well.
Are you sure you want to return a MemoryStream here?
I would change the code to this:
public static byte[] Foo()
{
using (var memStream = new MemoryStream())
using (var streamWriter = new StreamWriter(memStream))
{
for (int i = 0; i < 6; i++)
streamWriter.WriteLine("TEST");
streamWriter.Flush();
return memStream.ToArray();
}
}
[Test]
public void TestStreamRowCount()
{
var bytes = Foo();
using (var stream = new MemoryStream(bytes))
using (var reader = new StreamReader(stream))
{
var collection = new List<string>();
string input;
while ((input = reader.ReadLine()) != null)
collection.Add(input);
Assert.AreEqual(6, collection.Count);
}
}
Since you are not using "using" or streamWriter.Flush() the writer did not commit changes to the stream. As result Stream itslef does not have data yet. In general you want to wrap manipulation with Stream and StremaWriter instances with using.
You also should consider returning new instance of MemoryStream:
using(var memStream = new MemoryStream())
{
....
return new MemoryStream(memStream.ToArray(), false /*writable*/);
}
Try flushing streamWriter after writing your lines.
I am currently trying to serialize a List, it serializes (I think fine), but when it deserialize,
Sorry for the amount of code, but I am really stuck and have no idea why this is happening, i also tried to changed the struct into a class and no help.
THANKS.
i get the following error UPDATED
There was an error deserializing the object of type There was an error deserializing the object of type
`System.Collections.Generic.List`1[[A.B.C.DataValues, A.V, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]]. Unexpected end of file. Following elements are not closed: Time, DataValues, ArrayOfDataValues.`
I am serializing like this UPDATED
public void SerializeDataValue(List<DataValues> values)
{
DataContractSerializer serializer = new DataContractSerializer(typeof(List<DataValues>));
using (MemoryStream stream = new MemoryStream())
{
using (GZipStream compress = new GZipStream(stream, CompressionMode.Compress))
{
XmlDictionaryWriter w = XmlDictionaryWriter.CreateBinaryWriter(compress);
serializer.WriteObject(w, values);
}
_serializedData = stream.ToArray();
}
}
I am deserializing like this UPDATED
public List<DataValues> DeserializeDataValue()
{
if (SerializedData == null || SerializedData.Length == 0)
{
return new List<DataValues> ();
}
else
{
DataContractSerializer serializer = new DataContractSerializer(typeof(List<DataValues>));
using (MemoryStream stream = new MemoryStream(SerializedData))
{
using (GZipStream decompress = new GZipStream(stream, CompressionMode.Decompress))
{
XmlDictionaryReader r = XmlDictionaryReader.CreateBinaryReader(decompress, XmlDictionaryReaderQuotas.Max);
return serializer.ReadObject(r, true) as List<DataValues>;
}
}
}
}
Properties
private byte[] _serializedData;
[DataMember]
[Browsable(false)]
public byte[] SerializedData
{
get { return _serializedData; }
set { _serializedData = value; }
}
helper Methods
public static byte[] ReadFully(Stream input)
{
byte[] buffer = new byte[16 * 1024];
input.Position = 0;
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
return ms.ToArray();
}
}
Struct
[DataContract(Name = "DataValues", Namespace = "A.B.C")]
public struct DataValues
{
[DataMember]
public DateTime Time { get; set; }
[DataMember]
public Single Value { get; set; }
public DataValues(DateTime dateTime, Single value)
{
Time = dateTime;
Value = value;
}
}
It’s because you are not serialising the object(s) completely. You need to close the stream(s) after writing, especially when using gzip. Recommended practice is to use using:
public void SerializeDataValue(List<DataValues> values)
{
DataContractSerializer serializer = new DataContractSerializer(typeof(List<DataValues>));
using (MemoryStream stream = new MemoryStream())
{
using (GZipStream compress = new GZipStream(stream, CompressionMode.Compress))
{
XmlDictionaryWriter w = XmlDictionaryWriter.CreateBinaryWriter(compress);
serializer.WriteObject(w, values);
}
_serializedData = stream.ToArray();
}
}
Sorry to be late to this question.
The problem with the initial approach was simply that you weren't flushing (read: disposing) the XmlDictionaryWriter.
This should work (note the 2nd using clause):
using (GZipStream compress = new GZipStream(stream, CompressionMode.Compress))
using (XmlDictionaryWriter w = XmlDictionaryWriter.CreateBinaryWriter(compress))
{
serializer.WriteObject(w, values);
}
Hope this helps someone.
I can get the sample to work by removing the XmlDictionaryReader and instead directly feeding the input/output stream into the DataContractSerializer. It may be a defect in the XmlDictionaryReader for large compressed collections but I'm not sure.
Hope this helps:
public void SerializeDataValue(List<DataValues> values)
{
DataContractSerializer serializer = new DataContractSerializer(typeof(List<DataValues>));
using (MemoryStream stream = new MemoryStream())
{
using (GZipStream compress = new GZipStream(stream, CompressionMode.Compress))
{
serializer.WriteObject(compress , values);
}
_serializedData = stream.ToArray();
}
}
public List<DataValues> DeserializeDataValue()
{
if (SerializedData == null || SerializedData.Length == 0)
{
return new List<DataValues> ();
}
else
{
DataContractSerializer serializer = new DataContractSerializer(typeof(List<DataValues>));
using (MemoryStream stream = new MemoryStream(SerializedData))
{
using (GZipStream decompress = new GZipStream(stream, CompressionMode.Decompress))
{
return serializer.ReadObject(decompress , true) as List<DataValues>;
}
}
}
}
I ran exactly into the same problem and I finally found the solution : the XmlDictionaryWriter needs to be disposed/closed before the Stream you are writing into is itself closed. I discovered that thanks to the thorough example found at http://www.albahari.com/nutshell/ch15.aspx whiche are more complete than the MSDN ones.
In your sample code, that would be :
using (XmlDictionaryWriter w = XmlDictionaryWriter.CreateBinaryWriter(compress))
{
serializer.WriteObject(w, values);
}
On my own example, using the XmlDictionaryWriter instead of the plain and by default Xml writer only gave me a ~25% decrease in file size but a factor 3 when reading back the object.