Persisting MemoryCache Content to File - c#

I'm intending to use the powerful caching libraries that are introduced in .Net 4.0 in a Windows Forms application. So far, MemoryCache does everything I need except for persisting its contents to a file. What I'm trying to do is persisting the cache to a file on application exit, and then when the application is opened again, I should be able to write from the file and put its contents in the MemoryCache. I know that I can simply serialize the instance into binary on disk, but then again, I wouldn't know how to convert it back to a *MemoryCache*.
Your help is much appreciated.

I ended up implementing my own Cache project. Hopefully this will help someone somewhere:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.Serialization;
using System.Text;
namespace CachingDemo
{
class CachedMemory
{
System.Collections.Specialized.OrderedDictionary cache = null;
private String persistenceFilePath = null;
private int cacheSizeLimit;
public static readonly int CACHE_SIZE_NO_LIMIT = -1;
public CachedMemory(int initialCapacity, int cacheSizeLimit, string persistenceFilePath)
{
this.cache = new System.Collections.Specialized.OrderedDictionary(initialCapacity);
this.persistenceFilePath = persistenceFilePath;
this.cacheSizeLimit = cacheSizeLimit;
}
public int getCacheSize()
{
return this.cache.Count;
}
public CachedMemory(int cacheSizeLimit, string cacheFilePath)
{
initializeCache(cacheFilePath, cacheSizeLimit);
}
private void initializeCache(string cacheFilePath, int cacheSizeLimit)
{
this.cacheSizeLimit = cacheSizeLimit;
using (FileStream fileStream = new FileStream(cacheFilePath, FileMode.Open))
{
IFormatter bf = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
this.cache = (System.Collections.Specialized.OrderedDictionary)bf.Deserialize(fileStream);
fileStream.Close();
}
//In case the deserialized OrderedDictionary had more contents than the limit, we need to shrink it to make its size equal to the limit
if (this.cacheSizeLimit != CACHE_SIZE_NO_LIMIT && this.cache.Keys.Count > this.cacheSizeLimit)
{
int difference = this.cache.Keys.Count - this.cacheSizeLimit;
for (int i = 0; i < difference; i++)
{
cache.RemoveAt(0);
}
}
}
public string get(string key)
{
return cache[key] as string;
}
public string get(int index)
{
return cache[index] as string;
}
public void add(string key, string value)
{
//An ordered dictionary would throw an exception if we try to insert the same key again, so we have to make sure that the newly
//introduced key is not a duplicate.
if (this.cache.Contains(key))
{
this.cache.Remove(key);
}
else
if (this.cacheSizeLimit != CACHE_SIZE_NO_LIMIT && this.cache.Count == this.cacheSizeLimit)
{
this.cache.RemoveAt(0);
}
this.cache.Add(key, value);
}
public void persist()
{
using (FileStream fileStream = new FileStream(persistenceFilePath, FileMode.Create))
{
IFormatter bf = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
bf.Serialize(fileStream, this.cache);
fileStream.Close();
}
}
}
}

Related

Why can I no longer deserialize this data?

I am using the following code to serialize some data and save it to file:
MemoryStream stream = new MemoryStream();
DataContractJsonSerializer serializer = new DataContractJsonSerializer((typeof(Item)));
Item item = ((Item)list.SelectedItems[0].Tag);
serializer.WriteObject(stream, item);
var filepath = Program.appDataPath + list.SelectedItems[0].Group.Name + ".group";
stream.Position = 0;
using (FileStream fileStream = new FileStream(filepath, FileMode.Create))
{
stream.WriteTo(fileStream);
}
And later on, I'm trying to read back that data from file and insert it into ListView:
private void OpenFiles()
{
// DEBUG ONLY:
// Read into memorystream and filestream.
Console.WriteLine("Attempeting to open note.");
bool canLoad = false;
foreach (string file in Directory.GetFiles(Program.appDataPath))
{
if (file.EndsWith(".group"))
{
MemoryStream stream = new MemoryStream();
DataContractJsonSerializer serializer =
new DataContractJsonSerializer(
typeof(
List<Item>
)
);
using (FileStream fileStream =
new FileStream(
file,
FileMode.Open)
)
{
fileStream.CopyTo(stream);
}
stream.Position = 0;
//List<Withdrawal> tempWithList = new List<Withdrawal>();
foreach (Item item in (List<Item>)serializer.ReadObject(stream))
{
Console.WriteLine(item.Title + " " + item.Group.Name);
Item.Items.Add(item);
}
//Console.WriteLine("Got file \{file}");
//if (file.EndsWith(".group"))
//{
// Console.WriteLine("File is a group.");
// MemoryStream stream = new MemoryStream();
// DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(List<Item>));
// using (FileStream fileStream = new FileStream(file, FileMode.Open))
// {
// fileStream.CopyTo(stream);
// }
// Console.WriteLine("Got stream");
// stream.Position = 0;
// try
// {
// Item.Items = (List<Item>)serializer.ReadObject(stream);
// Console.WriteLine("WTF?");
// }
// catch(Exception exception)
// {
// Console.WriteLine(exception.Message);
// }
// Console.WriteLine(Item.Items.Count);
// canLoad = true;
//}
//else Console.WriteLine("File is not a group.");
}
if (canLoad)
{
//list.Items.Clear();
foreach (Item item in Item.Items)
{
ListViewGroup group = new ListViewGroup(item.Group.Name);
list.Groups.Add(group);
list.Items.Add(
new ListViewItem(
item.Title,
group)
);
Console.WriteLine(item.Title + " " + item.Group.Name);
}
}
}
}
Now, the above exact code works in an older program (few months old), but it's not working in a new program. I have no idea why. I have set breakpoints EVERYWHERE and it has proven to be kind of pointless in this case.
One thing I did learn from setting a breakpoint is that even though the stream contains the data expected, the very next second, when it gets added to list, it is NULL. There is nothing in the list. I've run out of ideas, and Google wasn't of much help.
Group.cs:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.Text;
using System.Threading.Tasks;
namespace Notes
{
[DataContract]
public class Group
{
[DataMember]
public string Name { get; set; }
}
}
Item.cs:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.Text;
using System.Threading.Tasks;
namespace Notes
{
[DataContract]
[Serializable]
public class Item : Note
{
[DataMember]
public static List<Item> Items = new List<Item>();
[DataContract]
public enum ItemType
{
Group,
Note
}
[DataMember]
public ItemType Type { get; set; }
[DataMember]
public int Index { get; set; }
}
}
Note.cs:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.Text;
using System.Threading.Tasks;
namespace Notes
{
[DataContract]
public class Note
{
[DataMember]
public string Title { get; set; }
[DataMember]
public string Content { get; set; }
[DataMember]
public Group Group;
[DataContract]
public enum NoteImportance
{
Important,
Neutral,
NotImportant
}
[DataMember]
public NoteImportance Importance { get; set; }
[DataMember]
public bool Protected { get; set; }
}
}
How can I deserialize these objects/read from file and get them into a List or ListView? I've already done this, but for some reason it's not working anymore.
Any help would be appreciated.
When you create a .group file, you serialize a single Item:
DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(Item));
// And later
serializer.WriteObject(stream, item);
But when you deserialize the contents of a .group file, you try to deserialize a List<Item>:
DataContractJsonSerializer serializer = new DataContractJsonSerializer(typeof(List<Item>));
// And later
foreach (Item item in (List<Item>)serializer.ReadObject(stream))
{
Item.Items.Add(item);
}
Those types don't match. But in order to deserialize the data you previously serialized, they need to match - or at least, the deserialized type cannot be a collection if the serialized type was, because collections are serialized as JSON arrays while other classes are serialized as JSON objects (name/value pairs).
Since it looks like each .group file has a single item, and there are many .group files in the directory you are scanning, you probably just want to do
var serializer = new DataContractJsonSerializer(typeof(Item));
// And later
var item = (Item)serializer.ReadObject(stream);
if (item != null)
Item.Items.Add(item);

Serializing Arrays in C#

Is there a way to Serialize an entire array in C# such as:
[Serializable()]
public class Data
{
public short Some_Short_Data = new short[100,100];
public string Some_String_Data = new string[100,100];
}
Then writting the file:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.Xml.Serialization;
using System.IO;
Data Write_Data = new Data();
XmlSerializer Writer = new XmlSerializer(typeof(Data));
using (FileStream file = File.OpenWrite("Myfile.xml"))
{
Writer.Serialize(file, Write_Data); //Writes data to the file
}
When I try this, it fails on:
XmlSerializer Writer = new XmlSerializer(typeof(Data));
Saying: "There was an error reflecting type 'Data' "
I am particularly new to both Stack and Xml/Xml Serialization so any help would be appreciated.
Multi-dimensional arrays are not supported by XmlSerialzier. But you can make a workaround by using a temp class something like this
public class Array100<T>
{
public T[] Data = new T[100];
}
public class Data
{
public Array100<short>[] Some_Short_Data = new Array100<short>[100];
public Array100<string>[] Some_String_Data = new Array100<string>[100];
}
BTW: No need for Serializable attribute. It is not used by XmlSerialzier
You cannot serialze a int[,] but you can serialize a int[][]. Before serializing your 2D array, just convert it into a jagged array like that :
var my2dArray = new int[2,5];
var myJaggedArray = new int [2][];
for(int i = 0 ; i < my2DArray.GetLength(0) ; i ++)
{
myJaggedArray[i] = new int[my2DArray.GetLength(1)];
for(int j = 0 ; j < my2DArray.GetLength(1) ; j ++)
myJaggedArray[i][j] = my2DArray[i,j];
}
I ran into this issue when attempting to serialize a multi-dimensional array that was one of the values in a dictionary of objects. The approach I took was to wrap all of the arrays in a class that is serializable before serialization and then unwrap them all afterwards.
If you don't care what the data looks like then you can serialize the object using the binary formatter, which can serialize anything marked as serializable, and then persist the binary data as a base 64 encoded string.
The wrapper implementation is below.
[Serializable]
public class UnserializableValueWrapper: IXmlSerializable
{
private static readonly BinaryFormatter Formatter = new BinaryFormatter();
public UnserializableValueWrapper([NotNull] object value)
{
if (value == null)
{
throw new ArgumentNullException("value");
}
this.Value = value;
}
public UnserializableValueWrapper()
{
}
public object Value { get; private set; }
public XmlSchema GetSchema()
{
return null;
}
public void ReadXml(XmlReader reader)
{
reader.ReadStartElement();
var length = int.Parse(reader.GetAttribute("length"));
reader.ReadStartElement("Data");
var buffer = new byte[length];
reader.ReadContentAsBase64(buffer, 0, length);
using (var stream = new MemoryStream(buffer))
{
this.Value = Formatter.Deserialize(stream);
}
reader.ReadEndElement();
reader.ReadEndElement();
}
public void WriteXml(XmlWriter writer)
{
writer.WriteStartElement("Data");
using (var stream = new MemoryStream())
{
Formatter.Serialize(stream, this.Value);
var buffer = stream.ToArray();
writer.WriteAttributeString("length", buffer.Length.ToString());
writer.WriteBase64(buffer, 0, buffer.Length);
}
writer.WriteEndElement();
}
}

Serialize/Deserilize array of objects/structs

I have read many codes on this but none happened to solve the problem. first the code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;
namespace Serialization
{
class Program
{
static void Main(string[] args)
{
using (MoveSaver objSaver = new MoveSaver(#"C:\1.bin"))
{
MoveAndTime mv1, mv2;
mv1.MoveStruc = "1";
mv1.timeHLd = DateTime.Now;
objSaver.SaveToFile(mv1);
mv2.MoveStruc = "2";
mv2.timeHLd = DateTime.Now;
objSaver.SaveToFile(mv2);
}
using (MoveSaver svrObj = new MoveSaver())
{
MoveAndTime[] MVTobjs = svrObj.DeSerializeObject(#"C:\1.bin");
foreach (MoveAndTime item in MVTobjs)
{
//Do Something
}
}
}
}
public class MoveSaver:IDisposable
{
public void Dispose()
{
if (fs != null)
{
fs.Close();
}
}
FileStream fs;
StreamWriter sw;
public string filename { get; set; }
public MoveSaver(string FileName)
{
this.filename = FileName;
fs = new FileStream(filename, FileMode.OpenOrCreate, FileAccess.ReadWrite);
}
public MoveSaver()
{
}
~MoveSaver()
{
if (fs != null)
{
fs.Close();
}
}
public MoveAndTime[] DeSerializeObject(string filename)
{
MoveAndTime[] objectToSerialize;
Stream stream = File.Open(filename, FileMode.Open);
BinaryFormatter bFormatter = new BinaryFormatter();
objectToSerialize = (MoveAndTime[])bFormatter.Deserialize(stream);
stream.Close();
return objectToSerialize;
}
public bool SaveToFile(MoveAndTime moveTime)
{
try
{
BinaryFormatter bformatter = new BinaryFormatter();
bformatter.Serialize(fs, moveTime);
return true;
}
catch (Exception)
{
return false;
}
}
}
[Serializable]
public struct MoveAndTime
{
public string MoveStruc;
public DateTime timeHLd;
}
}
The code mimics a need for saving all actions of user on the program. to be later shown on that program (say you play cards or so and you want to review :D what has happened). The problem is when DeSerializeObject function is called the line objectToSerialize = (MoveAndTime[])bFormatter.Deserialize(stream); throws an exception (definitely in runtime) that the cast from a single object to array is not valid:
Unable to cast object of type
'Serialization.MoveAndTime' to type
'Serialization.MoveAndTime[]'.
Any idea? Any improvement or total change of approach is appreciated.
You're saving a single MoveAndTime instance to the file, but you're trying to read an array of them.
Please modify your main block like this. I think it achieves what you want.
static void Main(string[] args)
{
using (MoveSaver objSaver = new MoveSaver(#"C:\1.bin"))
{
MoveAndTime[] MVobjects = new MoveAndTime[2];
MoveAndTime mv1, mv2;
mv2 = new MoveAndTime();
mv1 = new MoveAndTime();
mv1.MoveStruc = "1";
mv1.timeHLd = DateTime.Now;
mv2.MoveStruc = "2";
mv2.timeHLd = DateTime.Now;
MVobjects[0] = new MoveAndTime();
MVobjects[0] = mv1;
MVobjects[1] = new MoveAndTime();
MVobjects[1] = mv2;
objSaver.SaveToFile(MVobjects);
}
using (MoveSaver svrObj = new MoveSaver())
{
MoveAndTime[] MVTobjs = svrObj.DeSerializeObject(#"C:\1.bin");
foreach (MoveAndTime item in MVTobjs)
{
//Do Something
Console.WriteLine(item.MoveStruc);
Console.WriteLine(item.timeHLd);
}
}
}
Thanks

Transmitting complex objects using TCP

I have a client server application in which I need to transmit a user defined object from Client to Server using TCP connection. My object is of the following structure:
class Conversation
{
private string convName, convOwner;
public ArrayList convUsers;
public string getConvName()
{
return this.convName;
}
public string getConvOwner()
{
return this.convOwner;
}
}
Please help me how to transmit this object at from client and again de-serialize it into appropriate object at server side.
As answered, you should make your object serializable. Once you did that with the Serializable attribute, you can use the famous BinaryFormatter to convert your object into a byte array.
You can find many examples out there for using the BinaryFormatter, just use your favorite search engine. Here's a short example:
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;
public class SerializationUtils
{
public static byte[] SerializeToByteArray(object request)
{
byte[] result;
BinaryFormatter serializer = new BinaryFormatter();
using (MemoryStream memStream = new MemoryStream())
{
serializer.Serialize(memStream, request);
result = memStream.GetBuffer();
}
return result;
}
public static T DeserializeFromByteArray<T>(byte[] buffer)
{
BinaryFormatter deserializer = new BinaryFormatter();
using (MemoryStream memStream = new MemoryStream(buffer))
{
object newobj = deserializer.Deserialize(memStream);
return (T)newobj;
}
}
}
As for your class, it includes two private fields. I can't see where you set values for them, so I changed your code a bit, so that they can be set in the constructor. In addition, I added the needed Serializable attribute:
using System;
using System.Collections;
[Serializable]
public class Conversation
{
public Conversation(string convName, string convOwner)
{
this.convName = convName;
this.convOwner = convOwner;
}
public Conversation()
{
}
private string convName, convOwner;
public ArrayList convUsers;
public string getConvName()
{
return this.convName;
}
public string getConvOwner()
{
return this.convOwner;
}
}
Now let's put it all together, and see your class serialized and then deserialized, in a Console Application:
using System;
using System.Collections;
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;
namespace Capishi
{
[Serializable]
public class Conversation
{
public Conversation(string convName, string convOwner)
{
this.convName = convName;
this.convOwner = convOwner;
}
public Conversation()
{
}
private string convName, convOwner;
public ArrayList convUsers;
public string getConvName()
{
return this.convName;
}
public string getConvOwner()
{
return this.convOwner;
}
}
public class SerializationUtils
{
public static byte[] SerializeToByteArray(object request)
{
byte[] result;
BinaryFormatter serializer = new BinaryFormatter();
using (MemoryStream memStream = new MemoryStream())
{
serializer.Serialize(memStream, request);
result = memStream.GetBuffer();
}
return result;
}
public static T DeserializeFromByteArray<T>(byte[] buffer)
{
BinaryFormatter deserializer = new BinaryFormatter();
using (MemoryStream memStream = new MemoryStream(buffer))
{
object newobj = deserializer.Deserialize(memStream);
return (T)newobj;
}
}
}
class Program
{
static void Main(string[] args)
{
// create and initialize a conversation object
var convName = "Capishi";
var convOwner = "Ice Cream";
Conversation myConversation = new Conversation(convName, convOwner);
myConversation.convUsers = new ArrayList();
myConversation.convUsers.Add("Ron Klein");
myConversation.convUsers.Add("Rakesh K");
// serialize to a byte array
byte[] data = SerializationUtils.SerializeToByteArray(myConversation);
// print the resulting byte array if you want
// PrintArray(data);
// deserialize the object (on the other side of the communication
Conversation otherConversation = SerializationUtils.DeserializeFromByteArray<Conversation>(data);
// let's see if all of the members are really there
Console.WriteLine("*** start output ***");
Console.WriteLine("otherConversation.getConvName() = " + otherConversation.getConvName());
Console.WriteLine("otherConversation.getConvOwner() = " + otherConversation.getConvOwner());
Console.WriteLine("otherConversation.convUsers:");
foreach (object item in otherConversation.convUsers)
{
Console.WriteLine(item);
}
Console.WriteLine("*** done output ***");
// wait before close
Console.ReadLine();
}
/// <summary>
/// just a helper function to dump an array to the console's output
/// </summary>
/// <param name="data"></param>
private static void PrintArray(byte[] data)
{
for (int i = 0; i < data.Length; i++)
{
Console.Write("{0:000}", data[i]);
if (i < data.Length - 1)
Console.Write(", ");
}
Console.WriteLine();
}
}
}
The result is:
*** start output ***
otherConversation.getConvName() = Capishi
otherConversation.getConvOwner() = Ice Cream
otherConversation.convUsers:
Ron Klein
Rakesh K
*** done output ***
And a final note:
I'd use the generic List instead of the outdated ArrayList, unless you're bound to .NET 1.*.
One good course of action is to expose this object as a DataContract to a framework like WCF, and use the appropriate transports available in that framework.
For example:
[DataContract]
class Conversation
{
private string convName, convOwner;
public ArrayList convUsers;
[DataMember]
public string ConvName
{
get { return this.convName; }
}
[DataMember]
public string ConvOwner
{
get { return this.convOwner; }
}
}
You need to make your object serializable.

c# serialized data

I have been using BinaryFormatter to serialise data to disk but it doesn't seem very scalable. I've created a 200Mb data file but am unable to read it back in (End of Stream encountered before parsing was completed). It tries for about 30 minutes to deserialise and then gives up. This is on a fairly decent quad-cpu box with 8Gb RAM.
I'm serialising a fairly large complicated structure.
htCacheItems is a Hashtable of CacheItems. Each CacheItem has several simple members (strings + ints etc) and also contains a Hashtable and a custom implementation of a linked list. The sub-hashtable points to CacheItemValue structures which is currently a simple DTO which contains a key and a value. The linked list items are also equally simple.
The data file that fails contains about 400,000 CacheItemValues.
Smaller datasets work well (though takes longer than i'd expect to deserialize and use a hell of a lot of memory).
public virtual bool Save(String sBinaryFile)
{
bool bSuccess = false;
FileStream fs = new FileStream(sBinaryFile, FileMode.Create);
try
{
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(fs, htCacheItems);
bSuccess = true;
}
catch (Exception e)
{
bSuccess = false;
}
finally
{
fs.Close();
}
return bSuccess;
}
public virtual bool Load(String sBinaryFile)
{
bool bSuccess = false;
FileStream fs = null;
GZipStream gzfs = null;
try
{
fs = new FileStream(sBinaryFile, FileMode.OpenOrCreate);
if (sBinaryFile.EndsWith("gz"))
{
gzfs = new GZipStream(fs, CompressionMode.Decompress);
}
//add the event handler
ResolveEventHandler resolveEventHandler = new ResolveEventHandler(AssemblyResolveEventHandler);
AppDomain.CurrentDomain.AssemblyResolve += resolveEventHandler;
BinaryFormatter formatter = new BinaryFormatter();
htCacheItems = (Hashtable)formatter.Deserialize(gzfs != null ? (Stream)gzfs : (Stream)fs);
//remove the event handler
AppDomain.CurrentDomain.AssemblyResolve -= resolveEventHandler;
bSuccess = true;
}
catch (Exception e)
{
Logger.Write(new ExceptionLogEntry("Failed to populate cache from file " + sBinaryFile + ". Message is " + e.Message));
bSuccess = false;
}
finally
{
if (fs != null)
{
fs.Close();
}
if (gzfs != null)
{
gzfs.Close();
}
}
return bSuccess;
}
The resolveEventHandler is just a work around because i'm serialising the data in one application and loading it in another (http://social.msdn.microsoft.com/Forums/en-US/netfxbcl/thread/e5f0c371-b900-41d8-9a5b-1052739f2521)
The question is, how can I improve this? Is data serialisation always going to be inefficient, am i better off writing my own routines?
I would personally try to avoid the need for the assembly-resolve; that has a certain smell about it. If you must use BinaryFormatter, then I'd simply put the DTOs into a separate library (dll) that can be used in both applications.
If you don't want to share the dll, then IMO you shouldn't be using BinaryFormatter - you should be using a contract-based serializer, such as XmlSerializer or DataContractSerializer, or one of the "protocol buffers" implementations (and to repeat Jon's disclaimer: I wrote one of the others).
200MB does seem pretty big, but I wouldn't have expected it to fail. One possible cause here is the object tracking it does for the references; but even then, this surprises me.
I'd love to see a simplified object model to see if it is a "fit" for any of the above.
Here's an example that attempts to mirror your setup from the description using protobuf-net. Oddly enough there seems to be a glitch working with the linked-list, which I'll investigate; but the rest seems to work:
using System;
using System.Collections.Generic;
using System.IO;
using ProtoBuf;
[ProtoContract]
class CacheItem
{
[ProtoMember(1)]
public int Id { get; set; }
[ProtoMember(2)]
public int AnotherNumber { get; set; }
private readonly Dictionary<string, CacheItemValue> data
= new Dictionary<string,CacheItemValue>();
[ProtoMember(3)]
public Dictionary<string, CacheItemValue> Data { get { return data; } }
//[ProtoMember(4)] // commented out while I investigate...
public ListNode Nodes { get; set; }
}
[ProtoContract]
class ListNode // I'd probably expose this as a simple list, though
{
[ProtoMember(1)]
public double Head { get; set; }
[ProtoMember(2)]
public ListNode Tail { get; set; }
}
[ProtoContract]
class CacheItemValue
{
[ProtoMember(1)]
public string Key { get; set; }
[ProtoMember(2)]
public float Value { get; set; }
}
static class Program
{
static void Main()
{
// invent 400k CacheItemValue records
Dictionary<string, CacheItem> htCacheItems = new Dictionary<string, CacheItem>();
Random rand = new Random(123456);
for (int i = 0; i < 400; i++)
{
string key;
CacheItem ci = new CacheItem {
Id = rand.Next(10000),
AnotherNumber = rand.Next(10000)
};
while (htCacheItems.ContainsKey(key = rand.NextString())) {}
htCacheItems.Add(key, ci);
for (int j = 0; j < 1000; j++)
{
while (ci.Data.ContainsKey(key = rand.NextString())) { }
ci.Data.Add(key,
new CacheItemValue {
Key = key,
Value = (float)rand.NextDouble()
});
int tail = rand.Next(1, 50);
ListNode node = null;
while (tail-- > 0)
{
node = new ListNode
{
Tail = node,
Head = rand.NextDouble()
};
}
ci.Nodes = node;
}
}
Console.WriteLine(GetChecksum(htCacheItems));
using (Stream outfile = File.Create("raw.bin"))
{
Serializer.Serialize(outfile, htCacheItems);
}
htCacheItems = null;
using (Stream inFile = File.OpenRead("raw.bin"))
{
htCacheItems = Serializer.Deserialize<Dictionary<string, CacheItem>>(inFile);
}
Console.WriteLine(GetChecksum(htCacheItems));
}
static int GetChecksum(Dictionary<string, CacheItem> data)
{
int chk = data.Count;
foreach (var item in data)
{
chk += item.Key.GetHashCode()
+ item.Value.AnotherNumber + item.Value.Id;
foreach (var subItem in item.Value.Data.Values)
{
chk += subItem.Key.GetHashCode()
+ subItem.Value.GetHashCode();
}
}
return chk;
}
static string NextString(this Random random)
{
const string alphabet = "abcdefghijklmnopqrstuvwxyz0123456789 ";
int len = random.Next(4, 10);
char[] buffer = new char[len];
for (int i = 0; i < len; i++)
{
buffer[i] = alphabet[random.Next(0, alphabet.Length)];
}
return new string(buffer);
}
}
Serialization is tricky, particularly when you want to have some degree of flexibility when it comes to versioning.
Usually there's a trade-off between portability and flexibility of what you can serialize. For example, you might want to use Protocol Buffers (disclaimer: I wrote one of the C# ports) as a pretty efficient solution with good portability and versioning - but then you'll need to translate whatever your natural data structure is into something supported by Protocol Buffers.
Having said that, I'm surprised that binary serialization is failing here - at least in that particular way. Can you get it to fail with a large file with a very, very simple piece of serialization code? (No resolution handlers, no compression etc.)
Something that could help is cascade serializing.
You call mainHashtable.serialize(), which return a XML string for example. This method call everyItemInYourHashtable.serialize(), and so on.
You do the same with a static method in every class, called 'unserialize(String xml)', which unserialize your objetcs and return an object, or a list of objects.
You get the point ?
Of course, you need to implement this method in every of your class you want to be serializable.
Take a look at ISerializable interface, which represent exaclty what I'm describing. IMO, this interface looks too "Microsoft" (no use of DOM, etc), so i created mine, but principle is the same : cascade.

Categories

Resources