Develop Reference

Problems trying to pre-load objects to a text file for faster load times

I have been working on a Windows Form Control project to import into a 3rd party client software using their supplied SDK. The custom control written by yet another company I am trying to load requires sign on to a server before displaying information, which can take 20-30 seconds. In order to speed things up I had the idea to pre-load information needed by the control to a text file. Since it is not a known type it is throwing errors when trying to serialize the class.
I have a Dictionary I am using to reference back to the proper ICamera class. If I change "cam" from an ICamera type to a string, for example "cam.GetLiveURL()". It writes the text file without issue. This is the code I am using to populate the Dictionary.
foreach (ICamera cam in _adapter.Cameras())
{
OCCamera.Add(cam.GetDisplayName(), cam);
}
I have tried XMLSerializer, and it seems it has difficulty dealing with a Dictionary.
I have tried BinaryFormatter and get the error:
Type 'OCAdapter.OCCamera' in Assembly 'OCAdapter.dll' in not marked as serializable.
I have tried DataContractSerializer and get the error:
Type 'OCAdapter.OCCamera' with data contract name
'OCCamera:http://schemas.datacontract.org/2004/07/OCAdapter' is not
expected. Consider using a DataContractResolver or add ant types not
known statically to the list of known types - for example, by using
the KnownTypeAttribute attribute or by adding the to the list of known
types passed to DataContractSerializer.
I have tried playing around with the DataContractResolver and can not seem to get it to work, I do not understand it at all.
The code I am using for the BinaryFormatter and DataContractSerializer are straight from MSDN or elsewhere, and test fine without the custom type.
Maybe there is a better way to handle all this, and I am missing it. I am not opposed to ditching the Dictionary route for something else, or I can rewrite any amount of other code to make this work.

Mistake 1: trying to serialize your implementation rather than the *data.
Mistake 2: using BinaryFormatter... just about ever (except maybe AppDomain marshalling)
My advice: create a simple model ("DTO" model) that just represents the data you need, but not in terms of your specific implementation (no OCAdapter.OCCamera etc). You can construct this DTO model in whatever way is convenient for whatever serialization library you like. I'm partial to protobuf-net, but many others exist. Then map to/from your DTO model and your implementation model.
Advantages:
it'll work
changes to the implementation don't impact the data; it only impacts the mapping code
you can use just about any serializer you want
you can version the data sensibly

Serialize one object and deSerialize into a list of objects

I have a type which i need to do the following
zip, serialize and write to a file. This could happen multiple times.
I should be able to recreate a list of this type of object i.e. de-serialize and store in a list of collection.
I have tried few solution like given below but they are slow. Need a fast solution using .Net 4.0.
Soln:
Create a zip stream, use BinaryFormatter and then using StreamWriter write a line in the file and close it.
Same reverse way for creating the list. Reading a line unzip and then de-serialize one object at a time.

From what I understand, the BinaryFormatter + StreamWriter combo can become pretty slow and bloated because it adds metadata to the byte array about the object or file, properties, and datatypes.
One option you have, if you are willing to work with a third party library, is Protocol Buffers. According to the site, it is lightweight, fast serialization format that Google uses in their data communications. It's also recommended in this StackOverflow question: Fast and compact object serialization in .NET.
There are two libraries are available for .NET:
By Marc Gravell: https://code.google.com/p/protobuf-net/
By Jonathan Skeet: https://code.google.com/p/protobuf-csharp-port/
Here is a table of results comparing "protobuf-net" (first link) and "proto#" (second link) to other serialization techniques (more tests available here):
Serializer size serialize deserialize
-------------------------------------------------------------
protobuf-net 3 268 1,881
proto# 3 76 1,792
BinaryFormatter 153 6,694 8,420
SoapFormatter 687 28,609 55,125
XmlSerializer 153 14,594 19,819
DataContractSerializer 205 3,263 10,516
DataContractJsonSerializer 26 2,854 15,621
If you would prefer to have a little more control over it, though, (and if you are just serializing objects), then this link from Code Project contains a neat pattern for serializing them: http://www.codeproject.com/Articles/14164/A-Fast-Serialization-Technique
The idea is that you implement the ISerializable interface for whatever class you need to serialize. This forces you to add a the ISerializable.GetObjectData method, which provides a SerializationWriter that you use to write each property individually, which you then add to a SerializationInfo object. The syntax itself for this is actually incredibly straightforward.
Here is a quick, abbreviated, sample of the GetObjectData method from the site:
// Serialize the object. Write each field to the SerializationWriter
// then add this to the SerializationInfo parameter
public void GetObjectData (SerializationInfo info, StreamingContext ctxt) {
SerializationWriter sw = SerializationWriter.GetWriter ();
sw.Write (id1);
sw.Write (id2);
sw.Write (id3);
sw.Write (s1);
sw.Write (s2);
// more properties here
sw.AddToInfo (info);
}
Here are the results of this author's tests:
Formatter Size (bytes) Time (uS)
--------------------------------------------------------------------
Standard serialization Binary 2080 364
Fast serialization Binary 421 74
Fast serialization SOAP 1086 308

Is there such thing as a CSV Serializer? (similar to XmlSerializer)

I am toying around with serializing and deserializing CSV files and I am wondering if there is an existing library, similar in concept to the XmlSerializer, which can declaratively define objects and (de)serialize them to/from a file or stream. I have looked around a bit but have not found anything focused on serialization. I already have pretty solid code for parsing CSV documents per RFC 4180, but what would be really helpful is the serialization part. What I am not looking for is just a parser, advice to use String.Split(), etc.
Is there an existing project out there, or should I build one?
Bonus etiquette question: if I do end up rolling my own serializer, is it appropriate to answer this question with a link to the codeplex project?

I would highly recommend servicestack.text for this purpose. Avialable on nuget:
Install-package servicestack.text
It suports serialization to many data formats and unlike the built in XmlSerializer, you don't need to decorate all your properties with attributes. Here's an example to serialize to CSV.
using ServiceStack.Text;
...
var csv = CsvSerializer.SerializeToCsv(new[]{
new Dog () {
Bark = "Woof!",
Male = true,
Size = 10
}});

I've used this project (CsvHelper) in the past, and it works similar to the build in .NET serializer classes in the sense that you use attributes to craft the input/output.
There's really no need to roll your own, since there are tons out there. If you do end up rolling your own, feel free to post it. Most users, when answering a question with something they've written themselves (or are affiliated in some way) usually give a disclaimer saying so as a courtesy.

You should take a look into FileHelpers Library.
Some sample code from their site:
using FileHelpers;
// First declare the record class
[DelimitedRecord(",")]
public class SampleType
{
public string Field1;
public int Field2;
}
public void WriteExample()
{
FileHelperEngine engine = new FileHelperEngine(typeof(SampleType));
SampleType[] records = new SampleType[1];
records[0] = new SampleType();
records[0].Field1 = "Hello World";
records[0].Field2 = 12;
engine.WriteFile("destination.txt", records);
// Now the file contains the created record in this format:
//
// Hello World,12
}

Transferring data from C# to Java via serialization

I have a Java application for Android that will have some static data, which will be loaded at start up. This data is generated by a C# app as a bunch of classes with lists and Dictionaries, something like this:
public class DataContainer
{
public List<A> As = new List<A>();
public Dictionary<int, B> Bs = new Dictionary<int, B>();
// ...
}
public class A
{
public int IdA;
public int IdSomething;
public DateTime dt;
//...
}
public class B
{
public int IdB;
public string Name;
List<C> Cs = new List<C>();
//...
}
public class C
{
//...
}
(The DataContainer class is the root class that contains all the data to be transferred. I then have a similar class structure in the Java project into which I want the data to be imported.)
My issue is that I can't seem to find a combination of a .NET and Java tools/libraries that would be able to write and read the same format.
I've tried JSON with DataContractJsonSerializer and then loading with GSON but that doesn't work with Dictionaries/Maps (works fine with Lists but I need the dictionaries). The .NET exporter uses format like
"Bs":[{"Key":1,"Value":{...}}, ...]
while it looks like GSON uses
{key:value,key:value,...}
I've also looked at some XML but since there isn't a standard formatting all the libraries use something different by default and I really don't want to write all the writing/parsing rules by hand. Also, JSON has much less overhead compared to XML.
So, is there some pair of libraries for .NET export and Java import that can handle nested classes with Lists and Dictionaries? The things I've managed to find were either in the other direction or did just simple one class serialization.
BTW: I'm also using Tuples for Dictionary keys, but if nothing can handle that I'll just use nested Dictionaries, that is not a big deal.
EDIT: Since I can't answer my own question due to low rep here is the solution that worked for me:
So, I've tried a few more combinations and serializing using JSon.NET on the C#/.NET side and then loading using GSON on the Java side works great, except Tuples. For example, using Tuple as a key for dictionary will get serialized as:
{ "(False, 3)" : { ... }, "(True, 1)" : { ... }, ... }
So I have to use String keys in Java for now but that is not an issue. Possible solution is to write a custom class for the Tuple in both .NET and Java.
Example code:
string serializeObject = JsonConvert.SerializeObject(dataContainer, new IsoDateTimeConverter());
File.WriteAllText("test.json", serializeObject);
and Java:
InputStream is = getResources().openRawResource(R.raw.test); // open from /res/raw/test.json
Gson gson = new Gson();
InputStreamReader r = new InputStreamReader(is);
DataContainer dc = gson.fromJson(r, DataContainer.class);
Log.i("jsondebug", dc.Packers.get(0).Something);

You can have a look at Google Protocol Buffers. Protocol buffers are Google's lingua franca for data. Below is an excerpt from Protocol Buffers Developer Guide:
Protocol buffers are a flexible, efficient, automated mechanism for
serializing structured data – think XML, but smaller, faster, and
simpler. You define how you want your data to be structured once, then
you can use special generated source code to easily write and read
your structured data to and from a variety of data streams and using a
variety of languages. You can even update your data structure without
breaking deployed programs that are compiled against the "old" format.
Protocol buffers have many advantages over XML for serializing
structured data. Protocol buffers:
are simpler
are 3 to 10 times smaller
are 20 to 100 times faster
are less ambiguous
generate data access classes that are easier to use programmatically

An XML mapping would work just fine. Serialize as XML and send it over. You'll need marshallers and unmarshallers on either end.
Java has its JAXB standard for binding XML to objects. Perhaps you can manage it if you follow that standard.

I think you need to approach this from a different perspective. Think first about a JSON or XML format that will work well with both the C# and Java side. Think of that as an API or contract between the C# and Android apps. Instead of using libraries that marshall/unmarshall according to their own formats, make the code on each side read/write to the contract you defined.
Alternatively, you might want to look into using a binary format like Apache Thrift or Protocol Buffers.

The best way to store class instances to a file/database

What is the best way to store instances of a class to file/database?
We have a base class called Command and loads of derived classes.
Users create instances of these classes by adding commands to a graphical designer
where they can configure them. (Set the properties).
We then need a way to store these "commands" to a file without losing
any information.
One idea was to use db4o, but the GPL license is not acceptable for this project.
Any suggestions or code samples?
Update:
(In order to "de-blurryfie" my question :p)
The generated code might look something like:
command[i++] = new DelayInSecondsCommand(2);
command[i++] = new DaliRequestCommand(1, false, 254);
command[i++] = new DaliRequestCommand(2, false, 254);
command[i++] = new DaliRequestCommand(3, false, 254);
command[i++] = new WaitInSecondsCommand(2);
command[i++] = new DaliRequestCommand(1, false, 0);
command[i++] = new DaliRequestCommand(2, false, 0);
command[i++] = new DaliRequestCommand(3, false, 0);
command[i++] = new JumpCommand(0);
But then with loads of different commands.
I know it's possible with .NET serialization, altough I've never used it before,
but I was wondering if there are better alternatives, like I said db4o seems nice but the license doesn't fit the project.
Update 2:
Thank you for the replies. I'll probably go with the serialization solution now,
but I'll look into the other options as well. F.Y.I. data is stored in a SQL Compact database.

Are you trying to save the data in tables? or as blob/clob data? Since you mention files, I assume the latter: any of the standard .NET serializers should be fine - they all support inheritance etc. I'd consider for DataContractSerializer, as this combines the field-level support (like BinaryFormatter), and the assembly-independence of XmlSerializer.
You could also consider more esoteric things like protobuf-net.
So: what is it you need to do that won't work under the standard serializers?

serialization does the trick! Serialization is nothing more than converting an object or a connected graph of objects into a stream of bytes (in order to persist the current state of the object). This can be a binary stream, XML or whatever. You don't have to do this conversion by your own since .Net has great support for serialization. Once you serialized an object, you are free to store this data to a file or database. Likewise, a stream of bytes representing a serialized object can be deserialized into an object which will have the same state as the original one.
Btw: Once you have a serialized stream of bytes, you can apply some more functions on it, e.g. compression or encryption.

Pretty blurry question, why don't you just use .NET's built-in serialization possibilities (e.g. XmlSerializer).

db40 also provides a commerical license but it has been recently bought by versant so maybe you may want to look at that. This sort of database is known as object orientated database and is a way of creating persistant instances of classes which is very different to relational databases that work using tables.
This (wikpedia.org) is a good read on object orienated databases and this (also wikipedia) is a list of some of the available options.
In my opinion object databases are much better & more powerfull than relational and I will only use relational databases like mysql if I really have to (not very often).
I would recommomend you watch these videos and download the trial.

Serialization is a great way to store this type of data. See http://blog.paranoidferret.com/index.php/2008/06/20/csharp-tutorial-xml-serialization/

There is http://www.neodatis.org. Its LGPL, but the time I used there was only a implementation for Java. Now, there's a "beta" release for C#, but I didn't tested.