Develop Reference

Store bigger object data using without using a database

I'm writing a project most for fun, just to play around with .NET. I'm building an xaml project (or what you call the windows 8 apps) in C#.
In this case I will have a bigger object with some lists of other abojects and stuff. Is there any smart way of saving these object to disk for loading them later? That is something like GetMyOldSavesObectWithName("MyObject");
What i've read that the local storage is primary used to save smaller thing, such as settings. How much data is acceptale to save? Can it handle a bigger object, and what are the pros/cons? Should i save the objects to files, and how do i in that case do that? Is there any smart way of telling .net to "Save this object to MyObjectName.xml" or something?

In C#, you can use serialization to save your objects as files, commonly in XML format. Other libraries, such as JSON.net, can serialize into JSON.
You could also roll out your own saving/loading format, which will probably run faster and store data in a more compact way, but will take much more time on your part. This can be done with BinaryReaders and Writers.
Take a look at this StackOverflow answer if you wish to go the serialization route.
In most cases data will be so compact it will not use much space at all. Based on your comment, that "large" amount of data would really only take a few KBs.

C# : Serialize objects to XML without reflection

In an application, we can save the current state of the application and it's configuration(which can be huge). We are using the XmlSerializer.
We now have only what we need in the XML(all XmlIgnore are in place), and it's VERY slow to store the whole configuration(file of ~50-100MB).
We NEED to keep storing this configuration as XML, but we would like to avoid :
The reflection, which is to slow
To implement the IXmlSerializable interface
The idea was to have a method to implement in each object, in which we can register which fields/property we want to serialize, then having a SerializationManager which is able to read what we want to serialize, and then write them.
Like this, objects doesn't know the language (XML) in which they will be rendered, and if one day we want a binary serialization(or if we want to have the possibility to serialize in different format), we can.
But we don't want to reinvent the wheel, and I don't know if some library exists or if something like Linq to xml can help, or if this is natively possible, ...
So how do you think I can achieve this?

"The reflection, which is to slow"
Except, it doesn't use reflection at runtime. It performs metaprogramming on the first run (assuming you are using new XmlSerializer(type)) to inspect the type and generate static code that will work on the given type. Therefore, any volume-related performance issue is not related to reflection. There is a chance that the metaprogramming itself can take a measurable time, but a: this is unlikely unless your model is really complicated, and b: it can be avoided by using the sgen.exe tool to pre-generate the serialization assembly.
Any performance issue, therefore, is most likely due to the size of the model and the overhead of xml.
If you want to try a different serializer, consider something like protobuf-net. You won't be able to read the data (it will not be xml), but the output will be much smaller and faster.

As you mentioned
In an application, we can save the current state of the application and it's configuration
State, especially when it is big (100Mb is ... huge!), required its own way to serialize data. Many of us knows and hates that slow saving/loading game saves from past. Even now, game developers distinguish quicksave from ordinal saves. It is optimized to occurs faster (to example, by caching part of recently performed quicksave) than ordinal save.
First question is why XML? BinarySerializer is faster, but for sizes like this you better use manual serialization (as Marc Gravell suggested, use protobuf, it's ultimate superior to anything).
Second question is, do you really need serialize data (change their format)? The fastest way of saving state is to dump memory. Imagine you have all your data saved in one block of memory, then dumping this block into a file is a very quick save. You may (I am not sure, but it should be doable) construct your data in a way, what overriding this memory will be kind of loading game. This much faster of any conversion.
If you go with dumping, then consider to pack it (into zip). Packing and saving 10 mb should be faster than saving unpacked 100 mb (assuming, you are not using too slow or too good packing algorithm), memory operations and cpu are much faster than SSD.
To save configuration, you can still serialize it as usual. If you want it to be a single file, then define own format of this file, to example:
config_stream, separator["<<<>>>>"], memory block [100 Mb]
Serialize with XmlSerializer into memory, create file, save config, separator, dump.

Saving map data in a 2d ORPG

I'm trying to figure out how I can best save the map data for a 2d ORPG engine I am developing, the file would contain tile data (Is it blocked, what actual graphics would it use, and various other properties).
I am currently using a binary format but I think this might be a bit too limited and hard to debug, what alternatives are there, I was thinking about perhaps JSON or XML but I don't know if there are any other better options.
It has to work with C++ and C# and preferably also with Python.

Personally, I would stick with a binary format. Whatever method you choose, it's going to be a pain in the ass to edit by hand anyway, so you may as well stick to binary which gives you a size and speed advantage.
You're also going to want a map editor anyway so that you do not have to edit it by hand.

XML is well supported across basically every language. It may become verbose for large maps, however, depending on how you encode the map data in XML.
JSON might not be a good choice, simply because I don't think it supports multiline strings, which would be helpful (although not really necessary)
YAML is another alternative, though it's not as well-known.
You could just stick to binary - most maps would be a pain to edit by hand, no matter what format you pick (though I've heard of Starcraft maps being edited with hex editors...) Just use whatever seems easiest for you.
Additionally, check out the Tiled map editor (http://www.mapeditor.org/), which lets you edit maps (with custom tile properties, I think) and save it in an XML based format, including optional GZip for compression.

Lua is also a possibility which can be used as a config file with tables. It's been a while since I worked with Python but doesn't it also support a AJAX style data structure? You could simply use Python files if you are already using it.

serialize in .NET, deserialize in C++

I have a .NET application which serializes an object in binary format.
this object is a struct consisting of a few fields.
I must deserialize and use this object in a C++ application.
I have no idea if there are any serialization libraries for C++, a google search hasn't turned up much.
What is the quickest way to accomplish this?
Thanks in advance.
Roey.
Update :
I have serialized using Protobuf-net , in my .NET application, with relative ease.
I also get the .proto file that protobuf-net generated, using GetProto() command.
In the .proto file, my GUID fields get a type of "bcl.guid", but C++ protoc.exe compiler does not know how to interpret them!
What do I do with this?

If you are using BinaryFormatter, then it will be virtually impossible. Don't go there...
Protocol buffers is designed to be portable, cross platform and version-tolerant (so it won't explode when you add new fields etc). Google provide the C++ version, and there are several C# versions freely available (including my own) - see here for the full list.
Small, fast, easy.
Note that the v1 of protobuf-net won't handle structs directly (you'll need a DTO class), but v2 (very soon) does have tested struct support.

Can you edit the .NET app? If so why not use XML Serialization to output the data in a easy to import format?

Both boost and Google have libraries for serialization. However, if your struct is pretty trivial, you might consider managing the serialization yourself by writing bytes out from C# and then reading the data in C++ with fread.

Agree with others. You are making your app very vulnerable by doing this. Consider the situation if one of the classes you're serializing is changed in any way or built on a later version of the C# compiler: Your serialized classes could potentially change causing them to be unreadable.
An XML based solution might work well. Have you considered SOAP? A little out of fashion now but worth a look. The main issue is to decouple the implementation from the data. You can do this in binary if speed / efficiency is an issue, although in my experience, it rarely is.

Serializing in a binary format and expecting an application in another language to read the binary is a very brittle solution (ie it will tend to break on the smallest change to anything).
It would be more stable to serialize the data in a common standard format.

Do you have the option of changing the format? If so, consider choosing a non-binary format for greater interoperability. There are plenty of libraries for reading and writing XML. Json is popular as well.
Binary formats are efficient, but vulnerable to implementation details (does your C++ compiler pack data structures? how are ints and floats represented? what byte ordering is used?), and difficult to adjust if mangled. Text based formats are verbose, but tend to be much more robust. If you are uncertain about binary representations, text representations tend to be easier to understand (apart from challenges such as code pages and wide/narrow characters...).
For C++ XML libraries, the most capable (and perhaps also most complex) would still seem to be the Xerces library. But you should decide for yourself which library best fits your needs and skills.

Use XML Serialization its the best way to go, in fact is the cleanest way to go.
XmlSerializer s = new XmlSerializer( typeof( YourClassType ) );
TextWriter w = new StreamWriter( #"c:\list.xml" );
s.Serialize( w, yourClassListCollection );
w.Close();

What is the best way to implement precomputed data?

I have a computation that calculates a resulting percentage based on certain input. But these calculations can take quite some time, which can be annoying. Since there are about 12500 possible inputs, I thought it would be a good idea to precompute all the data, and look this up during normal program execution.
My first idea was to just create a simple file which is read at program initialization and populates some arrays. Although this will work, I would like to know if there are some other options? For example that the array is populated during compile time.
BTW, I'm writing my code in C#.

This tutorial here implements a serializer, which you can use to easily convert an object to a binary file and back. Once you have the serializer in hand, you can just create an object that holds all your data and serialize it; when you actually run your program, just deserialize the object and use it.
This has all the benefits of saving an object to the hard drive, with an implementation that is object-agnostic (meaning you don't have to write much code for any object you want to serialize) and outputs in binary (thus saving space, if that is a concern).

A file with data is probably the easiest and most flexible way to implement it.
If you wanted it in memory without having to read it from somewhere, I would write a program to output your data in C#-like CSV format suitable for copying and pasting into an array/collection initializer, and thereby generate the source code for your precomputed data.

Create a program that outputs valid C# code which initializes your lookup tables. Make this part of your build process so that it will automatically create the source file and then build the rest of your project.

As Daniel Lew said, serialize it into a binary file.
If you need speed, go for a Dictionary. A Dictionary is indexed on it's key, and should allow rapid lookup even with large amounts of data.

I would always start by considering if there was any way to avoid precomputing. If there's 12500 possible inputs, how many are required per user request ? Will all 12500 be needed at the same time or will they be spread out in time ? If you can get by with calculating a few at a time, I'd do that with lazy initialization. I prefer this solution simply because I'll have fewer issues with it in the long run. What do you do when the persistent format changes, or the data changes. How will you handle it when the file is missing or corrupted ? Persisting to a file does not create less code.
I would serialize such a file to a human-readable format if I had to persist a pre-loaded version. I'd probably use xml serialization since it's simple. But quite often there's issues of invalidation and recalculation. Do the values never change or only very infrequently ?

I agree with mquander and Trent. Use your favorite language or script to generate the whole C# file you need to define your data (no copy-pasting, that's a manual step and error-prone). Add it as a Pre-Build event in Visual Studio. You could even detect that you have an up-to-date file and avoid regeneration for most builds.
There is definitely a way to statically generate almost any data using template metaprogramming in C++, although it can be painful. It's not worth it unless you need many sets of different data in several parts of your program. I am not familiar enough with metaprogrammation in C# to evaluate the general effort in your case. You should look into that.