I'm trying to figure out how I can best save the map data for a 2d ORPG engine I am developing, the file would contain tile data (Is it blocked, what actual graphics would it use, and various other properties).
I am currently using a binary format but I think this might be a bit too limited and hard to debug, what alternatives are there, I was thinking about perhaps JSON or XML but I don't know if there are any other better options.
It has to work with C++ and C# and preferably also with Python.
Personally, I would stick with a binary format. Whatever method you choose, it's going to be a pain in the ass to edit by hand anyway, so you may as well stick to binary which gives you a size and speed advantage.
You're also going to want a map editor anyway so that you do not have to edit it by hand.
XML is well supported across basically every language. It may become verbose for large maps, however, depending on how you encode the map data in XML.
JSON might not be a good choice, simply because I don't think it supports multiline strings, which would be helpful (although not really necessary)
YAML is another alternative, though it's not as well-known.
You could just stick to binary - most maps would be a pain to edit by hand, no matter what format you pick (though I've heard of Starcraft maps being edited with hex editors...) Just use whatever seems easiest for you.
Additionally, check out the Tiled map editor (http://www.mapeditor.org/), which lets you edit maps (with custom tile properties, I think) and save it in an XML based format, including optional GZip for compression.
Lua is also a possibility which can be used as a config file with tables. It's been a while since I worked with Python but doesn't it also support a AJAX style data structure? You could simply use Python files if you are already using it.
Related
I'm writing a project most for fun, just to play around with .NET. I'm building an xaml project (or what you call the windows 8 apps) in C#.
In this case I will have a bigger object with some lists of other abojects and stuff. Is there any smart way of saving these object to disk for loading them later? That is something like GetMyOldSavesObectWithName("MyObject");
What i've read that the local storage is primary used to save smaller thing, such as settings. How much data is acceptale to save? Can it handle a bigger object, and what are the pros/cons? Should i save the objects to files, and how do i in that case do that? Is there any smart way of telling .net to "Save this object to MyObjectName.xml" or something?
In C#, you can use serialization to save your objects as files, commonly in XML format. Other libraries, such as JSON.net, can serialize into JSON.
You could also roll out your own saving/loading format, which will probably run faster and store data in a more compact way, but will take much more time on your part. This can be done with BinaryReaders and Writers.
Take a look at this StackOverflow answer if you wish to go the serialization route.
In most cases data will be so compact it will not use much space at all. Based on your comment, that "large" amount of data would really only take a few KBs.
I have a bunch of python objects with fields containing arrays of various dimensions and data types (ints or floats) and I want to write this to a file which I can read into C# objects elsewhere. I'm probably just going to write an XML file, but I thought there might be a quicker/easier way saving and reading it. Also, the resulting XML file will be rather large, which I would like to avoid if it is not too much hassle.
Is there a tried and tested file format that is compatible (and simple to use) with both languages?
What you are trying to do is called serialization. JSON is an excellent option for doing this with support in both languages.
Because you are working with floats etc.. I would consider looking at a format like BSON - "BSON is a binary format in which zero or more key/value pairs are stored as a single entity." It allows you to specify types, sizes etc...
http://bsonspec.org/#/specification
There are libraries for python, C# etc....
There are a heap of other compact easier to use than xml formats like bson out there. I only suggested this particular one as it was the first I remembered.
I am making a combinations generator. For small amounts of elements it's not a problem for the computer that the data is getting stored in the RAM memory instead in a file. But when the number of elements gets bigger, my computer runs out of memory (the exception OutOfMemoryException occurs). The combinations are numbers stored in Lists, that are currently getting stored in an another List.
But this only the first step- the generator work right. I want the data to be stored in file, from where a different program will be able to extract the combinations it needs. Mostly, I need to store the data in a separate file, because the generator will have to be able to create more and more and bigger combinations in the future. The computer will have to read certain parts of the data, without coping all of it in the temporary memory, because this is impossible.
I don't want to turn the data into text and when needed to convert the text back into data. I think this will make things slower because of the conversions. I want the lists to be stored into a custom made file, from where the program can directly extract the data without any converting.
There is a ton of options available I will briefly describe a few.
Use a database. From your description this does not look like a good choices but it will be the most flexible to all clients relativity fast and efficient storage.
Use one of the .net serializers from your description binary serializer will be your best choice. The serlizers offer a lot of advantages relativity fast and baked into .net with built in support and very easy to use.
Use a custom binary format. This will be the fastest option especially if you combine it with a memory mapped file. However binary formats can be hard to use and easy to screw up.
If you really want to store your data in file, you can use BinaryFormatter class. It is probably the most efficient way of serializing data objects into binary stream.
But I wouldn't recommend you generating combinations in this way if only don't you need to store them at one time and load long time after that. It's better to use lazy-generation of combinations. One by one, completely generated without the need to "generate bigger combinations in future" (generate "the biggest" needed combinations one by one - you might want to change your generation algorithm a bit - there are plenty answers how to do that already)
There's a good write up on how to serialize a List<> to a file at http://www.switchonthecode.com/tutorials/csharp-tutorial-serialize-objects-to-a-file
You can use something as a persistent data structure instead, this will reduce the amount of memory needed by your app, without changint too much the current code.
Have a look at this question:
Looking for a simple standalone persistent dictionary implementation in C#
there is a lot of resources doing that, in particular this answer seems to point to some really interesting links:
I have a computation that calculates a resulting percentage based on certain input. But these calculations can take quite some time, which can be annoying. Since there are about 12500 possible inputs, I thought it would be a good idea to precompute all the data, and look this up during normal program execution.
My first idea was to just create a simple file which is read at program initialization and populates some arrays. Although this will work, I would like to know if there are some other options? For example that the array is populated during compile time.
BTW, I'm writing my code in C#.
This tutorial here implements a serializer, which you can use to easily convert an object to a binary file and back. Once you have the serializer in hand, you can just create an object that holds all your data and serialize it; when you actually run your program, just deserialize the object and use it.
This has all the benefits of saving an object to the hard drive, with an implementation that is object-agnostic (meaning you don't have to write much code for any object you want to serialize) and outputs in binary (thus saving space, if that is a concern).
A file with data is probably the easiest and most flexible way to implement it.
If you wanted it in memory without having to read it from somewhere, I would write a program to output your data in C#-like CSV format suitable for copying and pasting into an array/collection initializer, and thereby generate the source code for your precomputed data.
Create a program that outputs valid C# code which initializes your lookup tables. Make this part of your build process so that it will automatically create the source file and then build the rest of your project.
As Daniel Lew said, serialize it into a binary file.
If you need speed, go for a Dictionary. A Dictionary is indexed on it's key, and should allow rapid lookup even with large amounts of data.
I would always start by considering if there was any way to avoid precomputing. If there's 12500 possible inputs, how many are required per user request ? Will all 12500 be needed at the same time or will they be spread out in time ? If you can get by with calculating a few at a time, I'd do that with lazy initialization. I prefer this solution simply because I'll have fewer issues with it in the long run. What do you do when the persistent format changes, or the data changes. How will you handle it when the file is missing or corrupted ? Persisting to a file does not create less code.
I would serialize such a file to a human-readable format if I had to persist a pre-loaded version. I'd probably use xml serialization since it's simple. But quite often there's issues of invalidation and recalculation. Do the values never change or only very infrequently ?
I agree with mquander and Trent. Use your favorite language or script to generate the whole C# file you need to define your data (no copy-pasting, that's a manual step and error-prone). Add it as a Pre-Build event in Visual Studio. You could even detect that you have an up-to-date file and avoid regeneration for most builds.
There is definitely a way to statically generate almost any data using template metaprogramming in C++, although it can be painful. It's not worth it unless you need many sets of different data in several parts of your program. I am not familiar enough with metaprogrammation in C# to evaluate the general effort in your case. You should look into that.
I'm working on Silverlight application that needs to display complex 2d vector graphics.
It downloads zipped XAML file from the server, parses it (XamlRead) and injects to the layout root on the page.
This works fine for fairly small xaml files. The problems is that I need to make it work with much bigger file (lots more content in it). For example one of my uncompressed xaml files is 20 MB large and XamlRead method takes tool long to parse it. My question is if is there a way to do all the parsing on the server side. It would best to just store serialized binary output of XamlRead method as BLOB in the database. However when I try to serialize it, I'm getting a message that "Canvas object is not marked as serializable". I will really appreciate any advices .
Silverlight doesn't have much binary serialization built in; however, protobuf-net works on Silverlight and may help plug this gap. In the current build you can only really serialize types you control (due to adding attributes) - however, I'm in the middle of a big refactor to (among other things) add support for serializing types without attributes.
I expect it to be about 2 more weeks before this is available as a (hopefully) stable build, but you're welcome to take a look at it then.
Note that you will still need to give it some help (telling it what you want it to serialize), but it may be useful.
In particular, the data format ("protocol buffers") is designed to be both dense and efficient to process, which should increase the parse speed. See here for more (numbers are from main .NET, not Silverlight)
I've found the SharpSerializer package very easy to use for fast binary serlization in Silverlight: http://www.sharpserializer.com/en/index.html. You do not need to use the Serializable attribute -- however it only serializes public members.
If parsing is really the problem, it might help to use pre-compiled XAMLs called 'BAML'. This is a binary representation of the XAML file. Since the binary format has a much much cheaper parser instead of the too generic XML, this helps a lot. BAML is also used internally by the .NET compiler to generate more compact files.
For more information, see also http://stuff.seans.com/2008/07/13/hello-wpf-world-part-2-why-xaml/