Develop Reference

Generic transform/map source JSON to different destination JSON's

I built a dynamic Survey type application where users can set up their template of questions/attributes that need to be captured. A template can have multiple sections, where each section contains multiple questions/attributes.
We write out the data in JSON according to the template, to be consumed or sent by webhooks, for external systems. Rather than developing a layer for each external integration to deserialize our system's object and transform it for the external system's consumption, I was thinking of writing dynamic mapper that uses a configuration schema(maybe JSON) that will map your source to your destination. Which will be great, as soon as either the source or destination changes, the configuration can just be adjusted accordingly, without having to change code and re-publish.
I'm using this system as an example, but I'm seeing multiple applications for different use cases, with the main one being a source API that needs to integrate with multiple external API's.
Source JSON example:
{
"Section1": {
"S1Attribute1": "S1Attribute1Answer",
"S1Attribute2": "S1Attribute2Answer",
"S1Attribute3": "S1Attribute3Answer"
},
"Section2": {
"S2Attribute1": "S2Attribute1Answer",
"S2Attribute2": "S2Attribute2Answer"
}
}
Destination 1 Example:
{
"SectionsFlattened": {
"S1Attribute1": "S1Attribute1Answer",
"S1Attribute2": "S1Attribute2Answer",
"S1Attribute3": "S1Attribute3Answer",
"S2Attribute1": "S2Attribute1Answer",
"S2Attribute2": "S2Attribute2Answer"
}
}
Destination 2 Example:
{
"NewSection1Name": {
"NewS1Attribute1Name": "S1Attribute1Answer",
"NewS1Attribute2Name": "S1Attribute2Answer",
"NewS1Attribute3Name": "S1Attribute3Answer",
"NewSection2Name": {
"NewS2Attribute1Name": "S2Attribute1Answer",
"NewS2Attribute2Name": "S2Attribute2Answer"
}
}
}
This is pretty simple examples of just moving properties around, but the possibilities are endless of how one would need to transform it.
It feels like a problem that somebody would have investigated or solved maybe, but in all my research I don't seem to find anything concrete, or I'm struggling to find a good starting point - maybe I'm approaching the problem incorrectly. Any suggestions/guidance/articles/libraries would be appreciated.

Loading level data from xml file in XNA

I am creating a tile-based game in XNA and will be loading the level information from xml files. I have no problem with loading xml data but I would like feedback on the approach I'm thinking of using for reading the xml file.
The code has a class hierarchy such that:
A Level class contains:
- a collection of TileLayers
- a collection of Tilesets
A TileLayer class contains
- a collection of Tiles
A Tileset class contains:
- a collection of TilsetTiles
A TilesetTile class contains:
- a TileId
- a collection of TileProperties
- a TileRectangle
Each of the above classes requires some information from the xml level file.
When I load a level I would like to simply call Level.Load();
My intention is that each class will have a Load() method that will:
1. Load any specific info it needs from the xml file.
2. Call Load() on its children.
The problem I see is that the code for processing the xml will be scattered around in different files making changes difficult to implement (for instance if I decide to encrypt the xml file), and no doubt breaks several aspects of the SOLID principles.
I have this idea that I could create an xmlLevelReader class whose sole purpose is to read an xml level file.
This class could then expose methods that can be called from the Load() method in each of the classes described above.
For example the TileLayer class Load() method could call xmlLevelReader.GetTiles() which would return an IEnumerable<Tile>
Do you think this approach will work?
Can you foresee any difficulties?
Is this approach too simplistic/complicated?
Any constructive criticism welcomed!
Thanks

Based on your comment, I see that you are using Tiled Map Editor. This lead me to suggest that you use TiledLib. Here is a brief explanation of how you can get up and running with importing your .tmx files for use in game.
Content Pipeline Overview
File -> Content Import -> Content Process -> Content Write -> .xnb -> ContentRead -> Game Object
TiledLib
TiledLib only handles the ContentImporter part of the above diagram. It will essentially handle reading the .tmx XML and allow you to process the data into whatever objects you need at run time. Fortunately, the TiledLib author has provided a Demos section in the download as well.
Basic Tiled Map Processor Demo
BasicDemo main game project which contains the ContentManager.Load call.
BasicDemoContent project which has the .tmx file from Tiled
BasicDemoContentPipeline project which has the ContentProcessor
TiledLib which has the ContentImporter
You really only need to worry about how the ContentProcessor works because TiledLib handles all the importing for you. Although I do suggest looking through the different classes to understand how it is deserializing the XML (for educational purposes).
The example ContentProcessor in the Basic Demo project takes in a MapContent object from the ContentImporter (TiledLib) and outputs a DemoMapContent object which is serialized to .xnb at build time and deserialized to a Map object at run time. Here are the classes that represent the map after being processed completely:
[ContentSerializerRuntimeType("BasicDemo.Map, BasicDemo")]
public class DemoMapContent
{
public int TileWidth;
public int TileHeight;
public List<DemoMapLayerContent> Layers = new List<DemoMapLayerContent>();
}
[ContentSerializerRuntimeType("BasicDemo.Layer, BasicDemo")]
public class DemoMapLayerContent
{
public int Width;
public int Height;
public DemoMapTileContent[] Tiles;
}
[ContentSerializerRuntimeType("BasicDemo.Tile, BasicDemo")]
public class DemoMapTileContent
{
public ExternalReference<Texture2DContent> Texture;
public Rectangle SourceRectangle;
public SpriteEffects SpriteEffects;
}
A Map contains a tile width, tile height, and a list of MapLayers.
A MapLayer contains a width, a height, and a list of Tiles.
A MapTile contains a texture, a source rectangle (proper rectangle in the tileset), and optional sprite effects (I've never used any).
How It's Made
I suggest reading the comments of the ContentProcessor to understand what is happening, but in brief, this is the basics:
Load texture data for tile set
Get source rectangles for each tile from within the texture
Iterate over all layers of the map
For each layer, iterate over all tiles
For each tile, get the proper texture and source rectangle
Assign all data from the input to the output properly
Caveats
As long as you stick to basic types (vague, I know), you also do not need to worry about ContentWriter and ContentReader parts of the content pipeline. It will automatically serialize and deserialize the .xnb files at build and run time, respectively. See a question I asked about this: here.
Also, you'll notice that if you're using object layers from within Tiled, the demo does not show you how to process those. TiledLib does properly import them, you just need to pull the data out and stick it in your proper classes. I'll try to edit this answer later with an example of how to do that.

If you are just wanting to load in XML without manipulating the data at all, you can just use the built in XNA Content Serializer.
Essentially you define a class which maps to your xml format, and then read the XML into an instance of that class.
For example. Here I define the class I want to load into:
SpriteRenderDefinition.cs
I chose this one because it has nested classes like the case you describe. Note that it goes into the ContentDefinitions project of you XNA solution.
Now here is the xml file that fills in the content of a SpriteRenderDefinition:
Sprite.xml
The format of that XML maps directly to the member names of SpriteRenderDefinition.
And finally, the code to actually load that XML data into an actual Object at runtime is very straight forward:
SpriteRenderDefinition def = GameObjectManager.pInstance.pContentManager.Load<SpriteRenderDefinition>(fileName);
After calling that line, you have a SpriteRenderDefintion object populated with all the content of the XML file! That is all the code I wrote. Everything else is built into XNA. It's really quite slick and useful if you take an hour or so to figure it out!

Serialization - Viewing the Object Graph from a Stream

I'm wondering if there's a way in which I can create a tree/view of a serialised object graph, and whether anyone has any pointers? EDIT The aim being that should we encounter a de-serialization problem for some reason, that we can actually view/produce a report on the serialized data to help us identify the cause of the problem before having to debug the code. Additionally I want to extend this in the future to take two streams (version 1, version 2) and highlight differences between the two of them to help ensure that we don't accidently remove interesting information during code changes. /EDIT
Traditionally we've used Soap or XML serialization, but these are becoming too restricted for our needs, and Binary serialization would generally do all that we need. The reason that this hasn't been adopted, is because it's much harder to view the serialized contents to help fix upgrade issues etc.
So I've started looking into trying to create a view on the serialized information. I can do this from an ISerializable constructor to a certain extent :
public A(SerializationInfo info, StreamingContext context)
{}
Given the serialization info I can reflect the m_data member and see the actual serialized contents. The problem with this approach is
It will only display a branch from the tree, I want to display the entire tree from the root and it's not really possible to do from this position.
It's not a convenient place to interrogate the information, I'd like to pass a stream to a class and do the work there.
I've seen the ObjectManager class but this works on an existing object graph, whereas I need to be able to work from the stream of data. I've looked through the BinaryFormatted which uses an ObjectReader and a __BinaryParser, hooking into the ObjectManager (which I think will then have the entire contents, just maybe in a flat list), but to replicate this or invoke it all via reflection (2 of those 3 classes are internal) seems like quite a lot of work, so I'm wondering if there's a better approach.

You could put a List<Child class> in every parent class (Even if there the same)
and when you create a child you immediately place it in that list or better yet declare it whilst adding it the list
For instance
ListName.Add(new Child(Constructer args));
Using this you would serialize them as one file which contains the hierarchy of the objects and the objects themselves.
If the parent and child classes are the same there is no reason why you cannot have dynamic and multi leveled hierarchy.

In order to achieve what you describe you would have to deserialize whole object graph from stream without knowing a type from which it was serialized. But this is not possible, because serializer doesn't store such information.
AFAIK it works in a following way. Suppose you have a couple of types:
class A { bool p1 }
class B { string p1; string p2; A p3}
// instantiate them:
var b = new B { p1 = "ppp1", p2 = "ppp2", p3 = new A { p1 = true} };
When serializer is writing this object, it starts walking object graph in some particular order (I assume in alphabetic order) and write object type and then it's contents. So your binary stream will like this:
[B:[string:ppp1][string:ppp2][A:[bool:true]]]
You see, here there are only values and their types. But order is implicit - like it is written.
So, if you change your object B, to suppose
class B { A p1; string p3; string p3;}
Serialzer will fail, because it will try to assing instance of string (which was serialized first) to pointer to A. You may try to reverse engineer how binary serialization works, then you may be able to create a dynamic tree of serialized objects. But this will require considerable effort.
For this purpose I would create class similar to this:
class Node
{
public string NodeType;
public List<Node> Children;
public object NodeValue;
}
Then while you will be reading from stream, you can create those nodes, and recreate whole serialized tree and analyze it.

Db4O activation depth, Faq, Best Practise for Web Application

Our database includes 4,000,000 records (sql server) and it's physical size is 550 MB .
Entities in database are related each other as graph style. When i load an entity from db with 5 level depth there is a problem (all records are loaded).
Is there any mechanism like Entity Framework( Include("MyProperty.ItsProperty"))
What is the best Types for using with db4O databases?
Is there any issue for Guid, Generic Collections?
Is there any best practise for WebApplication with db4o? Session Containers+EmbeddedDb4ODb or Client/ServerDb4O?
Thx for help..
Thx for good explanation. But i want to give my exact problem as a sample:
I have three entities: (N-N relationship. B is an intersection Entity. Concept:Graph)
class A
{
public B[] BList;
public int Number;
public R R;
}
class B
{
public A A;
public C C;
public D D;
public int Number;
}
class C
{
public B[] BList;
public E E;
public F F;
public int Number;
}
I want to query dbContext.A.Include("BList.C.BList.A").Include("BList.C.E.G").Where(....)
I want to get :A.BList.C.BList.A.R
But I dont want to get :A.R
I want to get :A.BList.C.E.G
But I dont want to get :A.BList.C.F
I want to get :A.BList.C.E.G
But I dont want get :A.BList.D
Note:this requirements can change a query to another query
Extra question is there any possibility to load
A.BList[#Number<120].C.BList.A[#Number>100] Super syntax :)

Activation: As you said db4o uses it's activation-mechanism to control which objects are loaded. To prevent that to many objects are loaded there are different strategies.
Lower the global default activation-depth: configuration.Common.ActivationDepth = 2 Then use the strategies below to activate objects on need.
Use class-specific activation configuration like cascading activation, minimum and maximun activation-depth etc.
Activate objects explicit on demand: container.Activate(theObject,5)
However all these stuff is rather painful on complex object graphs. The only strategy to get away from that pain is transparent activation. Create an attribute like TransparentlyActivated. Use this attribute to mark your stored classes. Then use the db4otool to enhance your classes. Add the db4otool-command to the Post-Build events in Visual Studio: Like 'PathTo\Db4oTool.exe -ta -debug -by-attribute:YourNamespace.TransparentlyActivated $(TargetPath)
Guid, Generic Collections:
No (in Version 7.12 or 8.0). However if you store your own structs: Those are handled very poorly by db4o
WebApplication: I recommend an embedded-container, and then a session-container for each request.
Update for extended question part
To your case. For such complex activation schema I would use transparent activation.
I assume you are using properties and not public fields in your real scenario, otherwise transparent persistence doesn't work.
The transparent activation basically loads an object in the moment a method/property is called the first. So when you access the property A.R then A itself it loaded, but not the referenced objects. I just go through a few of you access patterns to show what I mean:
Getting 'A.BList.C.BList.A.R'
A is loaded when you access A.BList. The BList array is filled with unactivate objects
You keep navigating further to BList.C. At this moment the BList object is loaded
Then you access C.BList. db4o loads the C-object
And so on and so forth.
So when you get 'A.BList.C.BList.A.R' then 'A.R' isn't loaded
A unloaded object is represented by an 'empty'-shell object, which has all values set to null or the default value. Arrays are always fully loaded, but first filled with unactivated objects.
Note that theres no real query syntax to do some kind of elaborate load requests. You load your start object and then pull stuff in as you need it.
I also need to mention that this kind of access will perform terrible over the network with db4o.
Yet another hint. If you want to do elaborate work on a graph-structure, you also should take a look at graph databases, like Neo4J or Sones Graph DB

LINQ to XML, ORM or something "Completely Different"?

I'm working on a Silverlight Project with all the features and limitations that entails. This is an update to a previous product. The intent, in order to be quick to market, is to maintain as much of the back-end (webservices, database, etc..) as at all possible. Our mandate it to only touch the back-end if there is no other way. We'll primarily be focused on re-writing the front-end. There's an important industry conference soon where we will want to demo the early look of the product. There may be time prior to the official release to do some re-work, but the back-end may need to wait until V2.
OK, so what I'm trying to do is use the MVVM pattern with data binding for the front-end for which I'm responsible (MVVM pattern is dictated from above). I have a pre-existig web service that serves up some XML. A sample of that XML looks like is below:
<CODEBOOKINDEX>
<ME Words="1" Score="25" Highscore="1">Main Entry Item
<NM>attack</NM>
<NM>cardiac</NM>
<NM>chest</NM>
<NM>effort</NM>
<NM>heart</NM>
<NM>pectoris</NM>
<NM>syndrome</NM>
<NM>vasomotor</NM>
<IE>413.9</IE>
<M1 Words="1" Score="25">An M1 Item (Same as ME, just first level Child)
<IE>557.1</IE>
</M1>
<M1 Words="1" Score="25">Another M1 Item
<IE>443.9</IE>
<M2 Words="1" Score="25">An M2 Item (again same as ME, just a child of an M1 item)
<CF>Arteriosclerosis,extremities</CF>
<IE>440.20</IE>
</M2>
</M1>
</ME></CODEBOOKINDEX>
So, my question, since I want to bind this to a UI using the MVVM pattern, it seems to me that I need to translate this into a custom object. As you can see there are a number of "Entry" items, MainEntry (ME) and Subentries (M1 or M2 in this example), these will all contain certain other nodes (they will all have an IE node, for example), they MAY contain 0 or more other node types (for example they MAY or may not contain one or more NM nodes, or they MAY contain one CF node or not). Whihc means (at least to me) that I can't really bind directly to XML because:
It violates the MVVM pattern (I could probably justify this for the demo, but would have to refactor later).
I can't really bind a UI element to an XML node that MAY not be there for a given item.
In some cases Ihave to translate a collection (a bunch of NM items, for example) into a formated strig for display purposes, which I don't THINK is a trivial thing.
So, I'm trying to understand the best way to translate this XML into a bindable object, which in my mind means transforming this XML into an object for the model and then overlaying a view-model on that model.
Can this be done easily with LINQ to XML queries, or am I really moving into the realm of an ORM such as NHibernate or Entity Framework (no holy wars about WHICH ORM please)?
I've only just established what controls I will be using for UI and I need to demonstrate to my manager rather quickly HOW I'm going to handle the translation.
So, the real questions:
Do I NEED an ORM? I'm not against using them, but I want to keep the size of the XAP file small and want to limit the amount of new tech I (and my teammates) need to learn in a single pass.
If I do need one, can I keep the file size down and can I ramp up quickly with either EF or NHibernatge and have a model to show very soon? I'm talking like a week here to have SOMETHING that will take output from the webservice and turn it into an object, even if the map isn't perfect initially, I need to demonstrate some progress.
Is there another option alltogether that I'm not considering that might be easier, limit the need to modify existing code (i.e. the webservice) and product usable results?

Do I NEED an ORM?
No. You aren't mapping to a relational source, so an object relational mapper won't help.
Get it done with Linq to Xml.
public CustomClass TranslateME(XElement source)
{
CustomClass result = new CustomClass();
result.Words = (int) source.Attribute("Words");
result.Score = (int) source.Attribute("Score");
XAttribute highScore = source.Attribute("HighScore");
result.HighScore = (highScore == null) ? 0 : (int) highScore;
result.NMs = source
.Elements("NM")
.Select(x => x.Value)
.ToList();
result.IE = source
.Element("IE").Value;
result.SubEntries = source
.Elements("M1")
.Select(x => TranslateM1(x))
.ToList();
return result;
}