I have a Java application for Android that will have some static data, which will be loaded at start up. This data is generated by a C# app as a bunch of classes with lists and Dictionaries, something like this:
public class DataContainer
{
public List<A> As = new List<A>();
public Dictionary<int, B> Bs = new Dictionary<int, B>();
// ...
}
public class A
{
public int IdA;
public int IdSomething;
public DateTime dt;
//...
}
public class B
{
public int IdB;
public string Name;
List<C> Cs = new List<C>();
//...
}
public class C
{
//...
}
(The DataContainer class is the root class that contains all the data to be transferred. I then have a similar class structure in the Java project into which I want the data to be imported.)
My issue is that I can't seem to find a combination of a .NET and Java tools/libraries that would be able to write and read the same format.
I've tried JSON with DataContractJsonSerializer and then loading with GSON but that doesn't work with Dictionaries/Maps (works fine with Lists but I need the dictionaries). The .NET exporter uses format like
"Bs":[{"Key":1,"Value":{...}}, ...]
while it looks like GSON uses
{key:value,key:value,...}
I've also looked at some XML but since there isn't a standard formatting all the libraries use something different by default and I really don't want to write all the writing/parsing rules by hand. Also, JSON has much less overhead compared to XML.
So, is there some pair of libraries for .NET export and Java import that can handle nested classes with Lists and Dictionaries? The things I've managed to find were either in the other direction or did just simple one class serialization.
BTW: I'm also using Tuples for Dictionary keys, but if nothing can handle that I'll just use nested Dictionaries, that is not a big deal.
EDIT: Since I can't answer my own question due to low rep here is the solution that worked for me:
So, I've tried a few more combinations and serializing using JSon.NET on the C#/.NET side and then loading using GSON on the Java side works great, except Tuples. For example, using Tuple as a key for dictionary will get serialized as:
{ "(False, 3)" : { ... }, "(True, 1)" : { ... }, ... }
So I have to use String keys in Java for now but that is not an issue. Possible solution is to write a custom class for the Tuple in both .NET and Java.
Example code:
string serializeObject = JsonConvert.SerializeObject(dataContainer, new IsoDateTimeConverter());
File.WriteAllText("test.json", serializeObject);
and Java:
InputStream is = getResources().openRawResource(R.raw.test); // open from /res/raw/test.json
Gson gson = new Gson();
InputStreamReader r = new InputStreamReader(is);
DataContainer dc = gson.fromJson(r, DataContainer.class);
Log.i("jsondebug", dc.Packers.get(0).Something);
You can have a look at Google Protocol Buffers. Protocol buffers are Google's lingua franca for data. Below is an excerpt from Protocol Buffers Developer Guide:
Protocol buffers are a flexible, efficient, automated mechanism for
serializing structured data – think XML, but smaller, faster, and
simpler. You define how you want your data to be structured once, then
you can use special generated source code to easily write and read
your structured data to and from a variety of data streams and using a
variety of languages. You can even update your data structure without
breaking deployed programs that are compiled against the "old" format.
Protocol buffers have many advantages over XML for serializing
structured data. Protocol buffers:
are simpler
are 3 to 10 times smaller
are 20 to 100 times faster
are less ambiguous
generate data access classes that are easier to use programmatically
An XML mapping would work just fine. Serialize as XML and send it over. You'll need marshallers and unmarshallers on either end.
Java has its JAXB standard for binding XML to objects. Perhaps you can manage it if you follow that standard.
I think you need to approach this from a different perspective. Think first about a JSON or XML format that will work well with both the C# and Java side. Think of that as an API or contract between the C# and Android apps. Instead of using libraries that marshall/unmarshall according to their own formats, make the code on each side read/write to the contract you defined.
Alternatively, you might want to look into using a binary format like Apache Thrift or Protocol Buffers.
Related
I have a requirement to take a library of C# classes that implement protobuf-net, and convert them into .proto files, which need to be converted using protoc into .py files. I understand that the .ToProto() function does this just fine, but I came up against an issue involving collections and generics when converting from .proto to .py files. When trying to serialize a list of DateTimes, for example I get the following error X.proto:64:13. "List_TimeSpan" is not defined. As this had not caused an issue upon serialization into a protobuf file, I wasn't aware of this situation at the time.
I am currently using proto-buf.net 2.3.2 for this project; it's the version some of my other work has been done with and I am aware that this could just be solved with a version upgrade. I'm just not sure if that is the answer with the digging I've done so far. If there's something else that I'm missing, I would truly appreciate any help that can be thrown my way.
If we consider:
[ProtoContract]
public class Foo {
[ProtoMember(12)]
public List<DateTime> Times { get; } = new List<DateTime>();
}
then GetProto<T>() in both v2.3.2 (the version mentioned in the question) and v2.4.4 (the current default version) generate:
syntax = "proto2";
import "protobuf-net/bcl.proto"; // schema for protobuf-net's handling of core .NET types
message Foo {
repeated .bcl.DateTime Times = 12;
}
So on the surface of it, it should already be just fine. If you're doing something more exotic (perhaps using a list in a dictionary value?), I'd be happy to help, but I'm going to need more of a clue as to what you're doing. Posting some C# that shows the thing you're seeing would be a great place to start.
Note that when protobuf-net first came around, there was no agreed transmission format for date/time-like values, so protobuf-net made something up, but it turns out to not be a convenient fit for cross-platform work; the following is a hard breaking change (it is not data compatible), but if possible, I would strongly recommend the well-known format that Google added later:
[ProtoContract]
public class Foo {
[ProtoMember(12, DataFormat = DataFormat.WellKnown)]
public List<DateTime> Times { get; } = new List<DateTime>();
}
which generates:
syntax = "proto2";
import "google/protobuf/timestamp.proto";
message Foo {
repeated .google.protobuf.Timestamp Times = 12;
}
I have to communicate with a IBM main frame using IBM WebSphere.
The service on the main frame side can only use flat files.
On my side I want to use CQRS (Command / Query)
In other words I want to serialize command / queries and deserialize query results
I could do it with standard reflection offcourse, but my question is if there is a nicer way of doing it?
Can I make use of dynamics?
Flatfile > ParsedObjectStructured > Dynamic type > static type
This would depend an awful lot on what the format of the flat file is, and how the schema works - is it self-describing, for example? However, it sounds to me like most of the work here would be in understanding the flat-file format (and the schema-binding). From there, the choice of "deserialize into the static type" vs "deserialize into a dynamic type" is kinda moot, and I would say that there is very little point deserializing into a dynamic type just to have to map it all to the static type. Additionally, the static type can (again, depending on the file-format specifics) be a handy place to decorate the types to say "here's how to interpret this", if the file-format needs specification. For example (and I'm totally making this up as I go along - don't expect this to relate to your format):
[Frobber(Offset = 4, Format = DataFormat.LittleEndianInt32)]
public int Id {get;set;}
[Frobber(Offset = 0, Format = DataFormat.LittleEndianInt32)]
public int Index {get;set;}
[Frobber(Offset = 8, Format = DataFormat.FixedAscii, Size = 20)]
public string Name {get;set;}
[Frobber(Offset = 28, Format = DataFormat.Blob)] // implicit Size=16 as Guid
public Guid UniqueKey {get;set;}
where FrobberAttribute is just something you might invent to specify the file format. Of course, if the schema is defined internally to the file, this may not be necessary.
Re reflection: basic reflection will work fine if the data is fairly light usage; but overall, reflection can be quite expensive. If you need it to be optimal, you would probably want the implementation to consider strategy-caching (i.e. only doing the discovery work once) and meta-programming (turning the strategy into ready-baked IL, rather than incurring the overhead of reflection at runtime).
If the file format is a common / popular one, you might find that there are existing tools for reading that format. If not, you can either roll your own, or find some crazy person who enjoys writing serialization and meta-programming tools. Such people do exist...
I have an object which
does not have the Serializable attribute set
Has properties, who's type, does not have the Serializable attribute set
I do not have control over (meaning i cannot edit the class)
I tried reading THIS, it talks about substitution classes to fix this when using Sharpserializer but frankly, I don't understand how to do this when I don't know the properties of my object.
are there some of serialization frameworks that can do this?
Edit: I'm looking into protobuf.net
I cannot figure out how to get it to work in my scenario though - Im hoping Marc will swing by to save the day? :) -
I read this which is the exact same problem as mine, but I'm still getting
"Type is not expected and no contract can be inferred"
when using
private static byte[] ClienToBytes(IScsClient client)
{
using (var memoryStream = new MemoryStream())
{
RuntimeTypeModel.Default.Add(typeof(IScsClient), true).SetSurrogate(typeof(BinaryFormatterSurrogate<IScsClient>));
Serializer.Serialize(memoryStream, client);
return memoryStream.ToArray();
}
}
am I using the RunTimeTypeModel wrong?
I would try protobuf-net. Take a look here:
http://code.google.com/p/protobuf-net/
Quote from Website:
protocol buffers is the name of the binary serialization format used by Google for much of their data communications. It is designed to be:
small in size - efficient data storage (far smaller than xml)
cheap to process - both at the client and server
platform independent - portable between different programming architectures
extensible - to add new data to old messages
I am toying around with serializing and deserializing CSV files and I am wondering if there is an existing library, similar in concept to the XmlSerializer, which can declaratively define objects and (de)serialize them to/from a file or stream. I have looked around a bit but have not found anything focused on serialization. I already have pretty solid code for parsing CSV documents per RFC 4180, but what would be really helpful is the serialization part. What I am not looking for is just a parser, advice to use String.Split(), etc.
Is there an existing project out there, or should I build one?
Bonus etiquette question: if I do end up rolling my own serializer, is it appropriate to answer this question with a link to the codeplex project?
I would highly recommend servicestack.text for this purpose. Avialable on nuget:
Install-package servicestack.text
It suports serialization to many data formats and unlike the built in XmlSerializer, you don't need to decorate all your properties with attributes. Here's an example to serialize to CSV.
using ServiceStack.Text;
...
var csv = CsvSerializer.SerializeToCsv(new[]{
new Dog () {
Bark = "Woof!",
Male = true,
Size = 10
}});
I've used this project (CsvHelper) in the past, and it works similar to the build in .NET serializer classes in the sense that you use attributes to craft the input/output.
There's really no need to roll your own, since there are tons out there. If you do end up rolling your own, feel free to post it. Most users, when answering a question with something they've written themselves (or are affiliated in some way) usually give a disclaimer saying so as a courtesy.
You should take a look into FileHelpers Library.
Some sample code from their site:
using FileHelpers;
// First declare the record class
[DelimitedRecord(",")]
public class SampleType
{
public string Field1;
public int Field2;
}
public void WriteExample()
{
FileHelperEngine engine = new FileHelperEngine(typeof(SampleType));
SampleType[] records = new SampleType[1];
records[0] = new SampleType();
records[0].Field1 = "Hello World";
records[0].Field2 = 12;
engine.WriteFile("destination.txt", records);
// Now the file contains the created record in this format:
//
// Hello World,12
}
I have to send information too a third party in an XML format they have specified, a very common task I'm sure.
I have set of XSD files and, using XSD.exe, I have created a set of types. To generate the XML I map the values from the types within my domain to the 3rd party types:
public ExternalBar Map(InternalFoo foo) {
var bar = new ExternalBar;
bar.GivenName = foo.FirstName;
bar.FamilyName = foo.LastName;
return bar;
}
I will then use the XMLSerializer to generate the files, probably checking them against the XSD before releasing them.
This method is very manual though and I wonder if there is a better way using the Framework or external tools to map the data and create the files.
LINQ to XML works quite well for this... e.g.
XElement results = new XElement("ExternalFoos",
from f in internalFoos
select new XElement("ExternalFoo", new XAttribute[] {
new XAttribute("GivenName", f.FirstName),
new XAttribute("FamilyName", f.LastName) } ));
Firstly, I'm assuming that the object properties in your existing domain map to the 3rd party types without much manipulation, except for the repetitive property assignments.
So I'd recommend just using standard XML serialization of your domain tree (generate an outbound schema for your classes using XSD), then post-processing the result via a set of XSLT stylesheets. Then after post-processing, validate the resulting XML documents against the 3rd party schemas.
It'll probably be more complicated than that, because it really depends on the complexity of the mapping between the object domains, but this is a method that I've used successfully in the past.
As far as GUI tools are concerned I've heard (but not used myself) that Stylus Studio is pretty good for schema-to-schema mappings (screenshot here).