I am facing a problem of inconsistency after deserialization using protobuf-net.
The class I would like to serialize/deserialize looks like:
[ProtoContract]
public class TSS
{
[ProtoMember(1, AsReference = true)]
public EventManager eventManager { get; private set; }
[ProtoMember(2)]
public DateTime referenceDateTime { get; private set; }
[ProtoMember(3)]
public Mode mode;
}
And inside EventManager class, it looks like:
[ProtoContract]
public class EventManager
{
[ProtoMember(1)]
public InputQueue inputQueue = new InputQueue();
[ProtoMember(2)]
public InputQueue InputQueue
{
get { return this.inputQueue; }
set { this.inputQueue = value; }
}
[ProtoMember(7, AsReference = true)]
public TSS tss;
}
The tss in class EventManager is a reference of TSS object, and eventManager in class TSS is a reference of EventManager object. This is the reason I put AsReference = true there (is this the right way?)
I do serialization like:
public void StateSaving(int time, TSS tss)
{
Stream memoryStream = new MemoryStream();
Serializer.Serialize(memoryStream, tss);
states.Add(time, memoryStream);
}
and do deserialization like:
public void Deserialize(int time, ref TSS tss)
{
Stream memoryStream = states[time];
memoryStream.Position = 0;
tss = Serializer.Deserialize<TSS>(memoryStream);
}
The problem is that whenever I do deserialization, the data structures like inputQueue in the EventManager is populated with NULL values instead of actual values at that point. I am a newbie to protobuf-net, so please point out any mistakes (I believe there are a lot).
Thanks in advance!!
(from comments)
I have located the problem, basically there's a list that needs to be deserialized. And this list is a list of events, in which the constructors of the events have parameters, and when it tries to deserialize, the program will run the parameterless constructors (I manually added these constructors in order to eliminate the exceptions) instead of the right ones (with parameters). I know this is how serialization/deserialization work, but is there a way I can serialize and deserialize this list correctly?
Ah, indeed. There are various approaches when it comes to object construction:
use the parameterless constructor
look for a constructor which matches all the defined members
skip the constructor completely
use a custom object factory
use a surrogate object and custom conversion
Things like XmlSerializer use the first; things like DataContractSerializer and BinaryFormatter use the 3rd; the good news is that protobuf-net supports all 5. I suggest that in your case the best option is to use the 3rd option for this type, which you can do by:
[ProtoContract(SkipConstructor=true)]
Related
I'm using protobuf-net in an application that does a lot of binary serialization of objects from (for all intents and purposes) 3rd party dlls. As a result, I can't use the [Proto-] attributes on the contracts themselves, and I'm instead using the RuntimeTypeModel to prepare the serializer at runtime as it encounters new types. Example serializer wrapper:
public static class ProtobufSerializer
{
public static byte[] Serialize<T>(T obj)
{
PrepareSerializer(typeof(T));
ProtoBuf.Serialize(memoryStream, obj);
}
public static T Deserialize<T>(byte[] bytes)
{
PrepareSerializer(typeof(T));
ProtoBuf.Serialize(memoryStream, obj);
}
}
Where we can safely assume that PrepareSerializer is capable of preparing RuntimeTypeModel to serialize any given type. I'm having some issues with the deserialization of objects where I have to leverage DynamicType=true though. For instance, given the following interface:
public interface IFoo
{
string Name {get;}
}
And the implementation:
public class Foo : IFoo
{
public string Name {get;set;}
public Bar Bar {get;set;}
[OnDeserializing]
public void OnDeserializing(Type t)
{
PrepareSerializer(typeof(Foo));
}
}
public class Bar
{
public int Baz {get;set;}
}
The PrepareSerializer method essentially would use a surrogate and generate a model roughly equivalent to:
// registered surrogate for IFoo
[ProtoContract]
public class IFooSurrogate
{
[ProtoMember(1, DynamicType=true)]
public object Value
[OnSerializing]
public void OnSerializing(Type t)
{
PrepareSerializer(this.Value.GetType());
}
}
Where Value is set by the implicit converters to equal the instance of IFoo. This works fine during serialize (where the event is fired and gives me a chance to prepare the serializer for the specific interface implementation type). It would also work fine in a non-distributed system where I would have to run through a serialize method before ever trying to deserialize that type. During deserialization in a distributed system though, where the current node has never seen Foo before, the ProtoBuf.Serializer throws an InvalidOperationException complaining about a lack of serializer for type Bar before it runs the Foo.OnDeserializing event (giving me a chance to tell it how to deserialize Bar).
Is there any way to attach a hook to ensure my code is given a chance to know about 'Foo' before protobuf-net complains about a lack of serializers?
I haven't tried exactly this situation, but: in order to allow some flexibility, all Type storage and rehydration goes via the TypeModel.DynamicTypeFormatting event; so, you could in theory hook this event on RuntimeTypeModel.Default, with something like:
RuntimeTypeModel.DynamicTypeFormatting += (sender, args) => {
if (args.FormattedName != null) { // meaning: rehydrating
lock(SomeSyncLock) {
if(NotYetKnown(args.FormattedName))
Prepare(args.FormattedName);
}
}
};
The intent of this API is to allow you to control how types are resolved, but... I guess it would work for this too?
I can, however, get behind the idea of an event that is more deliberately targeted at the first time a new Type is seen, essentially replacing / supplementing the "apply default behaviour" code. I don't think it exists today, though.
I'm trying to come up with a workaround that would accommodate an abstract base class's constructor being initialized when a client consuming my WCF service performs a new() over a DataContract object. I'm aware that the DataContract objects are created as raw, uninitialized objects thus no constructors are called. I ran across the user of the [OnSerializing], [OnSerialized], [OnDeserializing], and [OnDeserialized] attributes, and I've discovered that they are not honored by the serialization engine of WCF unless you explicitly force it to use XML, which is not desired in this specific case. Here's a very simplified coding example of what I'm trying to use.
[DataContract(Namespace = "http://somenamespace/Data/ContractBase/v1")]
public abstract ContractBase
{
[DataMember(IsRequired = true)]
public SomeDataContract BaseClassObject { get; set; }
public string Name { get; set; }
public ContractBase()
{
BaseClassObject = new SomeDataContract("randomConstructorArgument");
Name = "Ezra";
}
}
[DataContract(Namespace = "http://somenamespace/Data/TheClass/v1")]
[KnownType(typeof(ContractBase))]
public sealed class TheClass : ContractBase
{
[DataMember]
public PetDataContract MyPet { get; set; }
[DataMember]
public int SomeIntProperty { get; set; }
public TheClass()
: base()
{
MyPet = new PetDataContract ("Fido");
SomeIntProperty = -1;
}
}
I'm aware that the client performing TheClass myClass = new TheClass(); will not initialize the base constructor since the constructor of TheClass is never called. I attempted to add in methods such as the following to trigger when serialization occurs without any success.
private void Initialize()
{
MyPet = new PetDataContract ("Fido");
SomeIntProperty = -1;
base.Initialize();
}
[OnSerializing]
private void OnSerializing(StreamingContext c)
{
Initialize();
}
The base class would have the Initialize method as well so that the "constructors" would be chained. The constructors themselves would be updated to include the Initialize(); call to use the same common source of code.
Is there a way to handle this without forcing the serialization to be done through the XmlSerializer? My current workaround is to provide a method in the WCF service to create the object on the server and return the post-constructor version.
public TheClass CreateTheClass(TheClass contract)
{
// Calls the constructor of TheClass and its base constructor.
return new TheClass();
}
This does work as expected, but it's an extra service call that I'd rather avoid because of the network I/O cost. Any help would be extremely appreciated.
Thanks!
According to this article the attributes you mentioned should work nicely with DataContractSerializer. Your last example is a little bit strange - you are trying to use OnSerializing attribute while saying that constructors are not called during deserialization by WCF.
I would suggest to use your approach with Initialize methods marked by OnDeserializing (or OnDeserialized if you wish to call your code after deserialization was completed) attribute.
When I have a class with no default constructor, i.e. using dependency injection to pass its dependencies, can Newtonsoft.Json create such an object?
For example:
public class SomeFoo
{
private readonly IFooDependency _dependency;
public SomeFoo(IFooDependency dependency){
if(dependency == null)
throw new ArgumentNullException("dependency");
_dependency = dependency;
}
public string Data { get; set; }
public int MoreData { get; set; }
public void DoFoo(){
Data = _dependency.GetFooData();
MoreData = _dependency.GetMoreFooDate();
}
}
During serialization, I only care of storing Data and MoreData (and the type of the object, but let's don't complicate things for the moment). Now, to deserialize, can I call something like
var obj = JsonConvert.DeserializeObject<SomeFoo>(jsonText);
How can I let JsonConvert know about my DI container?
(Note: A work-around would be to always have default constructors in my classes, and call Service Locator in there to get any dependencies I need. I'm just looking for some more clean solution without poluting my classes with such constructors).
You shouldn't let JsonConvert know anything about your DI container. The problems you're experiencing are caused by a flaw in the design of your application. The flaw here is that you mix data and behavior.
If you separate the data from the behavior your problem (and many other problems) will simply go away. You can do this by creating two classes: one for the data, and one for the behavior:
public class SomeFoo
{
public string Data { get; set; }
public int MoreData { get; set; }
}
public class SomeFooHandler
{
private readonly IFooDependency _dependency;
public SomeFooHandler(IFooDependency dependency) {
_dependency = dependency;
}
public void Handle(SomeFoo foo) {
foo.Data = _dependency.GetFooData();
foo.MoreData = _dependency.GetMoreFooDate();
}
}
Since now data and behavior are separated, SomeFoo can be serialized without any problem and SomeFooHandler can simply be injected. SomeFoo has becomes a Parameter Object.
I agree with the separation of concerns posted by Steven, and the answer Mark Seemann has posted here. However, if you still want to go this way, here is a solution that may help:
Inherit a CustomCreationConverter<T>:
internal class NinjectCustomConverter<T> : CustomCreationConverter<T> where T : class
{
private readonly IResolutionRoot _serviceLocator;
public NinjectCustomConverter(IResolutionRoot serviceLocator)
{
_serviceLocator = serviceLocator;
}
public override T Create(Type objectType)
{
return _serviceLocator.Get(objectType) as T;
}
}
Then make sure you retrieve this converter instance via your DI container as well. The code below will deserialize and perform DI on your object:
var ninjectConverter = kernel.Get<NinjectCustomConverter<SerializedObject>>();
var settings = new JsonSerializerSettings();
settings.Converters.Add(ninjectConverter);
var instance = JsonConvert.DeserializeObject<SerializedObject>(json, settings);
Here is a complete working example.
If your objective is to use the injected dependency to modify the data, then you can create a custom Converter.
With this, you should be able to inject your dependency. Similar to the code below:
var settings = new JsonSerializerSettings
{
Converters = { new FooConverter<T>(injectedDependency) }
};
return JsonConvert.DeserializeObject<Dto>(json, settings);
There're many samples of how to create a custom Converters, so you can refer to them.
Assume you have two classes, one inherits the other and the child needs to be serialized / deserialized with XmlSerializer. However, the parent contains a member that is not serializeable, say a dictionary.
public class Parent {
public Dictionary<string, int> dictionary;
}
The parent class is a library used for many other scripts. It cannot be modified. Now the child class contains only serializable members:
public class Child : Parent {
[XmlElement]
public int foo;
}
When trying to call the serializer, I receive an error saying that the dictionary is not serializable. When trying to serialize in JSON, I managed to get away by the price of a warning. I just created another member with the same name and type, and used ScriptIgnore:
public class Child : Parent {
public int foo;
[ScriptIgnore]
public Dictionary<string, int> dictionary;
}
I tried the same trick again here (by using XmlIgnore) but that didn't work on very well, the error was the same. The only way I managed to go through this is create separate classes that will only serve xml de/serializing and then copy the values back into the appropriate place.
Does anyone know a better way around this? Can I make XmlSerializer forget about the parent dictionary in any way?
The very first thing I would say, and always say: if serializing an existing model gets tricky - even remotely awkward, then stop doing that. Take 2 minutes to create a separate DTO model, i.e. a model created solely for the purposes of serialization (and indeed, perhaps even tailored to a specific serializer). Now you put the exact right types, right members, right attributes, and right layout. All you need to do is add some conversion methods - static conversion operators work great here. So what I would say is: create a ParentDto and ChildDto (your names may vary); it'll take 3 minutes, and it'll work great.
Now, back to the question...
XmlSerializer looks at the declaring class for input; for both attributes and conditional serialization, no: we can't add those into the type model at this point. But there is another option - you can use XmlAttributeOverrides to pretend that there was an [XmlIgnore] on the dictionary member. However, some important caveats:
the XmlAttributeOverrides API is a bit of a faff to use (see MSDN for an example)
it is critical that you only do this once, and then store and re-use the XmlSerializer that you create this way; basically, if you don't do this, it will create a new dynamic assembly every time you new a serializer, and assemblies never unload, so you will haemorrhage memory; note that the simple usage (new XmlSerializer(someType) etc) has an inbuilt cache for this; but the XmlAttributeOverrides usage does not
But again, all this messing with XmlAttributeOverrides is a lot more work than just creating a basic DTO
Example of using XmlAttributeOverrides:
using System;
using System.Collections.Generic;
using System.Xml.Serialization;
public class Parent {
public Dictionary<string, int> WantToIgnoreThis { get; set; }
}
public class Child : Parent {
public int Foo { get; set; }
}
static class Program
{
static readonly XmlSerializer customSerializer;
static Program()
{
var xao = new XmlAttributeOverrides();
xao.Add(typeof(Parent), "WantToIgnoreThis", new XmlAttributes {
XmlIgnore = true
});
customSerializer = new XmlSerializer(typeof(Child), xao);
}
static void Main()
{
//var ser = new XmlSerializer(typeof(Child));
// ^^ this would fail
customSerializer.Serialize(Console.Out, new Child {
Foo = 123
});
}
}
Note in particular how the static field is used to cache the serializer.
You could implement IXmlSerializable yourself and handle the specifics in ReadXml(XmlReader reader) and WriteXml(XmlWriter writer). The XmlSerializer will call those methods if your class implements them instead of generating its own serializer.
public class Child : Parent, IXmlSerializable
{
public int Foo { get; set; }
public Dictionary<string, int> Dictionary { get; set; }
public void WriteXml(XmlWriter writer)
{
writer.WriteStartElement("Foo");
writer.WriteValue(this.Foo);
writer.WriteEndElement();
}
void ReadXml(XmlReader reader)
{
var wasEmpty = reader.IsEmptyElement;
reader.Read();
if (wasEmpty)
{
return;
}
reader.ReadStartElement("Foo");
this.Foo = reader.ReadContentAsInt();
reader.ReadEndElement();
}
}
I have developed an application that is meant to send data from client to server and back etc. using serialized objects.
For this application, I decided that protobuf-net would be a good option (especially as it handles variable-length objects so well).
However, when sending an object from client to server or vica-versa, all I know is that the object will be some child class of 'ReplicableObject'. Hence, I am using:
Serializer.SerializeWithLengthPrefix(stream, ro, PrefixStyle.Base128);
Where 'ro' is an object of a type that subclasses from ReplicableObject.
However, I get this exception:
An unhandled exception of type 'ProtoBuf.ProtoException' occurred in
protobuf-net.dll
Additional information: Unexpected type found during serialization;
types must be included with ProtoIncludeAttribute; found MessageObject
passed as ReplicableObject
In this particular instance, I'm trying to send a MessageObject.
As there is precious little documentation for protobuf-net, I am stuck on what to do. I've tried a few attributes here and there to no avail.
Any help appreciated.
Edit: I should make it clear that the subclasses might not even be ones that I've written.
Protobuf is a contract-based serialization format, designed to be platform independent. As such, no type metadata is included on the wire as it would not apply between platforms. Even inheritance is not part of the core protobuf spec.
protobuf-net as a specific implementation introduces support for inheritance (via some smoke and mirrors), but ideally it should still be possible to define the expected types in advance - exactly the same as other serializers such as XmlSerializer or DataContractSerializer. This can be done by using [ProtoInclude(...)] to specify the anticipated concrete types.
If you genuinely can't tell the actual types in advance, there is also a DynamicType option, which writes the AssemblyQualifiedName into the stream. If you are interested in this route, then note that the "cross-platform" features of the format start to break down, but it is very useful for .NET-to-.NET purposes.
At the simplest, a wrapper such as:
[ProtoContract]
public class SomeWrapper {
[ProtoMember(1, DynamicType = true)]
public object Value {get;set;}
}
Wrap your object in that and it should behave (in v2 at least; DynamicType did not exist in v1). Full example:
[TestFixture]
public class SO7218127
{
[Test]
public void Test()
{
var orig = new SomeWrapper {Value = new SubType { Foo = 123, Bar = "abc"}};
var clone = Serializer.DeepClone(orig);
Assert.AreEqual(123, orig.Value.Foo);
Assert.AreEqual("abc", ((SubType) clone.Value).Bar);
}
[ProtoContract]
public class SomeWrapper
{
[ProtoMember(1, DynamicType = true)]
public BaseType Value { get; set; }
}
[ProtoContract]
public class BaseType
{
[ProtoMember(1)]
public int Foo { get; set; }
}
[ProtoContract]
public class SubType : BaseType
{
[ProtoMember(2)]
public string Bar { get; set; }
}
}