Develop Reference

Protobuf-net dynamic types in a load-balanced system

I'm using protobuf-net in an application that does a lot of binary serialization of objects from (for all intents and purposes) 3rd party dlls. As a result, I can't use the [Proto-] attributes on the contracts themselves, and I'm instead using the RuntimeTypeModel to prepare the serializer at runtime as it encounters new types. Example serializer wrapper:
public static class ProtobufSerializer
{
public static byte[] Serialize<T>(T obj)
{
PrepareSerializer(typeof(T));
ProtoBuf.Serialize(memoryStream, obj);
}
public static T Deserialize<T>(byte[] bytes)
{
PrepareSerializer(typeof(T));
ProtoBuf.Serialize(memoryStream, obj);
}
}
Where we can safely assume that PrepareSerializer is capable of preparing RuntimeTypeModel to serialize any given type. I'm having some issues with the deserialization of objects where I have to leverage DynamicType=true though. For instance, given the following interface:
public interface IFoo
{
string Name {get;}
}
And the implementation:
public class Foo : IFoo
{
public string Name {get;set;}
public Bar Bar {get;set;}
[OnDeserializing]
public void OnDeserializing(Type t)
{
PrepareSerializer(typeof(Foo));
}
}
public class Bar
{
public int Baz {get;set;}
}
The PrepareSerializer method essentially would use a surrogate and generate a model roughly equivalent to:
// registered surrogate for IFoo
[ProtoContract]
public class IFooSurrogate
{
[ProtoMember(1, DynamicType=true)]
public object Value
[OnSerializing]
public void OnSerializing(Type t)
{
PrepareSerializer(this.Value.GetType());
}
}
Where Value is set by the implicit converters to equal the instance of IFoo. This works fine during serialize (where the event is fired and gives me a chance to prepare the serializer for the specific interface implementation type). It would also work fine in a non-distributed system where I would have to run through a serialize method before ever trying to deserialize that type. During deserialization in a distributed system though, where the current node has never seen Foo before, the ProtoBuf.Serializer throws an InvalidOperationException complaining about a lack of serializer for type Bar before it runs the Foo.OnDeserializing event (giving me a chance to tell it how to deserialize Bar).
Is there any way to attach a hook to ensure my code is given a chance to know about 'Foo' before protobuf-net complains about a lack of serializers?

I haven't tried exactly this situation, but: in order to allow some flexibility, all Type storage and rehydration goes via the TypeModel.DynamicTypeFormatting event; so, you could in theory hook this event on RuntimeTypeModel.Default, with something like:
RuntimeTypeModel.DynamicTypeFormatting += (sender, args) => {
if (args.FormattedName != null) { // meaning: rehydrating
lock(SomeSyncLock) {
if(NotYetKnown(args.FormattedName))
Prepare(args.FormattedName);
}
}
};
The intent of this API is to allow you to control how types are resolved, but... I guess it would work for this too?
I can, however, get behind the idea of an event that is more deliberately targeted at the first time a new Type is seen, essentially replacing / supplementing the "apply default behaviour" code. I don't think it exists today, though.

NetDataContractSerializer Deserialization With New Property

Lacking any real foresight, I've serialized a large set of data decorated only with Serializable using NetDataContractSerializer, and now I'd like to add a new field. What are my options?
The original class looks something like this (with a few levels of inheritance and quite a few fields):
[Serializable]
public class InheritedClass : BaseClass
{
public string StringId { get; set; }
}
And now I'd like to add another property, say something like:
[Serializable]
public class InheritedClass : BaseClass
{
public string StringId { get; set; }
public int IntId { get; set; }
}
Now when I update the class and go to deserialize, I receive an exception since the new field is not present, something like:
Exception thrown: 'System.Runtime.Serialization.SerializationException' in System.Runtime.Serialization.dll
Additional information: Error in line 1 position 601. 'Element' '_x003C_StringId_x003E_k__BackingField' from namespace 'http://schemas.datacontract.org/2004/07/QT' is not expected. Expecting element '_x003C_IntId_x003E_k__BackingField'.
Ok, so this makes sense since NetDataContractSerializer requires the same class. I can get around that using a DataMember attribute like:
[DataMember(IsRequired = false)]
The problem then is that switching to DataMember (as I should have done upfront, or used a different serializer) changes the implicit alphabetical ordering, and then most of my fields will silently not deserialize as is well known.
I've attempted to add an ordering that's inline with the ordering on disk manually (via Order properties on the attribute), but that doesn't appear to be respected either. (I don't see an order value I could match in the raw xml either.)
Are there any other options beyond writing something to load the xml and insert the missing node? (Or equivalently setup a parallel type and deserialize from one an re-serialize to another?) If not, I'll probably just load up with the current type and deserialize to JsonNet or protobuf, but am I missing anything more straightforward with DataMember/etc?

Marking a type with [Serializable] means that the type can be serialized by serializing its public and private fields -- not its properties. NetDataContractSerializer respects this attribute when present, serializing the fields as indicated. For an auto-implemented property the secret backing field is what is actually serialized.
When adding a new field, what one generally does to handle legacy data is to mark it with [OptionalField] to indicate that it won't always be present in serialization streams. In c# 7.3 and later, it's possible to do this to the secret backing field of an auto-implemented property by using a field-targeted attribute:
[Serializable]
public class InheritedClass : BaseClass
{
public string StringId { get; set; }
[field: OptionalField]
public int IntId { get; set; }
}
Prior to c# 7.3 there is no way to apply an attribute to the backing field of an auto-implemented property. Thus you need to make the backing field be explicit and add the attribute to it:
[Serializable]
public class InheritedClass : BaseClass
{
public string StringId { get; set; }
[OptionalField]
int intId;
public int IntId { get { return intId; } set { intId = value; } }
}
Notes:
As noted in the question, if a type is marked with data contract attributes then NetDataContractSerializer will use those in preference to the default [Serializable] contract and allow you to explicitly indicate properties to serialize (and provide names clearer than the secret backing field names).
Unfortunately it is not always practical to add data contract attributes to legacy types.
NetDataContractSerializer has not been ported to .NET Core / .NET 5 and likely never will be.

Inconsistencies after deserialization using protobuf-net

I am facing a problem of inconsistency after deserialization using protobuf-net.
The class I would like to serialize/deserialize looks like:
[ProtoContract]
public class TSS
{
[ProtoMember(1, AsReference = true)]
public EventManager eventManager { get; private set; }
[ProtoMember(2)]
public DateTime referenceDateTime { get; private set; }
[ProtoMember(3)]
public Mode mode;
}
And inside EventManager class, it looks like:
[ProtoContract]
public class EventManager
{
[ProtoMember(1)]
public InputQueue inputQueue = new InputQueue();
[ProtoMember(2)]
public InputQueue InputQueue
{
get { return this.inputQueue; }
set { this.inputQueue = value; }
}
[ProtoMember(7, AsReference = true)]
public TSS tss;
}
The tss in class EventManager is a reference of TSS object, and eventManager in class TSS is a reference of EventManager object. This is the reason I put AsReference = true there (is this the right way?)
I do serialization like:
public void StateSaving(int time, TSS tss)
{
Stream memoryStream = new MemoryStream();
Serializer.Serialize(memoryStream, tss);
states.Add(time, memoryStream);
}
and do deserialization like:
public void Deserialize(int time, ref TSS tss)
{
Stream memoryStream = states[time];
memoryStream.Position = 0;
tss = Serializer.Deserialize<TSS>(memoryStream);
}
The problem is that whenever I do deserialization, the data structures like inputQueue in the EventManager is populated with NULL values instead of actual values at that point. I am a newbie to protobuf-net, so please point out any mistakes (I believe there are a lot).
Thanks in advance!!

(from comments)
I have located the problem, basically there's a list that needs to be deserialized. And this list is a list of events, in which the constructors of the events have parameters, and when it tries to deserialize, the program will run the parameterless constructors (I manually added these constructors in order to eliminate the exceptions) instead of the right ones (with parameters). I know this is how serialization/deserialization work, but is there a way I can serialize and deserialize this list correctly?
Ah, indeed. There are various approaches when it comes to object construction:
use the parameterless constructor
look for a constructor which matches all the defined members
skip the constructor completely
use a custom object factory
use a surrogate object and custom conversion
Things like XmlSerializer use the first; things like DataContractSerializer and BinaryFormatter use the 3rd; the good news is that protobuf-net supports all 5. I suggest that in your case the best option is to use the 3rd option for this type, which you can do by:
[ProtoContract(SkipConstructor=true)]

c# - DataContract Serialization - Base Classes, Inheritance and Overrides

Good day everyone,
I'm an independent game developer who has, in the past, primarily worked with XNA and, at the other extreme, commercial toolsets. The reach of XNA is pretty limited, however, and I'm building a cross-platform abstraction layer to target multiple platforms.
To cut a long story short, I've needed xml serialization that's accessible more broadly than [Serializable], and I've been pointed to data contracts. I've been doing a lot of research, but can't find any good information about some of the basics of the system, pertaining to inheritance and overrides.
The crux of my question is...
[DataContract]
public class Node
{
private string name;
public string Name { get { return name; } set { name = value; } }
public virtual float Rotation { get { return 0f; } set { } }
}
[DataContract]
public class FancyNode : Node
{
private float rotation;
public override float Rotation { get { return rotation; } set { rotation = value; } }
}
If I serialize a 'FancyNode', will 'Rotation' be properly serialized, and will 'Name' be serialised?
Follow-up Question:
I meant to ask earlier, but couldn't recall at the time. How does the serializer handler overriden [IgnoreDataMember] properties? For example...
[DataContract]
public class Simple
{
[IgnoreDataMember]
public virtual string Value { get { return ""; } set { } }
}
[DataContract]
public class Complex : Simple
{
private string value;
public override string Value { get { return value; } set { this.value = value; } }
}
Would 'Value' in 'Complex' be serialized? Some answers are suggesting that if no [DataMember] tags are used, all properties will be serialized. If so, does the [IgnoreDataMember] attribute of the base class have any bearing?

Some thoughts presented in a few answers are unclear. Please let me clear them up. First, the serializer that uses DataContract can be used to serialize types that are not decorated with DataContract. You only need to inform the serailizer about them.
For example, the DataContractJsonSerializer class will serialize a type into a JSON object (usually a string.) All you need to tell it is the type of object you're serializing, and then any other types it may reference:
var serializer = new System.Runtime.Serialization.Json.DataContractJsonSerializer(typeof(MyClass), new Type[] { Type1, Type2, Type3 });
The serializer will automatically serialize each public field and property of the type. For more information, see http://msdn.microsoft.com/en-us/ms733127
DataContractAttribute Marked Types
Once you mark a type with the DataContractAttribute, you turn that type into an automatically known type (that you don't need to provide as a child type) and you turn it into the properties and fields into opt-in mode.
Therefore, you must decorate each field or property you want to be serialized with the DataMemberAttribute. This means that the IgnoreDataMemberAttribute is useless. It is, the serializer will not look for it, since anything not marked with the DataMemberAttribute is automatically ignored.
Unmarked DataContractAttribute Types
When serializing a type that does not have the DataContractAttribute applied, as previously stated, each public property or field will be serialized. Therefore, the IgnoreDataMemberAttribute is used here to prevent a property or field from being serialized.

As far as I know, DataContract is an 'opt-in' serialization method, i.e. stuff isn't serialized unless you decorate it (tell the serializer you want to serialize it)
So for the above example, you would need to add [DataMember] to the properties you wanted to serialize
With the standard Serializable attribute, the serializer looks at all fields and only ignores those which you mark as NonSerialized
Some examples here:
http://jamescbender.azurewebsites.net/?p=651
Check the notes section on this for info on what gets serialized and a rough outline of what happens:
http://msdn.microsoft.com/en-us/library/ms733127.aspx
Edit: Also I can't see any reason why any of the fields once marked as [DataMember] wouldn't be serialized properly. The DataContract method of serialization can also deal with circular references - something that other serialization sometimes has trouble with:
http://msdn.microsoft.com/en-us/library/hh241056.aspx

You should include the [DataMember] attribute on properties that should be exposed by the contract.
[DataContract]
public class FancyNode : Node
{
private float rotation;
[DataMember]
public override float Rotation { get { return rotation; } set { rotation = value; } }
}
Note that Windows Communication Foundation (WCF) uses the Data Contract Serializer to serialize and deserialize data (convert it to and from XML). So actually you are still using Xml Serialization.

No it wont serialize because of Opt-In approach, you have to explicitly apply DataMember to your base class. Data Serialization does not work automatically in case you Inherit in Child Class

Serialization system.object

I'm trying to create a webservice to return a generic. The return class looks like this:
[Serializable]
public class RenderReturn
{
public RenderReturnStatus StatusReturn { get; set; }
public string MessageReturn { get; set; }
public string MessageTitle { get; set; }
public object **ObjectReturn** { get; set; }
}
Where ObjectReturn can be an object or a list of application objects, like cars, customers, etc..
But the webservice returns the following error:
System.InvalidOperationException: There was an error generating the XML document. --->
System.InvalidOperationException: The type Environment was not expected. Use the XmlInclude or SoapInclude attribute to specify types that are not known statically.
at System.Xml.Serialization.XmlSerializationWriter.WriteTypedPrimitive(String name, String ns, Object o, Boolean xsiType)
This is possible or method should always return specific types?

Explicitly name your types. Otherwise one can put something in that isn't serializable.

Try something along these lines:
public class RenderReturn<T>
{
public T ObjectReturn {get;set;}
}
That way, at run-time, you'll have a concrete type rather than just System.Object.

Type of object is too generic for .NET to know what kind of object it is dealing with and how to deserialise. So it asks you to give it some hint by using XmlInclude attributes telling .NET the types to expect. In WCF you do the same: you use KnownType attribute to decorate properties.
Type object is not a good candidate for DTO objects that need to cross process boundaries.
In WCF or Web Services, try not to think in object-oriented fashion but think in WSDL. As far as WSDL is concerned you have a contract which explicitly defines the type of messages passed between client and server.