I'm using protobuf-net's serialization library and my team came across an interesting use case. In our application, we're deserializing a byte array onto an existing instance. The existing instance has properties, some are nullable and some aren't. The issue that I'm facing is that if a nullable property on the existing instance has a value, but the serialized instance just received over the wire doesn't have a value for that property. Since Protocol Buffers doesn't have a concept of null, null isn't serialized into the byte array and thus the deserialize call doesn't overwrite the nullable property with null.
In case you couldn't follow that, I wrote a quick unit test to illustrate my point/question:
[TestFixture]
public class WhenSerializingNullableProperties
{
private RuntimeTypeModel _runtimeTypeModel;
[TestFixtureSetUp]
public void FixtureSetup()
{
_runtimeTypeModel = TypeModel.Create();
_runtimeTypeModel.Add(typeof (Foo), false).Add("Id", "Name");
}
internal class Foo
{
public string Name { get; set; }
public int? Id { get; set; }
}
[Test]
public void ShouldSetNullablePropertiesToNull()
{
var foo = new Foo { Name = "name", Id = null };
byte[] serialized;
using (var stream = new MemoryStream())
{
_runtimeTypeModel.Serialize(stream, security);
serialized = stream.ToArray();
}
foo.Id = 18;
foo.Name = "other";
using (var stream = new MemoryStream(serialized))
{
_runtimeTypeModel.Deserialize(stream, foo, typeof(Foo));
}
foo.Name.Should().Be("name"); // this PASSES
foo.Id.Should().NotHaveValue(); // this FAILS
}
}
I understand (and am grateful for) protobuf's compactness with excluding nulls. I don't want to throw in a default value and have my streams be larger. Instead, I'm looking for protobuf-net to set a nullable property to null if it's not included in the stream being deserialized.
Thanks in advance for any and all help provided.
Related
I am trying to get protobuf serialization working with the RedisSessionStateProvider. I have specified the redisSerializerType as a custom class which implements Microsoft.Web.Redis.ISerializer - here is the deserialization code:
public object Deserialize(byte[] data)
{
return DeserializeDirect(data);
}
private object DeserializeDirect(byte[] data)
{
using (var memoryStream = new MemoryStream(data))
{
return Serializer.Deserialize<object>(memoryStream);
}
return null;
}
As I need to implement Microsoft.Web.Redis.ISerializer the signature for deserialize uses a return type of object and there is no way to pass in the actual type being returned. So when DeserializeDirect tries to use the Protobuf.Serializer to deserialize it (as expected) says "Type is not expected, and no contract can be inferred: System.Object". I am using a web app with .NET framework 4.6.1 and I was hoping somebody could point out what I am doing wrong.
Thanks!
Normally, protobuf-net really wants to know the exact type. You can, however, cheat using DynamicType. This tells protobuf-net to include additional type metadata - something it doesn't usually include.
Note that this can make you code brittle - it may fail if the type changes in you code!
I will be implementing Any soon (as part of 2.3.0), which is another option here.
public static void Main()
{
// the actual object we care about
object obj = new Foo { X = 1 };
// serialize and deserialize via stub
var stub = new Stub { Data = obj };
var clone = Serializer.DeepClone(stub);
// prove it worked
Console.WriteLine(clone.Data);
// prove it is a different instance
Console.WriteLine(ReferenceEquals(obj, clone.Data));
}
[ProtoContract]
public class Foo
{
[ProtoMember(1)]
public int X { get; set; }
public override string ToString() => $"X={X}";
}
[ProtoContract]
public sealed class Stub
{
[ProtoMember(1, DynamicType = true)]
public object Data { get; set; }
}
1. NullReference Exception on null values inside nullable collection.
[ProtoContract]
public class ProtoTest
{
[ProtoMember(1)]
public List<string> Collecton { get; set; }
}
static void Main(string[] args)
{
ProtoTest test = new ProtoTest();
test.Collecton = new List<string>();
test.Collecton.Add("A");
test.Collecton.Add(null);
test.Collecton.Add("B");
test.Collecton.Add(null);
//This code works fine on Protobuf 1.0.0.282
//But throws exception on Protobuf 2.0.0.668
byte[] buffer = Serialize<ProtoTest>(test);
ProtoTest converted = Deserialize<ProtoTest>(buffer);
//In 1.0.0.282 converted.Collection is having 2 items excluded null values.
//In 2.0.0.668 Serialization fails with NullReference exception.
Console.Read();
}
2. Trying to Creates default instance on null values in side reference type collection
[ProtoContract]
public abstract class NullableBase
{
public NullableBase()
{
}
[ProtoMember(1)]
public int? Value { get; set; }
}
[ProtoContract]
public class NullableChild:NullableBase
{
[ProtoMember(2)]
public string StrValue { get; set; }
}
[ProtoContract]
public class ProtoTest
{
[ProtoMember(1)]
public List<NullableBase> RefCollecton { get; set; }
}
static void Main(string[] args)
{
var nullableBaseType=ProtoBuf.Meta.RuntimeTypeModel.Default.Add(typeof(NullableBase), true);
nullableBaseType.AddSubType(100, typeof(NullableChild));
ProtoTest test = new ProtoTest();
test.RefCollecton = new List<NullableBase>();
test.RefCollecton.Add(new NullableChild() { StrValue = "A" });
test.RefCollecton.Add(new NullableChild() { StrValue = "B" });
test.RefCollecton.Add(null);
byte[] buffer = Serialize<ProtoTest>(test);
//For null values on reference type Protobuf is trying to create default instance.
//Here the type is NullBase and its an abstract class. Protobuf wont be able to create it and throwing exception.
//Why Protobuf creates default instance for null values.
ProtoTest converted = Deserialize<ProtoTest>(buffer);
Console.Read();
}
We were using Protobuf 1.0.0.282 and it was excluding null values for reference types and nullable collections on serializing. But in V 2.0.0.668, its behaving in differently for nullable types and reference types . For nullable types it throws NullReferenceException on null values while serializing. But for reference types it got serialized and while deserializing its trying to create default instance for null values.
Why they removed this functionality?. Is there any specific reason for that?.
If there is any good reason, then for reference and nullable type collection it should throw nullreferenece exception right? We have tried this solution of "SupportNull" https://code.google.com/p/protobuf-net/issues/detail?id=217.
I am using the following class to serialize individual objects (T) with a prefix, and then attempting to deserialize back to a list w/ the DeserializeItems() method:
public class ProtobufSerializationProvider<T> : ISerializationProvider<T>
{
readonly Type type;
readonly TypeModel model;
public ProtobufSerializationProvider()
{
this.type = typeof(T);
this.model = createModel(this.type);
}
public byte[] Serialize(T instance)
{
byte[] buffer;
using (MemoryStream strm = new MemoryStream())
{
model.SerializeWithLengthPrefix
(strm, instance, type, PrefixStyle.Base128, 1);
buffer = strm.GetBuffer();
}
return buffer;
}
// here is the problem method
public IEnumerable<T> DeserializeAll(MemoryStream stream)
{
return model.DeserializeItems<T>(stream, PrefixStyle.Base128, 1);
}
TypeModel createModel(Type type)
{
try
{
RuntimeTypeModel runtimeModel = TypeModel.Create();
this.addTypeToModel(runtimeModel, type);
this.addTypePropertiesToModel(runtimeModel, type);
return runtimeModel.Compile();
}
catch (Exception e)
{
throw e.InnerException;
}
}
void addTypePropertiesToModel(RuntimeTypeModel typeModel, Type type)
{
PropertyInfo[] properties = type.GetProperties();
for (int i = 0; i < properties.Length; i++)
{
Type innerType = properties[i].PropertyType;
if (!innerType.IsPrimitive && !(innerType == typeof(string)))
{
addTypeToModel(typeModel, properties[i].PropertyType);
}
}
}
MetaType addTypeToModel(RuntimeTypeModel typeModel, Type t)
{
var properties = t.GetProperties()
.Select(p => p.Name)
.OrderBy(name => name);
return typeModel
.Add(t, true)
.Add(properties.ToArray());
}
}
I am getting a default InvalidOperationException "Operation is not valid due to the current state of the object" when I try to enumerate the IEnumerable, whether by casting ToList(), or counting it Count(), etc. Specifically the MoveNext() method will throw the error:
enumerator.MoveNext()
Also the stream has to be open until DeserializeItems(stream) returns, and I ensured that was the case. But once the IEnumerable successfully returns I cannot use it.
Not sure if there is a problem with the serialized items or not. Also I noticed that each of my items 256 bytes, although a large portion of those bytes are just trailing nulls.
This is the class I am serializing as a test. Note that I am not using attributes since I create the model manually:
public class NestedFoo
{
public string NestedFooStr { get; set; }
}
public class Foo
{
public string Foo1 { get; set; }
public string Foo2 { get; set; }
public string Foo3 { get; set; }
public string Foo4 { get; set; }
public NestedFoo NestedFoo { get; set; }
}
Thanks.
UPDATE
After switching from MemoryStream.GetBuffer() to MemoryStream.ToArray(), the message is shortened but trying to cast the IEnumerable .ToList() or any other operation produces the same error.
However, I have noticed that after switching to .ToArray() the 'Current' property of the IEnumerable after enumeration reaches the last element in the IEnumerable before throwing the error, whereas when using .GetBuffer() the 'Current' property on the first element when the error is thrown.
I suspect the problem could be that the IEnumerable<> doesn't know when its out of elements because its reached the end of the stream? I append several items serialized with SerializeWithLengthPrefix() to a single byte[], and then there may be nulls left over trailing the array. If I read that entire array into the DeserializeItems method, does it need to know how to terminate or does it only continue if it detects a length prefix (as I would expect?).
Also one other question I had was to what extent using the SerializeWithLengthPrefix() method is equivalent to just serializing a List in one operation, using the DataFormat.Group enumeration, like:
[ProtoContract]
public class ProtoList<T>
{
[ProtoMember(1, DataFormat = DataFormat.Group)]
public List<T> List { get; set; }
public ProtoList() { }
public ProtoList(IEnumerable<T> items)
{
this.List = new List<T>(items);
}
}
Using a container such as this seems to perform very well, so I may just go with this route in cases where I want to serialize a list in one shot.
However for serializing items individually as they are added I am currently using my own prefixing function, so if I can get the Protobuf prefixing methods to work that would be good too.
The 256/trailing nulls is because you are using MemoryStream.GetBuffer(). This method is fine as long as it is used in combination with the Length, as the buffer is oversized. If you want a right-sized array, use ToArray() instead. Alternatively use ArraySegment<byte> to avoid copying any data.
It is possible this also fixes the exception (0 is not a valid value for a field-header). If not, leave a comment so I know to look in more detail. If it does fix it, please also let me know, and I'll try to make the error message (for the 0 field-header case) more helpful.
Just to confirm, ran into same problem.
Do not use tags / packet ids as 0. Solved everything.
I want to copy values from one object to another object. Something similar to pass by value but with assignment.
For example:
PushPin newValPushPin = oldPushPin; //I want to break the reference here.
I was told to write a copy constructor for this. But this class has a lot of properties, it will probably take an hour to write a copy constructor by hand.
Is there a better way to assign an object to another object by value?
If not, is there a copy constructor generator?
Note: ICloneable is not available in Silverlight.
If you can mark the object that is to be cloned as Serializable then you can use in-memory serialization to create a copy. Check the following code, it has the advantage that it will work on other kinds of objects as well and that you don't have to change your copy constructor or copy code each time an property is added, removed or changed:
class Program
{
static void Main(string[] args)
{
var foo = new Foo(10, "test", new Bar("Detail 1"), new Bar("Detail 2"));
var clonedFoo = foo.Clone();
Console.WriteLine("Id {0} Bar count {1}", clonedFoo.Id, clonedFoo.Bars.Count());
}
}
public static class ClonerExtensions
{
public static TObject Clone<TObject>(this TObject toClone)
{
var formatter = new BinaryFormatter();
using (var memoryStream = new MemoryStream())
{
formatter.Serialize(memoryStream, toClone);
memoryStream.Position = 0;
return (TObject) formatter.Deserialize(memoryStream);
}
}
}
[Serializable]
public class Foo
{
public int Id { get; private set; }
public string Name { get; private set; }
public IEnumerable<Bar> Bars { get; private set; }
public Foo(int id, string name, params Bar[] bars)
{
Id = id;
Name = name;
Bars = bars;
}
}
[Serializable]
public class Bar
{
public string Detail { get; private set; }
public Bar(string detail)
{
Detail = detail;
}
}
There is a protected member called "MemberwiseClone", you can write this in your class...
public MyClass Clone(){
return (MyClass)this.MemberwiseClone();
}
then you can access..
MyClass newObject = oldObject.Clone();
The only way (that I'm aware of) to do this, and do it correctly, is to implement the copy yourself. Take for example:
public class FrobAndState
{
public Frob Frobber { get; set;}
public bool State { get; set; }
}
public class Frob
{
public List<int> Values { get; private set; }
public Frob(int[] values)
{
Values = new List<int>(values);
}
}
In this example you'd need to know how Frob was implemented, i.e. the fact that you need to call the constructor to create a copy of it as Values is read-only, to be able to make a copy of a given instance of FrobAndState.
Also - you couldn't just implement FrobAndState.Copy thusly:
public class FrobAndState
{
// ... Properties
public FrobAndState Copy()
{
var new = new FrobAndState();
new.State = this.State;
new.Frobber = this.Frobber;
}
}
Because both the instance of FrobAndState that you called .Copy() on, and the new instance would both have a reference to the same instance of Frobber.
In short, copying things is hard and any Copy implementation is difficult to get right.
C# does not have a copy constructor. There are different ways to tackle this. At the OOP level you could use inheritance or aggregation. AutoMapper might also be worth a try.
I want to copy values from one object
to another object. Something similiar
to pass by value but with assignment.
What do you mean by "with assignment"? If you mean that you want people to be able to say:
a = b;
And for you to define what = means, the only way you can do that in C# is if b is a different type to a and you've defined an implicit conversion (or more tenuously, if a stands for something of the form x.Y where Y is a property with a setter). You can't override = for a simple assignment between identical types in C#.
I was told to write a copy constructor
for this. But this class has alot of
properties, it will probably take an
hour to write a copy constructor by
hand.
If that's really true, then I would guess that you have a different problem. Your class is too big.
If you make your class Serializable you could Serialize it to a MemoryStream and Deserialize to a new instance.
If you want copy-on-assignment you should be using a struct instead of a class. But be careful, it is easy to make subtle mistakes. It is highly recommended that all stucts be immmutable to reduce the chance for error.
Though, this may not answer your question directly, but to add a cent; usually the term Clone is linked with shallow copy(referenced objects). To have a deep copy, I believe you will need to look into the some creational pattern(prototype?). The answer to this question might help.
You implement Justin Angel's method of cloning objects in Silverlight
using System;
using System.Reflection;
using System.Windows;
namespace JustinAngelNet.Silverlight.Framework
{
public static class SilverlightExtensions
{
public static T Clone<T>(T source)
{
T cloned = (T) Activator.CreateInstance(source.GetType());
foreach (PropertyInfo curPropInfo in source.GetType().GetProperties())
{
if (curPropInfo.GetGetMethod() != null
&& (curPropInfo.GetSetMethod() != null))
{
// Handle Non-indexer properties
if (curPropInfo.Name != "Item")
{
// get property from source
object getValue = curPropInfo.GetGetMethod().Invoke(source, new object[] {});
// clone if needed
if (getValue != null && getValue is DependencyObject)
getValue = Clone((DependencyObject) getValue);
// set property on cloned
if (getValue != null)
curPropInfo.GetSetMethod().Invoke(cloned, new object[] {getValue});
}
// handle indexer
else
{
// get count for indexer
int numberofItemInColleciton =
(int)
curPropInfo.ReflectedType.GetProperty("Count").GetGetMethod().Invoke(source, new object[] {});
// run on indexer
for (int i = 0; i < numberofItemInColleciton; i++)
{
// get item through Indexer
object getValue = curPropInfo.GetGetMethod().Invoke(source, new object[] {i});
// clone if needed
if (getValue != null && getValue is DependencyObject)
getValue = Clone((DependencyObject) getValue);
// add item to collection
curPropInfo.ReflectedType.GetMethod("Add").Invoke(cloned, new object[] {getValue});
}
}
}
}
return cloned;
}
}
}
Then you can do this
MyClass newObject = SilverlightExtensions.Clone(oldObject);
Suppose I have the following (trivially simple) base class:
public class Simple
{
public string Value { get; set; }
}
I now want to do the following:
public class PathValue : Simple
{
[XmlAttribute("path")]
public string Value { get; set; }
}
public class ObjectValue : Simple
{
[XmlAttribute("object")]
public string Value { get; set; }
}
But without actually redefining the property. I want to apply attributes to members of the base class. Is this possible?
The real problem is that in my serialization mechanism from/to XML (which works brilliantly btw), I find a lot of similar elements where only the names of the attributes differ (they're not consistent, and I don't control the format). Right now I need to create a different class for every such element, whereas they're like 100% the same (apart from the attributes).
I don't think it's possible, but you might never know.
UPDATE:
I tried Marc's approach, but to no avail:
public class Document
{
public PathValue Path;
public ObjectValue Object;
}
class Program
{
static void Main(string[] args)
{
var doc = new Document()
{
Path = new PathValue() { Value = "some path" },
Object = new ObjectValue() { Value = "some object" }
};
XmlAttributeOverrides overrides = new XmlAttributeOverrides();
overrides.Add(typeof(PathValue), "Value", new XmlAttributes() { XmlAttribute = new XmlAttributeAttribute("path") });
overrides.Add(typeof(ObjectValue), "Value", new XmlAttributes() { XmlAttribute = new XmlAttributeAttribute("object") });
XmlSerializer serializer = new XmlSerializer(typeof(Document), overrides);
serializer.Serialize(Console.Out, doc);
Console.WriteLine();
Console.ReadLine();
}
}
...doesn't do the trick.
I'm going to answer this question myself, so that I can accept this answer. I don't like the answer, but I suppose it's the only valid answer.
The answer is: No, you can't do it.
Could you perhaps use the overload XmlSerializer constructor that lets you pass in the attributes to apply at runtime? Then you don't have to worry about it...
caveat: you want to cache the serializer instance and re-use it; otherwise (with the complex constructors) it does dynamic type generation each time.
Example:
using System;
using System.Xml.Serialization;
public class Simple {
public string Value { get; set; }
static void Main() {
XmlAttributeOverrides overrides = new XmlAttributeOverrides();
overrides.Add(typeof(Simple), "Value", new XmlAttributes {
XmlAttribute = new XmlAttributeAttribute("path")
});
XmlSerializer pathSerializer = new XmlSerializer(
typeof(Simple), overrides);
// cache and re-use pathSerializer!!!
Simple obj = new Simple();
obj.Value = "abc";
pathSerializer.Serialize(Console.Out, obj);
}
}
Output:
<Simple xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema" path="abc" />
How about this:
public class Simple
{
[XmlIgnore]
public string Value { get; set; }
}
public class PathValue : Simple
{
[XmlAttribute("path")]
public string Path {
get { return base.Value != null ? base.Value : null; }
set { base.Value = value != null ? value : null; }
}
}
public class ObjectValue : Simple
{
[XmlAttribute("object")]
public string Object {
get { return base.Value != null ? base.Value : null; }
set { base.Value = value != null ? value : null; }
}
}
This is the same technique used to serialize an unserializable type like a Uri that takes a serializable type in the constructor.
You are probably aware of this, but as an idea (although the code structure would completely change in that case):
One way would be to serialize the base class as a collection of name-value pairs, using custom serialization (there is also XDocument and similar helpful stuff to make it easier). Although it doesn't enforce type safety, it would spare you from doing lots of manual work.
I also prefer going for custom serialization because it allows a wider range of possibilities (serializing immutable classes, for example). XmlSerializer is also really nasty sometimes (e.g. I hate adding the "MyFieldSpecified" property to create optional attributes).
Perhaps you can mark the base class property with a common mapping, than you only override the property in inherited classes where it should be different. At least you would save some overriding.