Develop Reference

Protobuf-net deserialize string field to c# guid

In a simplified scenario, assume we have a custom c# type which contains a guid field and the corresponding proto type is,
message HelloReply {
string message = 1;
string guid_id = 2; // this is defined GUID in corresponding c# type
repeated string names = 3;
map<string, double> prices = 4;
}
When I try to deserialize this proto to c# type, I get exception stating 'Invalid wire-type' and a link to explanation which is not helpful to me. Is there a work around for this or am I overlooking something ?

In protobuf-net v3, this changes; there is now a concept called CompatibilityLevel, that works for this scenario; if a Guid member or containing type has CompatibilityLevel of 300 or more, then it is treated as a string. This topic is discussed in depth here: https://github.com/protobuf-net/protobuf-net/blob/main/docs/compatibilitylevel.md

Protobuf-net has opinions on guids. Opinions that were forged back in the depth of time, and that I now regret, but which are hard to revert without breaking people. If I was writing this today with hindsight, yes: it would probably just serialize as a string. But: that isn't what it expects today!
Frankly I'd hack around it with a shadow property. So instead of
[ProtoMember(42)]
public Guid Foo {get;set;}
You could use:
public Guid Foo {get;set;}
[ProtoMember(42)]
private string FooSerialized {
get => Foo.ToString(); // your choice of formatting
set => Foo = Guid.Parse(value);
}

How does a wrapper work for JsonUtility

So I've been looking to convert my Json array into an array of objects in Unity. I've found my solution from a 2 yeard old thread without explanation but I'm curious as to how it actually works.
If I use Visual Studio to look for the definition of FromJson it shows me this
public static T FromJson<T>(string json);
As I understand is that FromJson asks for an object to be filled, I give the class MyWrapper but besides telling MyWrapper that he contains a list of Question I never ask it to create a new item in the list. So how does it actually fill the list?
C#
MyWrapper wrappedQuestions = JsonUtility.FromJson<MyWrapper>(jsonString);
[Serializable]
public class MyWrapper
{
public List<Question> questions;
}
[Serializable]
public class Question
{
public int questionType;
public string questionString;
public int questionAnswer;
}
Json
{
"questions": [
{
"questionType": 1,
"questionString": "4^2",
"questionAnswer": 16
},
{
"questionType": 2,
"questionString": "√(25)",
"questionAnswer": 5
}
]
}
I'm still a beginner programmer so I hope I'am able to ask such questions here.

If you wonder why you need a wrapper for that, that's simply because Unity engineers did not add direct support for primitive types or arrays. It's just how they programmed it. Most Json API are not like this.
So how does it actually fill the list?
Reflection.
1.It reads the json you passed to it. It detects the questions variable in the json. It detects that the questions variable is an array due to the format such as [] and the commas that separates each item.
2.It finds the type of that questions variable which is Question or List of Question.
3.It uses Activator.CreateInstance to create new instance of Question, read each value from each variable in the json and then fill them up with reflection with that new instance it created.
4. It returns the new instance that is filled.
If you read and understand how to do basic stuff with reflection in C#, you may be able to make your own simple Json parser with the Queue class.
Finally, you can use the JsonHelper wrapper from this answer to serialize/de-serialize arrays easily without having to make a wrapper for each class.

As JSON stands for Javascript Object Notation, JSON objects (strings) follow a pattern. For any string to parsed and converted to the object, it has to be a valid JSON. It has certain rules/syntax,
for example
[ ] is for array/list,
{ } is for objects
and every object can contain key-value pairs (separated by colon :) to represent variables along with their values
{ "Key" : "Value" }
Now JSON parser is aware of these rules so it can check if any string is valid JSON.
What does it need to know to convert a JSON into a class object?
The class type you provide here :
JsonUtility.FromJson<MyWrapper>(jsonString);
is MyWrapper.
It should have the same structure as your jsonString.
Let's break down your jsonString to map it with the class structure:
This represents a MyWrapper object. which contains only one property called questions, which is an empty list.
{
"questions": [ ]
}
if questions have any element in it, it would be of type Question we can write it as following in JSON:
{
"questionType": 1, // int value
"questionString": "4^2", // string value
"questionAnswer": 16 // int value
}
now if questions have more than one elements they would be separated by a comma:
"questions": [
{
"questionType": 1,
"questionString": "4^2",
"questionAnswer": 16
},
{
"questionType": 2,
"questionString": "√(25)",
"questionAnswer": 4
},
...
]
JSON parser is aware of all these rules so by looking at class structure it can parse the string and create an object of the stated class provided that JSON is valid and class structure matches with JSON structure.
That's all I could say about this, hope you got the basic understanding.

How do I correctly use derived classes as fields of a Microsoft Bond object

So there's no confusion, when I talk through my issue I am doing so as someone who is using the compiled classes that result from Bond schemas (that is to say I use "class" instead of "struct", etc.). I feel like it makes more cognitive sense to think of it this way.
I am using Microsoft Bond and I have a main class that has several properties, one of which is an instance of a derived class.
When creating an instance of the main class I have no problem setting the property to an instance of the derived class; however when I deserialize from binary back into the main class the property is now seen as its base class.
I have tried to cast it as the derived class but that throws a runtime exception.
The examples for using derived classes in the Bond documentation/manual have you specifying the derived class at the time of deserialization, but I am not deserializing just the derived class but the main class.
Here's an example of how I have the bond schema set up
struct BaseExample
{
0: int property1;
}
struct DerivedExample : BaseExample
{
0: int property2;
}
struct MainExample
{
0: BaseExample mainProperty;
}
In usage I am setting mainProperty to an instance of the DerivedExample class.
What I'd expect is that after deserialization, mainProperty is still of type DerivedExample (containing property2) but what I am seeing instead is mainProperty is of type BaseExample (and doesn't contain property2)
Am I forced to use generics to do this or is there something I am missing?
EDIT: Adding examples
My code that uses the classes generated from the Bond schemas is like this.
We have a calling service that creates a message of this type and uses Bond to serialize it into a byte array before sending it on a stream.
var message = new MainExample();
var derivedExample = new DerivedExample()
{
property1 = 1,
property2 = 2,
};
message.mainProperty = derivedExample;
// This block is all from the Bond examples
var output = new OutputBuffer();
var writer = new CompactBinaryWriter<OutputBuffer>(output);
Serialize.To(writer, message);
SendMessage(output.Data.Array);
Now we have a receiving service that is going to take this message off the stream and use Bond to deserialize it back into an object.
void HandleMessage(byte[] messageBA)
{
// This block is all from the Bond examples
var input = new InputBuffer(messageBA);
var reader = new CompactBinaryReader<InputBuffer>(input);
MainExample message = Deserialize<BondEvent>.From(reader);
// mainProperty is now of type BaseExample and not DerivedExample
message.mainProperty.property1; // is accessable
message.mainProperty.property2; // will not compile
DerivedExample castedProperty = message.mainProperty as DerivedExample; // fails at runtime
}
Full disclosure: I am actually using F# but I figured it would be better to do these in C#

The slicing behavior that you are observing when deserializing is expected with the schemas as written. The MainExample.mainProperty field is of type BaseExample, so when it is serialized, only the BaseExample fields are written. It doesn't matter which runtime type is used. Additionally, when it is deserialized, only BaseExample fields will be realized.
When dealing with inheritance and polymorphism, Bond does not include any type information in serialized payloads: it leaves the decision about how to model this up to the schema designer. This stems from Bond's philosophy of only paying for what you use.
Depending on the data you are modeling, I see two ways to design your schema:
generics
bonded
Generics
As mentioned in the question, the MainExample struct could be made generic:
struct MainExample<T>
{
0: T mainProperty;
}
This essentially allows you to easily create a bunch of different structs with similar shapes. But these structs will not have an "is a" relationship. Methods like HandleMessage likely will also have to be generic, causing a generic cascade.
Bonded
To include a field in another struct that is polymorphic in type, make the field a bonded field. Bonded fields do not slice when serialized. Also, they are not immediately deserialized, so the receiving side has a chance to pick the appropriate type to deserialize into.
In the .bond file, we'd have this:
struct MainExample
{
0: bonded<BaseExample> mainProperty;
}
To serialize, the following:
var message = new MainExample();
var derivedExample = new DerivedExample()
{
property1 = 1,
property2 = 2,
};
message.mainProperty = new Bonded<DerivedExample>(derivedExample);
// NB: new Bonded<BaseExample>(derivedExample) WILL slice
And to deserialize:
void HandleMessage(byte[] messageBA)
{
// This block is all from the Bond examples
var input = new InputBuffer(messageBA);
var reader = new CompactBinaryReader<InputBuffer>(input);
MainExample message = Deserialize<BondEvent>.From(reader);
DerivedExample de = message.mainProperty.Deserialize<DerivedExample>();
}
When using bonded fields for polymorphism, we will need to have some way of knowing which most-derived type to deserialize into. Sometimes this is known from context external to the payload (e.g., perhaps each of the messages handled only has one type). Other times, we need to embed this information in the common part of the payload. A common way to do this is with an enum:
enum PropertyKind
{
Base;
Derived;
}
struct MainExample
{
0: bonded<BaseExample> mainProperty;
1: PropertyKind mainPropertyKind = Base;
}
There's a fully worked example of doing this kind of dispatch in the C# polymorphic_container example in the Bond repository.
OutputBuffer.Data.Array
I notice that in the code to send a message, there's the following line, which contains a bug:
SendMessage(output.Data.Array);
The OutputBuffer.Data property is an ArraySegment<byte>, which is used to represent a slice of some other array. This slice may be shorter than the entire array (the Count property), and it may start at an offset other than 0 (the Offset property). Most I/O libraries have an overload like SendMessage(byte[] buf, int offset, int count) that can be used in cases like this.
The default array backing an OutputBuffer is 65K, so there's almost certainly a bunch of extra data being sent.

Can I access fields of an Object in C#

I am using Newtonsoft.Json.JsonConvert.DeserializeObject(string str) to translate a string into a .Net Object. The real type of this string can be multiple, and there's no other information to indicate the type of this string. But I can confirm that the string message is a derived class object of a common class like Message, and fields in Message can however tell the real type, Message has a field like int type. The string is MessageA or MessageB or whatever all with a different type.
If I translate it into an Object, I can saw in the visual studio debugger that this Object have exactly the fields described in the Json string. But I cant access these fields. And cast this Object into a Message will failed with a bad cast.
What I am doing now is first translate the string into a Message and to see the type, then I translate again. It's not preferred. So can I just translate it into something that I can read all of the data? Another option is Dictionary, but I have some numerical fields. Any other suggestions?

JsonConvert.DeserializeObject(string str), when used on a JSON object, returns a JObject. You can use this directly (e.g. use DeserializeObject<JObject> to explicitly type it), or as a dynamic, to access its properties, e.g.
var data = #"{""type"": 1, ""otherProperty"": ""Hello!""}";
dynamic obj = JsonConvert.DeserializeObject(data);
if (obj.type == 1)
{
Console.WriteLine(obj.otherProperty); // prints Hello!
}
Also, you mention numeric fields as being a problem with working with a dictionary, but if you make it a Dictionary<string, dynamic> you might find it easier to work with:
var data = #"{""type"": 2, ""otherProperty"": 5}";
var dict = JsonConvert.DeserializeObject<Dictionary<string, dynamic>>(data);
if (dict["type"] == 2)
{
int i = (int)dict["otherProperty"]; // type is long, so cast is required if you want an int
Console.WriteLine(i); // prints 5
}

Decimal precision of zero when serializing doubles to JSON

Consider the following JSON object:
{
"value": 0
}
Now suppose I'm mapping this to a .NET type Foo:
class Foo
{
public double Value { get; set; }
}
The type of Foo.Value is double, because Value isn't always an integer value.
Using JSON.NET, this works beautifully:
Foo deserialized = JsonConvert.DeserializeObject<Foo>(json);
However, observe what happens when I try to convert the object back to its JSON representation:
string serialized = JsonConvert.SerializeObject(deserialized, Formatting.Indented);
Output:
{
"Value": 0.0
}
Notice the trailing zero? How do I get rid of it?
EDIT
I suspect that the answer will be write your own converter. If it is, then that's fine and I guess I'll accept that as the answer. I'm just wondering if perhaps there exists an attribute that I don't know of that lets you specify the output format (or similar).

It appears that this is a hard-coded behavior of the library:
https://github.com/JamesNK/Newtonsoft.Json/blob/master/Src/Newtonsoft.Json/JsonConvert.cs#L300
If you want to alter the behavior you'll need to edit the library and recompile from source (or choose another JSON library)