Develop Reference

How to use (pack) Google.Protobuf.WellknownTypes.Any in protobuf-net code first gRPC

I am creating an gRPC service and we decided to choose the code first approach with protobuf-net.
Now I am running into a scenario where we have a couple of classes that need to be wrapped.
We do not want to define KnownTypes in the MyMessage class (just a sample name to illustrate the problem).
So I am trying to use the Any type which currently gives me some struggle with packing.
The sample code has the MyMessage which defines some header values and has to possiblity to deliver any type as payload.
[ProtoContract]
public class MyMessage
{
[ProtoMember(1)] public int HeaderValue1 { get; set; }
[ProtoMember(2)] public string HeaderValue2 { get; set; }
[ProtoMember(3)] public Google.Protobuf.WellknownTypes.Any Payload { get; set; }
}
[ProtoContract]
public class Payload1
{
[ProtoMember(1)] public bool Data1 { get; set; }
[ProtoMember(2)] public string Data2 { get; set; }
}
[ProtoContract]
public class Payload2
{
[ProtoMember(1)] public string Data1 { get; set; }
[ProtoMember(2)] public string Data2 { get; set; }
}
Somewhere in the code I construct my message with a payload ...
Payload2 payload = new Payload2 {
Data1 = "abc",
Data2 = "def"
};
MyMessage msg = new MyMessage
{
HeaderValue1 = 123,
HeaderValue2 = "iAmHeaderValue2",
Payload = Google.Protobuf.WellknownTypes.Any.Pack(payload)
};
Which doesn't work because Payload1 and Payload2 need to implement Google.Protobuf.IMessage.
Since I can't figure out how and do not find a lot information how to do it at all I am wondering if I am going a wrong path.
How is it intedend to use Any in protobuf-net?
Is there a simple (yet compatible) way to pack a C# code first class into Google.Protobuf.WellknownTypes.Any?
Do I really need to implement Google.Protobuf.IMessage?

Firstly, since you say "where we have a couple of classes that need to be wrapped" (emphasis mine), I wonder if what you actually want here is oneof rather than Any. protobuf-net has support for the oneof concept, although it isn't obvious from a code-first perspective. But imagine we had (in a contract-first sense):
syntax = "proto3";
message SomeType {
oneof Content {
Foo foo = 1;
Bar bar = 2;
Blap blap = 3;
}
}
message Foo {}
message Bar {}
message Blap {}
This would be implemented (via the protobuf-net schema tools) as:
private global::ProtoBuf.DiscriminatedUnionObject __pbn__Content;
[global::ProtoBuf.ProtoMember(1, Name = #"foo")]
public Foo Foo
{
get => __pbn__Content.Is(1) ? ((Foo)__pbn__Content.Object) : default;
set => __pbn__Content = new global::ProtoBuf.DiscriminatedUnionObject(1, value);
}
public bool ShouldSerializeFoo() => __pbn__Content.Is(1);
public void ResetFoo() => global::ProtoBuf.DiscriminatedUnionObject.Reset(ref __pbn__Content, 1);
[global::ProtoBuf.ProtoMember(2, Name = #"bar")]
public Bar Bar
{
get => __pbn__Content.Is(2) ? ((Bar)__pbn__Content.Object) : default;
set => __pbn__Content = new global::ProtoBuf.DiscriminatedUnionObject(2, value);
}
public bool ShouldSerializeBar() => __pbn__Content.Is(2);
public void ResetBar() => global::ProtoBuf.DiscriminatedUnionObject.Reset(ref __pbn__Content, 2);
[global::ProtoBuf.ProtoMember(3, Name = #"blap")]
public Blap Blap
{
get => __pbn__Content.Is(3) ? ((Blap)__pbn__Content.Object) : default;
set => __pbn__Content = new global::ProtoBuf.DiscriminatedUnionObject(3, value);
}
public bool ShouldSerializeBlap() => __pbn__Content.Is(3);
public void ResetBlap() => global::ProtoBuf.DiscriminatedUnionObject.Reset(ref __pbn__Content, 3);
optionally with an enum to help:
public ContentOneofCase ContentCase => (ContentOneofCase)__pbn__Content.Discriminator;
public enum ContentOneofCase
{
None = 0,
Foo = 1,
Bar = 2,
Blap = 3,
}
This approach may be easier and preferable to Any.
On Any:
Short version: protobuf-net has not, to date, had any particular request to implement Any. It probably isn't a huge amount of work - simply: it hasn't yet happened. It looks like you're referencing both protobuf-net and the Google libs here, and using the Google implementation of Any. That's fine, but protobuf-net isn't going to use it at all - it doesn't know about the Google APIs in this context, so: implementing IMessage won't actually help you.
I'd be more than happy to look at Any with you, from the protobuf-net side. Ultimately, time/availability is always the limiting factor, so I prioritise features that are seeing demand. I think you may actually be the first person asking me about Any in protobuf-net.

My Any implementation.
[ProtoContract(Name = "type.googleapis.com/google.protobuf.Any")]
public class Any
{
/// <summary>Pack <paramref name="value"/></summary>
public static Any Pack(object? value)
{
// Handle null
if (value == null) return new Any { TypeUrl = null!, Value = Array.Empty<byte>() };
// Get type
System.Type type = value.GetType();
// Write here
MemoryStream ms = new MemoryStream();
// Serialize
RuntimeTypeModel.Default.Serialize(ms, value);
// Create any
Any any = new Any
{
TypeUrl = $"{type.Assembly.GetName().Name}/{type.FullName}",
Value = ms.ToArray()
};
// Return
return any;
}
/// <summary>Unpack any record</summary>
public object? Unpack()
{
// Handle null
if (TypeUrl == null || Value == null || Value.Length == 0) return null;
// Find '/'
int slashIx = TypeUrl.IndexOf('/');
// Convert to C# type name
string typename = slashIx >= 0 ? $"{TypeUrl.Substring(slashIx + 1)}, {TypeUrl.Substring(0, slashIx)}" : TypeUrl;
// Get type (Note security issue here!)
System.Type type = System.Type.GetType(typename, true)!;
// Deserialize
object value = RuntimeTypeModel.Default.Deserialize(type, Value.AsMemory());
// Return
return value;
}
/// <summary>Test type</summary>
public bool Is(System.Type type) => $"{type.Assembly.GetName().Name}/{type.FullName}" == TypeUrl;
/// <summary>Type url (using C# type names)</summary>
[ProtoMember(1)]
public string TypeUrl = null!;
/// <summary>Data serialization</summary>
[ProtoMember(2)]
public byte[] Value = null!;
/// <summary></summary>
public static implicit operator Container(Any value) => new Container(value.Unpack()! );
/// <summary></summary>
public static implicit operator Any(Container value) => Any.Pack(value.Value);
/// <summary></summary>
public struct Container
{
/// <summary></summary>
public object? Value;
/// <summary></summary>
public Container()
{
this.Value = null;
}
/// <summary></summary>
public Container(object? value)
{
this.Value = value;
}
}
}
'System.Object' can be used as a field or property in a surrounding Container record:
[DataContract]
public class Container
{
/// <summary></summary>
[DataMember(Order = 1, Name = nameof(Value))]
public Any.Container Any { get => new Any.Container(Value); set => Value = value.Value; }
/// <summary>Object</summary>
public object? Value;
}
Serialization
RuntimeTypeModel.Default.Add(typeof(Any.Container), false).SetSurrogate(typeof(Any));
var ms = new MemoryStream();
RuntimeTypeModel.Default.Serialize(ms, new Container { Value = "Hello world" });
Container dummy = RuntimeTypeModel.Default.Deserialize(typeof(Container), ms.ToArray().AsMemory()) as Container;

Why is my game serializing this class?

So I'm making a game, and it saves users' progress on the computer in a binary file. The User class stores a few things:
Integers for stat values (Serializable)
Strings for the Username and the skin assets
Lists of both the Achievement class and the InventoryItem class, which I have created myself.
Here are the User fields:
public string Username = "";
// ID is used for local identification, as usernames can be changed.
public int ID;
public int Coins = 0;
public List<Achievement> AchievementsCompleted = new List<Achievement>();
public List<InventoryItem> Inventory = new List<InventoryItem>();
public List<string> Skins = new List<string>();
public string CurrentSkinAsset { get; set; }
The Achievement class stores ints, bools, and strings, which are all serializable. The InventoryItem class stores its name (a string) and an InventoryAction, which is a delegate that is called when the item is used.
These are the Achievement class's fields:
public int ID = 0;
public string Name = "";
public bool Earned = false;
public string Description = "";
public string Image;
public AchievmentDifficulty Difficulty;
public int CoinsOnCompletion = 0;
public AchievementMethod OnCompletion;
public AchievementCriteria CompletionCriteria;
public bool Completed = false;
And here are the fields for the InventoryItem class:
InventoryAction actionWhenUsed;
public string Name;
public string AssetName;
The source of the InventoryAction variables are in my XNAGame class. What I mean by this is that the XNAGame class has a method called "UseSword()" or whatever, which it passes into the InventoryItem class. Previously, the methods were stored in the Game1 class, but the Game class, which Game1 inherits from, is not serializable, and there's no way for me to control that. This is why I have an XNAGame class.
I get an error when trying to serialize: "The 'SpriteFont' class is not marked as serializable", or something like that. Well, there is a SpriteFont object in my XNAGame class, and some quick tests showed that this is the source of the issue. Well, I have no control over whether or not the SpriteFont class is Serializable.
Why is the game doing this? Why must all the fields in the XNAGame class be serializable, when all I need is a few methods?
Keep in mind when answering that I'm 13, and may not understand all the terms you're using. If you need any code samples, I'll be glad to provide them for you. Thanks in advance!
EDIT: One solution I have thought of is to store the InventoryAction delegates in a Dictionary, except that this will be a pain and isn't very good programming practice. If this is the only way, I'll accept it, though (Honestly at this point I think this is the best solution).
EDIT 2: Here's the code for the User.Serialize method (I know what I'm doing in inefficient, and I should use a database, blah, blah, blah. I'm fine with what I'm doing now, so bear with me.):
FileStream fileStream = null;
List<User> users;
BinaryFormatter binaryFormatter = new BinaryFormatter();
try
{
if (File.Exists(FILE_PATH) && !IsFileLocked(FILE_PATH))
{
fileStream = File.Open(FILE_PATH, FileMode.Open);
users = (List<User>)binaryFormatter.Deserialize(fileStream);
}
else
{
fileStream = File.Create(FILE_PATH);
users = new List<User>();
}
for (int i = 0; i < users.Count; i++)
{
if (users[i].ID == this.ID)
{
users.Remove(users[i]);
}
}
foreach (Achievement a in AchievementsCompleted)
{
if (a.CompletionCriteria != null)
{
a.CompletionCriteria = null;
}
if (a.OnCompletion != null)
{
a.OnCompletion = null;
}
}
users.Add(this);
fileStream.Position = 0;
binaryFormatter.Serialize(fileStream, users);

You cannot serialize a SpriteFont by design, actually this is possible (.XNB file) but it hasn't been made public.
Solution:
Strip it off your serialized class.
Alternatives:
If for some reasons you must serialize some font, the first thing that comes to my mind would be to roll-out your own font system such as BMFont but that's a daunting task since you'll have to use it everywhere else where you might already do ...
Generate a pre-defined amount of fonts (i.e. Arial/Times/Courier at size 10/11/12 etc ...) using XNA Content app (can't recall its exact name); then store this user preference as two strings. With a string.Format(...) you should be able to load the right font back quite easily.
Alternative 2 is certainly the easiest and won't take more than a few minutes to roll-out.
EDIT
Basically, instead of saving a delegate I do the following:
inventory items have their own type
each type name is de/serialized accordingly
their logic does not happen in the main game class anymore
you don't have to manually match item type / action method
So while you'll end up with more classes, you have concerns separated and you can keep your main loop clean and relatively generic.
Code:
public static class Demo
{
public static void DemoCode()
{
// create new profile
var profile = new UserProfile
{
Name = "Bill",
Gold = 1000000,
Achievements = new List<Achievement>(new[]
{
Achievement.Warrior
}),
Inventory = new Inventory(new[]
{
new FireSpell()
})
};
// save it
using (var stream = File.Create("profile.bin"))
{
var formatter = new BinaryFormatter();
formatter.Serialize(stream, profile);
}
// load it
using (var stream = File.OpenRead("profile.bin"))
{
var formatter = new BinaryFormatter();
var deserialize = formatter.Deserialize(stream);
var userProfile = (UserProfile) deserialize;
// set everything on fire :)
var fireSpell = userProfile.Inventory.Items.OfType<FireSpell>().FirstOrDefault();
if (fireSpell != null) fireSpell.Execute("whatever");
}
}
}
[Serializable]
public sealed class UserProfile
{
public string Name { get; set; }
public int Gold { get; set; }
public List<Achievement> Achievements { get; set; }
public Inventory Inventory { get; set; }
}
public enum Achievement
{
Warrior
}
[Serializable]
public sealed class Inventory : ISerializable
{
public Inventory() // for serialization
{
}
public Inventory(SerializationInfo info, StreamingContext context) // for serialization
{
var value = (string) info.GetValue("Items", typeof(string));
var strings = value.Split(';');
var items = strings.Select(s =>
{
var type = Type.GetType(s);
if (type == null) throw new ArgumentNullException(nameof(type));
var instance = Activator.CreateInstance(type);
var item = instance as InventoryItem;
return item;
}).ToArray();
Items = new List<InventoryItem>(items);
}
public Inventory(IEnumerable<InventoryItem> items)
{
if (items == null) throw new ArgumentNullException(nameof(items));
Items = new List<InventoryItem>(items);
}
public List<InventoryItem> Items { get; }
#region ISerializable Members
public void GetObjectData(SerializationInfo info, StreamingContext context)
{
var strings = Items.Select(s => s.GetType().AssemblyQualifiedName).ToArray();
var value = string.Join(";", strings);
info.AddValue("Items", value);
}
#endregion
}
public abstract class InventoryItem
{
public abstract void Execute(params object[] objects);
}
public abstract class Spell : InventoryItem
{
}
public sealed class FireSpell : Spell
{
public override void Execute(params object[] objects)
{
// using 'params object[]' a simple and generic way to pass things if any, i.e.
// var world = objects[0];
// var strength = objects[1];
// now do something with these !
}
}

Okay, so I figured it out.
The best solution was to use a Dictionary in the XNAGame class, which stores two things: an ItemType (an enumeration), and an InventoryAction. Basically, when I use an item, I check it's type and then look up it's method. Thanks to everyone who tried, and I'm sorry if the question was confusing.

Force binary deserialization to fail when type modified

I'm looking for a non-intrusive way to enforce deserialization to fail under the following circumstances:
The type is not defined in a strongly named assembly.
BinaryFormatter is used.
Since serialized, the type has been modified (e.g. a property has been added).
Below is an illustration/repro of the problem in form of a failing NUnit test. I'm looking for a generic way to make this pass without modifying the Data class, preferably by just setting up the BinaryFormatter during serialization and/or deserialization. I also don't want to involve serialization surrogates, as this is likely to require specific knowledge for each affected type.
Can't find anything in the MSDN docs that helps me though.
[Serializable]
public class Data
{
public string S { get; set; }
}
public class DataSerializationTests
{
/// <summary>
/// This string contains a Base64 encoded serialized instance of the
/// original version of the Data class with no members:
/// [Serializable]
/// public class Data
/// { }
/// </summary>
private const string Base64EncodedEmptyDataVersion =
"AAEAAAD/////AQAAAAAAAAAMAgAAAEtTc2MuU3Rvcm0uRGF0YS5UZXN0cywgV"+
"mVyc2lvbj0xLjAuMC4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1YmxpY0tleVRva2"+
"VuPW51bGwFAQAAABlTc2MuU3Rvcm0uRGF0YS5UZXN0cy5EYXRhAAAAAAIAAAAL";
[Test]
public void Deserialize_FromOriginalEmptyVersionFails()
{
var binaryFormatter = new BinaryFormatter();
var memoryStream = new MemoryStream(Convert.FromBase64String(Base64EncodedEmptyDataVersion));
memoryStream.Seek(0L, SeekOrigin.Begin);
Assert.That(
() => binaryFormatter.Deserialize(memoryStream),
Throws.Exception
);
}
}

I'd recommend a "Java" way here - declare int field in every single serializable class like private int _Serializable = 0; and check that your current version & serialized version match; manually increase when you change properties. If you insist on automated way you'll have to store a lot of metadata and check if current metadata & persisted metadata matches (extra burden on performance/size of serialized data).
Here is the automatic descriptor. Basically you'll have to store TypeDescriptor instance as a part of your binary data & on retrieve check if persisted TypeDescriptor is valid for serialization (IsValidForSerialization) against current TypeDescriptor.
var persistedDescriptor = ...;
var currentDescriptor = Describe(typeof(Foo));
bool isValid = persistedDescriptor.IsValidForSerialization(currentDescriptor);
[Serializable]
[DataContract]
public class TypeDescriptor
{
[DataMember]
public string TypeName { get; set; }
[DataMember]
public IList<FieldDescriptor> Fields { get; set; }
public TypeDescriptor()
{
Fields = new List<FieldDescriptor>();
}
public bool IsValidForSerialization(TypeDescriptor currentType)
{
if (!string.Equals(TypeName, currentType.TypeName, StringComparison.Ordinal))
{
return false;
}
foreach(var field in Fields)
{
var mirrorField = currentType.Fields.FirstOrDefault(f => string.Equals(f.FieldName, field.FieldName, StringComparison.Ordinal));
if (mirrorField == null)
{
return false;
}
if (!field.Type.IsValidForSerialization(mirrorField.Type))
{
return false;
}
}
return true;
}
}
[Serializable]
[DataContract]
public class FieldDescriptor
{
[DataMember]
public TypeDescriptor Type { get; set; }
[DataMember]
public string FieldName { get; set; }
}
private static TypeDescriptor Describe(Type type, IDictionary<Type, TypeDescriptor> knownTypes)
{
if (knownTypes.ContainsKey(type))
{
return knownTypes[type];
}
var descriptor = new TypeDescriptor { TypeName = type.FullName, Fields = new List<FieldDescriptor>() };
knownTypes.Add(type, descriptor);
if (!type.IsPrimitive && type != typeof(string))
{
foreach (var field in type.GetFields(BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public).OrderBy(f => f.Name))
{
var attributes = field.GetCustomAttributes(typeof(NonSerializedAttribute), false);
if (attributes != null && attributes.Length > 0)
{
continue;
}
descriptor.Fields.Add(new FieldDescriptor { FieldName = field.Name, Type = Describe(field.FieldType, knownTypes) });
}
}
return descriptor;
}
public static TypeDescriptor Describe(Type type)
{
return Describe(type, new Dictionary<Type, TypeDescriptor>());
}
I also though about some mechanism of shortening size of persisted metadata - like calculating MD5 from xml-serialized or json-serialized TypeDescriptor; but in that case new property/field will mark your object as incompatible for serialization.

Interaction VS and other program

I've written a DLL that parses XML and returns Dictionary with tag name and its value. I'm using it in other program called ZennoPoster Project Maker. Here's the code:
XMLWork.XMLWorker worker = new XMLWork.XMLWorker(); // My parse class
string path = #"Z:\New\test.xml";
Dictionary<string, string> data = worker.GetData(path); // GetData - method, that returns
// data from XML
project.Variables["second_name"].Value = data["second_name"];
This block of code I must remake into XMLWorker class method and return a project type and in ZennoPoster I have to with 1 line of code return data. How can I do it?

Assuming that you have the following simplified types:
namespace Objects
{
public class Project
{
public Dictionary<string, Variable> Variables { get; set; }
}
public class Variable
{
public object Value { get; set; }
}
}
You could structure your XMLWorker class like this:
using Objects;
public class XMLWorker
{
public Project Project { get; private set; }
public XMLWorker(string path)
{
Project = new Project();
Dictionary<string, string> data = GetData(path);
Project.Variables["second_name"].Value = data["second_name"];
}
internal Dictionary<string, string> GetData(string path)
{
// method implementation
}
}
Sample usage:
var project = (new XMLWork.XMLWorker(#"Z:\New\test.xml")).Project;

C# problem with object array

I am fairly new to arrays in C# and am used to storing a mass of data in a string and in INI files and then breaking it down into basic arrays using delimiters...so yeh, my knowledge is almost none existent.
My main form class begin this definition:
public CAirportData[] _AirportData; //size not known
This is the method I am using to create the array:
...string[] airports = possibleAirports.Split(','); //size is known
foreach (string airport in airports)
{
string[] rwys = inif.Read(airport, "rwys").Split(':'); //size is known (2)
_AirportData = new CAirportData[] { new CAirportData() { icao=airport, depRwy=rwys[0], arrRwy=rwys[1] } };
}
I know this just boils down to my limited knowledge of objects and arrays. But I can't seem to find anything on the internet that uses this sort of thing. I have tried to combine other peoples code with little success.
I need the _AirportData array to be available outside of the form hence public and declared outside of any methods. I supose the main problem is that I am overwriting array and foreach airport I am creating a new array hence loosing the previous. I had tried moving the ..= new CAirportData[] to all sorts of places but Visual Studio doesn't like it.
Below is the class definition for CAirportData:
public class CAirportData
{
public string icao { get; set; }
public string depRwy { get; set; }
public string arrRwy { get; set; }
public override string ToString()
{
string result = string.Format("ICAO: {0}, Dep: {1}, Arr: {2}", this.icao, this.depRwy, this.arrRwy);
return result;
}
}
public class CMRunways
{
public string icao { get; set; }
public string depRwy { get; set; }
public string arrRwy { get; set; }
}
Many thanks in advance for any help!

What you're looking for is generic List. Change the definition to:
public List<CAirportData> _AirportData = new List<CAirportData>();
Then the code in the loop to:
_AirportData.Add(new CAirportData { icao=airport, depRwy=rwys[0], arrRwy=rwys[1] });

This is what I would do...Create a static class, with a static property (airports) and add a static constructor to load the airports from file at the begining.
public static class Session
{
public static CAirportData[] _AirportData;
static Session()
{
string airports = possibleAirports.Split(",");
foreach (string airport in airports)
{
string[] rwys = inif.Read(airport, "rwys").Split(':'); //size is known (2)
_AirportData = new CAirportData[] { new CAirportData() { icao=airport, depRwy=rwys[0], arrRwy=rwys[1] } };
}
}
}
Now you can access the array anywhere in the project like
MessageBox.Show(Session.CAirportData[0].depRwy);