Develop Reference

How to use (pack) Google.Protobuf.WellknownTypes.Any in protobuf-net code first gRPC

I am creating an gRPC service and we decided to choose the code first approach with protobuf-net.
Now I am running into a scenario where we have a couple of classes that need to be wrapped.
We do not want to define KnownTypes in the MyMessage class (just a sample name to illustrate the problem).
So I am trying to use the Any type which currently gives me some struggle with packing.
The sample code has the MyMessage which defines some header values and has to possiblity to deliver any type as payload.
[ProtoContract]
public class MyMessage
{
[ProtoMember(1)] public int HeaderValue1 { get; set; }
[ProtoMember(2)] public string HeaderValue2 { get; set; }
[ProtoMember(3)] public Google.Protobuf.WellknownTypes.Any Payload { get; set; }
}
[ProtoContract]
public class Payload1
{
[ProtoMember(1)] public bool Data1 { get; set; }
[ProtoMember(2)] public string Data2 { get; set; }
}
[ProtoContract]
public class Payload2
{
[ProtoMember(1)] public string Data1 { get; set; }
[ProtoMember(2)] public string Data2 { get; set; }
}
Somewhere in the code I construct my message with a payload ...
Payload2 payload = new Payload2 {
Data1 = "abc",
Data2 = "def"
};
MyMessage msg = new MyMessage
{
HeaderValue1 = 123,
HeaderValue2 = "iAmHeaderValue2",
Payload = Google.Protobuf.WellknownTypes.Any.Pack(payload)
};
Which doesn't work because Payload1 and Payload2 need to implement Google.Protobuf.IMessage.
Since I can't figure out how and do not find a lot information how to do it at all I am wondering if I am going a wrong path.
How is it intedend to use Any in protobuf-net?
Is there a simple (yet compatible) way to pack a C# code first class into Google.Protobuf.WellknownTypes.Any?
Do I really need to implement Google.Protobuf.IMessage?

Firstly, since you say "where we have a couple of classes that need to be wrapped" (emphasis mine), I wonder if what you actually want here is oneof rather than Any. protobuf-net has support for the oneof concept, although it isn't obvious from a code-first perspective. But imagine we had (in a contract-first sense):
syntax = "proto3";
message SomeType {
oneof Content {
Foo foo = 1;
Bar bar = 2;
Blap blap = 3;
}
}
message Foo {}
message Bar {}
message Blap {}
This would be implemented (via the protobuf-net schema tools) as:
private global::ProtoBuf.DiscriminatedUnionObject __pbn__Content;
[global::ProtoBuf.ProtoMember(1, Name = #"foo")]
public Foo Foo
{
get => __pbn__Content.Is(1) ? ((Foo)__pbn__Content.Object) : default;
set => __pbn__Content = new global::ProtoBuf.DiscriminatedUnionObject(1, value);
}
public bool ShouldSerializeFoo() => __pbn__Content.Is(1);
public void ResetFoo() => global::ProtoBuf.DiscriminatedUnionObject.Reset(ref __pbn__Content, 1);
[global::ProtoBuf.ProtoMember(2, Name = #"bar")]
public Bar Bar
{
get => __pbn__Content.Is(2) ? ((Bar)__pbn__Content.Object) : default;
set => __pbn__Content = new global::ProtoBuf.DiscriminatedUnionObject(2, value);
}
public bool ShouldSerializeBar() => __pbn__Content.Is(2);
public void ResetBar() => global::ProtoBuf.DiscriminatedUnionObject.Reset(ref __pbn__Content, 2);
[global::ProtoBuf.ProtoMember(3, Name = #"blap")]
public Blap Blap
{
get => __pbn__Content.Is(3) ? ((Blap)__pbn__Content.Object) : default;
set => __pbn__Content = new global::ProtoBuf.DiscriminatedUnionObject(3, value);
}
public bool ShouldSerializeBlap() => __pbn__Content.Is(3);
public void ResetBlap() => global::ProtoBuf.DiscriminatedUnionObject.Reset(ref __pbn__Content, 3);
optionally with an enum to help:
public ContentOneofCase ContentCase => (ContentOneofCase)__pbn__Content.Discriminator;
public enum ContentOneofCase
{
None = 0,
Foo = 1,
Bar = 2,
Blap = 3,
}
This approach may be easier and preferable to Any.
On Any:
Short version: protobuf-net has not, to date, had any particular request to implement Any. It probably isn't a huge amount of work - simply: it hasn't yet happened. It looks like you're referencing both protobuf-net and the Google libs here, and using the Google implementation of Any. That's fine, but protobuf-net isn't going to use it at all - it doesn't know about the Google APIs in this context, so: implementing IMessage won't actually help you.
I'd be more than happy to look at Any with you, from the protobuf-net side. Ultimately, time/availability is always the limiting factor, so I prioritise features that are seeing demand. I think you may actually be the first person asking me about Any in protobuf-net.

My Any implementation.
[ProtoContract(Name = "type.googleapis.com/google.protobuf.Any")]
public class Any
{
/// <summary>Pack <paramref name="value"/></summary>
public static Any Pack(object? value)
{
// Handle null
if (value == null) return new Any { TypeUrl = null!, Value = Array.Empty<byte>() };
// Get type
System.Type type = value.GetType();
// Write here
MemoryStream ms = new MemoryStream();
// Serialize
RuntimeTypeModel.Default.Serialize(ms, value);
// Create any
Any any = new Any
{
TypeUrl = $"{type.Assembly.GetName().Name}/{type.FullName}",
Value = ms.ToArray()
};
// Return
return any;
}
/// <summary>Unpack any record</summary>
public object? Unpack()
{
// Handle null
if (TypeUrl == null || Value == null || Value.Length == 0) return null;
// Find '/'
int slashIx = TypeUrl.IndexOf('/');
// Convert to C# type name
string typename = slashIx >= 0 ? $"{TypeUrl.Substring(slashIx + 1)}, {TypeUrl.Substring(0, slashIx)}" : TypeUrl;
// Get type (Note security issue here!)
System.Type type = System.Type.GetType(typename, true)!;
// Deserialize
object value = RuntimeTypeModel.Default.Deserialize(type, Value.AsMemory());
// Return
return value;
}
/// <summary>Test type</summary>
public bool Is(System.Type type) => $"{type.Assembly.GetName().Name}/{type.FullName}" == TypeUrl;
/// <summary>Type url (using C# type names)</summary>
[ProtoMember(1)]
public string TypeUrl = null!;
/// <summary>Data serialization</summary>
[ProtoMember(2)]
public byte[] Value = null!;
/// <summary></summary>
public static implicit operator Container(Any value) => new Container(value.Unpack()! );
/// <summary></summary>
public static implicit operator Any(Container value) => Any.Pack(value.Value);
/// <summary></summary>
public struct Container
{
/// <summary></summary>
public object? Value;
/// <summary></summary>
public Container()
{
this.Value = null;
}
/// <summary></summary>
public Container(object? value)
{
this.Value = value;
}
}
}
'System.Object' can be used as a field or property in a surrounding Container record:
[DataContract]
public class Container
{
/// <summary></summary>
[DataMember(Order = 1, Name = nameof(Value))]
public Any.Container Any { get => new Any.Container(Value); set => Value = value.Value; }
/// <summary>Object</summary>
public object? Value;
}
Serialization
RuntimeTypeModel.Default.Add(typeof(Any.Container), false).SetSurrogate(typeof(Any));
var ms = new MemoryStream();
RuntimeTypeModel.Default.Serialize(ms, new Container { Value = "Hello world" });
Container dummy = RuntimeTypeModel.Default.Deserialize(typeof(Container), ms.ToArray().AsMemory()) as Container;

datetime field in json deserialization

I have a .json like this:
[
{
"number":"00000001",
"dt_doc":"2019-09-26T17:39.000Z",
"address":"complete address"
}
]
But I've got problem with the field dt_doc, this is my deserialization code...
I have this in the main:
public override void CreateNewOutputRows()
{
String jsonFileContent = File.ReadAllText(Variables.JsonFilePath);
JavaScriptSerializer js = new JavaScriptSerializer();
List<Testata> testata = js.Deserialize<List<Testata>>(jsonFileContent);
foreach(Testata test in testata)
{
Output0Buffer.AddRow();
Output0Buffer.number= test.number;
Output0Buffer.dtdoc = test.dt_doc;
Output0Buffer.address= test.address;
}
}
and in my class Testata.cs I have defined the field in this way:
public DateTime dt_doc { get; set; }
But I got an exception on this field, probably related to 8601 standard, is there any way to solve?
This is the exception:
Error: System.FormatException: 2019-09-26T17:39.000Z it's not a valid value for DateTime. ---> System.FormatException: String not recognized as valid DateTime value.

The error is because you are missing seconds in date
"dt_doc":"2019-09-26T17:39.000Z"
should be
"dt_doc":"2019-09-26T17:39.00.000Z"
If this is intentional then you can specify the format. I have tried this using Newtonsoft.Json
public class Testata
{
[JsonConverter(typeof(DateFormatConverter), "yyyy-MM-ddTHH:mm.fffZ")]
public DateTime dt_doc { get; set; }
}
public class DateFormatConverter : IsoDateTimeConverter
{
public DateFormatConverter(string format)
{
DateTimeFormat = format;
}
}
List<Testata> testata = JsonConvert.DeserializeObject<List<Testata>>(jsonString);

you could read it in your class as a string and then:
DateTime.ParseExact(test.dt_doc,"yyyy-MM-ddTHH:mm.fffZ");

Why is the date returned as "0001-01-01" when calling a web service?

I have a very peculiar problem.
I have this code:
DT_PMNotificationRequestRecords request = new DT_PMNotificationRequestRecords();
request.NOTIF_DATE = DateTime.ParseExact(txtNotifData.Text, "yyyy-MM-dd", CultureInfo.InvariantCulture);
request.REQ_START = DateTime.ParseExact(txtReqStart.Text, "yyyy-MM-dd", CultureInfo.InvariantCulture);
Now both Notif_DATE and REQ_START have proper date values.
But when the request object is passed to the web service that I have written using the same class as an object in array.
requestRecords = new DT_PMNotificationRequestRecords[]{request, request2};
Notif = new Notifications.SI_PMNotification_Out_SyncResponseSoapClient(bindingHttp, endPointHttp);
Notif.SI_PMNotification_Out_Sync(requestRecords);
Web service:
public DT_PMNotificationResponseRecords[] SI_PMNotification_Out_Sync(DT_PMNotificationRequestRecords[] request)
{
try
{
Here in the request object the REQ_Start date is returned as "0001-01-01" but notif_date is correct, e.g. 10-10-2019.
In class the fields are defined like this:
[System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified, DataType="date", Order=6)]
public System.DateTime NOTIF_DATE
{
get { return this.nOTIF_DATEField; }
set {
this.nOTIF_DATEField = value;
this.RaisePropertyChanged("NOTIF_DATE");
}
}
[System.Xml.Serialization.XmlElementAttribute(Form=System.Xml.Schema.XmlSchemaForm.Unqualified, DataType="date", Order=10)]
public System.DateTime REQ_START
{
get { return this.rEQ_STARTField; }
set {
this.rEQ_STARTField = value;
this.RaisePropertyChanged("REQ_START");
}
}

XmlSerializer does not account for local timezone when deserializing property annotated with XmlElement with DataType time

When I deserialize a time string, using XmlSerializer.Deserialize, I expect it to take my local timezone into account so that a time string in the format
00:00:00.0000000+01:00
was parsed as 00:00, because I am in the timezone GMT+1.
Did I get that wrong?
Here is the code I am running to test xml deserialization:
using System;
using System.IO;
using System.Xml.Serialization;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace Testing
{
[TestClass]
public class FooTest
{
[TestMethod]
public void Test()
{
var serializer = new XmlSerializer(typeof(Foo),
new XmlRootAttribute("Foo"));
var xml = "<Foo><TheTime>00:00:00.0000000+01:00</TheTime></Foo>";
var stream = new MemoryStream();
var writer = new StreamWriter(stream);
writer.Write(xml);
writer.Flush();
stream.Position = 0;
var f = (Foo) serializer.Deserialize(stream);
Assert.AreEqual("00:00", f.TheTime.ToShortTimeString()); // actual: 01:00
}
[Serializable]
public class Foo
{
[XmlElement(DataType = "time")]
public DateTime TheTime { get; set; }
}
}
}

Unfortunately, there is no built-in type that you can deserialize a xs:time value into when it includes an offset (which is optional in the XSD spec).
Instead, you'll need to define a custom type and implement the appropriate interfaces for custom serialization and deserialization. Below is a minimal TimeOffset struct that will do just that.
[XmlSchemaProvider("GetSchema")]
public struct TimeOffset : IXmlSerializable
{
public DateTime Time { get; set; }
public TimeSpan Offset { get; set; }
public static XmlQualifiedName GetSchema(object xs)
{
return new XmlQualifiedName("time", "http://www.w3.org/2001/XMLSchema");
}
XmlSchema IXmlSerializable.GetSchema()
{
// this method isn't actually used, but is required to be implemented
return null;
}
void IXmlSerializable.ReadXml(XmlReader reader)
{
var s = reader.NodeType == XmlNodeType.Element
? reader.ReadElementContentAsString()
: reader.ReadContentAsString();
if (!DateTimeOffset.TryParseExact(s, "HH:mm:ss.FFFFFFFzzz",
CultureInfo.InvariantCulture, DateTimeStyles.None, out var dto))
{
throw new FormatException("Invalid time format.");
}
this.Time = dto.DateTime;
this.Offset = dto.Offset;
}
void IXmlSerializable.WriteXml(XmlWriter writer)
{
var dto = new DateTimeOffset(this.Time, this.Offset);
writer.WriteString(dto.ToString("HH:mm:ss.FFFFFFFzzz", CultureInfo.InvariantCulture));
}
public string ToShortTimeString()
{
return this.Time.ToString("HH:mm", CultureInfo.InvariantCulture);
}
}
With this defined, you can now change the type of Foo.TheTime in your code to be a TimeOffset and your test will pass. You can also remove the DataType="time" in the attribute, as it's declared in the object itself via the GetSchema method.

Mapping TimeSpan in SQLite and Dapper

I'm attempting to use Dapper to interface to an existing database format that has a table with a duration encoded as ticks in a BIGINT column. How do I tell Dapper to map my POCO's TimeSpan-typed property to ticks when inserting into and reading from the database?
I've tried to set the type map for TimeSpan to DbType.Int64:
SqlMapper.AddTypeMap(typeof(TimeSpan), DbType.Int64);
And I've also created an ITypeHandler, but the SetValue method is never called:
public class TimeSpanToTicksHandler : SqlMapper.TypeHandler<TimeSpan>
{
public override TimeSpan Parse(object value)
{
return new TimeSpan((long)value);
}
public override void SetValue(IDbDataParameter parameter, TimeSpan value)
{
parameter.Value = value.Ticks;
}
}
Here's my POCO:
public class Task
{
public TimeSpan Duration { get; set; }
// etc.
}
When executing a simple insert statement like this:
string sql = "INSERT INTO Tasks (Duration) values (#Duration);";
And passing the POCO as the object to insert:
Task task = new Task { Duration = TimeSpan.FromSeconds(20) };
connection.Execute(sql, task);
I get this exception:
System.InvalidCastException : Unable to cast object of type 'System.TimeSpan' to type 'System.IConvertible'.
at System.Convert.ToInt64(Object value, IFormatProvider provider)
at System.Data.SQLite.SQLiteStatement.BindParameter(Int32 index, SQLiteParameter param)
at System.Data.SQLite.SQLiteStatement.BindParameters()
at System.Data.SQLite.SQLiteCommand.BuildNextCommand()
at System.Data.SQLite.SQLiteCommand.GetStatement(Int32 index)
at System.Data.SQLite.SQLiteDataReader.NextResult()
at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery()
at Dapper.SqlMapper.ExecuteCommand(IDbConnection cnn, ref CommandDefinition command, Action`2 paramReader) in SqlMapper.cs: line 3310
at Dapper.SqlMapper.ExecuteImpl(IDbConnection cnn, ref CommandDefinition command) in SqlMapper.cs: line 1310
at Dapper.SqlMapper.Execute(IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Nullable`1 commandTimeout, Nullable`1 commandType) in SqlMapper.cs: line 1185
If I leave the TimeSpan type mapping as-is (it defaults to DbType.Time), it writes the string version of the TimeSpan, i.e. `00:00:20.000", which isn't helpful as it does not match the format of the other data in the column.

Could you do the following instead?
public class Task
{
public TimeSpan Duration { get; set; }
public long Ticks
{
get { return Duration.Ticks; }
set { Duration = new TimeSpan(value); }
}
// etc.
}
string sql = "INSERT INTO Tasks (Duration) values (#Ticks);";

Solutions for LinqToDB:
MappingSchema.SetDataType(typeof(TimeSpan), DataType.NText);
Or:
MappingSchema.SetDataType(typeof(TimeSpan), DataType.Int64);
Example:
public class Program
{
private const string ConnectionString = "Data Source=:memory:;Version=3;New=True;";
public static void Main()
{
var dataProvider = new SQLiteDataProvider();
var connection = dataProvider.CreateConnection(ConnectionString);
connection.Open();
var dataConnection = new DataConnection(dataProvider, connection);
dataConnection.MappingSchema.SetDataType(typeof(TimeSpan), DataType.Int64);
dataConnection.CreateTable<Category>();
dataConnection.GetTable<Category>()
.DataContextInfo
.DataContext
.Insert(new Category
{
Id = 2,
Time = new TimeSpan(10, 0, 0)
});
foreach (var category in dataConnection.GetTable<Category>())
{
Console.WriteLine($#"Id: {category.Id}, Time: {category.Time}");
}
}
private class Category
{
public int Id { get; set; }
public TimeSpan Time { get; set; }
}
}