In the following code i want to use a predefined protobuf message in c#. I found that I was able to write and use the method to take a method that has been created and make a byte[]:
ContainerMessage containerMessage = new ContainerMessage();
containerMessage.Type = CommandType.Connect;
containerMessage.Connect = new Connect();
containerMessage.Connect.ClientName = "TemporaryClientName";
byte[] stream = new byte[containerMessage.CalculateSize()];
using (Google.Protobuf.CodedOutputStream outstream = new Google.Protobuf.CodedOutputStream(stream))
{
containerMessage.WriteTo(outstream);
}
This works as expected and i can inspect the message and the values are as expected as are the values in the byte[]. But if I try to Deserialize even this simple byte[] that i have just created:
using (Google.Protobuf.CodedInputStream instream = new Google.Protobuf.CodedInputStream(stream))
{
instream.ReadMessage(containerMessage);
}
It fails with:
An unhandled exception of type 'Google.Protobuf.InvalidProtocolBufferException' occurred in Google.Protobuf.dll
Additional information: Protocol message contained an invalid tag (zero).
Is this way of deserializing from a byte[] correct for protobuf?
The Protobuf Definition is:
message ContainerMessage {
CommandType type = 1;
bool commandSuccess = 2;
oneof message {
Connect connect = 3;
}
}
enum CommandType {
START = 0;
CONNECT = 2;
}
message Connect {
string ClientName = 1;
uint32 PushPullPort = 2;
}
And the CS file is generated with the command line:
protoc.exe -I=%CD% --csharp_out=..\GeneratedCsMessaging\ Connect.proto
The CodedOutputStream and CodedInputStream are mainly intended to be used by the compiled proto classes. The API for CodedOutputStream states such and mentions that if you want to have manually-written code calling either of both classes you need to use their WriteTag method before each value.
However, since you want to use the Google Protobuf for serializing and parsing any System.IO.Stream will do the job just like intended. This is very well documented and described in the Parsing and serialization section of the Protocol Buffer Basics for C#. The examples which can be found in Google Protobuf's Github can be quite helpful for getting the hang of Google Protobuf quickly. There you can see that a MemoryStream is used to serialize the object while the Parse.ParseFrom method can be used to parse an object out of the serialized data.
As you've mentioned in the comments to your question using Google.Protobuf; is an essential part to be able to use Google Protobuf's features.
EDIT: A sample usage in your case could look something like this
byte[] serializedBytes;
ContainerMessage containerMessage = new ContainerMessage()
{
Connect = new Connect()
{
ClientName = "TemporaryClientName",
},
Type = CommandType.Connect,
};
using( MemoryStream stream = new MemoryStream())
{
containerMessage.WriteTo(stream);
serializedBytes = stream.ToArray();
}
ContainerMessage parsedCopy = ContainerMessage.Parser.ParseFrom(serializedBytes);
Related
I have a protobuf object that I am sending from a C# application (using clrZmq) to a C++ service (using the zmq C++ bindings) on a local machine (for testing). I attempt to send my object from C# using the following
Taurus.Odds odds = Util.GetFakeOdds();
using (var context = ZmqContext.Create())
using (var socket = context.CreateSocket(SocketType.REQ))
{
byte[] buffer = null;
socket.Connect(TARGET); // TARGET = "tcp://127.0.0.1:6500"
Taurus.FeedMux mux = new Taurus.FeedMux();
mux.type = Taurus.FeedMux.Type.ODDS;
mux.odds = odds;
SendStatus status = socket.Send(mux.ToByteArray());
if (status == SendStatus.Sent)
{
int i;
byte[] arr = socket.Receive(buffer, SocketFlags.None, out i);
Taurus.Bet bet = buffer.ToObject<Taurus.Bet>();
}
...
}
Where I am serializing to my Taurus.Odds object to byte[] via the extension method
public static byte[] ToByteArray(this object o)
{
if(o == null)
return null;
BinaryFormatter bf = new BinaryFormatter();
using (MemoryStream ms = new MemoryStream())
{
bf.Serialize(ms, o);
return ms.ToArray();
}
}
I see in my C++ application that the code receives the message, but the C++ ZMQ classes fail to de-serialize it correctly. I have some Java code that send to the C++ code in the same way without issue. My question is, am I sending my object via ZMQ correctly in the above and if not what am I doing wrong?
Thanks for your time.
Here's your error:
I am serializing to my Taurus.Odds object to byte[] via the extension method
...
BinaryFormatter bf = new BinaryFormatter();
...
You seem to be unaware of what BinaryFormatter is. It is in no way related to ProtoBuf. The docs say the following:
Serializes and deserializes an object, or an entire graph of connected objects, in binary format.
This binary format is a .NET-specific implementation detail. And it's very rigid at that, with poor versioning support. It was mainly used in the .NET remoting days, and it's generally considered a bad idea to use it today, as there are much better serializers around.
As you can see, there's no way your C++ app could be able to read that, as it's not in protobuf format.
So throw this method away and replace it with some proper protobuf serializing code, as explained in the protobuf-net docs. You'll need to add [ProtoContract] and [ProtoMember] attributes in your objects. Then you could write something like:
public static byte[] ToByteArray<T>(this T o)
{
if (o == null)
return null;
using (MemoryStream ms = new MemoryStream())
{
ProtoBuf.Serializer.Serialize(ms, o);
return ms.ToArray();
}
}
I'm accessing the tomtom json api, and the api either returns me an array of objects, or a single object, when an error has happen.
Example:
[{"driverno": "...
Error Example:
{"errorCode": "8011","errorMsg": "request quota reached, error code: 8011"}
The data is accessed WebRequest, WebResponse and they return a stream, which can then be passed to a DataContractJsonSerializer. However, I can't create a serialization class, which accepts both forms of JSON, and the stream can't be passed twice, because the seek function is not supported.
Is there a way, to create a serialization class which supports both types of JSON input?
I found a workaround, where I copy the Stream to a MemoryStream, which enables seeking. I'm not completly settisfied with th solution, becuase it does a Stream copying and the DataContractJsonSerializer twice.
Sample:
string text = File.ReadAllText(PAHT);
text = Regex.Replace(text, "\\{[\\n\\r ]*\"__type", "{\"__type");
// copy to MemoryStream
using (MemoryStream dataStream = new MemoryStream(Encoding.UTF8.GetBytes(text)))
{
DataContractJsonSerializer errorDeserializer = new DataContractJsonSerializer(typeof(RequestError));
RequestError errorSerilaized = (RequestError)errorDeserializer.ReadObject(dataStream);
// check if an error happened
if (errorSerilaized.errorCode == null)
{
// seek the stream to position 0
dataStream.Position = 0;
DataContractJsonSerializer _deserializer = new DataContractJsonSerializer(typeof(NoneErrorSerializationClass));
NoneErrorSerializationClass tripReportsSerialized = (NoneErrorSerializationClass)_deserializer.ReadObject(dataStream);
// ...
}
else
{
MessageBox.Show(errorSerilaized.errorMsg);
}
}
I am creating a stream in C# and trying to read it in java, but I receive the error: "Protocol message tag had invalid wire type." when i read it in my java code the object created in c#.
Details:
I started from an equal .proto file (see below) to create the correspondent .java file and .cs file (compiling using the protoc for java in version "protobuf-2.4.1" and the protobuf-csharp-port-2.4.1.473-full-binaries for c#).
I succeed to create the addressbook.java and the addressbook.cs.
The object is created in c# and written to a file using the following c# code:
[...]
byte[] bytes;
//Create a builder to start building a message
Person.Builder newContact = Person.CreateBuilder();
//Set the primitive properties
newContact.SetId(1)
.SetName("Foo")
.SetEmail("foo#bar");
//Now add an item to a list (repeating) field
newContact.AddPhone(
//Create the child message inline
Person.Types.PhoneNumber.CreateBuilder().SetNumber("555-1212").Build()
);
//Now build the final message:
Person person = newContact.Build();
newContact = null;
using(MemoryStream stream = new MemoryStream())
{
//Save the person to a stream
person.WriteTo(stream);
bytes = stream.ToArray();
//save this to a file (by me)
ByteArrayToFile("personStreamFromC#", bytes);
[...]
I copy the created file "personStreamFromC#" to my java solution and try to read it using the following java code:
AddressBook.Builder addressBook = AddressBook.newBuilder();
// Read the existing address book.
try {
FileInputStream input = new FileInputStream(args[0]);
byte[] data = IOUtils.toByteArray(input);
addressBook.mergeFrom(data);
// Read the existing address book.
AddressBook addressBookToReadFrom =
AddressBook.parseFrom(new FileInputStream(args[0]));
Print(addressBookToReadFrom);
}
But I get the following message:
Exception in thread "main" com.google.protobuf.InvalidProtocolBufferException: Protocol message
tag had invalid wire type. at
com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:78)
at
com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:498)
at
com.google.protobuf.GeneratedMessage$Builder.parseUnknownField(GeneratedMessage.java:438)
at
com.example.tutorial.AddressBookProtos$Person$Builder.mergeFrom(AddressBookProtos.java:1034)
at
com.example.tutorial.AddressBookProtos$Person$Builder.mergeFrom(AddressBookProtos.java:1)
at
com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:275)
at
com.example.tutorial.AddressBookProtos$AddressBook$Builder.mergeFrom(AddressBookProtos.java:1715)
at
com.example.tutorial.AddressBookProtos$AddressBook$Builder.mergeFrom(AddressBookProtos.java:1)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:300)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
at
com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:162)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:716)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
at
com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:153)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:709)
at AddPerson.main(test.java:104)
Below the .proto file:
package tutorial;
message Person {
required string name = 1;
required int32 id = 2; // Unique ID number for this person.
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phone = 4;
}
message AddressBook {
repeated Person person = 1;
}
Any ideas ??
You write Person object to file in C#, but then read AddressBook in Java, I don't think this is correct. Try following in your Java code:
Person.parseFrom(new FileInputStream(args[0]));
One common mistake that can cause invalid wire-type errors (especially when using files) is: over-writing an existing file without truncating it. We can't see your ByteArrayToFile, but frankly File.WriteAllBytes may be an easier option. The problem is that if the new data is smaller than the original contents, any remaining extra bytes are essentially garbage.
My advice:
check if you can deserialize it in c#; if you can't, the error is certainly in the file handling
if it works in c#, check how you are getting the file to the java code: are you copying it around anywhere?
and check you are using binary (not text) processing at all stages
I have an object I'd like to serialize to a memory buffer, which is then sent via UART to an embedded device.
I'm working in a C# environment on windows.
What I'd like to do is to create two classes that look like this:
class StatusElement
{
byte statusPart1;
byte statusPart2;
}
class DeviceCommand
{
byte Address;
byte Length;
StatusElement[] statusElements; // Can have an arbitrary number of elements in it
}
I'd like to use a serialize, preferably something based on c# serialization, to convert the second class to a byte stream.
The problem is that the embedded device is hard-coded to accept an exact sequence (AddressByte, LengthByte .... ErrorCorrectionByte) so I cannot use the regular C# serialization, which adds serialization metadata in the stream. This also rules out other serializes like Protobuf.
So my question is:
Is it possible to customize the c# serialization to get the output I need? How?
--- Update ---
Thanks everyone for the help.
After consideration I’ve decided to implement my own mini-serializer, using reflection and per-type handler. More complex but gives me more flexibility and automation capabilities.
use a MemoryStream to manully serialize your object.
private byte[] Serialize()
{
using (var ms = new MemoryStream())
{
ms.WriteByte(Address);
ms.WriteByte(Length);
foreach (var element in statusElements)
{
ms.WriteByte(element.statusPart1);
ms.WriteByte(element.statusPart2);
}
return ms.ToArray();
}
}
Likewise for deserialization:
private static DeviceCommand Deserialize(byte[] input)
{
DeviceCommand result = new DeviceCommand();
using (var ms = new MemoryStream(input))
{
result.Address = ms.ReadByte();
result.Length = ms.ReadByte();
//assuming .Length contains the number of statusElements:
result.statusElemetns = new StatusElement[result.Length];
for (int i = 0; i < result.Length; i++)
{
result.statusElements[i] = new StatusElement();
result.statusElements[i].statusPart1 = ms.ReadByte();
result.statusElements[i].statusPart2 = ms.ReadByte();
}
}
return result;
}
If you need only to write bytes or byte arrays, you can use the MemoryStream directly. If you want to use other .NET base types, access your Stream with a System.IO.BinaryWriter / BinaryReader. This class is used by the System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
for binary serialization and deserialization.
I am trying to store some objects in an SQL Server Compact (SQL CE) database by serializing them with a SOAP formatter. Serializing seems to work just fine, but when I try to deserialize the object I get an error saying
There is an unclosed literal string. Line 53, position 72.
Furthermore, after restarting the application on attempting to fill the dataset I get the following error:
Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.
All my columns (except the ID) allow for null values and are non-unique, so I have no idea where this comes from. Here is the code of my serializer:
public static class Serializer
{
static public string Serialize(AssessmentReport theObject)
{
MemoryStream mStream = new MemoryStream();
SoapFormatter formatter = new SoapFormatter();
formatter.Serialize(mStream, theObject);
byte[] buffer = mStream.ToArray();
mStream.Close();
string value = Encoding.UTF8.GetString(buffer);
return value;
}
static public AssessmentReport Deserialize(string value)
{
byte[] buffer = Encoding.UTF8.GetBytes(value);
MemoryStream mStream = new MemoryStream(buffer);
SoapFormatter formatter = new SoapFormatter();
mStream.Position = 0;
AssessmentReport theReport = (AssessmentReport)formatter.Deserialize(mStream);
mStream.Close();
return theReport;
}
}
Here is how I call the serializer (theReport is an instance of the object to be serialized):
examTableAdapter.UpdateAsmFile(Serializer.Serialize(theReport), examID);
And here is how I am calling the deserializing method:
string value = Convert.ToString(examTableAdapter.GetAsmFile(2));
AsmReport theReport = Serializer.Deserialize(value)
The field in the SQL Server Compact database where the string is saved is of type nvarchar with a limit of 3500.
I tried using a binary formatter, but when serializing it seems to always return an empty byte[] buffer. I really need deep serializing, that's why the XML serializer is out of question.
Ok so this one I have been trying to figure out for more than two months now.
While I could not find a logical solution to the problem, it seems that changing the encoding to utf7 fixed the problem. I can't think of a reason why this would be the issue, but it seems to be specific to my machine (I finally had the opportunity to run the code on another computer and it worked perfectly with utf8).