I have an object I'd like to serialize to a memory buffer, which is then sent via UART to an embedded device.
I'm working in a C# environment on windows.
What I'd like to do is to create two classes that look like this:
class StatusElement
{
byte statusPart1;
byte statusPart2;
}
class DeviceCommand
{
byte Address;
byte Length;
StatusElement[] statusElements; // Can have an arbitrary number of elements in it
}
I'd like to use a serialize, preferably something based on c# serialization, to convert the second class to a byte stream.
The problem is that the embedded device is hard-coded to accept an exact sequence (AddressByte, LengthByte .... ErrorCorrectionByte) so I cannot use the regular C# serialization, which adds serialization metadata in the stream. This also rules out other serializes like Protobuf.
So my question is:
Is it possible to customize the c# serialization to get the output I need? How?
--- Update ---
Thanks everyone for the help.
After consideration I’ve decided to implement my own mini-serializer, using reflection and per-type handler. More complex but gives me more flexibility and automation capabilities.
use a MemoryStream to manully serialize your object.
private byte[] Serialize()
{
using (var ms = new MemoryStream())
{
ms.WriteByte(Address);
ms.WriteByte(Length);
foreach (var element in statusElements)
{
ms.WriteByte(element.statusPart1);
ms.WriteByte(element.statusPart2);
}
return ms.ToArray();
}
}
Likewise for deserialization:
private static DeviceCommand Deserialize(byte[] input)
{
DeviceCommand result = new DeviceCommand();
using (var ms = new MemoryStream(input))
{
result.Address = ms.ReadByte();
result.Length = ms.ReadByte();
//assuming .Length contains the number of statusElements:
result.statusElemetns = new StatusElement[result.Length];
for (int i = 0; i < result.Length; i++)
{
result.statusElements[i] = new StatusElement();
result.statusElements[i].statusPart1 = ms.ReadByte();
result.statusElements[i].statusPart2 = ms.ReadByte();
}
}
return result;
}
If you need only to write bytes or byte arrays, you can use the MemoryStream directly. If you want to use other .NET base types, access your Stream with a System.IO.BinaryWriter / BinaryReader. This class is used by the System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
for binary serialization and deserialization.
Related
The websocket client is returning a ReadOnlyMemory<byte>.
The issue is that JsonDocument.Parse fails due to the fact that the buffer has been compressed. I've got to decompress it somehow before I parse it. How do I do that? I cannot really change the websocket library code.
What I want is something like public Func<ReadOnlyMemory<byte>> DataInterpreterBytes = () => which optionally decompresses these bytes out of this class. How do I do that? Is it possible to decompress ReadOnlyMemory<byte> and if the handler is unused to basically to do nothing.
private static string DecompressData(byte[] byteData)
{
using var decompressedStream = new MemoryStream();
using var compressedStream = new MemoryStream(byteData);
using var deflateStream = new GZipStream(compressedStream, CompressionMode.Decompress);
deflateStream.CopyTo(decompressedStream);
decompressedStream.Position = 0;
using var streamReader = new StreamReader(decompressedStream);
return streamReader.ReadToEnd();
}
Snippet
private void OnMessageReceived(object? sender, MessageReceivedEventArgs e)
{
var timestamp = DateTime.UtcNow;
_logger.LogTrace("Message was received. {Message}", Encoding.UTF8.GetString(e.Message.Buffer.Span));
// We dispose that object later on
using var document = JsonDocument.Parse(e.Message.Buffer);
var tokenData = document.RootElement;
So, if you had a byte array, you'd do this:
private static JsonDocument DecompressData(byte[] byteData)
{
using var compressedStream = new MemoryStream(byteData);
using var deflateStream = new GZipStream(compressedStream, CompressionMode.Decompress);
return JsonDocument.Parse(deflateStream);
}
This is similar to your snippet above, but no need for the intermediate copy: just read straight from the GzipStream. JsonDocument.Parse also has an overload that takes a stream, so you can use that and avoid yet another useless copy.
Unfortunately, you don't have a byte array, you have a ReadOnlyMemory<byte>. There is no way out of the box to create a memory stream out of a ReadOnlyMemory<byte>. Honestly, it feels like an oversight, like they forgot to put that feature into .NET.
So here are your options instead.
The first option is to just convert the ReadOnlyMemory<byte> object to an array with ToArray():
// assuming e.Message.Buffer is a ReadOnlyMemory<byte>
using var document = DecompressData(e.Message.Buffer.ToArray());
This is really straightforward, but remember it actually copies the data, so for large documents it might not be a good idea if you want to avoid using too much memory.
The second is to try and extract the underlying array from the memory. This can be achieved with MemoryMarshal.TryGetArray, which gives you an ArraySegment (but might fail if the memory isn't actually a managed array).
private static JsonDocument DecompressData(ReadOnlyMemory<byte> byteData)
{
if(MemoryMarshal.TryGetArray(byteData, out var segment))
{
using var compressedStream = new MemoryStream(segment.Array, segment.Offset, segment.Count);
// rest of the code goes here
}
else
{
// Welp, this memory isn't actually an array, so... tough luck?
}
}
The third way might feel dirty, but if you're okay with using unsafe code, you can just pin the memory's span and then use UnmanagedMemoryStream:
private static unsafe JsonDocument DecompressData(ReadOnlyMemory<byte> byteData)
{
fixed (byte* ptr = byteData.Span)
{
using var compressedStream = new UnmanagedMemoryStream(ptr, byteData.Length);
using var deflateStream = new GZipStream(compressedStream, CompressionMode.Decompress);
return JsonDocument.Parse(deflateStream);
}
}
The other solution is to write your own Stream class that supports this. The Windows Community Toolkit has an extension method that returns a Stream wrapper around the memory object. If you're not okay with using an entire third party library just for that, you can probably just roll your own, it's not that much code.
In the following code i want to use a predefined protobuf message in c#. I found that I was able to write and use the method to take a method that has been created and make a byte[]:
ContainerMessage containerMessage = new ContainerMessage();
containerMessage.Type = CommandType.Connect;
containerMessage.Connect = new Connect();
containerMessage.Connect.ClientName = "TemporaryClientName";
byte[] stream = new byte[containerMessage.CalculateSize()];
using (Google.Protobuf.CodedOutputStream outstream = new Google.Protobuf.CodedOutputStream(stream))
{
containerMessage.WriteTo(outstream);
}
This works as expected and i can inspect the message and the values are as expected as are the values in the byte[]. But if I try to Deserialize even this simple byte[] that i have just created:
using (Google.Protobuf.CodedInputStream instream = new Google.Protobuf.CodedInputStream(stream))
{
instream.ReadMessage(containerMessage);
}
It fails with:
An unhandled exception of type 'Google.Protobuf.InvalidProtocolBufferException' occurred in Google.Protobuf.dll
Additional information: Protocol message contained an invalid tag (zero).
Is this way of deserializing from a byte[] correct for protobuf?
The Protobuf Definition is:
message ContainerMessage {
CommandType type = 1;
bool commandSuccess = 2;
oneof message {
Connect connect = 3;
}
}
enum CommandType {
START = 0;
CONNECT = 2;
}
message Connect {
string ClientName = 1;
uint32 PushPullPort = 2;
}
And the CS file is generated with the command line:
protoc.exe -I=%CD% --csharp_out=..\GeneratedCsMessaging\ Connect.proto
The CodedOutputStream and CodedInputStream are mainly intended to be used by the compiled proto classes. The API for CodedOutputStream states such and mentions that if you want to have manually-written code calling either of both classes you need to use their WriteTag method before each value.
However, since you want to use the Google Protobuf for serializing and parsing any System.IO.Stream will do the job just like intended. This is very well documented and described in the Parsing and serialization section of the Protocol Buffer Basics for C#. The examples which can be found in Google Protobuf's Github can be quite helpful for getting the hang of Google Protobuf quickly. There you can see that a MemoryStream is used to serialize the object while the Parse.ParseFrom method can be used to parse an object out of the serialized data.
As you've mentioned in the comments to your question using Google.Protobuf; is an essential part to be able to use Google Protobuf's features.
EDIT: A sample usage in your case could look something like this
byte[] serializedBytes;
ContainerMessage containerMessage = new ContainerMessage()
{
Connect = new Connect()
{
ClientName = "TemporaryClientName",
},
Type = CommandType.Connect,
};
using( MemoryStream stream = new MemoryStream())
{
containerMessage.WriteTo(stream);
serializedBytes = stream.ToArray();
}
ContainerMessage parsedCopy = ContainerMessage.Parser.ParseFrom(serializedBytes);
I have a protobuf object that I am sending from a C# application (using clrZmq) to a C++ service (using the zmq C++ bindings) on a local machine (for testing). I attempt to send my object from C# using the following
Taurus.Odds odds = Util.GetFakeOdds();
using (var context = ZmqContext.Create())
using (var socket = context.CreateSocket(SocketType.REQ))
{
byte[] buffer = null;
socket.Connect(TARGET); // TARGET = "tcp://127.0.0.1:6500"
Taurus.FeedMux mux = new Taurus.FeedMux();
mux.type = Taurus.FeedMux.Type.ODDS;
mux.odds = odds;
SendStatus status = socket.Send(mux.ToByteArray());
if (status == SendStatus.Sent)
{
int i;
byte[] arr = socket.Receive(buffer, SocketFlags.None, out i);
Taurus.Bet bet = buffer.ToObject<Taurus.Bet>();
}
...
}
Where I am serializing to my Taurus.Odds object to byte[] via the extension method
public static byte[] ToByteArray(this object o)
{
if(o == null)
return null;
BinaryFormatter bf = new BinaryFormatter();
using (MemoryStream ms = new MemoryStream())
{
bf.Serialize(ms, o);
return ms.ToArray();
}
}
I see in my C++ application that the code receives the message, but the C++ ZMQ classes fail to de-serialize it correctly. I have some Java code that send to the C++ code in the same way without issue. My question is, am I sending my object via ZMQ correctly in the above and if not what am I doing wrong?
Thanks for your time.
Here's your error:
I am serializing to my Taurus.Odds object to byte[] via the extension method
...
BinaryFormatter bf = new BinaryFormatter();
...
You seem to be unaware of what BinaryFormatter is. It is in no way related to ProtoBuf. The docs say the following:
Serializes and deserializes an object, or an entire graph of connected objects, in binary format.
This binary format is a .NET-specific implementation detail. And it's very rigid at that, with poor versioning support. It was mainly used in the .NET remoting days, and it's generally considered a bad idea to use it today, as there are much better serializers around.
As you can see, there's no way your C++ app could be able to read that, as it's not in protobuf format.
So throw this method away and replace it with some proper protobuf serializing code, as explained in the protobuf-net docs. You'll need to add [ProtoContract] and [ProtoMember] attributes in your objects. Then you could write something like:
public static byte[] ToByteArray<T>(this T o)
{
if (o == null)
return null;
using (MemoryStream ms = new MemoryStream())
{
ProtoBuf.Serializer.Serialize(ms, o);
return ms.ToArray();
}
}
I've seen this type of question asked before but not sure what the root cause of the problem was or how to fix it.
I am modifying an existing class to be able to load data into the member variables from flash. Right now, the class loads data from a file through the load function. This function has been overloaded to take in byte array.
The data read back from the flash is put into this byte array.
The error that is thrown is (happens at the line ... = formatter.Deserialize(stream)):
The input stream is not a valid binary format. The starting contents (in bytes) are: 93-E3-E6-3F-C3-F5-E4-41-00-C0-8D-C3-14-EE-4A-C3-00 ...
The interesting thing here is that the contents are exactly the contents of the byte array that is being passed into the stream. In other words, this is the data from the flash and this is exactly what I want serialized. I'm not sure why the error is being thrown.
Or a better question is what is a is a valid binary format for a BinaryFormatter? Does it need a certain size? Is there specific end value needed? Are certain values invalid? The current size of the byte array input is 24 bytes.
Code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Drawing.Imaging;
using System.Windows.Media.Imaging;
using System.IO;
using Galileo.Data;
using System.Xml.Serialization;
using System.Reflection;
using System.Runtime.InteropServices;
using UComm;
using System.Runtime.Serialization.Formatters.Binary;
using ULog;
public void Load(byte[] spCalfromPrimary)
{
try
{
Type settingsType = this.GetType();
object tmp = Activator.CreateInstance(settingsType);
Stream stream = new MemoryStream();
stream.Write(spCalfromPrimary, 0, spCalfromPrimary.Length);
stream.Position = 0;
BinaryFormatter formatter = new BinaryFormatter();
//tmp = formatter.Deserialize(stream);
formatter.Deserialize(stream); //**<--- throws error here**
// Use reflection to copy all public properties from the temporary object into this one.
PropertyInfo[] properties = settingsType.GetProperties();
foreach (PropertyInfo property in properties)
{
object value = property.GetValue(tmp, null);
if (value == null)
{
throw new FileFormatException("Null value encountered in settings file");
}
else
{
property.SetValue(this, value, null);
}
}
}
catch (Exception ex)
{
_logger.DebugException("Failed to load spatial cal value from FW", ex);
Console.WriteLine(ex.Message);
}
}
// <summary>
/// Loads the setting from file
/// </summary>
public void Load()
{
Type settingsType = this.GetType();
XmlSerializer xser = new XmlSerializer(settingsType);
object tmp = Activator.CreateInstance(settingsType);
using (StreamReader reader = new StreamReader(_filename)) { tmp = xser.Deserialize(reader); }
// Use reflection to copy all public properties from the temporary object into this one.
PropertyInfo[] properties = settingsType.GetProperties();
foreach (PropertyInfo property in properties)
{
object value = property.GetValue(tmp, null);
if (value == null)
{
throw new FileFormatException("Null value encountered in settings file");
}
else
{
property.SetValue(this, value, null);
}
}
}
Note that I have also tried the a Convert byte array to object function (I found on stackoverflow). When I used this function, an exception was still thrown at .Deserialize(memStream).
// Convert a byte array to an Object
private Object ByteArrayToObject(byte[] arrBytes)
{
MemoryStream memStream = new MemoryStream();
BinaryFormatter binForm = new BinaryFormatter();
memStream.Write(arrBytes, 0, arrBytes.Length);
memStream.Seek(0, SeekOrigin.Begin);
Object obj = (Object) binForm.Deserialize(memStream);
return obj;
}
Apparently I left out some important information.
Serialization happens in a different application from deserialization. Serialization uses a bitconverter to take the data, convert to a byte array and upload it to flash. Let me explain. The data that is being serialized / deserialized & stored in flash is calibration data. Calibration is performed at the factory with Application1 by production. This uses a bitconverter to put every field into a stream and then serialize the stream.
CFlatInterface.FloatToStream(bData, ref i, rtsMappingData.ScaleTrackingDMD);
CFlatInterface.FloatToStream(bData, ref i, rtsMappingData.RotationAngle);
CFlatInterface.FloatToStream(bData, ref i, rtsMappingData.CenterOfRotation.dx);
where the function FloatToStream is defined as:
public static void FloatToStream(byte[] buf, ref int index, float val)
{
Buffer.BlockCopy(BitConverter.GetBytes(val), 0, buf, index, sizeof(float));
index += sizeof(float);
}
So every field that makes up the Calibration is put into the stream this way. The data is put into the stream and a byte array is constructed and sent to flash.
On the other side, once the product is out of the factory and in use, Application2 (user application) has a Calibration object that has all the calibration fields. This reads the flash, and gets the data that was written by Application1. Application2 is trying to deserialize the calibration data using BinaryFormatter and the code above. I am coming to the conclusion this is not possible (Thanks Rotem). The correct course of action is to use the same formatter for both serialization / deserialization - i will implement it this way and indicate if that makes a difference.
Following your update, the obvious issue is that you are serializing and deserializing with different formatters.
BinaryFormatter serializes more than just field data. It also serializes type information and metadata so it knows how to deserialize the objects, so it is expecting more than just raw data bytes as input.
Either use a BinaryFormatter on the serializing end as well, or use a manual deserialization technique on the receiving end.
I need to generate an XML file and i need to stick as much data into it as possible BUT there is a filesize limit. So i need to keep inserting data until something says no more. How do i figure out the XML file size without repeatably writing it to file?
I agree with John Saunders. Here's some code that will basically do what he's talking about but as an XmlSerializer except as a FileStream and uses a MemoryStream as intermediate storage. It may be more effective to extend stream though.
public class PartitionedXmlSerializer<TObj>
{
private readonly int _fileSizeLimit;
public PartitionedXmlSerializer(int fileSizeLimit)
{
_fileSizeLimit = fileSizeLimit;
}
public void Serialize(string filenameBase, TObj obj)
{
using (var memoryStream = new MemoryStream())
{
// serialize the object in the memory stream
using (var xmlWriter = XmlWriter.Create(memoryStream))
new XmlSerializer(typeof(TObj))
.Serialize(xmlWriter, obj);
memoryStream.Seek(0, SeekOrigin.Begin);
var extensionFormat = GetExtensionFormat(memoryStream.Length);
var buffer = new char[_fileSizeLimit];
var i = 0;
// split the stream into files
using (var streamReader = new StreamReader(memoryStream))
{
int readLength;
while ((readLength = streamReader.Read(buffer, 0, _fileSizeLimit)) > 0)
{
var filename
= Path.ChangeExtension(filenameBase,
string.Format(extensionFormat, i++));
using (var fileStream = new StreamWriter(filename))
fileStream.Write(buffer, 0, readLength);
}
}
}
}
/// <summary>
/// Gets the a file extension formatter based on the
/// <param name="fileLength">length of the file</param>
/// and the max file length
/// </summary>
private string GetExtensionFormat(long fileLength)
{
var numFiles = fileLength / _fileSizeLimit;
var extensionLength = Math.Ceiling(Math.Log10(numFiles));
var zeros = string.Empty;
for (var j = 0; j < extensionLength; j++)
{
zeros += "0";
}
return string.Format("xml.part{{0:{0}}}", zeros);
}
}
To use it, you'd initialize it with the max file length and then serialize using the base file path and then the object.
public class MyType
{
public int MyInt;
public string MyString;
}
public void Test()
{
var myObj = new MyType { MyInt = 42,
MyString = "hello there this is my string" };
new PartitionedXmlSerializer<MyType>(2)
.Serialize("myFilename", myObj);
}
This particular example will generate an xml file partitioned into
myFilename.xml.part001
myFilename.xml.part002
myFilename.xml.part003
...
myFilename.xml.part110
In general, you cannot break XML documents at arbitrary locations, even if you close all open tags.
However, if what you need is to split an XML document over multiple files, each of no more than a certain size, then you should create your own subtype of the Stream class. This "PartitionedFileStream" class could write to a particular file, up to the size limit, then create a new file, and write to that file, up to the size limit, etc.
This would leave you with multiple files which, when concatenated, make up a valid XML document.
In the general case, closing tags will not work. Consider an XML format that must contain one element A followed by one element B. If you closed the tags after writing element A, then you do not have a valid document - you need to have written element B.
However, in the specific case of a simple site map file, it may be possible to just close the tags.
You can ask the XmlTextWriter for it's BaseStream, and check it's Position.
As the other's pointed out, you may need to reserve some headroom to properly close the Xml.