I'm creating a program which has to send data between a client and server efficiently. To organize packets clearly, I'm using serialization. However, when I serialize these packets the data is unnecessarily large. I'll explain what I'm doing so that you can understand what I need.
My packet classes work like this. I have a Packet object:
using System;
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;
[Serializable]
public class Packet
{
public static byte[] Serialize(Object o)
{
MemoryStream ms = new MemoryStream();
BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(ms, o);
return ms.ToArray();
}
public static Object Deserialize(byte[] bt)
{
MemoryStream ms = new MemoryStream();
BinaryFormatter bf = new BinaryFormatter();
ms.Write(bt, 0, bt.Length);
ms.Position = 0;
object obj = bf.Deserialize(ms);
ms.Close();
return obj;
}
}
I can then create other classes that inherit from the Packet class, here's an example:
using System;
[Serializable]
public class PacketUserInfo : Packet
{
public string Name;
public int Age;
}
Then, it's very simple to put this into a byte array and send it (Of course the above packet is merely an example). However, the size of the resulting array is at least 10 times larger than it would be if I was to use a BinaryWriter and manually write the information.
Why is the serialized data so large? Is there any way to decrease it while still keeping everything organized with packets as their own classes?
Note: I'm only intending to serialize simple properties like this, nothing fancy.
Where you say "Why is the serialized data [...] larger than it would be if I was to use a BinaryWriter and manually write the information", with information you mean property values. The serializer you use however, serializes not only the data, but also some information about the class. You can see this by viewing the serialized data in a text editor.
Is there any way to decrease it while still keeping everything organized with packets as their own classes?
Use more specialized serialization, like protobuf or the library suggested by #Piotr.
Also I think your serialization code should not reside in the Packet base class, but rather in a separate class, like PacketEncoder.
Related
I have a protobuf object that I am sending from a C# application (using clrZmq) to a C++ service (using the zmq C++ bindings) on a local machine (for testing). I attempt to send my object from C# using the following
Taurus.Odds odds = Util.GetFakeOdds();
using (var context = ZmqContext.Create())
using (var socket = context.CreateSocket(SocketType.REQ))
{
byte[] buffer = null;
socket.Connect(TARGET); // TARGET = "tcp://127.0.0.1:6500"
Taurus.FeedMux mux = new Taurus.FeedMux();
mux.type = Taurus.FeedMux.Type.ODDS;
mux.odds = odds;
SendStatus status = socket.Send(mux.ToByteArray());
if (status == SendStatus.Sent)
{
int i;
byte[] arr = socket.Receive(buffer, SocketFlags.None, out i);
Taurus.Bet bet = buffer.ToObject<Taurus.Bet>();
}
...
}
Where I am serializing to my Taurus.Odds object to byte[] via the extension method
public static byte[] ToByteArray(this object o)
{
if(o == null)
return null;
BinaryFormatter bf = new BinaryFormatter();
using (MemoryStream ms = new MemoryStream())
{
bf.Serialize(ms, o);
return ms.ToArray();
}
}
I see in my C++ application that the code receives the message, but the C++ ZMQ classes fail to de-serialize it correctly. I have some Java code that send to the C++ code in the same way without issue. My question is, am I sending my object via ZMQ correctly in the above and if not what am I doing wrong?
Thanks for your time.
Here's your error:
I am serializing to my Taurus.Odds object to byte[] via the extension method
...
BinaryFormatter bf = new BinaryFormatter();
...
You seem to be unaware of what BinaryFormatter is. It is in no way related to ProtoBuf. The docs say the following:
Serializes and deserializes an object, or an entire graph of connected objects, in binary format.
This binary format is a .NET-specific implementation detail. And it's very rigid at that, with poor versioning support. It was mainly used in the .NET remoting days, and it's generally considered a bad idea to use it today, as there are much better serializers around.
As you can see, there's no way your C++ app could be able to read that, as it's not in protobuf format.
So throw this method away and replace it with some proper protobuf serializing code, as explained in the protobuf-net docs. You'll need to add [ProtoContract] and [ProtoMember] attributes in your objects. Then you could write something like:
public static byte[] ToByteArray<T>(this T o)
{
if (o == null)
return null;
using (MemoryStream ms = new MemoryStream())
{
ProtoBuf.Serializer.Serialize(ms, o);
return ms.ToArray();
}
}
BinaryFormatter is able to handle serialization simply:
private byte[] TokenToBytes(SessionSecurityToken token)
{
if (token == null)
{
return null;
}
using (var memoryStream = new MemoryStream())
{
var binaryFormatter = new BinaryFormatter();
binaryFormatter.Serialize(memoryStream, token);
return memoryStream.ToArray();
}
}
When I tried replacing BinaryFormatter with protobuf-net:
using (var memoryStream = new MemoryStream())
{
Serializer.Serialize(memoryStream, token);
return memoryStream.ToArray();
}
I get the following exception:
Type is not expected, and no contract can be inferred:
System.IdentityModel.Tokens.SessionSecurityToken
I tried adding:
RuntimeTypeModel.Default.Add(typeof(SessionSecurityToken), true);
Which gets past the exception but I now get a zero byte array.
How do I properly configure protobuf-net to serialize a SessionSecurityToken?
On the flipside, SessionSecurityToken does not have a parameterless constructor.
using (var memoryStream = new MemoryStream(tokenAsBytes))
{
return Serializer.Deserialize<SessionSecurityToken>(memoryStream);
}
Throws a ProtoException:
No parameterless constructor found for SessionSecurityToken
BinaryFormatter is able to do it without any fuss:
using (var memoryStream = new MemoryStream(bytes))
{
var binaryFormatter = new BinaryFormatter();
return (SessionSecurityToken)binaryFormatter.Deserialize(memoryStream);
}
How do I properly configure protobuf-net to deserialize a SessionSecurityToken?
protobuf-net does not claim to be able to serialize every single type; indeed, you would have great difficulty serializing that via most serializers (XmlSerializer, any of the json serializers, DataContractSerializer, etc). BinaryFormatter is in a different category of serializers - and in this particular case, implements custom serialization via ISerializable.GetObjectData(SerializationInfo, StreamingContext).
The constructor thing is a red herring; actually, protobuf-net can bypass constructors completely, and in this particular scenario BinaryFormatter is using a custom serialization constructor via .ctor(SerializationInfo, StreamingContext).
For simple cases, protobuf-net can be configured via attributes or runtime options; for more complex scenarios, surrogates can be used to map between representations - however, in this case I would suggest (looking at the implementation of SessionSecurityToken) that this is more complex than you probably want to maintain.
I would step back a step or two here; most serializers are designed to work with data, not implementation - and work great with DTOs etc. SessionSecurityToken is very much not a DTO, and there is no simple way of switching between them. My strong suggestion here would be: serialize what this represents, not what it is. However, if this is part of an existing complex model and is really really hard to separate out, you could switch back to BinaryFormatter for those bits. I haven't tested this, but consider:
RuntimeTypeModel.Default.Add(typeof(SessionSecurityToken), false)
.SetSurrogate(typeof(BinaryFormatterSurrogate<SessionSecurityToken>));
With:
[ProtoContract]
public class BinaryFormatterSurrogate<T>
{
[ProtoMember(1)]
public byte[] Raw { get; set; }
public static explicit operator T(BinaryFormatterSurrogate<T> value)
{
if(value==null || value.Raw == null) return default(T);
using(var ms = new MemoryStream(value.Raw))
{
return (T)new BinaryFormatter().Deserialize(ms);
}
}
public static explicit operator BinaryFormatterSurrogate<T>(T value)
{
object obj = value;
if (obj == null) return null;
using (var ms = new MemoryStream())
{
new BinaryFormatter().Serialize(ms, obj);
return new BinaryFormatterSurrogate<T> { Raw = ms.ToArray() };
}
}
}
Keep in mind that this simply embeds the output of one serializer as raw data inside another. Fortunately protobuf-net is happy talking binary, so this won't add any noticeable overhead (just the header and length-prefix for the blob) - but it also won't do anything particularly smart or clever with the SessionSecurityToken instances. If this is the only thing you are serializing, it really isn't worth it. If this is just one ugly bump in a larger DTO model, where most of it can serialize nicely - then it might get the job done for you.
Using binary formatting for 1st time in .net C#
Code from MSDN is like this:
IFormatter formatter = new BinaryFormatter();
Stream stream = new FileStream("MyFile.lvl", FileMode.Create, FileAccess.Write,FileShare.None);
formatter.Serialize(stream, Globals.CurrentLevel);
stream.Close();
Just wondering should I store an IFormatter in a field in my class and use it over and over again or should I do as above and instantiate a new one every time I save/load something?
I noticed it is not IDisposable.
There's very little overhead in re-creating a BinaryFormatter, most of the properties it sets in the constructor are enums, see here (thanks to Reflector):
public BinaryFormatter()
{
this.m_typeFormat = FormatterTypeStyle.TypesAlways;
this.m_securityLevel = TypeFilterLevel.Full;
this.m_surrogates = null;
this.m_context = new StreamingContext(StreamingContextStates.All);
}
If you were going to re-use it though, you'd need to synchronize access to the Serialize and Deserialize methods to keep them thread-safe.
I have an object I'd like to serialize to a memory buffer, which is then sent via UART to an embedded device.
I'm working in a C# environment on windows.
What I'd like to do is to create two classes that look like this:
class StatusElement
{
byte statusPart1;
byte statusPart2;
}
class DeviceCommand
{
byte Address;
byte Length;
StatusElement[] statusElements; // Can have an arbitrary number of elements in it
}
I'd like to use a serialize, preferably something based on c# serialization, to convert the second class to a byte stream.
The problem is that the embedded device is hard-coded to accept an exact sequence (AddressByte, LengthByte .... ErrorCorrectionByte) so I cannot use the regular C# serialization, which adds serialization metadata in the stream. This also rules out other serializes like Protobuf.
So my question is:
Is it possible to customize the c# serialization to get the output I need? How?
--- Update ---
Thanks everyone for the help.
After consideration I’ve decided to implement my own mini-serializer, using reflection and per-type handler. More complex but gives me more flexibility and automation capabilities.
use a MemoryStream to manully serialize your object.
private byte[] Serialize()
{
using (var ms = new MemoryStream())
{
ms.WriteByte(Address);
ms.WriteByte(Length);
foreach (var element in statusElements)
{
ms.WriteByte(element.statusPart1);
ms.WriteByte(element.statusPart2);
}
return ms.ToArray();
}
}
Likewise for deserialization:
private static DeviceCommand Deserialize(byte[] input)
{
DeviceCommand result = new DeviceCommand();
using (var ms = new MemoryStream(input))
{
result.Address = ms.ReadByte();
result.Length = ms.ReadByte();
//assuming .Length contains the number of statusElements:
result.statusElemetns = new StatusElement[result.Length];
for (int i = 0; i < result.Length; i++)
{
result.statusElements[i] = new StatusElement();
result.statusElements[i].statusPart1 = ms.ReadByte();
result.statusElements[i].statusPart2 = ms.ReadByte();
}
}
return result;
}
If you need only to write bytes or byte arrays, you can use the MemoryStream directly. If you want to use other .NET base types, access your Stream with a System.IO.BinaryWriter / BinaryReader. This class is used by the System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
for binary serialization and deserialization.
I am trying to learn udp sockets etc.... I created two programs server and client. The client sends a packet to the server, the server bounces it back.
This is the code I use in both programs for converting the data to and from a byte[]
but I am getting an error when converting from byte[]
public static Packet Open(byte[] b)
{
MemoryStream memStream = new MemoryStream();
BinaryFormatter binForm = new BinaryFormatter();
memStream.Write(b, 0, b.Length);
memStream.Seek(0, SeekOrigin.Begin);
object obj = new object();
try
{
// this line here is where the error is occurring
obj = (object)binForm.Deserialize(memStream);
}
catch (Exception er)
{
MessageBox.Show(er.Message);
}
if (obj is Packet)
return (Packet)obj;
else
return null;
}
public byte[] Bundle()
{
BinaryFormatter bf = new BinaryFormatter();
MemoryStream ms = new MemoryStream();
bf.Serialize(ms, this);
return ms.ToArray();
}
If I do this, all from one program it works
Packet p =new Packet();
p.Message="hello";
byte[] data = p.Bundle();
Packet p2 = Packet.Open(data);
MessageBox.Show(p2.Message);
The error I am receiving is "unable to find assembly in "the name of my client program"
AnyIdeas?
It sounds to me like you are serializing a type that is not shared via a reference between both ends. Note: it is not sufficient to have the same class compiled into both, since BinaryFormatter includes the full type name including the assembly, so: it will still count as an unrelated type. The common fix there (and I use the word "fix" entirely incorrectly) is to write an assembly for the DTO and reference that assembly from both client and server. This approach still has many issues, though.
For info, there are other serializers that are compatible with just having a similar class at each end. I'm biased, but I would suggest having a look at protobuf-net; the output is usually significantly smaller, and it isn't tied to the type, meaning the class just has to be broadly similar at each end (it is very version tolerant). Plus it is faster (CPU-wise) too!