Develop Reference

c# - More efficient serialization for packets

I'm creating a program which has to send data between a client and server efficiently. To organize packets clearly, I'm using serialization. However, when I serialize these packets the data is unnecessarily large. I'll explain what I'm doing so that you can understand what I need.
My packet classes work like this. I have a Packet object:
using System;
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;
[Serializable]
public class Packet
{
public static byte[] Serialize(Object o)
{
MemoryStream ms = new MemoryStream();
BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(ms, o);
return ms.ToArray();
}
public static Object Deserialize(byte[] bt)
{
MemoryStream ms = new MemoryStream();
BinaryFormatter bf = new BinaryFormatter();
ms.Write(bt, 0, bt.Length);
ms.Position = 0;
object obj = bf.Deserialize(ms);
ms.Close();
return obj;
}
}
I can then create other classes that inherit from the Packet class, here's an example:
using System;
[Serializable]
public class PacketUserInfo : Packet
{
public string Name;
public int Age;
}
Then, it's very simple to put this into a byte array and send it (Of course the above packet is merely an example). However, the size of the resulting array is at least 10 times larger than it would be if I was to use a BinaryWriter and manually write the information.
Why is the serialized data so large? Is there any way to decrease it while still keeping everything organized with packets as their own classes?
Note: I'm only intending to serialize simple properties like this, nothing fancy.

Where you say "Why is the serialized data [...] larger than it would be if I was to use a BinaryWriter and manually write the information", with information you mean property values. The serializer you use however, serializes not only the data, but also some information about the class. You can see this by viewing the serialized data in a text editor.
Is there any way to decrease it while still keeping everything organized with packets as their own classes?
Use more specialized serialization, like protobuf or the library suggested by #Piotr.
Also I think your serialization code should not reside in the Packet base class, but rather in a separate class, like PacketEncoder.

Why does my protobuf-net stream not work?

I have an object that can be serialized and deserialized but upon deserialization it throws me an error:
Invalid field in source data: 0
I don't know why this is happening
code for de-serialization and receiving:
public void listenUDP()
{
EndPoint ep = (EndPoint)groupEP;
//BinaryFormatter bf = new BinaryFormatter();
recieving_socket.Bind(ep);
while (true)
{
byte[] objData = new byte[65535];
recieving_socket.ReceiveFrom(objData, ref ep);
MemoryStream ms = new MemoryStream();
ms.Write(objData, 0, objData.Length);
ms.Seek(0, SeekOrigin.Begin);
messageHandle(ProtoBuf.Serializer.Deserialize<SimplePacket>(ms));
ms.Dispose();
}
}
Code for serialization:
public void sendDataUDP(Vec2f[] data)
{
SimplePacket packet = new SimplePacket(DateTime.UtcNow, data);
//IFormatter formatter = new BinaryFormatter();
MemoryStream stream = new MemoryStream();
System.Diagnostics.Stopwatch st = System.Diagnostics.Stopwatch.StartNew();
//formatter.Serialize(stream, data);
ProtoBuf.Serializer.Serialize<SimplePacket>(stream, packet);
//Console.WriteLine(st.ElapsedTicks);
stream.Close();
st.Restart();
sending_socket.SendTo(stream.ToArray(), sending_end_point);
//Console.WriteLine(st.ElapsedTicks);
st.Stop();
}

The root object in a protobuf message, as defined by the google specification, does not include any notion of the end of the message. This is intentional, so that concatenation is identical to merging two fragments. Consequently, the consuming code needs to restrict itself to a single message. This is identical between all protobuf implementations, and is not specific for protobuf-net.
What is happening is that your buffer is currently oversized, with garbage at the end. Currently (because you are reading one message) that garbage is most likely all zeros, and a zero is not a valid marker for a field. However, when re-using the buffer the garbage could be... anything.
In your case, probably the easiest way to do this is to use the SerializeWithLengthPrefix / DeserializeWithLengthPrefix methods, which handle all this for you by prepending the payload length at the start of the message, and only processing that much data.
As a final thought: it is not clear to me that your code will guarantee that is has read an entire message; a single receive could (on TCP, at least) return part of a message - or 2 and a bit messages, etc: TCP is stream-based, not message-based.

How do you configure protobuf-net's RuntimeModel.Default to support serializing/deserializing a SessionSecurityToken?

BinaryFormatter is able to handle serialization simply:
private byte[] TokenToBytes(SessionSecurityToken token)
{
if (token == null)
{
return null;
}
using (var memoryStream = new MemoryStream())
{
var binaryFormatter = new BinaryFormatter();
binaryFormatter.Serialize(memoryStream, token);
return memoryStream.ToArray();
}
}
When I tried replacing BinaryFormatter with protobuf-net:
using (var memoryStream = new MemoryStream())
{
Serializer.Serialize(memoryStream, token);
return memoryStream.ToArray();
}
I get the following exception:
Type is not expected, and no contract can be inferred:
System.IdentityModel.Tokens.SessionSecurityToken
I tried adding:
RuntimeTypeModel.Default.Add(typeof(SessionSecurityToken), true);
Which gets past the exception but I now get a zero byte array.
How do I properly configure protobuf-net to serialize a SessionSecurityToken?
On the flipside, SessionSecurityToken does not have a parameterless constructor.
using (var memoryStream = new MemoryStream(tokenAsBytes))
{
return Serializer.Deserialize<SessionSecurityToken>(memoryStream);
}
Throws a ProtoException:
No parameterless constructor found for SessionSecurityToken
BinaryFormatter is able to do it without any fuss:
using (var memoryStream = new MemoryStream(bytes))
{
var binaryFormatter = new BinaryFormatter();
return (SessionSecurityToken)binaryFormatter.Deserialize(memoryStream);
}
How do I properly configure protobuf-net to deserialize a SessionSecurityToken?

protobuf-net does not claim to be able to serialize every single type; indeed, you would have great difficulty serializing that via most serializers (XmlSerializer, any of the json serializers, DataContractSerializer, etc). BinaryFormatter is in a different category of serializers - and in this particular case, implements custom serialization via ISerializable.GetObjectData(SerializationInfo, StreamingContext).
The constructor thing is a red herring; actually, protobuf-net can bypass constructors completely, and in this particular scenario BinaryFormatter is using a custom serialization constructor via .ctor(SerializationInfo, StreamingContext).
For simple cases, protobuf-net can be configured via attributes or runtime options; for more complex scenarios, surrogates can be used to map between representations - however, in this case I would suggest (looking at the implementation of SessionSecurityToken) that this is more complex than you probably want to maintain.
I would step back a step or two here; most serializers are designed to work with data, not implementation - and work great with DTOs etc. SessionSecurityToken is very much not a DTO, and there is no simple way of switching between them. My strong suggestion here would be: serialize what this represents, not what it is. However, if this is part of an existing complex model and is really really hard to separate out, you could switch back to BinaryFormatter for those bits. I haven't tested this, but consider:
RuntimeTypeModel.Default.Add(typeof(SessionSecurityToken), false)
.SetSurrogate(typeof(BinaryFormatterSurrogate<SessionSecurityToken>));
With:
[ProtoContract]
public class BinaryFormatterSurrogate<T>
{
[ProtoMember(1)]
public byte[] Raw { get; set; }
public static explicit operator T(BinaryFormatterSurrogate<T> value)
{
if(value==null || value.Raw == null) return default(T);
using(var ms = new MemoryStream(value.Raw))
{
return (T)new BinaryFormatter().Deserialize(ms);
}
}
public static explicit operator BinaryFormatterSurrogate<T>(T value)
{
object obj = value;
if (obj == null) return null;
using (var ms = new MemoryStream())
{
new BinaryFormatter().Serialize(ms, obj);
return new BinaryFormatterSurrogate<T> { Raw = ms.ToArray() };
}
}
}
Keep in mind that this simply embeds the output of one serializer as raw data inside another. Fortunately protobuf-net is happy talking binary, so this won't add any noticeable overhead (just the header and length-prefix for the blob) - but it also won't do anything particularly smart or clever with the SessionSecurityToken instances. If this is the only thing you are serializing, it really isn't worth it. If this is just one ugly bump in a larger DTO model, where most of it can serialize nicely - then it might get the job done for you.

Highly customized serializing

I have an object I'd like to serialize to a memory buffer, which is then sent via UART to an embedded device.
I'm working in a C# environment on windows.
What I'd like to do is to create two classes that look like this:
class StatusElement
{
byte statusPart1;
byte statusPart2;
}
class DeviceCommand
{
byte Address;
byte Length;
StatusElement[] statusElements; // Can have an arbitrary number of elements in it
}
I'd like to use a serialize, preferably something based on c# serialization, to convert the second class to a byte stream.
The problem is that the embedded device is hard-coded to accept an exact sequence (AddressByte, LengthByte .... ErrorCorrectionByte) so I cannot use the regular C# serialization, which adds serialization metadata in the stream. This also rules out other serializes like Protobuf.
So my question is:
Is it possible to customize the c# serialization to get the output I need? How?
--- Update ---
Thanks everyone for the help.
After consideration I’ve decided to implement my own mini-serializer, using reflection and per-type handler. More complex but gives me more flexibility and automation capabilities.

use a MemoryStream to manully serialize your object.
private byte[] Serialize()
{
using (var ms = new MemoryStream())
{
ms.WriteByte(Address);
ms.WriteByte(Length);
foreach (var element in statusElements)
{
ms.WriteByte(element.statusPart1);
ms.WriteByte(element.statusPart2);
}
return ms.ToArray();
}
}
Likewise for deserialization:
private static DeviceCommand Deserialize(byte[] input)
{
DeviceCommand result = new DeviceCommand();
using (var ms = new MemoryStream(input))
{
result.Address = ms.ReadByte();
result.Length = ms.ReadByte();
//assuming .Length contains the number of statusElements:
result.statusElemetns = new StatusElement[result.Length];
for (int i = 0; i < result.Length; i++)
{
result.statusElements[i] = new StatusElement();
result.statusElements[i].statusPart1 = ms.ReadByte();
result.statusElements[i].statusPart2 = ms.ReadByte();
}
}
return result;
}

If you need only to write bytes or byte arrays, you can use the MemoryStream directly. If you want to use other .NET base types, access your Stream with a System.IO.BinaryWriter / BinaryReader. This class is used by the System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
for binary serialization and deserialization.

Have trouble deserializing an object when it is sent as a packet

I am trying to learn udp sockets etc.... I created two programs server and client. The client sends a packet to the server, the server bounces it back.
This is the code I use in both programs for converting the data to and from a byte[]
but I am getting an error when converting from byte[]
public static Packet Open(byte[] b)
{
MemoryStream memStream = new MemoryStream();
BinaryFormatter binForm = new BinaryFormatter();
memStream.Write(b, 0, b.Length);
memStream.Seek(0, SeekOrigin.Begin);
object obj = new object();
try
{
// this line here is where the error is occurring
obj = (object)binForm.Deserialize(memStream);
}
catch (Exception er)
{
MessageBox.Show(er.Message);
}
if (obj is Packet)
return (Packet)obj;
else
return null;
}
public byte[] Bundle()
{
BinaryFormatter bf = new BinaryFormatter();
MemoryStream ms = new MemoryStream();
bf.Serialize(ms, this);
return ms.ToArray();
}
If I do this, all from one program it works
Packet p =new Packet();
p.Message="hello";
byte[] data = p.Bundle();
Packet p2 = Packet.Open(data);
MessageBox.Show(p2.Message);
The error I am receiving is "unable to find assembly in "the name of my client program"
AnyIdeas?

It sounds to me like you are serializing a type that is not shared via a reference between both ends. Note: it is not sufficient to have the same class compiled into both, since BinaryFormatter includes the full type name including the assembly, so: it will still count as an unrelated type. The common fix there (and I use the word "fix" entirely incorrectly) is to write an assembly for the DTO and reference that assembly from both client and server. This approach still has many issues, though.
For info, there are other serializers that are compatible with just having a similar class at each end. I'm biased, but I would suggest having a look at protobuf-net; the output is usually significantly smaller, and it isn't tied to the type, meaning the class just has to be broadly similar at each end (it is very version tolerant). Plus it is faster (CPU-wise) too!