I searched for an answer to this question but I didn't see many specific answers to it. My question is relatively simple: given data serialized in the protobuf format, is it possible to deserialize it without knowing the data type?
It seems that I can serialize any type without knowing what it is by doing:
public static byte[] SerializeData<T>(T arg)
{
try
{
using (var testStream = new MemoryStream())
{
Serializer.Serialize(testStream, arg);
....
Is there a way to do this such that Serializer.Deserialize(stream) (or something of that sort)?
Related
I have a protobuf object that I am sending from a C# application (using clrZmq) to a C++ service (using the zmq C++ bindings) on a local machine (for testing). I attempt to send my object from C# using the following
Taurus.Odds odds = Util.GetFakeOdds();
using (var context = ZmqContext.Create())
using (var socket = context.CreateSocket(SocketType.REQ))
{
byte[] buffer = null;
socket.Connect(TARGET); // TARGET = "tcp://127.0.0.1:6500"
Taurus.FeedMux mux = new Taurus.FeedMux();
mux.type = Taurus.FeedMux.Type.ODDS;
mux.odds = odds;
SendStatus status = socket.Send(mux.ToByteArray());
if (status == SendStatus.Sent)
{
int i;
byte[] arr = socket.Receive(buffer, SocketFlags.None, out i);
Taurus.Bet bet = buffer.ToObject<Taurus.Bet>();
}
...
}
Where I am serializing to my Taurus.Odds object to byte[] via the extension method
public static byte[] ToByteArray(this object o)
{
if(o == null)
return null;
BinaryFormatter bf = new BinaryFormatter();
using (MemoryStream ms = new MemoryStream())
{
bf.Serialize(ms, o);
return ms.ToArray();
}
}
I see in my C++ application that the code receives the message, but the C++ ZMQ classes fail to de-serialize it correctly. I have some Java code that send to the C++ code in the same way without issue. My question is, am I sending my object via ZMQ correctly in the above and if not what am I doing wrong?
Thanks for your time.
Here's your error:
I am serializing to my Taurus.Odds object to byte[] via the extension method
...
BinaryFormatter bf = new BinaryFormatter();
...
You seem to be unaware of what BinaryFormatter is. It is in no way related to ProtoBuf. The docs say the following:
Serializes and deserializes an object, or an entire graph of connected objects, in binary format.
This binary format is a .NET-specific implementation detail. And it's very rigid at that, with poor versioning support. It was mainly used in the .NET remoting days, and it's generally considered a bad idea to use it today, as there are much better serializers around.
As you can see, there's no way your C++ app could be able to read that, as it's not in protobuf format.
So throw this method away and replace it with some proper protobuf serializing code, as explained in the protobuf-net docs. You'll need to add [ProtoContract] and [ProtoMember] attributes in your objects. Then you could write something like:
public static byte[] ToByteArray<T>(this T o)
{
if (o == null)
return null;
using (MemoryStream ms = new MemoryStream())
{
ProtoBuf.Serializer.Serialize(ms, o);
return ms.ToArray();
}
}
For I can have direct access to a particular object in a serialized generic list, I need to know position of it from de-serialized stream.
private static void Serialze(object obj, FileStream Stream)
{
BinaryFormatter bin = new BinaryFormatter();
bin.FilterLevel = System.Runtime.Serialization.Formatters.TypeFilterLevel.Low;
bin.Serialize(Stream, obj);
}
private static object DeSerialize(FileStream Stream)
{
if (Stream.Position >= Stream.Length) return null;
BinaryFormatter bin = new BinaryFormatter();
bin.FilterLevel = System.Runtime.Serialization.Formatters.TypeFilterLevel.Low;
object x = bin.Deserialize(Stream);
return x;
}
Suppose I have a generic list like this:
List<MyClass> L1=new List<MyClass>();
How can I to find position of L1[10] in my FileStreem, for using it, in DeSerialize method?
The format that BinaryFormatter uses is not seekable. You cannot extract sub-objects.
Probably, this question is an instance of the XY-Problem. Tell us what you want to achieve and why you need this. As asked, there is no answer.
Maybe you can make use of a database instead. Or, use Protocol Buffers, which are more flexible.
BinaryFormatter is able to handle serialization simply:
private byte[] TokenToBytes(SessionSecurityToken token)
{
if (token == null)
{
return null;
}
using (var memoryStream = new MemoryStream())
{
var binaryFormatter = new BinaryFormatter();
binaryFormatter.Serialize(memoryStream, token);
return memoryStream.ToArray();
}
}
When I tried replacing BinaryFormatter with protobuf-net:
using (var memoryStream = new MemoryStream())
{
Serializer.Serialize(memoryStream, token);
return memoryStream.ToArray();
}
I get the following exception:
Type is not expected, and no contract can be inferred:
System.IdentityModel.Tokens.SessionSecurityToken
I tried adding:
RuntimeTypeModel.Default.Add(typeof(SessionSecurityToken), true);
Which gets past the exception but I now get a zero byte array.
How do I properly configure protobuf-net to serialize a SessionSecurityToken?
On the flipside, SessionSecurityToken does not have a parameterless constructor.
using (var memoryStream = new MemoryStream(tokenAsBytes))
{
return Serializer.Deserialize<SessionSecurityToken>(memoryStream);
}
Throws a ProtoException:
No parameterless constructor found for SessionSecurityToken
BinaryFormatter is able to do it without any fuss:
using (var memoryStream = new MemoryStream(bytes))
{
var binaryFormatter = new BinaryFormatter();
return (SessionSecurityToken)binaryFormatter.Deserialize(memoryStream);
}
How do I properly configure protobuf-net to deserialize a SessionSecurityToken?
protobuf-net does not claim to be able to serialize every single type; indeed, you would have great difficulty serializing that via most serializers (XmlSerializer, any of the json serializers, DataContractSerializer, etc). BinaryFormatter is in a different category of serializers - and in this particular case, implements custom serialization via ISerializable.GetObjectData(SerializationInfo, StreamingContext).
The constructor thing is a red herring; actually, protobuf-net can bypass constructors completely, and in this particular scenario BinaryFormatter is using a custom serialization constructor via .ctor(SerializationInfo, StreamingContext).
For simple cases, protobuf-net can be configured via attributes or runtime options; for more complex scenarios, surrogates can be used to map between representations - however, in this case I would suggest (looking at the implementation of SessionSecurityToken) that this is more complex than you probably want to maintain.
I would step back a step or two here; most serializers are designed to work with data, not implementation - and work great with DTOs etc. SessionSecurityToken is very much not a DTO, and there is no simple way of switching between them. My strong suggestion here would be: serialize what this represents, not what it is. However, if this is part of an existing complex model and is really really hard to separate out, you could switch back to BinaryFormatter for those bits. I haven't tested this, but consider:
RuntimeTypeModel.Default.Add(typeof(SessionSecurityToken), false)
.SetSurrogate(typeof(BinaryFormatterSurrogate<SessionSecurityToken>));
With:
[ProtoContract]
public class BinaryFormatterSurrogate<T>
{
[ProtoMember(1)]
public byte[] Raw { get; set; }
public static explicit operator T(BinaryFormatterSurrogate<T> value)
{
if(value==null || value.Raw == null) return default(T);
using(var ms = new MemoryStream(value.Raw))
{
return (T)new BinaryFormatter().Deserialize(ms);
}
}
public static explicit operator BinaryFormatterSurrogate<T>(T value)
{
object obj = value;
if (obj == null) return null;
using (var ms = new MemoryStream())
{
new BinaryFormatter().Serialize(ms, obj);
return new BinaryFormatterSurrogate<T> { Raw = ms.ToArray() };
}
}
}
Keep in mind that this simply embeds the output of one serializer as raw data inside another. Fortunately protobuf-net is happy talking binary, so this won't add any noticeable overhead (just the header and length-prefix for the blob) - but it also won't do anything particularly smart or clever with the SessionSecurityToken instances. If this is the only thing you are serializing, it really isn't worth it. If this is just one ugly bump in a larger DTO model, where most of it can serialize nicely - then it might get the job done for you.
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 3 years ago.
Improve this question
If it helps, I'm using the CampBX API to get funds in my account. I wrote the following code to make the API Call:
using (var wb = new WebClient())
{
String url = "https://CampBX.com/api/myfunds.php";
var data = new NameValueCollection();
data["user"] = "USERNAME";
data["pass"] = "PASSWORD";
var response = wb.UploadValues(url, "POST", data);
}
WebClient.UploadValues() returns a byte[] and I have no idea how to correctly parse that.
Here is the CampBX info, under Account Balances.
Simply put, you need to use a JSON parser. Personally I like Newtonsoft.Json and it is what I will use in this example.
The first step is converting the byte[] to a sequence of characters, either a string object or a TextReader. The second step is to pass this information to the parser. So, in your case, the code would look something like this:
JToken parsedToken;
using (var responseReader = new StreamReader(new MemoryStream(response))) {
parsedToken = JToken.ReadFrom(responseReader);
}
The parsedToken object can then be used to extract whatever data you need. (See the documentation for information on extracting data from a JToken object.)
Note that WebClient.UploadValues() discards the information regarding the response entity's character encoding. StreamReader will use UTF-8 encoding by default, which is sufficient to parse UTF-8 or ASCII. Depending on the JSON encoder used by the server, the response may always be ASCII-compatible, so you might not have to worry about it. Nevertheless, it's something you should investigate.
DataContractJsonSerializer built-in object will be your friend here, provided you know what is the internal structure of the returned object (or can at least guess it, from the JSON).
The steps are:
Define a contract class to hold the deserialized JSON object
namespace AppNameSpace
{
[DataContract] /* Place this inside your app namespace */
internal class iResponse /*Name this class appropriately */
{
[DataMember]
internal string field1;
[DataMember]
internal string field2;
[DataMember]
internal Int32 field3;
}
...
}
The actual parsing itself is about three lines
DataContractJsonSerializer ser = new DataContractJsonSerializer(typeof(iOpenParams));
MemoryStream stream1 = new MemoryStream(response);
iResponse resp_json = (iResponse)ser.ReadObject(stream1);
For more details and examples, refer to: http://msdn.microsoft.com/en-us/library/bb412179.aspx
My solution is simpler:
Object retorno;
var response = wb.UploadValues(url, "POST", data);
using (var responseReader = new StreamReader(new MemoryStream(response)))
{
retorno = JsonConvert.DeserializeObject<Object>(responseReader.ReadToEnd());
}
I have a legacy app that uses SoapFormatter to persist a graph of objects (maybe 50 different classes). I want to move away from using this as it is deprecated, and increasingly hard to continue to support deserializing from old files as the classes change.
I want to use DataContractSerializer going forward. Does anyone have any suggestions as to a good strategy for migration? I need to continue to be able to deserializing old files written by SoapFormatter...
Thanks
I don't think you want to be limited to a backward-compatible format.
So you will need to distinguish old and new content. And easy method would be :
Old Format: <soapdata>
New Format: <header> <newdata>
And in your new Load() method:
(Try to) Read the header
If a header is found, continue to read the new format
else re-position to begin and use the SOAP formatter to read.
The simplest code would be to try deserialize with DataContractSerializer and fallback to SoapFormatter if it fails.
The save part will always use the DataContractSerializer, so that your new objects or the updated ones will use your new supported version.
public MyContract Deserialize(string file)
{
try
{
using (var stream = loadFile())
{
return loadWithDataContractSerializer(stream);
}
}
catch (SerializationException)
{
using (var stream = openForRead(file))
{
return convertToContract(loadWithSoapFormatter(stream));
}
}
}
private MyContract loadWithDataContractSerializer(Stream s);
private MyOldObject loadWithSoapFormatter(Stream s);
private MyContract convertToContract(MyOldObject obj);
public void Serialize(string file, MyContract data)
{
using (var stream = openForWrite(file))
{
writeWithDataContractSerializer(stream, data);
}
}
Of course, it might be possible to implement a custom logic to allow DataContractSerializer to understant the SoapFormatter structure, but you will have to provide a lot more work.