Develop Reference

C# FlatBufferBuilder create String from Stream

Suppose you need to read a large string from a stream and you want to put that string into a flatbuffer.
Currently what I do is read the stream into a string and then use the FlatbufferBuilder.CreateString(string s) function.
This works fine but it does have as a drawback that the string is copied and loaded into memory twice: once by reading it from the stream into the string; and then a second time the string is copied into the flatbuffer.
I was wondering if there is a way to fill the flatbuffer string directly from a stream?
For a more concrete example:
Suppose your flatbuffer schema looks like:
table Message
{
_Data: string;
}
root_type Message;
We can then create a flatbuffer like this (with myData a string)
var fbb = new FlatBufferBuilder(myData.Length);
var dataOffset = fbb.CreateString(myData);
var message = Message.CreateMessage(fbb, dataOffset);
Message.FinishMessageBuffer(fbb, message);
So the question is can we somehow do the same thing, where myData is a System.IO.Stream?
Obviously the following works, but I'd like to avoid first reading the Stream into memory.
using (var reader = new StreamReader(myStream)
{
var myData = reader.ReadToEnd();
var fbb = new FlatBufferBuilder(myData.Length);
var dataOffset = fbb.CreateString(myData);
var message = Message.CreateMessage(fbb, dataOffset);
Message.FinishMessageBuffer(fbb, message);
}

There is currently no way to avoid that copy twice, afaik.. it should be relatively simple to implement a version of CreateString that takes a stream and reduces it to one copy. You could have a go at that and open a PR on github with the result.

Show error message by uploading file more than 5 MB [duplicate]

I send json file to server and want to read that twice.
[HttpPost]
public ActionResult CreateCases(string fileFormat, Guid key)
{
var file = Request.Files[0];
CheckFile(file);
Create(file);
return Json();
}
public object Check(HttpPostedFileBase file)
{
var stream = file.InputStream;
var serializer = new JsonSerializer();
using (var sr = new StreamReader(stream))
using (var jsonTextReader = new JsonTextReader(sr))
{
dynamic json = serializer.Deserialize(jsonTextReader);
...
}
}
public object Create(HttpPostedFileBase file)
{
var stream = file.InputStream;
var serializer = new JsonSerializer();
using (var sr = new StreamReader(stream))
using (var jsonTextReader = new JsonTextReader(sr))
{
dynamic json = serializer.Deserialize(jsonTextReader);
...
}
}
In Check method file.ContentLength = right value
In Create method file.ContentLength = 0 and json variable already = null
What am I doing wrong?
Thanks in advance.

What am I doing wrong?
This:
I [...] want to read that [file] twice
Your client only sends the file to your web application once, so you should only read it once.
Sure, you can rewind the input stream and appear to solve the immediate problem, but that just introduces new problems, because now you have the entire file in memory at once - and your code can only continue once the entire request has been read.
You don't want to read the file twice.
If you want to validate, then process the JSON, then obtain the JSON, store it in a variable, and then validate and process that variable. Yes, this still requires you to read the entire request body, but then that's your requirement.

Using protobuf CodedInputStream to read from byte[]

In the following code i want to use a predefined protobuf message in c#. I found that I was able to write and use the method to take a method that has been created and make a byte[]:
ContainerMessage containerMessage = new ContainerMessage();
containerMessage.Type = CommandType.Connect;
containerMessage.Connect = new Connect();
containerMessage.Connect.ClientName = "TemporaryClientName";
byte[] stream = new byte[containerMessage.CalculateSize()];
using (Google.Protobuf.CodedOutputStream outstream = new Google.Protobuf.CodedOutputStream(stream))
{
containerMessage.WriteTo(outstream);
}
This works as expected and i can inspect the message and the values are as expected as are the values in the byte[]. But if I try to Deserialize even this simple byte[] that i have just created:
using (Google.Protobuf.CodedInputStream instream = new Google.Protobuf.CodedInputStream(stream))
{
instream.ReadMessage(containerMessage);
}
It fails with:
An unhandled exception of type 'Google.Protobuf.InvalidProtocolBufferException' occurred in Google.Protobuf.dll
Additional information: Protocol message contained an invalid tag (zero).
Is this way of deserializing from a byte[] correct for protobuf?
The Protobuf Definition is:
message ContainerMessage {
CommandType type = 1;
bool commandSuccess = 2;
oneof message {
Connect connect = 3;
}
}
enum CommandType {
START = 0;
CONNECT = 2;
}
message Connect {
string ClientName = 1;
uint32 PushPullPort = 2;
}
And the CS file is generated with the command line:
protoc.exe -I=%CD% --csharp_out=..\GeneratedCsMessaging\ Connect.proto

The CodedOutputStream and CodedInputStream are mainly intended to be used by the compiled proto classes. The API for CodedOutputStream states such and mentions that if you want to have manually-written code calling either of both classes you need to use their WriteTag method before each value.
However, since you want to use the Google Protobuf for serializing and parsing any System.IO.Stream will do the job just like intended. This is very well documented and described in the Parsing and serialization section of the Protocol Buffer Basics for C#. The examples which can be found in Google Protobuf's Github can be quite helpful for getting the hang of Google Protobuf quickly. There you can see that a MemoryStream is used to serialize the object while the Parse.ParseFrom method can be used to parse an object out of the serialized data.
As you've mentioned in the comments to your question using Google.Protobuf; is an essential part to be able to use Google Protobuf's features.
EDIT: A sample usage in your case could look something like this
byte[] serializedBytes;
ContainerMessage containerMessage = new ContainerMessage()
{
Connect = new Connect()
{
ClientName = "TemporaryClientName",
},
Type = CommandType.Connect,
};
using( MemoryStream stream = new MemoryStream())
{
containerMessage.WriteTo(stream);
serializedBytes = stream.ToArray();
}
ContainerMessage parsedCopy = ContainerMessage.Parser.ParseFrom(serializedBytes);

Serialized data on tcpclient needs to state amount?

I have sent data as byte using TcpClient and I wanted to send my own class instead bytes of data.
By bytes of data, what I meant is that I am sending the data converted into bytes like this:
using (MemoryStream bufferStream = new MemoryStream())
{
using (BinaryWriter bufferData = new BinaryWriter(bufferStream))
{
// Simple PONG Action
bufferData.Write((byte)10);
}
_logger.Info("Received PING request, Sending PONG");
return bufferStream.ToArray();
}
And instead I would like to send it like this, without having to declare its size or w/e
public class MyCommunicationData
{
public ActionType Action { get; set; }
public Profile User { get; set; }
...
}
Normally, when I send my data as bytes the first 5 bytes I use to indicate the action and the message size.
But if I migrate to serialize all the data as a single class, do I still need to send what action and size it is or using serialized messages the client and server would know what to read etc or is there a way to do so I can send it without having to specify things out of the serialization object ?
Not sure if this matters here, I am using AsyncCallback to read and write to the network stream:
_networkStream = _client.tcpClient.GetStream();
_callbackRead = new AsyncCallback(_OnReadComplete);
_callbackWrite = new AsyncCallback(_OnWriteComplete);
Let me know if you need me to post any other functions.

If you use a text based serializer(for ex, Json), you can utilize StreamReader's ReadLine and StreamWriter's WriteLine (created from tcpClient.GetStream).
Your code would be something like
writer.WriteLine(JsonConvert.SerializeObject(commData))
and to get the data on the other end
var myobj = JsonConvert.DeserializeObject<MyCommunicationData>(reader.ReadLine())
--EDIT--
//**Server**
Task.Factory.StartNew(() =>
{
var reader = new StreamReader(tcpClient.GetStream());
var writer = new StreamReader(tcpClient.GetStream());
while (true)
{
var myobj = JsonConvert.DeserializeObject<MyCommunicationData>(reader.ReadLine());
//do work with obj
//write response to client
writer.WriteLine(JsonConvert.SerializeObject(commData));
}
},
TaskCreationOptions.LongRunning);

Highly customized serializing

I have an object I'd like to serialize to a memory buffer, which is then sent via UART to an embedded device.
I'm working in a C# environment on windows.
What I'd like to do is to create two classes that look like this:
class StatusElement
{
byte statusPart1;
byte statusPart2;
}
class DeviceCommand
{
byte Address;
byte Length;
StatusElement[] statusElements; // Can have an arbitrary number of elements in it
}
I'd like to use a serialize, preferably something based on c# serialization, to convert the second class to a byte stream.
The problem is that the embedded device is hard-coded to accept an exact sequence (AddressByte, LengthByte .... ErrorCorrectionByte) so I cannot use the regular C# serialization, which adds serialization metadata in the stream. This also rules out other serializes like Protobuf.
So my question is:
Is it possible to customize the c# serialization to get the output I need? How?
--- Update ---
Thanks everyone for the help.
After consideration I’ve decided to implement my own mini-serializer, using reflection and per-type handler. More complex but gives me more flexibility and automation capabilities.

use a MemoryStream to manully serialize your object.
private byte[] Serialize()
{
using (var ms = new MemoryStream())
{
ms.WriteByte(Address);
ms.WriteByte(Length);
foreach (var element in statusElements)
{
ms.WriteByte(element.statusPart1);
ms.WriteByte(element.statusPart2);
}
return ms.ToArray();
}
}
Likewise for deserialization:
private static DeviceCommand Deserialize(byte[] input)
{
DeviceCommand result = new DeviceCommand();
using (var ms = new MemoryStream(input))
{
result.Address = ms.ReadByte();
result.Length = ms.ReadByte();
//assuming .Length contains the number of statusElements:
result.statusElemetns = new StatusElement[result.Length];
for (int i = 0; i < result.Length; i++)
{
result.statusElements[i] = new StatusElement();
result.statusElements[i].statusPart1 = ms.ReadByte();
result.statusElements[i].statusPart2 = ms.ReadByte();
}
}
return result;
}

If you need only to write bytes or byte arrays, you can use the MemoryStream directly. If you want to use other .NET base types, access your Stream with a System.IO.BinaryWriter / BinaryReader. This class is used by the System.Runtime.Serialization.Formatters.Binary.BinaryFormatter
for binary serialization and deserialization.