Develop Reference

Why does my protobuf-net stream not work?

I have an object that can be serialized and deserialized but upon deserialization it throws me an error:
Invalid field in source data: 0
I don't know why this is happening
code for de-serialization and receiving:
public void listenUDP()
{
EndPoint ep = (EndPoint)groupEP;
//BinaryFormatter bf = new BinaryFormatter();
recieving_socket.Bind(ep);
while (true)
{
byte[] objData = new byte[65535];
recieving_socket.ReceiveFrom(objData, ref ep);
MemoryStream ms = new MemoryStream();
ms.Write(objData, 0, objData.Length);
ms.Seek(0, SeekOrigin.Begin);
messageHandle(ProtoBuf.Serializer.Deserialize<SimplePacket>(ms));
ms.Dispose();
}
}
Code for serialization:
public void sendDataUDP(Vec2f[] data)
{
SimplePacket packet = new SimplePacket(DateTime.UtcNow, data);
//IFormatter formatter = new BinaryFormatter();
MemoryStream stream = new MemoryStream();
System.Diagnostics.Stopwatch st = System.Diagnostics.Stopwatch.StartNew();
//formatter.Serialize(stream, data);
ProtoBuf.Serializer.Serialize<SimplePacket>(stream, packet);
//Console.WriteLine(st.ElapsedTicks);
stream.Close();
st.Restart();
sending_socket.SendTo(stream.ToArray(), sending_end_point);
//Console.WriteLine(st.ElapsedTicks);
st.Stop();
}

The root object in a protobuf message, as defined by the google specification, does not include any notion of the end of the message. This is intentional, so that concatenation is identical to merging two fragments. Consequently, the consuming code needs to restrict itself to a single message. This is identical between all protobuf implementations, and is not specific for protobuf-net.
What is happening is that your buffer is currently oversized, with garbage at the end. Currently (because you are reading one message) that garbage is most likely all zeros, and a zero is not a valid marker for a field. However, when re-using the buffer the garbage could be... anything.
In your case, probably the easiest way to do this is to use the SerializeWithLengthPrefix / DeserializeWithLengthPrefix methods, which handle all this for you by prepending the payload length at the start of the message, and only processing that much data.
As a final thought: it is not clear to me that your code will guarantee that is has read an entire message; a single receive could (on TCP, at least) return part of a message - or 2 and a bit messages, etc: TCP is stream-based, not message-based.

Data loss TCP IP C# [duplicate]

This question already has answers here:
Receiving data in TCP
(10 answers)
Closed 2 years ago.
Here's my code:
private void OnReceive(IAsyncResult result)
{
NetStateObject state = (NetStateObject)result.AsyncState;
Socket client = state.Socket;
int size = client.EndReceive(result);
byte[] data = state.Buffer;
object data = null;
using (MemoryStream stream = new MemoryStream(data))
{
BinaryFormatter formatter = new BinaryFormatter();
data = formatter.Deserialize(stream);
}
//todo: something with data
client.BeginReceive(
state.Buffer,
0,
NetStateObject.BUFFER_SIZE,
SocketFlags.None,
OnReceive,
state
);
}
state.Buffer has a maximum size of NetStateObject.BUFFER_SIZE (1024). Firstly, is this too big or too small? Second, if I send something larger than that, my deserialize messes up because the object it is trying to deserialize doesnt have all the information (because not all the data was sent). How do I make sure that all my data has been received before I try to construct it and do something with it?
Completed Working Code
private void OnReceive(IAsyncResult result)
{
NetStateObject state = (NetStateObject)result.AsyncState;
Socket client = state.Socket;
try
{
//get the read data and see how many bytes we received
int bytesRead = client.EndReceive(result);
//store the data from the buffer
byte[] dataReceived = state.Buffer;
//this will hold the byte data for the number of bytes being received
byte[] totalBytesData = new byte[4];
//load the number byte data from the data received
for (int i = 0; i < 4; i++)
{
totalBytesData[i] = dataReceived[i];
}
//convert the number byte data to a numan readable integer
int totalBytes = BitConverter.ToInt32(totalBytesData, 0);
//create a new array with the length of the total bytes being received
byte[] data = new byte[totalBytes];
//load what is in the buffer into the data[]
for (int i = 0; i < bytesRead - 4; i++)
{
data[i] = state.Buffer[i + 4];
}
//receive packets from the connection until the number of bytes read is no longer less than we need
while (bytesRead < totalBytes + 4)
{
bytesRead += state.Socket.Receive(data, bytesRead - 4, totalBytes + 4 - bytesRead, SocketFlags.None);
}
CommandData commandData;
using (MemoryStream stream = new MemoryStream(data))
{
BinaryFormatter formatter = new BinaryFormatter();
commandData = (CommandData)formatter.Deserialize(stream);
}
ReceivedCommands.Enqueue(commandData);
client.BeginReceive(
state.Buffer,
0,
NetStateObject.BUFFER_SIZE,
SocketFlags.None,
OnReceive,
state
);
dataReceived = null;
totalBytesData = null;
data = null;
}
catch(Exception e)
{
Console.WriteLine("***********************");
Console.WriteLine(e.Source);
Console.WriteLine("***********************");
Console.WriteLine(e.Message);
Console.WriteLine("***********************");
Console.WriteLine(e.InnerException);
Console.WriteLine("***********************");
Console.WriteLine(e.StackTrace);
}
}

TCP is a stream protocol. It has no concept of packets. A single write call can be sent in multiple packets, and multiple write calls can be put into the same packet. So you need to implement your own packetizing logic on top of TCP.
There are two common ways to packetize:
Delimiter characters, this is usually used in text protocols, with the new-line being a common choice
Prefix the length to each packet, usually a good choice with binary protocols.
You store the size of a logical packet at the beginning of that packet. Then you read until you received enough bytes to fill the packet and start deserializing.

How do I make sure that all my data has been received before I try to construct it and do something with it?
You have to implement some protocol so you know.
While TCP is reliable, it does not guarantee that the data from single write at one end of the socket will appear as a single read at the other end: retries, packet fragmentation and MTU can all lead to data being received in different sized units by the receiver. You will get the data in the right order.
So you need to include some information when sending that allows the receiver to know when it has the complete message. I would also recommend including what kind of message and what version of the data (this will form the basis of being able to support different client and server versions together).
So the sender sends:
- Message type
- Message version
- Message size (in bytes)
And the receiver will loop, performing a read with a buffer and appending this to a master buffer (MemoryStream is good for this). Once the complete header is received it knows when the complete data has been received.
(Another route is to include some pattern as an "end of message" marker, but then you need to handle the same sequence of bytes occurring in the content—hard to do if the data is binary rather than text.)

C# Why doesn't "Flush" force the bytes down the network stream?

I have a project where I'm trying to send a serialized object to the server, then wait for an "OK" or "ERROR" message to come back.
I seem to be having a similar problem to th poster of : TcpClient send/close problem
The issue is that the only way I seem to be able to send the original object is to close the connection, but then (of course) I can't wait to see if the object was processed successfully by the server.
private void button4_Click(object sender, EventArgs e)
{
RequestPacket req = new RequestPacket();
/// ... Fill out request packet ...
/// Connect to the SERVER to send the message...
TcpClient Client = new TcpClient("localhost", 10287);
using (NetworkStream ns = Client.GetStream())
{
XmlSerializer xml = new XmlSerializer(typeof(RequestPacket));
xml.Serialize(ns, req);
/// NOTE: This doesn't seem to do anything....
/// The server doesn't get the object I just serialized.
/// However, if I use ns.Close() it does...
/// but then I can't get the response.
ns.Flush();
// Get the response. It should be "OK".
ResponsePacket resp;
XmlSerializer xml2 = new XmlSerializer(typeof(ResponsePacket));
resp = (ResponsePacket)xml2.Deserialize(ns);
/// ... EVALUATE RESPONSE ...
}
Client.Close()
}
UPDATE: In response to one commenter, I don't think the client can be at fault. It is simply waiting for the object, and the object never comes until I close the socket.... however, if I'm wrong, I'll GLADLY eat crow publicly. =) Here's the client:
static void Main(string[] args)
{
// Read the port from the command line, use 10287 for default
CMD cmd = new CMD(args);
int port = 10287;
if (cmd.ContainsKey("p")) port = Convert.ToInt32(cmd["p"]);
TcpListener l = new TcpListener(port);
l.Start();
while (true)
{
// Wait for a socket connection.
TcpClient c = l.AcceptTcpClient();
Thread T = new Thread(ProcessSocket);
T.Start(c);
}
}
static void ProcessSocket(object c)
{
TcpClient C = (TcpClient)c;
try
{
RequestPacket rp;
//// Handle the request here.
using (NetworkStream ns = C.GetStream())
{
XmlSerializer xml = new XmlSerializer(typeof(RequestPacket));
rp = (RequestPacket)xml.Deserialize(ns);
}
ProcessPacket(rp);
}
catch
{
// not much to do except ignore it and go on.
}
}
Yeah.... it's that simple.

Uh oh, you can blame Nagle's algorithm. It has nothing to do with C# though, it is a default behavior for TCP/IP stack. Enable NoDelay socket option using SetSocketOption method. But be careful, disabling Nagle's algorithm will downgrade the throughput.
I'm also not sure about that stream you are using on top of the socket, as I am not a C# developer at all, but try to drop its instance so it does write for sure :-)

The short version is apparently, when using XmlSerializer (or any other big blob) to shove data down a NetworkStream, it will simply hold the line open indefinitely waiting for more information to be written. It only flushes the connection once you close it. This creates a situation where this method is great for sending, but not receiving. Or vice-versa. It becomes a one-way communication, and useless for continued back-and-forth communication over the same connection.
It's kind of crappy that I had to work around something that seemed so elegant on the surface, but dropping back to my old C days, I've resorted to sending a "number of bytes" packet first, then the actual packet. This enables me to READ at the other end the exact number of bytes so I never get caught in a blocking pattern.
To simplify my life, I created a class that holds some static methods for both sending and receiving. This class can send ANY XML-serializable class across the network, so it does what I need it to do.
If anyone has a more elegant solution, I'd be open to hearing it.
public class PacketTransit
{
public static void SendPacket(TcpClient C, object Packet)
{
MemoryStream ms = new MemoryStream();
XmlSerializer xml = new XmlSerializer(Packet.GetType());
xml.Serialize(ms, Packet);
ms.Position = 0;
byte[] b = ms.GetBuffer();
ms.Dispose();
byte [] sizePacket = BitConverter.GetBytes(b.Length);
// Send the 4-byte size packet first.
C.Client.Send(sizePacket, sizePacket.Length, SocketFlags.None);
C.Client.Send(b, b.Length, SocketFlags.None);
}
/// The string is the XML file that needs to be converted.
public static string ReceivePacket(TcpClient C, Type PacketType)
{
byte [] FirstTen = new byte[1024];
int size = 0;
byte[] sizePacket = BitConverter.GetBytes(size);
// Get the size packet
int sp = C.Client.Receive(sizePacket, sizePacket.Length, SocketFlags.None);
if (sp <= 0) return "";
size = BitConverter.ToInt32(sizePacket, 0);
// read until "size" is met
StringBuilder sb = new StringBuilder();
while (size > 0)
{
byte[] b = new byte[1024];
int x = size;
if (x > 1024) x = 1024;
int r = C.Client.Receive(b, x, SocketFlags.None);
size -= r;
sb.Append(UTF8Encoding.UTF8.GetString(b));
}
return sb.ToString();
}
/// The XML data that needs to be converted back to the appropriate type.
public static object Decode(string PacketData, Type PacketType)
{
MemoryStream ms = new MemoryStream(UTF8Encoding.UTF8.GetBytes(PacketData));
XmlSerializer xml = new XmlSerializer(PacketType);
object obj = xml.Deserialize(ms);
ms.Dispose();
return obj;
}
public static RequestPacket GetRequestPacket(TcpClient C)
{
string str = ReceivePacket(C, typeof(RequestPacket));
if (str == "") return new RequestPacket();
RequestPacket req = (RequestPacket) Decode(str, typeof(RequestPacket));
return req;
}
public static ResponsePacket GetResponsePacket(TcpClient C)
{
string str = ReceivePacket(C, typeof(ResponsePacket));
if (str == "") return new ResponsePacket();
ResponsePacket res = (ResponsePacket)Decode(str, typeof(ResponsePacket));
return res;
}
}
To use this class, I simply need to call PacketTransit.SendPacket(myTcpClient, SomePacket) to send any given XML-Serializable object. I can then use PacketTransit.GetResponsePacket or PacketTransit.GetRequestPacket to receive it at the other end.
For me, this is working very well, but it was alot more of a workout than originally expected.

you should use a StreamWriter/Reader linked to your network stream, .Flush does nothing on a NetworkStream, see here:
http://www.c-sharpcorner.com/UploadFile/dottys/SocketProgDTRP11222005023030AM/SocketProgDTRP.aspx

I believe the real problem here may be that the XmlDeserializer may not return until it has read EOS from the stream. You may need to shutdown the sending stream for output to force this to happen.

Sending and receiving custom objects using Tcpclient class in C#

I have a client server application in which the server and the client need to send and receive objects of a custom class over the network. I am using TcpClient class for transmitting the data. I am serializing the object at the sender side and sending the resulting stream of bytes to the receiver. But at the receiver, when I try to de-serialize the bytes received, it throws Serialization Exception and the details are :
The input stream is not a valid
binary format. The starting contents
(in bytes) are:
0D-0A-00-01-00-00-00-FF-FF-FF-FF-01-00-00-00-00-00
...
My server code that serializes the object is:
byte[] userDataBytes;
MemoryStream ms = new MemoryStream();
BinaryFormatter bf1 = new BinaryFormatter();
bf1.Serialize(ms, new DataMessage());
userDataBytes = ms.ToArray();
netStream.Write(userDataBytes, 0, userDataBytes.Length);
The client code that de-serializes it is:
readNetStream.Read(readMsgBytes, 0, (int)tcpServer.ReceiveBufferSize);
MemoryStream ms = new MemoryStream(readMsgBytes);
BinaryFormatter bf1 = new BinaryFormatter();
ms.Position = 0;
object rawObj = bf1.Deserialize(ms);
DataMessage msgObj = (DataMessage)rawObj;
Please help me to solve this problem and possibly suggest any other method to transmit objects of custom classes across network using TcpClient in C#.
Thanks,
Rakesh.

Have a look at this code. It takes a slightly different approach.
Example given by the link above: - Note: there was another problem he was facing which he solved here (keep-alive). It's in the link after the initial sample code.
Object class to send (remember the [Serializable]):
[serializable]
public class Person {
private string fn;
private string ln;
private int age;
...
public string FirstName {
get {
return fn;
}
set {
fn=value;
}
}
...
...
public Person (string firstname, string lastname, int age) {
this.fn=firstname;
...
}
}
Class to send object:
using System;
using System.Net;
using System.Net.Sockets;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
class DataSender
{
public static void Main()
{
Person p=new Person("Tyler","Durden",30); // create my serializable object
string serverIp="192.168.0.1";
TcpClient client = new TcpClient(serverIp, 9050); // have my connection established with a Tcp Server
IFormatter formatter = new BinaryFormatter(); // the formatter that will serialize my object on my stream
NetworkStream strm = client.GetStream(); // the stream
formatter.Serialize(strm, p); // the serialization process
strm.Close();
client.Close();
}
}
Class to receive object:
using System;
using System.Net;
using System.Net.Sockets;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
class DataRcvr
{
public static void Main()
{
TcpListener server = new TcpListener(9050);
server.Start();
TcpClient client = server.AcceptTcpClient();
NetworkStream strm = client.GetStream();
IFormatter formatter = new BinaryFormatter();
Person p = (Person)formatter.Deserialize(strm); // you have to cast the deserialized object
Console.WriteLine("Hi, I'm "+p.FirstName+" "+p.LastName+" and I'm "+p.age+" years old!");
strm.Close();
client.Close();
server.Stop();
}
}

When receiving on client side you do not know how much data you want to read.
You are only relying on the ReceiveBufferSize, while your data can be larger
or smaller then that.
I think the best approach here is to send 4 bytes that tells your client about the length of incoming data:
byte[] userDataLen = BitConverter.GetBytes((Int32)userDataBytes.Length);
netStream.Write(userDataLen, 0, 4);
netStream.Write(userDataBytes, 0, userDataBytes.Length);
and on the recieving end you first read the data length and then read
exact amount of data.
byte[] readMsgLen = new byte[4];
readNetStream.Read(readMsgLen, 0, 4);
int dataLen = BitConverter.ToInt32(readMsgLen);
byte[] readMsgData = new byte[dataLen];
readNetStream.Read(readMsgData, 0, dataLen);
Infact, I just realized, that you might has to do a little more to assure you read all data (just an idea because I haven't tried it, but just incase you run into problem again you can try this).
The NetworkStream.Read() method returns a number indicating the amount of data it has read. It might be possible that the incoming data is larger then the RecieveBuffer. In that case you have to loop until you read all of the data. You have to do something like this:
SafeRead(byte[] userData, int len)
{
int dataRead = 0;
do
{
dataRead += readNetStream.Read(readMsgData, dataRead, len - dataRead);
} while(dataRead < len);
}

TCP is stream-based protocol (as opposed to datagram protocol) so it's possible to receive only part of sended data via Read method call.
To solve this problem you may use DataLength field (as cornerback84 suggested) or you may use your own "application-level packet" structure.
For example, you may use something like this
|-------------------------------|
|Begin|DataLength| Data |End|
| 4b | 4b | 1..MaxLen|4b |
|-------------------------------|
where
Begin - start packet identifier (for example 0x0A, 0x0B, 0x0C, 0x0D)
DataLength - Data field length (for example, from 0 to MaxLength)
Data - actual data (serialized Person class or some other data)
End - end packet identifier (for example 0x01, 0x05, 0x07, 0x0F).
That is, on client side you would wait not only for incoming data, after receiving data you would search you Application level packets, and you may deserialized Data part only after receiving valid packet.

TCP Framing with Binary Protocol

Hey, I'm having an issue seperating packets using a custom binary protocol.
Currently the server side code looks like this.
public void HandleConnection(object state)
{
TcpClient client = threadListener.AcceptTcpClient();
NetworkStream stream = client.GetStream();
byte[] data = new byte[4096];
while (true)
{
int recvCount = stream.Read(data, 0, data.Length);
if (recvCount == 0) break;
LogManager.Debug(Utility.ToHexDump(data, 0, recvCount));
//processPacket(new MemoryStream(data, 0, recvCount));
}
LogManager.Debug("Client disconnected");
client.Close();
Dispose();
}
I've been watching the hex dumps of the packets, and sometimes the entire packet comes in one shot, let's say all 20 bytes. Other times it comes in fragmented, how do I need to buffer this data to be able to pass it to my processPacket() method correctly. I'm attempting to use a single byte opcode header only, should I add something like a (ushort)contentLength to the header aswell? I'm trying to make the protocol as lightweight as possible, and this system won't be sending very large packets(< 128 bytes).
The client side code I'm testing with is, as follows.
public void auth(string user, string password)
{
using (TcpClient client = new TcpClient())
{
client.Connect(IPAddress.Parse("127.0.0.1"), 9032);
NetworkStream networkStream = client.GetStream();
using (BinaryWriter writer = new BinaryWriter(networkStream))
{
writer.Write((byte)0); //opcode
writer.Write(user.ToUpper());
writer.Write(password.ToUpper());
writer.Write(SanitizationMgr.Verify()); //App hash
writer.Write(Program.Seed);
}
}
}
I'm not sure if that could be what's messing it up, and binary protocol doesn't seem to have much info on the web, especially where C# is involved. Any comment's would be helpful. =)
Solved with this, not sure if it's correct, but it seems to give my handlers just what they need.
public void HandleConnection(object state)
{
TcpClient client = threadListener.AcceptTcpClient();
NetworkStream stream = client.GetStream();
byte[] data = new byte[1024];
uint contentLength = 0;
var packet = new MemoryStream();
while (true)
{
int recvCount = stream.Read(data, 0, data.Length);
if (recvCount == 0) break;
if (contentLength == 0 && recvCount < headerSize)
{
LogManager.Error("Got incomplete header!");
Dispose();
}
if(contentLength == 0) //Get the payload length
contentLength = BitConverter.ToUInt16(data, 1);
packet.Write(data, (int) packet.Position, recvCount); //Buffer the data we got into our MemStream
if (packet.Length < contentLength + headerSize) //if it's not enough, continue trying to read
continue;
//We have a full packet, pass it on
//LogManager.Debug(Utility.ToHexDump(packet));
processPacket(packet);
//reset for next packet
contentLength = 0;
packet = new MemoryStream();
}
LogManager.Debug("Client disconnected");
client.Close();
Dispose();
}

You should just treat it as a stream. Don't rely on any particular chunking behaviour.
Is the amount of data you need always the same? If not, you should change the protocol (if you can) to prefix the logical "chunk" of data with the length in bytes.
In this case you're using BinaryWriter on one side, so attaching a BinaryReader to the NetworkStream returned by TcpClient.GetStream() would seem like the easiest approach. If you really want to capture all the data for a chunk at a time though, you should go back to my idea of prefixing the data with its length. Then just loop round until you've got all the data.
(Make sure you've got enough data to read the length though! If your length prefix is 4 bytes, you don't want to read 2 bytes and miss the next 2...)