Develop Reference

Receiving entire data from socket before reading contents

I'm currently trying to setup a server that accepts multiple clients and can receive and respond to messages.
The client and server use a common library at the core, which contains a Request class that gets serialized and sent from client to server and similar in reverse.
The server listens asyncronously to clients on each of their sockets and attempts to take the data received and deserialize the data into the Request class.
Data is sent via a NetworkStream using a BinaryFormatter to send directly on the socket. The received data is then parsed using a Network Stream on the other end.
I've tried using a MemoryStream to store the data to a buffer and then deserialize it as shown below, however this hasn't worked. Directly deserializing the NetworkStream didn't work either.
Searching around I haven't found much information that has worked for my use case.
This is the active code after the sockets are successfully connected:
On the request class, sending from the client:
public void SendData(Socket socket)
{
IFormatter formatter = new BinaryFormatter();
Stream stream = new NetworkStream(socket, false);
formatter.Serialize(stream, this);
stream.Close();
}
Server Code receiving this data:
public void Receive(Socket socket)
{
try
{
ReceiveState state = new ReceiveState(socket);
state.Stream.BeginRead(state.Buffer, 0, ReceiveState.BUFFER_SIZE, new AsyncCallback(DataReceived), state);
}
catch (Exception e)
{
Logger.LogError(e.ToString());
}
}
private void DataReceived(IAsyncResult ar)
{
ReceiveState state = (ReceiveState)ar.AsyncState;
int bytesRead = state.Stream.EndRead(ar);
//Resolve Message
try
{
IFormatter formatter = new BinaryFormatter();
MemoryStream memoryStream = new MemoryStream(state.Buffer, 0, bytesRead);
Request request = (Request)formatter.Deserialize(memoryStream);
Logger.Log("Request Received Successfully");
ResolveRequest(request, state.Socket);
}
catch (Exception e)
{
Logger.LogError(e.ToString());
}
//Resume listening
Receive(state.Socket);
}
public class ReceiveState
{
public byte[] Buffer;
public const int BUFFER_SIZE = 1024;
public Socket Socket;
public NetworkStream Stream;
public ReceiveState(Socket socket)
{
Buffer = new byte[BUFFER_SIZE];
Socket = socket;
Stream = new NetworkStream(Socket, false);
}
}
Currently, when BeginRead() is called on the NetworkStream I get a single byte of data, then the remaining data when the next BeginRead() is called.
e.g. The Serialized data should be: 00-01-00-00-00-FF-FF-FF-FF-01-...
I receive: 00 followed by 01-00-00-00-FF-FF-FF-FF-01-... which fails to deserialize.
I take it that the issue is that the DataReceived() method is called as soon as any data appears, which is the single byte taken, then the remainder arrives before listening is resumed.
Is there a way to make sure each message is received in full before deserializing? I'd like to be able to deserialize the object as soon as the last byte is received.

TCP is a stream protocol, not a packet protocol. That means you are only guaranteed to get the same bytes in the same order (or a network failure); you are not guaranteed to get them in the same chunk configurations. So: you need to implement your own framing protocol. A frame is how you partition messages. For binary messages, a simple framing protocol might be "length = 4 bytes little-endian int32, followed by {length} bytes of payload", in which case the correct decode is to buffer until you have 4 bytes, decode the length, buffer {length} bytes, then decode the payload. YOU NEED TO WRITE the code that buffers the correct amounts, and at every point you need to deal with over-reading, back-buffers, etc. It is a complex topic. Frankly, a lot of the nuances are solved by using the "pipelines" API (I have a multi-part discussion on that API here).
However, additional guidance:
never ever use BinaryFormatter, especially for scenarios like this; it will hurt you, and it is not a good fit for most use-cases (it also isn't a particularly good serializer); my recommendation would be something like protobuf (perhaps protobuf-net), but I'm arguably biased
network code is subtle and complex, and RPC is largely a "solved" problem; consider trying tools like gRPC instead of rolling it yourself; this can be very easy

C# waiting for data from a NetworkStream

I'm working on a asynchronous TCP server class that uses a TcpListener object. I'm using the BeginAcceptTcpClient method for the TcpListener, and when the callback fires and EndAcceptTcpClient, I get a TcpClient object. In order to receive and send with this TcpClient I need to use the NetworkStream provided by the client object.
The way I've been using the NetworkStream feels wrong though. I call BeginRead and a callback to eventually use EndRead, but this requires that I use byte[] buffer. This has worked fine for me so far, but I have to wonder if there is a cleaner way of doing things. My current flow is so as follows: receive data into a byte[] buffer, throw data into a MemoryStream, use a BinaryReader to get the data that I'm passing, and then I can ultimately get what I need for my protocol.
Is there a more elegant way to get from NetworkStream to BinaryReader (and ultimately BinaryWriter as I'm going to pass data back similarly as I received it)? It feels wasteful that I must first dump it into a byte[], then into a MemoryStream (does that copy the data?), then finally be able to create a reader/writer object.
I've looked into simply creating a BinaryReader/BinaryWriter using the NetworkStream, but from what I've gathered, those objects require a stream with data previously available. It seems like I just need to have some way to be notified that the NetworkStream has data available without reading into a buffer. Perhaps I am mistaken and this is exactly how NetworkStreams are supposed to be used. It just seems like things could be a lot more streamlined if I didn't have to copying buffers from one stream into another.
EDIT:
Here is an example of the source in question:
public class Server
{
TcpListener listener;
const int maxBufferSize = 0xFFFF;
byte[] clientBuffer = new byte[maxBuffersize];
public Server(IPAddress address, int port)
{
listener = new TcpListener(address, port);
}
public void Start()
{
listener.Start();
listener.BeginAcceptTcpClient(OnAccept, listener);
}
public void Stop()
{
listener.Stop();
}
private void OnAccept(IAsyncResult ar)
{
TcpListener listener = ar.AsyncState as TcpListener;
TcpClient client = listener.EndAcceptTcpClient(ar);
client.GetStream().BeginRead(clientBuffer, 0, maxBufferSize, OnReceive, client);
listener.BeginAcceptTcpClient(OnAccept, listener);
}
private void OnReceive(IAsyncResult ar)
{
TcpClient client = ar.AsyncState as TcpClient;
int len = client.GetStream().EndRead(ar);
if (len == 0)
{
client.Close();
return;
}
else
{
MemoryStream inStream = new MemoryStream(len == maxBufferSize ? clientBuffer : clientBuffer.Take(len).ToArray());
MemoryStream outStream = DoStuff(inStream); //Data goes off to the app at this point and returns a response stream
client.GetStream().Write(outStream.ToArray(), 0, outStream.Length);
inStream.Dispose();
outStream.Dispose();
}
}
}
My question revolves around what happens in OnReceive. You'll see that I finish the read operation with EndRead, at which point I can now retrieve the data from the byte[] field of the Server class. My concern is that the time spent copying data from the NetworkStream into an array, and then into a MemoryStream is wasteful (at least it feels that way, perhaps C# handles this stuff efficiently?)
Thanks in advance.

SslStream EndRead gets first 1 Byte

I've written a TcpClient and Server which are communicating via an SslStream.
The communication works, but when i send a message from the Client to the Server, first the Server reads 1 Byte, and in the next step the rest. Example: I want to send "test" via Client, and the Server receives first "t", and then "est"
Here is the code for the Client to send
public void Send(string text) {
byte[] message = Encoding.UTF8.GetBytes(text);
SecureStream.BeginWrite(message, 0, message.Length, new AsyncCallback(WriteCallback), null);
}
private void WriteCallback(IAsyncResult AR) {
}
And here the code the Server uses to read
private SslStream CryptedStream = ...;
private byte[] buffer = new byte[1024];
public void BeginReadCallback(IAsyncResult AsyncCall) {
// initialize variables
int bytesRead = 0;
try {
// retrieve packet
bytesRead = CryptedStream.EndRead(AsyncCall);
// check if client has disconnected
if (bytesRead > 0) {
// copy buffer to a temporary one
var temporaryBuffer = buffer;
Array.Resize(ref temporaryBuffer, bytesRead);
string read = Encoding.ASCII.GetString(temporaryBuffer);
SetText(read);
// read more data
CryptedStream.BeginRead(buffer, 0, 1024, new AsyncCallback(BeginReadCallback), null);
// client is still connected, read data from buffer
//ProcessPacket(temporaryBuffer, temporaryBuffer.Length, helper);
} else {
// client disconnected, do everything to disconnect the client
//DisconnectClient(helper);
}
} catch (Exception e) {
// encountered an error, closing connection
// Program.log.Add(e.ToString(), Logger.LogLevel.Error);
// DisconnectClient(helper);
}
}
Did i miss something?
Thanks for your help

As Lasse explained streaming APIs do not promise you to return a specific number of bytes per read.
The best fix for this is to not use sockets. Use a higher level API such as WCF, SignalR, HTTP, ...
If you insist you probably should use BinaryReader/Writer to send your data. That makes it quite easy. For example, it has string sending built-in. You also can manually length-prefix easily with those classes.
Probably, you don't need async IO and should not use it. If you insist you can at least get rid of the callbacks by using await.

How to asynchronously receive complex object in C#?

EDIT:
A more concise explanation of what I was trying to do here and with answers
I'm using c# asynchronous sockets to receive data from a source.
My problem is how and where to store received data if there are more to be received?
When a string is received, I can just use a string builder to receive and store just like this from msdn:
private void ReceiveCallback_onQuery(IAsyncResult ar)
{
try
{
// Retrieve the state object and the client socket
// from the asynchronous state object.
StateObject state = (StateObject)ar.AsyncState;
Socket client = state.workSocket;
// Read data from the remote device.
int bytesRead = client.EndReceive(ar);
if (bytesRead > 0)
{
// There might be more data, so store the data received so far.
dataReceived += state.buffer; //Does not work (dataReceived is byte[] type)
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback_onQuery), state);
}
else
{
// All the data has arrived; put it in response.
if (dataReceived > 1)
{
response_onQueryHistory = ByteArrayToObject(dataReceived)
}
// Signal that all bytes have been received.
receiveDoneQuery.Set();
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
I dont want to convert my received data into a string as in my case, I am receiving a complex object.
The data sent was serialized and I'm also fine with deserializing.
My question how to 'continually' receive data from the socket without using the string builder to store it.
Thanks!

This depends on how the complex thing is serialized prior to being pushed down the wire in the broken down bytes, you will receive those bytes and use the same algorithm/technique used to serialize the thing to deserialize it back to its original state.
For a more specific answer I'd ask that you be more specific yourself.

My problem is how and where to store received data if there are more to be received?
Can use Buffer.BlockCopy and queue it up, for eg,
int rbytes = client.EndReceive(ar);
if (rbytes > state.buffer)
{
byte[] bytesReceived = new byte[rbytes];
Buffer.BlockCopy(state.buffer, 0, bytesReceived, 0, rbytes);
state.myQueue.Enqueue(bytesReceived);
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback_onQuery), state)
}

C# asynchronous socket Begin/EndReceive questions

I've implemented a async function in order to read large socket data, mostly files will be transferred. Code is based on this question:
public class StateObject
{
public Socket workSocket = null;
public const int BUFFER_SIZE = 1024;
public byte[] buffer = new byte[BUFFER_SIZE];
public StringBuilder sb = new StringBuilder();
}
public static void Read_Callback(IAsyncResult ar)
{
StateObject so = (StateObject) ar.AsyncState;
Socket s = so.workSocket;
int read = s.EndReceive(ar);
if (read > 0)
{
so.sb.Append(Encoding.ASCII.GetString(so.buffer, 0, read));
if (read == StateObject.BUFFER_SIZE)
{
s.BeginReceive(so.buffer, 0, StateObject.BUFFER_SIZE, 0,
new AyncCallback(Async_Send_Receive.Read_Callback), so);
return;
}
}
if (so.sb.Length > 0)
{
//All of the data has been read, so displays it to the console
string strContent;
strContent = so.sb.ToString();
Console.WriteLine(String.Format("Read {0} byte from socket" +
"data = {1} ", strContent.Length, strContent));
}
s.Close();
}
After all of the data has been captured in the stringbuilder, I decrypt it and convert the base64 string to a file using another method. There will be multiple socket receiving\sending files so every workersocket has an ID.
Now here's the case; I want some status on the receiving file. For example a progress bar showing how much is received and the total amount to be received. The Read_Callback method uses a recursive call in order to get all the data from the socket, which prevents me from getting the total amount to be received. Does anyone know how to get the total amount the socket has to receive? Or is my only option to send the size fixed to the listener before transferring the data?
As mentioned before, my stateobject class contains an ID for identifying the socket which the data is for. Upon first connection, the program stores the connection and received info in an arraylist.
In order to get the amount of bytes already read, I have to report the status to one of the initiated worker classes stored in the arraylist. If I want to do this during the recursive call, how will I perform this without losing performance when receiving data?
As last, which buffer size should be used? should this be the MTU of tcp which is 1500 bytes in most cases?

I would suggest sending the total file size as the first step in your communication, before sending along the actual file. For example your StateObject class could have an additional property that holds the total size of the thing your transmitting, then in a different thread (the UI thread) you can create a progress bar and report based of the total transmitted vs the total size of the file as a percentage or whatever you like. Just track how much data has been transmitted as it occurs and update your object appropriately as it does so.

In addition to CG's answer, I would also skip the stringbuilder and use something like:
FileStream fs = File.OpenRead(filePath);
Byte[] bytes = new Byte[fs.Length];
fs.Read(bytes, 0, Convert.ToInt32(fs.Length));
fs.Close();