I'm currently trying to setup a server that accepts multiple clients and can receive and respond to messages.
The client and server use a common library at the core, which contains a Request class that gets serialized and sent from client to server and similar in reverse.
The server listens asyncronously to clients on each of their sockets and attempts to take the data received and deserialize the data into the Request class.
Data is sent via a NetworkStream using a BinaryFormatter to send directly on the socket. The received data is then parsed using a Network Stream on the other end.
I've tried using a MemoryStream to store the data to a buffer and then deserialize it as shown below, however this hasn't worked. Directly deserializing the NetworkStream didn't work either.
Searching around I haven't found much information that has worked for my use case.
This is the active code after the sockets are successfully connected:
On the request class, sending from the client:
public void SendData(Socket socket)
{
IFormatter formatter = new BinaryFormatter();
Stream stream = new NetworkStream(socket, false);
formatter.Serialize(stream, this);
stream.Close();
}
Server Code receiving this data:
public void Receive(Socket socket)
{
try
{
ReceiveState state = new ReceiveState(socket);
state.Stream.BeginRead(state.Buffer, 0, ReceiveState.BUFFER_SIZE, new AsyncCallback(DataReceived), state);
}
catch (Exception e)
{
Logger.LogError(e.ToString());
}
}
private void DataReceived(IAsyncResult ar)
{
ReceiveState state = (ReceiveState)ar.AsyncState;
int bytesRead = state.Stream.EndRead(ar);
//Resolve Message
try
{
IFormatter formatter = new BinaryFormatter();
MemoryStream memoryStream = new MemoryStream(state.Buffer, 0, bytesRead);
Request request = (Request)formatter.Deserialize(memoryStream);
Logger.Log("Request Received Successfully");
ResolveRequest(request, state.Socket);
}
catch (Exception e)
{
Logger.LogError(e.ToString());
}
//Resume listening
Receive(state.Socket);
}
public class ReceiveState
{
public byte[] Buffer;
public const int BUFFER_SIZE = 1024;
public Socket Socket;
public NetworkStream Stream;
public ReceiveState(Socket socket)
{
Buffer = new byte[BUFFER_SIZE];
Socket = socket;
Stream = new NetworkStream(Socket, false);
}
}
Currently, when BeginRead() is called on the NetworkStream I get a single byte of data, then the remaining data when the next BeginRead() is called.
e.g. The Serialized data should be: 00-01-00-00-00-FF-FF-FF-FF-01-...
I receive: 00 followed by 01-00-00-00-FF-FF-FF-FF-01-... which fails to deserialize.
I take it that the issue is that the DataReceived() method is called as soon as any data appears, which is the single byte taken, then the remainder arrives before listening is resumed.
Is there a way to make sure each message is received in full before deserializing? I'd like to be able to deserialize the object as soon as the last byte is received.
TCP is a stream protocol, not a packet protocol. That means you are only guaranteed to get the same bytes in the same order (or a network failure); you are not guaranteed to get them in the same chunk configurations. So: you need to implement your own framing protocol. A frame is how you partition messages. For binary messages, a simple framing protocol might be "length = 4 bytes little-endian int32, followed by {length} bytes of payload", in which case the correct decode is to buffer until you have 4 bytes, decode the length, buffer {length} bytes, then decode the payload. YOU NEED TO WRITE the code that buffers the correct amounts, and at every point you need to deal with over-reading, back-buffers, etc. It is a complex topic. Frankly, a lot of the nuances are solved by using the "pipelines" API (I have a multi-part discussion on that API here).
However, additional guidance:
never ever use BinaryFormatter, especially for scenarios like this; it will hurt you, and it is not a good fit for most use-cases (it also isn't a particularly good serializer); my recommendation would be something like protobuf (perhaps protobuf-net), but I'm arguably biased
network code is subtle and complex, and RPC is largely a "solved" problem; consider trying tools like gRPC instead of rolling it yourself; this can be very easy
Related
I'm working on a asynchronous TCP server class that uses a TcpListener object. I'm using the BeginAcceptTcpClient method for the TcpListener, and when the callback fires and EndAcceptTcpClient, I get a TcpClient object. In order to receive and send with this TcpClient I need to use the NetworkStream provided by the client object.
The way I've been using the NetworkStream feels wrong though. I call BeginRead and a callback to eventually use EndRead, but this requires that I use byte[] buffer. This has worked fine for me so far, but I have to wonder if there is a cleaner way of doing things. My current flow is so as follows: receive data into a byte[] buffer, throw data into a MemoryStream, use a BinaryReader to get the data that I'm passing, and then I can ultimately get what I need for my protocol.
Is there a more elegant way to get from NetworkStream to BinaryReader (and ultimately BinaryWriter as I'm going to pass data back similarly as I received it)? It feels wasteful that I must first dump it into a byte[], then into a MemoryStream (does that copy the data?), then finally be able to create a reader/writer object.
I've looked into simply creating a BinaryReader/BinaryWriter using the NetworkStream, but from what I've gathered, those objects require a stream with data previously available. It seems like I just need to have some way to be notified that the NetworkStream has data available without reading into a buffer. Perhaps I am mistaken and this is exactly how NetworkStreams are supposed to be used. It just seems like things could be a lot more streamlined if I didn't have to copying buffers from one stream into another.
EDIT:
Here is an example of the source in question:
public class Server
{
TcpListener listener;
const int maxBufferSize = 0xFFFF;
byte[] clientBuffer = new byte[maxBuffersize];
public Server(IPAddress address, int port)
{
listener = new TcpListener(address, port);
}
public void Start()
{
listener.Start();
listener.BeginAcceptTcpClient(OnAccept, listener);
}
public void Stop()
{
listener.Stop();
}
private void OnAccept(IAsyncResult ar)
{
TcpListener listener = ar.AsyncState as TcpListener;
TcpClient client = listener.EndAcceptTcpClient(ar);
client.GetStream().BeginRead(clientBuffer, 0, maxBufferSize, OnReceive, client);
listener.BeginAcceptTcpClient(OnAccept, listener);
}
private void OnReceive(IAsyncResult ar)
{
TcpClient client = ar.AsyncState as TcpClient;
int len = client.GetStream().EndRead(ar);
if (len == 0)
{
client.Close();
return;
}
else
{
MemoryStream inStream = new MemoryStream(len == maxBufferSize ? clientBuffer : clientBuffer.Take(len).ToArray());
MemoryStream outStream = DoStuff(inStream); //Data goes off to the app at this point and returns a response stream
client.GetStream().Write(outStream.ToArray(), 0, outStream.Length);
inStream.Dispose();
outStream.Dispose();
}
}
}
My question revolves around what happens in OnReceive. You'll see that I finish the read operation with EndRead, at which point I can now retrieve the data from the byte[] field of the Server class. My concern is that the time spent copying data from the NetworkStream into an array, and then into a MemoryStream is wasteful (at least it feels that way, perhaps C# handles this stuff efficiently?)
Thanks in advance.
I've written a TcpClient and Server which are communicating via an SslStream.
The communication works, but when i send a message from the Client to the Server, first the Server reads 1 Byte, and in the next step the rest. Example: I want to send "test" via Client, and the Server receives first "t", and then "est"
Here is the code for the Client to send
public void Send(string text) {
byte[] message = Encoding.UTF8.GetBytes(text);
SecureStream.BeginWrite(message, 0, message.Length, new AsyncCallback(WriteCallback), null);
}
private void WriteCallback(IAsyncResult AR) {
}
And here the code the Server uses to read
private SslStream CryptedStream = ...;
private byte[] buffer = new byte[1024];
public void BeginReadCallback(IAsyncResult AsyncCall) {
// initialize variables
int bytesRead = 0;
try {
// retrieve packet
bytesRead = CryptedStream.EndRead(AsyncCall);
// check if client has disconnected
if (bytesRead > 0) {
// copy buffer to a temporary one
var temporaryBuffer = buffer;
Array.Resize(ref temporaryBuffer, bytesRead);
string read = Encoding.ASCII.GetString(temporaryBuffer);
SetText(read);
// read more data
CryptedStream.BeginRead(buffer, 0, 1024, new AsyncCallback(BeginReadCallback), null);
// client is still connected, read data from buffer
//ProcessPacket(temporaryBuffer, temporaryBuffer.Length, helper);
} else {
// client disconnected, do everything to disconnect the client
//DisconnectClient(helper);
}
} catch (Exception e) {
// encountered an error, closing connection
// Program.log.Add(e.ToString(), Logger.LogLevel.Error);
// DisconnectClient(helper);
}
}
Did i miss something?
Thanks for your help
As Lasse explained streaming APIs do not promise you to return a specific number of bytes per read.
The best fix for this is to not use sockets. Use a higher level API such as WCF, SignalR, HTTP, ...
If you insist you probably should use BinaryReader/Writer to send your data. That makes it quite easy. For example, it has string sending built-in. You also can manually length-prefix easily with those classes.
Probably, you don't need async IO and should not use it. If you insist you can at least get rid of the callbacks by using await.
I'm using C# and 32feet (version 3.5) to send blocks of 250 bytes over bluetooth (SPP) to an embedded device I'm currently writing firmware for.
I'm setting up my connection with the following code:
var client = new BluetoothClient();
client.Encrypt = true;
client.Connect(bluetoothAddress, ServiceClassId);
NetworkStream stream = client.GetStream();
var writer = new BinaryWriter(stream);
I got some problem with very low throughput and each block took about 100 ms to be transferred with the following code:
public void SendData(List<byte> data)
{
try
{
foreach (byte d in data)
{
writer.Write(d);
}
writer.Flush();
}
catch (Exception)
{
// The connection was lost
ConnectionCleanup();
}
}
After changing the code block above to the code below each block is transmitted within 4 ms.
try
{
writer.Write(data.ToArray());
writer.Flush();
}
catch (Exception)
{
// The connection was lost
ConnectionCleanup();
}
I'm struggling to understand how this "simple" code change can have such large impact on the throughput. Can anyone help me explain what's going on? I guess it has something to do with the underlying mechanism of 32feet?
I've changed the code back and forth and the result is the same every time. The transmitted data is also the same.
I've also connected to the device directly from Windows and then opened the COM-port in Realterm to send the same data. In this case I get similar throughput as using writer.Write(data.ToArray()).
I'm using the Microsoft Bluetooth Stack.
Take a look at the reference source for BinaryWriter, Write(byte) calls the underlying stream's WriteByte(byte), while Write(byte[]) calls Write(byte[], int, int). Looking further, we see that NetworkStream does not override the virtual method WriteByte so the base implementation is used:
// Writes one byte from the stream by calling Write(byte[], int, int).
// This implementation does not perform well because it allocates a new
// byte[] each time you call it, and should be overridden by any
// subclass that maintains an internal buffer. Then, it can help perf
// significantly for people who are writing one byte at a time.
public virtual void WriteByte(byte value)
{
byte[] oneByteArray = new byte[1];
oneByteArray[0] = value;
Write(oneByteArray, 0, 1);
}
Also, NetworkStream has no internal buffers, it simply passes the Write calls to the underlying Socket. You are making 250 network calls in the first case and 1 in the second, so the reason for the performance difference should be obvious.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How to asynchronously receive complex object in C#?
My complex object is of type IOrderedQueryable
It has 4 attributes, all of type List
I'm sending my object using asynchronous socket through this:
private void SendDatabaseObj(Socket handler, IOrderedQueryable<BuildHistory1> buildHistoryQueryResult)
{
byte[] byteData = ObjectToByteArray(buildHistoryQueryResult);
// Begin sending the data to the remote device.
handler.BeginSend(byteData, 0, byteData.Length, 0,
new AsyncCallback(SendCallback), handler);
}
ObjectToByteArray() function (Serializing the object before sending):
private byte[] ObjectToByteArray(Object obj)
{
BinaryFormatter bf = new BinaryFormatter();
MemoryStream ms = new MemoryStream();
bf.Serialize(ms, obj);
return ms.ToArray();
}
I'm receiving the object I sent through this:
private void ReceiveCallback_onQuery(IAsyncResult ar)
{
try
{
// Retrieve the state object and the client socket
// from the asynchronous state object.
StateObject state = (StateObject)ar.AsyncState;
Socket client = state.workSocket;
// Read data from the remote device.
int bytesRead = client.EndReceive(ar);
if (bytesRead > 0)
{
// There might be more data, so store the data received so far. But how to store?
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback_onQuery), state);
}
else
{
// All the data has arrived; put it in response.
if (dataReceived > 1)
{
//Use the deserializing function here to retrieve the object to its normal form
}
// Signal that all bytes have been received.
receiveDoneQuery.Set();
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
My de-serializing function:
private Object ByteArrayToObject(byte[] arrayBytes)
{
MemoryStream ms = new MemoryStream();
BinaryFormatter bf = new BinaryFormatter();
ms.Write(arrayBytes, 0, arrayBytes.Length);
ms.Seek(0, SeekOrigin.Begin);
Object obj = (Object)bf.Deserialize(ms);
return obj;
}
Now my question is in the receiving function "ReceiveCallback_onQuery()". If there are more data to be received, how to store previously received data?
EDIT:
I am aware of doing the code below but is there other ways of just storing the data received in a byte[] variable so I could convert them back to IOrderedQueryable
state.sb.Append(Encoding.ASCII.GetString(state.buffer, 0, bytesRead));
When streaming data, such as with sockets, you have to have, built-in to your protocol, some way to know where the beginning and ending of each message is in the data. This is handled automatically by tools for standard protocols (such as WCF for SOAP), but if you are going to design your own protocol, you'll need to implement it yourself. The simplest method would be to add a known delimiter character or string between each message, but then you need to be careful that the delimiter never appears within a message's data. Another alternative would be to send the length of the message in a message header, or to simply use fixed-length messages.
Don't pass a IQueryable over the network. You can't use the querying features on the other side. Instead create a class which represents a request (i.e. contains information about which items to receive) and a response (an object which contains an array of matching objects).
If you do not want to take care of the networking layer by yourself and want's a more lightweight approach than WCF you can use my Griffin.Networking library.
I've just uploaded a video that demonstrates how to create a simple chat client/server in 20 minutes: https://www.youtube.com/watch?v=DjJOaV2y1Lo
Sample code: https://github.com/jgauffin/Samples/tree/master/Griffin.Networking/ChatServerClient
I have recently started getting into NetworkStreams, and I had a question. I am currently creating a thread, and processing all incoming messages as they come in.
Here is the code to illustrate this:
client.Connect(serverEndPoint);
clientStream = client.GetStream();
client.NoDelay = true;
ctThread = new Thread(getMessage);
ctThread.Start();
private void getMessage()
{
while (true)
{
Byte[] data = new Byte[800];
String responseData = String.Empty;
Int32 bytes = clientStream.Read(data, 0, data.Length);
responseData = System.Text.Encoding.ASCII.GetString(data, 0, bytes);
MessageReceived(this, new ClientMessageEventArgs(responseData));
}
}
In the above, I raise an event "MessageReceived" which is handled according the the packet data. This works great, but also have a seperate case where I need to retrieve data immediately after I send my request.
Is it ok to have two streams per client? Is this even possible to do on the same port? How should this be handled? Essentially, I want to be able to Send and then Receive data immediately after (blocking way).
You can read and write from network streams independently and in a thread safe manner. i.e. reading from one thread and writing from another.
If you checkout the open source network communication library networkComms.net you can see how this is achieved independently in the sending method SendPacket() (line 1304) and receiving method IncomingPacketHandler() (line 802).
Mx