Develop Reference

Sockets starts to slow down and not respond

i am developing a server (with c#) and a client (with flash, actionscript 3.0) application. Server sends data (datas are arround 90 bytes) to clients continuously and clients behave according to data they received (data is json formatted)
for a while, everything works as expected but after some time passed, clients start to receive messages laggy. they keep waiting for some time and then they behave according to last message (some messages lost). after some time passed clients starts to wait and process all the messages at the same time. I could not figured out what causing this. My network condition is stable.
here is some part of my c# code, sending message:
public void Send(byte[] buffer)
{
if (ClientSocket != null && ClientSocket.Connected)
{
ClientSocket.BeginSend(buffer, 0, buffer.Length, 0, WriteCallback, ClientSocket);
}
}
private void WriteCallback(IAsyncResult result)
{
//
}
and some part of my client, receiving message (actionscript)
socket.addEventListener(ProgressEvent.SOCKET_DATA, onResponse);
function onResponse(e:ProgressEvent):void {
trace(socket.bytesAvailable);
if(socket.bytesAvailable > 0) {
try
{
var serverResponse:String = socket.readUTFBytes(socket.bytesAvailable);
....
I hope i could explain my problem. How should i optimize my code? What can be causing lags. Thanks.

You really need to give more detail as to how you're setting up the socket (is it TCP or UDP?)
Assuming it's a TCP socket, then it would appear that your client relies on each receive call returning the same number of bytes that were sent by the server's Send() call. This is however not the case, and could well be the cause of your issues if a message is only being partially received on the client, or multiple messages are received at once.
For example, the server may send a 90 byte message in a single call, but your client may receive it in one 90-byte receive, or two 45-byte chunks, or even 90 x 1-byte chunks, or anything in between. Multiple messages sent by the server may also be partially combined when received by the client. E.g. two 90-byte messages may be received in a single 180-byte chunk, or a 150-byte and a 30-byte chunk, etc. etc.
You need therefore to provide some kind of framing on your messages so that when the stream of data is received by the client, it can be reliably reconstructed into individual messages.
The most basic framing mechanism would be to prefix each message sent with a fixed-length field indicating the message size. you may be able to get away with a single byte if you can guarantee that your messages will never be > 255 bytes long, which will simplify the receiving code.
On the client side, you first need to receive the length prefix, and then read up to that many bytes off the socket to construct the message data. If you receive fewer than the required number of bytes, your receiving code must wait for more data (appending it to the partially-received message when it is eventually received) until it has a complete message of the.
Once the full message is received it can be processed as you are currently.
Unfortunately I don't know ActionScript, so can't give you an example of the client-side code, but here's how you might write the server and client framing in C#:
Server side:
public void SendMessage(string message)
{
var data = Encoding.UTF8.GetBytes(message);
if (data.Length > byte.MaxValue) throw new Exception("Data exceeds maximum size");
var bufferList = new[]
{
new ArraySegment<byte>(new[] {(byte) data.Length}),
new ArraySegment<byte>(data)
};
ClientSocket.Send(bufferList);
}
Client side:
public string ReadMessage()
{
var header = new byte[1];
// Read the header indicating the data length
var bytesRead = ServerSocket.Receive(header);
if (bytesRead > 0)
{
var dataLength = header[0];
// If the message size is zero, return an empty string
if (dataLength == 0) return string.Empty;
var buffer = new byte[dataLength];
var position = 0;
while ((bytesRead = ServerSocket.Receive(buffer, position, buffer.Length - position, SocketFlags.None)) > 0)
{
// Advance the position by the number of bytes read
position += bytesRead;
// If there's still more data to read before we have a full message, call Receive again
if (position < buffer.Length) continue;
// We have a complete message - return it.
return Encoding.UTF8.GetString(buffer);
}
}
// If Receive returns 0, the socket has been closed, so return null to indicate this.
return null;
}

Best way to accept multiple tcp clients?

I have a client/server infrastructure. At present they use a TcpClient and TcpListener to send a receive data between all the clients and server.
What I currently do is when data is received (on it's own thread), it is put in a queue for another thread to process in order to free the socket so it is ready and open to receive new data.
// Enter the listening loop.
while (true)
{
Debug.WriteLine("Waiting for a connection... ");
// Perform a blocking call to accept requests.
using (client = server.AcceptTcpClient())
{
data = new List<byte>();
// Get a stream object for reading and writing
using (NetworkStream stream = client.GetStream())
{
// Loop to receive all the data sent by the client.
int length;
while ((length = stream.Read(bytes, 0, bytes.Length)) != 0)
{
var copy = new byte[length];
Array.Copy(bytes, 0, copy, 0, length);
data.AddRange(copy);
}
}
}
receivedQueue.Add(data);
}
However I wanted to find out if there is a better way to do this. For example if there are 10 clients and they all want to send data to the server at the same time, one will get through while all the others will fail.Or if one client has a slow connection and hogs the socket all other communication will halt.
Is there not some way to be able to receive data from all clients at the same time and add the received data in the queue for processing when it has finished downloading?

So here is an answer that will get you started - which is more beginner level than my blog post.
.Net has an async pattern that revolves around a Begin* and End* call. For instance - BeginReceive and EndReceive. They nearly always have their non-async counterpart (in this case Receive); and achieve the exact same goal.
The most important thing to remember is that the socket ones do more than just make the call async - they expose something called IOCP (IO Completion Ports, Linux/Mono has these two but I forget the name) which is extremely important to use on a server; the crux of what IOCP does is that your application doesn't consume a thread while it waits for data.
How to Use The Begin/End Pattern
Every Begin* method will have exactly 2 more arguments in comparisson to it's non-async counterpart. The first is an AsyncCallback, the second is an object. What these two mean is, "here is a method to call when you are done" and "here is some data I need inside that method." The method that gets called always has the same signature, inside this method you call the End* counterpart to get what would have been the result if you had done it synchronously. So for example:
private void BeginReceiveBuffer()
{
_socket.BeginReceive(buffer, 0, buffer.Length, BufferEndReceive, buffer);
}
private void EndReceiveBuffer(IAsyncResult state)
{
var buffer = (byte[])state.AsyncState; // This is the last parameter.
var length = _socket.EndReceive(state); // This is the return value of the method call.
DataReceived(buffer, 0, length); // Do something with the data.
}
What happens here is .Net starts waiting for data from the socket, as soon as it gets data it calls EndReceiveBuffer and passes through the 'custom data' (in this case buffer) to it via state.AsyncResult. When you call EndReceive it will give you back the length of the data that was received (or throw an exception if something failed).
Better Pattern for Sockets
This form will give you central error handling - it can be used anywhere where the async pattern wraps a stream-like 'thing' (e.g. TCP arrives in the order it was sent, so it could be seen as a Stream object).
private Socket _socket;
private ArraySegment<byte> _buffer;
public void StartReceive()
{
ReceiveAsyncLoop(null);
}
// Note that this method is not guaranteed (in fact
// unlikely) to remain on a single thread across
// async invocations.
private void ReceiveAsyncLoop(IAsyncResult result)
{
try
{
// This only gets called once - via StartReceive()
if (result != null)
{
int numberOfBytesRead = _socket.EndReceive(result);
if(numberOfBytesRead == 0)
{
OnDisconnected(null); // 'null' being the exception. The client disconnected normally in this case.
return;
}
var newSegment = new ArraySegment<byte>(_buffer.Array, _buffer.Offset, numberOfBytesRead);
// This method needs its own error handling. Don't let it throw exceptions unless you
// want to disconnect the client.
OnDataReceived(newSegment);
}
// Because of this method call, it's as though we are creating a 'while' loop.
// However this is called an async loop, but you can see it the same way.
_socket.BeginReceive(_buffer.Array, _buffer.Offset, _buffer.Count, SocketFlags.None, ReceiveAsyncLoop, null);
}
catch (Exception ex)
{
// Socket error handling here.
}
}
Accepting Multiple Connections
What you generally do is write a class that contains your socket etc. (as well as your async loop) and create one for each client. So for instance:
public class InboundConnection
{
private Socket _socket;
private ArraySegment<byte> _buffer;
public InboundConnection(Socket clientSocket)
{
_socket = clientSocket;
_buffer = new ArraySegment<byte>(new byte[4096], 0, 4096);
StartReceive(); // Start the read async loop.
}
private void StartReceive() ...
private void ReceiveAsyncLoop() ...
private void OnDataReceived() ...
}
Each client connection should be tracked by your server class (so that you can disconnect them cleanly when the server shuts down, as well as search/look them up).

You should use asynchronous socket programming to achieve this. Take a look at the example provided by MSDN.

You should use asynchronous method of reading the data, an example is:
// Enter the listening loop.
while (true)
{
Debug.WriteLine("Waiting for a connection... ");
client = server.AcceptTcpClient();
ThreadPool.QueueUserWorkItem(new WaitCallback(HandleTcp), client);
}
private void HandleTcp(object tcpClientObject)
{
TcpClient client = (TcpClient)tcpClientObject;
// Perform a blocking call to accept requests.
data = new List<byte>();
// Get a stream object for reading and writing
using (NetworkStream stream = client.GetStream())
{
// Loop to receive all the data sent by the client.
int length;
while ((length = stream.Read(bytes, 0, bytes.Length)) != 0)
{
var copy = new byte[length];
Array.Copy(bytes, 0, copy, 0, length);
data.AddRange(copy);
}
}
receivedQueue.Add(data);
}
Also you should consider using AutoResetEvent or ManualResetEvent to be notified when new data is added to the collection so the thread that handle the data will know when data is received, and if you are using 4.0 you better switch off to using BlockingCollection instead of Queue.

What I do usually is using a thread pool with several threads.
Upon each new connection I'm running the connection handling (in your case - everything you do in the using clause) in one of the threads from the pool.
By that you achieve both performance since you're allowing several simultaneously accepted connection and you also limiting the number of resources (threads, etc') you allocate for handling incoming connections.
You have a nice example here
Good Luck

C# Socket BeginReceive / EndReceive capturing multiple messages

Problem:
When I do something like this:
for (int i = 0; i < 100; i++)
{
SendMessage( sometSocket, i.ToString());
Thread.Sleep(250); // works with this, doesn't work without
}
With or without the sleep the server logs sending of separate messages. However without the sleep the client ends up receiving multiple messages in single OnDataReceived so the client will receive messages like:
0,
1,
2,
34,
5,
678,
9 ....
Server sending Code:
private void SendMessage(Socket socket, string message)
{
logger.Info("SendMessage: Preparing to send message:" + message);
byte[] byteData = Encoding.ASCII.GetBytes(message);
if (socket == null) return;
if (!socket.Connected) return;
logger.Info("SendMessage: Sending message to non " +
"null and connected socket with ip:" + socket.RemoteEndPoint);
// Record this message so unit testing can very this works.
socket.Send(byteData);
}
Client receiving code:
private void OnDataReceived(IAsyncResult asyn)
{
logger.Info("OnDataReceived: Data received.");
try
{
SocketPacket theSockId = (SocketPacket)asyn.AsyncState;
int iRx = theSockId.Socket.EndReceive(asyn);
char[] chars = new char[iRx + 1];
System.Text.Decoder d = System.Text.Encoding.UTF8.GetDecoder();
int charLen = d.GetChars(theSockId.DataBuffer, 0, iRx, chars, 0);
System.String szData = new System.String(chars);
logger.Info("OnDataReceived: Received message:" + szData);
InvokeMessageReceived(new SocketMessageEventArgs(szData));
WaitForData(); // .....
Socket Packet:
public class SocketPacket
{
private Socket _socket;
private readonly int _clientNumber;
private byte[] _dataBuffer = new byte[1024]; ....
My hunch is it's something to do with the buffer size or its just the between the OnDataReceived and EndReceive we're getting multiple messages.
Update: It turns out when I put a Thread.Sleep at the start of OnDataReceived it gets every message. Is the only solution to this wrapping my message in a prefix of length and an string to signify the end?

This is expected behaviour. A TCP socket represents a linear stream of bytes, not a sequence of well-delimited “packets”. You must not assume that the data you receive is chunked the same way it was when it was sent.
Notice that this has two consequences:
Two messages may get merged into a single callback call. (You noticed this one.)
A single message may get split up (at any point) into two separate callback calls.
Your code must be written to handle both of these cases, otherwise it has a bug.

There is no need to abandon Tcp because it is stream oriented.
You can fix the problems that you are having by implementing message framing.
See
http://blogs.msdn.com/malarch/archive/2006/06/26/647993.aspx
also:
http://nitoprograms.blogspot.com/2009/04/message-framing.html

TCP sockets don't always send data right away -- in order to minimize network traffic, TCP/IP implementations will often buffer the data for a bit and send it when it sees there's a lull (or when the buffer's full).
If you want to ensure that the messages are processed one by one, you'll need to either set socket.NoDelay = true (which might not help much, since data received may still be bunched up together in the receive buffer), implement some protocol to separate messages in the stream (like prefixing each message with its length, or perhaps using CR/LF to separate them), or use a message-oriented protocol like SCTP (which might not be supported without additional software) or UDP (if you can deal with losing messages).

What is stopping data flow with .NET 3.5 asynchronous System.Net.Sockets.Socket?

I have a .NET 3.5 client/server socket interface using the asynchronous methods. The client connects to the server and the connection should remain open until the app terminates. The protocol consists of the following pattern:
send stx
receive ack
send data1
receive ack
send data2 (repeat 5-6 while more data)
receive ack
send etx
So a single transaction with two datablocks as above would consist of 4 sends from the client. After sending etx the client simply waits for more data to send out, then begins the next transmission with stx. I do not want to break the connection between individual exchanges or after each stx/data/etx payload.
Right now, after connection, the client can send the first stx, and get a single ack, but I can't put more data onto the wire after that. Neither side disconnects, the socket is still intact. The client code is seriously abbreviated as follows - I'm following the pattern commonly available in online code samples.
private void SendReceive(string data) {
// ...
SocketAsyncEventArgs completeArgs;
completeArgs.Completed += new EventHandler<SocketAsyncEventArgs>(OnSend);
clientSocket.SendAsync(completeArgs);
// two AutoResetEvents, one for send, one for receive
if ( !AutoResetEvent.WaitAll(autoSendReceiveEvents , -1) )
Log("failed");
else
Log("success");
// ...
}
private void OnSend( object sender , SocketAsyncEventArgs e ) {
// ...
Socket s = e.UserToken as Socket;
byte[] receiveBuffer = new byte[ 4096 ];
e.SetBuffer(receiveBuffer , 0 , receiveBuffer.Length);
e.Completed += new EventHandler<SocketAsyncEventArgs>(OnReceive);
s.ReceiveAsync(e);
// ...
}
private void OnReceive( object sender , SocketAsyncEventArgs e ) {}
// ...
if ( e.BytesTransferred > 0 ) {
Int32 bytesTransferred = e.BytesTransferred;
String received = Encoding.ASCII.GetString(e.Buffer , e.Offset , bytesTransferred);
dataReceived += received;
}
autoSendReceiveEvents[ SendOperation ].Set(); // could be moved elsewhere
autoSendReceiveEvents[ ReceiveOperation ].Set(); // releases mutexes
}
The code on the server is very similar except that it receives first and then sends a response - the server is not doing anything (that I can tell) to modify the connection after it sends a response. The problem is that the second time I hit SendReceive in the client, the connection is already in a weird state.
Do I need to do something in the client to preserve the SocketAsyncEventArgs, and re-use the same object for the lifetime of the socket/connection? I'm not sure which eventargs object should hang around during the life of the connection or a given exchange.
Do I need to do something, or Not do something in the server to ensure it continues to Receive data?
The server setup and response processing looks like this:
void Start() {
// ...
listenSocket.Bind(...);
listenSocket.Listen(0);
StartAccept(null); // note accept as soon as we start. OK?
mutex.WaitOne();
}
void StartAccept(SocketAsyncEventArgs acceptEventArg) {
if ( acceptEventArg == null )
{
acceptEventArg = new SocketAsyncEventArgs();
acceptEventArg.Completed += new EventHandler<SocketAsyncEventArgs>(OnAcceptCompleted);
}
Boolean willRaiseEvent = this.listenSocket.AcceptAsync(acceptEventArg);
if ( !willRaiseEvent )
ProcessAccept(acceptEventArg);
// ...
}
private void OnAcceptCompleted( object sender , SocketAsyncEventArgs e ) {
ProcessAccept(e);
}
private void ProcessAccept( SocketAsyncEventArgs e ) {
// ...
SocketAsyncEventArgs readEventArgs = new SocketAsyncEventArgs();
readEventArgs.SetBuffer(dataBuffer , 0 , Int16.MaxValue);
readEventArgs.Completed += new EventHandler<SocketAsyncEventArgs>(OnIOCompleted);
readEventArgs.UserToken = e.AcceptSocket;
dataReceived = ""; // note server is degraded for single client/thread use
// As soon as the client is connected, post a receive to the connection.
Boolean willRaiseEvent = e.AcceptSocket.ReceiveAsync(readEventArgs);
if ( !willRaiseEvent )
this.ProcessReceive(readEventArgs);
// Accept the next connection request.
this.StartAccept(e);
}
private void OnIOCompleted( object sender , SocketAsyncEventArgs e ) {
// switch ( e.LastOperation )
case SocketAsyncOperation.Receive:
ProcessReceive(e); // similar to client code
// operate on dataReceived here
case SocketAsyncOperation.Send:
ProcessSend(e); // similar to client code
}
// execute this when a data has been processed into a response (ack, etc)
private SendResponseToClient(string response) {
// create buffer with response
// currentEventArgs has class scope and is re-used
currentEventArgs.SetBuffer(sendBuffer , 0 , sendBuffer.Length);
Boolean willRaiseEvent = currentClient.SendAsync(currentEventArgs);
if ( !willRaiseEvent )
ProcessSend(currentEventArgs);
}
A .NET trace shows the following when sending ABC\r\n:
Socket#7588182::SendAsync()
Socket#7588182::SendAsync(True#1)
Data from Socket#7588182::FinishOperation(SendAsync)
00000000 : 41 42 43 0D 0A
Socket#7588182::ReceiveAsync()
Exiting Socket#7588182::ReceiveAsync() -> True#1
And it stops there. It looks just like the first send from the client but the server shows no activity.
I think that could be info overload for now but I'll be happy to provide more details as required.
Thanks!

Firstly, I'm sure you know, but it's worth repeating; a TCP connection is a stream of bytes. Each read can return between 1 byte and the size of the buffer that you used depending on the amount of data that has arrived. Just because you send the data with 3 sends doesn't mean that you'll need 3 recvs to read it, you could get it all in one, or each recv could return a single byte. It's up to YOU to deal with message framing as you're the only one that knows about it. TCP is a stream of bytes.
Also WHY are you using those events? If you don't want to use asynchronous data flow then don't use the async functions, write something very simple with the sync socket functions and remove all the pointless complexity of using an async API and then using synchronisation primitives to hobble it.

Windows socket.Send data isn't received until socket.Close

I'm developing a server application that asynchronously accepts TCP connections (BeginAccept/EndAccept) and data (BeginReceive/EndReceive). The protocol requires an ACK to be sent whenever the EOM character is found before it will send the next message. The accept and receive are working but the sending app is not receiving the ACK (sent synchronously).
private void _receiveTransfer(IAsyncResult result)
{
SocketState state = result.AsyncState as SocketState;
int bytesReceived = state.Socket.EndReceive(result);
if (bytesReceived == 0)
{
state.Socket.Close();
return;
}
state.Offset += bytesReceived;
state.Stream.Write(state.Buffer, 0, bytesReceived);
if (state.Buffer[bytesReceived - 1] == 13)
{
// process message
Messages.IMessage message = null;
try
{
var value = state.Stream.ToArray();
// do some work
var completed = true;
if (completed)
{
// send positive ACK
var ackMessage = string.Format(ack, message.TimeStamp.ToString("yyyyMMddhhmm"), message.MessageType, message.Id, "AA", message.Id);
var buffer = ASCIIEncoding.ASCII.GetBytes(ackMessage);
int bytesSent = state.Socket.Send(buffer, 0, buffer.Length, SocketFlags.None);
}
else
{
// send rejected ACK
var ackMessage = string.Format(ack, message.TimeStamp.ToString("yyyyMMddhhmm"), message.MessageType, message.Id, "AR", message.Id);
state.Socket.Send(ASCIIEncoding.ASCII.GetBytes(ackMessage));
}
}
catch (Exception e)
{
// log exception
// send error ACK
if (message != null)
{
var ackMessage = string.Format(ack, DateTime.Now.ToString("yyyyMMddhhmm"), message.MessageType, message.Id, "AE", message.Id);
state.Socket.Send(ASCIIEncoding.ASCII.GetBytes(ackMessage));
}
}
}
state.Socket.BeginReceive(state.Buffer, 0, state.Buffer.Length, SocketFlags.None, new AsyncCallback(_receiveTransfer), state);
}
The state.Socket.Send returns the correct number of bytes but the data isn't received until the socket is disposed.
Suggestions are appreciated.

you shouldn't do anything synchronous from async completion routines. Under load you can end up hijacking all IO completion threads from the thread pool and severly hurt performance, up to and including complete IO deadlock. So don't send ACKs synchronously from async callback.
protocols and formats that use preambles are easier to manage that those that use terminators. Ie. write the length of the message in the fixed size message header as opposed to detecting a terminator \0x13. Of course, this applies if the protocol is under your control to start with.
As for your question, you didn't specify if the same code as you posted is also on the client side too.

How long are you giving it? The network stack can buffer, and that could delay transmition. From MSDN:
To increase network efficiency, the
underlying system may delay
transmission until a significant
amount of outgoing data is collected.
A successful completion of the Send
method means that the underlying
system has had room to buffer your
data for a network send.
You might want to try flushing using the IOControl method.
edit
Actually, the IOControl flush will kill the buffer. You may want to check out the Two Generals Problem to see if your protocol will have some inherent problems.

try setting TCP_NODELAY socket option

Have you set the NoDelay property on the socket to true? When set to false (the default), data is buffered for up to 200 milliseconds before it's sent. The reason is to reduce network traffic by limiting the number of packets that are sent. Setting NoDelay to true will force the data to be sent sooner.