I am in the design phase of writing a new Windows service application that accepts TCP/IP connections for long running connections (i.e., this is not like HTTP where there are many short connections, but rather a client connects and stays connected for hours or days or even weeks).
I'm looking for ideas for the best way to design the network architecture. I'm going to need to start at least one thread for the service. I am considering using the Asynch API (BeginRecieve, etc.) since I don't know how many clients I will have connected at any given time (possibly hundreds). I definitely do not want to start a thread for each connection.
Data will primarily flow out to the clients from my server, but there will be some commands sent from the clients on occasion. This is primarily a monitoring application in which my server sends status data periodically to the clients.
What is the best way to make this as scalable as possible? Basic workflow?
To be clear, I'm looking for .NET-based solutions (C# if possible, but any .NET language will work).
I would need a working example of a solution, either as a pointer to something I could download or a short example in-line. And it must be .NET and Windows based (any .NET language is acceptable).
I've written something similar to this in the past. From my research years ago showed that writing your own socket implementation was the best bet, using the asynchronous sockets. This meant that clients not really doing anything actually required relatively few resources. Anything that does occur is handled by the .NET thread pool.
I wrote it as a class that manages all connections for the servers.
I simply used a list to hold all the client connections, but if you need faster lookups for larger lists, you can write it however you want.
private List<xConnection> _sockets;
Also you need the socket actually listening for incoming connections.
private System.Net.Sockets.Socket _serverSocket;
The start method actually starts the server socket and begins listening for any incoming connections.
public bool Start()
{
System.Net.IPHostEntry localhost = System.Net.Dns.GetHostEntry(System.Net.Dns.GetHostName());
System.Net.IPEndPoint serverEndPoint;
try
{
serverEndPoint = new System.Net.IPEndPoint(localhost.AddressList[0], _port);
}
catch (System.ArgumentOutOfRangeException e)
{
throw new ArgumentOutOfRangeException("Port number entered would seem to be invalid, should be between 1024 and 65000", e);
}
try
{
_serverSocket = new System.Net.Sockets.Socket(serverEndPoint.Address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
}
catch (System.Net.Sockets.SocketException e)
{
throw new ApplicationException("Could not create socket, check to make sure not duplicating port", e);
}
try
{
_serverSocket.Bind(serverEndPoint);
_serverSocket.Listen(_backlog);
}
catch (Exception e)
{
throw new ApplicationException("An error occurred while binding socket. Check inner exception", e);
}
try
{
//warning, only call this once, this is a bug in .net 2.0 that breaks if
// you're running multiple asynch accepts, this bug may be fixed, but
// it was a major pain in the rear previously, so make sure there is only one
//BeginAccept running
_serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
}
catch (Exception e)
{
throw new ApplicationException("An error occurred starting listeners. Check inner exception", e);
}
return true;
}
I'd just like to note the exception handling code looks bad, but the reason for it is I had exception suppression code in there so that any exceptions would be suppressed and return false if a configuration option was set, but I wanted to remove it for brevity sake.
The _serverSocket.BeginAccept(new AsyncCallback(acceptCallback)), _serverSocket) above essentially sets our server socket to call the acceptCallback method whenever a user connects. This method runs from the .NET threadpool, which automatically handles creating additional worker threads if you have many blocking operations. This should optimally handle any load on the server.
private void acceptCallback(IAsyncResult result)
{
xConnection conn = new xConnection();
try
{
//Finish accepting the connection
System.Net.Sockets.Socket s = (System.Net.Sockets.Socket)result.AsyncState;
conn = new xConnection();
conn.socket = s.EndAccept(result);
conn.buffer = new byte[_bufferSize];
lock (_sockets)
{
_sockets.Add(conn);
}
//Queue receiving of data from the connection
conn.socket.BeginReceive(conn.buffer, 0, conn.buffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), conn);
//Queue the accept of the next incoming connection
_serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
}
catch (SocketException e)
{
if (conn.socket != null)
{
conn.socket.Close();
lock (_sockets)
{
_sockets.Remove(conn);
}
}
//Queue the next accept, think this should be here, stop attacks based on killing the waiting listeners
_serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
}
catch (Exception e)
{
if (conn.socket != null)
{
conn.socket.Close();
lock (_sockets)
{
_sockets.Remove(conn);
}
}
//Queue the next accept, think this should be here, stop attacks based on killing the waiting listeners
_serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
}
}
The above code essentially just finished accepting the connection that comes in, queues BeginReceive which is a callback that will run when the client sends data, and then queues the next acceptCallback which will accept the next client connection that comes in.
The BeginReceive method call is what tells the socket what to do when it receives data from the client. For BeginReceive, you need to give it a byte array, which is where it will copy the data when the client sends data. The ReceiveCallback method will get called, which is how we handle receiving data.
private void ReceiveCallback(IAsyncResult result)
{
//get our connection from the callback
xConnection conn = (xConnection)result.AsyncState;
//catch any errors, we'd better not have any
try
{
//Grab our buffer and count the number of bytes receives
int bytesRead = conn.socket.EndReceive(result);
//make sure we've read something, if we haven't it supposadly means that the client disconnected
if (bytesRead > 0)
{
//put whatever you want to do when you receive data here
//Queue the next receive
conn.socket.BeginReceive(conn.buffer, 0, conn.buffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), conn);
}
else
{
//Callback run but no data, close the connection
//supposadly means a disconnect
//and we still have to close the socket, even though we throw the event later
conn.socket.Close();
lock (_sockets)
{
_sockets.Remove(conn);
}
}
}
catch (SocketException e)
{
//Something went terribly wrong
//which shouldn't have happened
if (conn.socket != null)
{
conn.socket.Close();
lock (_sockets)
{
_sockets.Remove(conn);
}
}
}
}
EDIT: In this pattern I forgot to mention that in this area of code:
//put whatever you want to do when you receive data here
//Queue the next receive
conn.socket.BeginReceive(conn.buffer, 0, conn.buffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), conn);
Generally, in the whatever you want code, I would do reassembly of packets into messages, and then create them as jobs on the thread pool. This way the BeginReceive of the next block from the client isn't delayed while whatever message processing code is running.
The accept callback finishes reading the data socket by calling end receive. This fills the buffer provided in the begin receive function. Once you do whatever you want where I left the comment, we call the next BeginReceive method which will run the callback again if the client sends any more data.
Now here's the really tricky part: When the client sends data, your receive callback might only be called with part of the message. Reassembly can become very very complicated. I used my own method and created a sort of proprietary protocol to do this. I left it out, but if you request, I can add it in. This handler was actually the most complicated piece of code I had ever written.
public bool Send(byte[] message, xConnection conn)
{
if (conn != null && conn.socket.Connected)
{
lock (conn.socket)
{
//we use a blocking mode send, no async on the outgoing
//since this is primarily a multithreaded application, shouldn't cause problems to send in blocking mode
conn.socket.Send(bytes, bytes.Length, SocketFlags.None);
}
}
else
return false;
return true;
}
The above send method actually uses a synchronous Send call. For me that was fine due to the message sizes and the multithreaded nature of my application. If you want to send to every client, you simply need to loop through the _sockets List.
The xConnection class you see referenced above is basically a simple wrapper for a socket to include the byte buffer, and in my implementation some extras.
public class xConnection : xBase
{
public byte[] buffer;
public System.Net.Sockets.Socket socket;
}
Also for reference here are the usings I include since I always get annoyed when they aren't included.
using System.Net.Sockets;
I hope that's helpful. It may not be the cleanest code, but it works. There are also some nuances to the code which you should be weary about changing. For one, only have a single BeginAccept called at any one time. There used to be a very annoying .NET bug around this, which was years ago so I don't recall the details.
Also, in the ReceiveCallback code, we process anything received from the socket before we queue the next receive. This means that for a single socket, we're only actually ever in ReceiveCallback once at any point in time, and we don't need to use thread synchronization. However, if you reorder this to call the next receive immediately after pulling the data, which might be a little faster, you will need to make sure you properly synchronize the threads.
Also, I hacked out a lot of my code, but left the essence of what's happening in place. This should be a good start for you're design. Leave a comment if you have any more questions around this.
There are many ways of doing network operations in C#. All of them use different mechanisms under the hood, and thus suffer major performance issues with a high concurrency. Begin* operations are one of these that many people often mistake for being the faster/fastest way of doing networking.
To solve these issues, they introduced the Async set of methods: From MSDN, SocketAsyncEventArgs Class -
The SocketAsyncEventArgs class is part of a set of enhancements to the System.Net.Sockets..::.Socket class that provide an alternative asynchronous pattern that can be used by specialized high-performance socket applications. This class was specifically designed for network server applications that require high performance. An application can use the enhanced asynchronous pattern exclusively or only in targeted hot areas (for example, when receiving large amounts of data).
The main feature of these enhancements is the avoidance of the repeated allocation and synchronization of objects during high-volume asynchronous socket I/O. The Begin/End design pattern currently implemented by the System.Net.Sockets..::.Socket class requires a System..::.IAsyncResult object be allocated for each asynchronous socket operation.
Under the covers, the *Async API uses I/O completion ports which is the fastest way of performing networking operations, see Windows Sockets 2.0: Write Scalable Winsock Apps Using Completion Ports
And just to help you out, I am including the source code for a telnet server I wrote using the *Async API. I am only including the relevant portions. Also to note, instead of processing the data inline, I instead opt to push it onto a lock free (wait free) queue that is processed on a separate thread. Note that I am not including the corresponding Pool class which is just a simple pool which will create a new object if it is empty, and the Buffer class which is just a self-expanding buffer which is not really needed unless you are receiving an indeterministic amount of data.
public class Telnet
{
private readonly Pool<SocketAsyncEventArgs> m_EventArgsPool;
private Socket m_ListenSocket;
/// <summary>
/// This event fires when a connection has been established.
/// </summary>
public event EventHandler<SocketAsyncEventArgs> Connected;
/// <summary>
/// This event fires when a connection has been shutdown.
/// </summary>
public event EventHandler<SocketAsyncEventArgs> Disconnected;
/// <summary>
/// This event fires when data is received on the socket.
/// </summary>
public event EventHandler<SocketAsyncEventArgs> DataReceived;
/// <summary>
/// This event fires when data is finished sending on the socket.
/// </summary>
public event EventHandler<SocketAsyncEventArgs> DataSent;
/// <summary>
/// This event fires when a line has been received.
/// </summary>
public event EventHandler<LineReceivedEventArgs> LineReceived;
/// <summary>
/// Specifies the port to listen on.
/// </summary>
[DefaultValue(23)]
public int ListenPort { get; set; }
/// <summary>
/// Constructor for Telnet class.
/// </summary>
public Telnet()
{
m_EventArgsPool = new Pool<SocketAsyncEventArgs>();
ListenPort = 23;
}
/// <summary>
/// Starts the telnet server listening and accepting data.
/// </summary>
public void Start()
{
IPEndPoint endpoint = new IPEndPoint(0, ListenPort);
m_ListenSocket = new Socket(endpoint.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
m_ListenSocket.Bind(endpoint);
m_ListenSocket.Listen(100);
//
// Post Accept
//
StartAccept(null);
}
/// <summary>
/// Not Yet Implemented. Should shutdown all connections gracefully.
/// </summary>
public void Stop()
{
//throw (new NotImplementedException());
}
//
// ACCEPT
//
/// <summary>
/// Posts a requests for Accepting a connection. If it is being called from the completion of
/// an AcceptAsync call, then the AcceptSocket is cleared since it will create a new one for
/// the new user.
/// </summary>
/// <param name="e">null if posted from startup, otherwise a <b>SocketAsyncEventArgs</b> for reuse.</param>
private void StartAccept(SocketAsyncEventArgs e)
{
if (e == null)
{
e = m_EventArgsPool.Pop();
e.Completed += Accept_Completed;
}
else
{
e.AcceptSocket = null;
}
if (m_ListenSocket.AcceptAsync(e) == false)
{
Accept_Completed(this, e);
}
}
/// <summary>
/// Completion callback routine for the AcceptAsync post. This will verify that the Accept occured
/// and then setup a Receive chain to begin receiving data.
/// </summary>
/// <param name="sender">object which posted the AcceptAsync</param>
/// <param name="e">Information about the Accept call.</param>
private void Accept_Completed(object sender, SocketAsyncEventArgs e)
{
//
// Socket Options
//
e.AcceptSocket.NoDelay = true;
//
// Create and setup a new connection object for this user
//
Connection connection = new Connection(this, e.AcceptSocket);
//
// Tell the client that we will be echo'ing data sent
//
DisableEcho(connection);
//
// Post the first receive
//
SocketAsyncEventArgs args = m_EventArgsPool.Pop();
args.UserToken = connection;
//
// Connect Event
//
if (Connected != null)
{
Connected(this, args);
}
args.Completed += Receive_Completed;
PostReceive(args);
//
// Post another accept
//
StartAccept(e);
}
//
// RECEIVE
//
/// <summary>
/// Post an asynchronous receive on the socket.
/// </summary>
/// <param name="e">Used to store information about the Receive call.</param>
private void PostReceive(SocketAsyncEventArgs e)
{
Connection connection = e.UserToken as Connection;
if (connection != null)
{
connection.ReceiveBuffer.EnsureCapacity(64);
e.SetBuffer(connection.ReceiveBuffer.DataBuffer, connection.ReceiveBuffer.Count, connection.ReceiveBuffer.Remaining);
if (connection.Socket.ReceiveAsync(e) == false)
{
Receive_Completed(this, e);
}
}
}
/// <summary>
/// Receive completion callback. Should verify the connection, and then notify any event listeners
/// that data has been received. For now it is always expected that the data will be handled by the
/// listeners and thus the buffer is cleared after every call.
/// </summary>
/// <param name="sender">object which posted the ReceiveAsync</param>
/// <param name="e">Information about the Receive call.</param>
private void Receive_Completed(object sender, SocketAsyncEventArgs e)
{
Connection connection = e.UserToken as Connection;
if (e.BytesTransferred == 0 || e.SocketError != SocketError.Success || connection == null)
{
Disconnect(e);
return;
}
connection.ReceiveBuffer.UpdateCount(e.BytesTransferred);
OnDataReceived(e);
HandleCommand(e);
Echo(e);
OnLineReceived(connection);
PostReceive(e);
}
/// <summary>
/// Handles Event of Data being Received.
/// </summary>
/// <param name="e">Information about the received data.</param>
protected void OnDataReceived(SocketAsyncEventArgs e)
{
if (DataReceived != null)
{
DataReceived(this, e);
}
}
/// <summary>
/// Handles Event of a Line being Received.
/// </summary>
/// <param name="connection">User connection.</param>
protected void OnLineReceived(Connection connection)
{
if (LineReceived != null)
{
int index = 0;
int start = 0;
while ((index = connection.ReceiveBuffer.IndexOf('\n', index)) != -1)
{
string s = connection.ReceiveBuffer.GetString(start, index - start - 1);
s = s.Backspace();
LineReceivedEventArgs args = new LineReceivedEventArgs(connection, s);
Delegate[] delegates = LineReceived.GetInvocationList();
foreach (Delegate d in delegates)
{
d.DynamicInvoke(new object[] { this, args });
if (args.Handled == true)
{
break;
}
}
if (args.Handled == false)
{
connection.CommandBuffer.Enqueue(s);
}
start = index;
index++;
}
if (start > 0)
{
connection.ReceiveBuffer.Reset(0, start + 1);
}
}
}
//
// SEND
//
/// <summary>
/// Overloaded. Sends a string over the telnet socket.
/// </summary>
/// <param name="connection">Connection to send data on.</param>
/// <param name="s">Data to send.</param>
/// <returns>true if the data was sent successfully.</returns>
public bool Send(Connection connection, string s)
{
if (String.IsNullOrEmpty(s) == false)
{
return Send(connection, Encoding.Default.GetBytes(s));
}
return false;
}
/// <summary>
/// Overloaded. Sends an array of data to the client.
/// </summary>
/// <param name="connection">Connection to send data on.</param>
/// <param name="data">Data to send.</param>
/// <returns>true if the data was sent successfully.</returns>
public bool Send(Connection connection, byte[] data)
{
return Send(connection, data, 0, data.Length);
}
public bool Send(Connection connection, char c)
{
return Send(connection, new byte[] { (byte)c }, 0, 1);
}
/// <summary>
/// Sends an array of data to the client.
/// </summary>
/// <param name="connection">Connection to send data on.</param>
/// <param name="data">Data to send.</param>
/// <param name="offset">Starting offset of date in the buffer.</param>
/// <param name="length">Amount of data in bytes to send.</param>
/// <returns></returns>
public bool Send(Connection connection, byte[] data, int offset, int length)
{
bool status = true;
if (connection.Socket == null || connection.Socket.Connected == false)
{
return false;
}
SocketAsyncEventArgs args = m_EventArgsPool.Pop();
args.UserToken = connection;
args.Completed += Send_Completed;
args.SetBuffer(data, offset, length);
try
{
if (connection.Socket.SendAsync(args) == false)
{
Send_Completed(this, args);
}
}
catch (ObjectDisposedException)
{
//
// return the SocketAsyncEventArgs back to the pool and return as the
// socket has been shutdown and disposed of
//
m_EventArgsPool.Push(args);
status = false;
}
return status;
}
/// <summary>
/// Sends a command telling the client that the server WILL echo data.
/// </summary>
/// <param name="connection">Connection to disable echo on.</param>
public void DisableEcho(Connection connection)
{
byte[] b = new byte[] { 255, 251, 1 };
Send(connection, b);
}
/// <summary>
/// Completion callback for SendAsync.
/// </summary>
/// <param name="sender">object which initiated the SendAsync</param>
/// <param name="e">Information about the SendAsync call.</param>
private void Send_Completed(object sender, SocketAsyncEventArgs e)
{
e.Completed -= Send_Completed;
m_EventArgsPool.Push(e);
}
/// <summary>
/// Handles a Telnet command.
/// </summary>
/// <param name="e">Information about the data received.</param>
private void HandleCommand(SocketAsyncEventArgs e)
{
Connection c = e.UserToken as Connection;
if (c == null || e.BytesTransferred < 3)
{
return;
}
for (int i = 0; i < e.BytesTransferred; i += 3)
{
if (e.BytesTransferred - i < 3)
{
break;
}
if (e.Buffer[i] == (int)TelnetCommand.IAC)
{
TelnetCommand command = (TelnetCommand)e.Buffer[i + 1];
TelnetOption option = (TelnetOption)e.Buffer[i + 2];
switch (command)
{
case TelnetCommand.DO:
if (option == TelnetOption.Echo)
{
// ECHO
}
break;
case TelnetCommand.WILL:
if (option == TelnetOption.Echo)
{
// ECHO
}
break;
}
c.ReceiveBuffer.Remove(i, 3);
}
}
}
/// <summary>
/// Echoes data back to the client.
/// </summary>
/// <param name="e">Information about the received data to be echoed.</param>
private void Echo(SocketAsyncEventArgs e)
{
Connection connection = e.UserToken as Connection;
if (connection == null)
{
return;
}
//
// backspacing would cause the cursor to proceed beyond the beginning of the input line
// so prevent this
//
string bs = connection.ReceiveBuffer.ToString();
if (bs.CountAfterBackspace() < 0)
{
return;
}
//
// find the starting offset (first non-backspace character)
//
int i = 0;
for (i = 0; i < connection.ReceiveBuffer.Count; i++)
{
if (connection.ReceiveBuffer[i] != '\b')
{
break;
}
}
string s = Encoding.Default.GetString(e.Buffer, Math.Max(e.Offset, i), e.BytesTransferred);
if (connection.Secure)
{
s = s.ReplaceNot("\r\n\b".ToCharArray(), '*');
}
s = s.Replace("\b", "\b \b");
Send(connection, s);
}
//
// DISCONNECT
//
/// <summary>
/// Disconnects a socket.
/// </summary>
/// <remarks>
/// It is expected that this disconnect is always posted by a failed receive call. Calling the public
/// version of this method will cause the next posted receive to fail and this will cleanup properly.
/// It is not advised to call this method directly.
/// </remarks>
/// <param name="e">Information about the socket to be disconnected.</param>
private void Disconnect(SocketAsyncEventArgs e)
{
Connection connection = e.UserToken as Connection;
if (connection == null)
{
throw (new ArgumentNullException("e.UserToken"));
}
try
{
connection.Socket.Shutdown(SocketShutdown.Both);
}
catch
{
}
connection.Socket.Close();
if (Disconnected != null)
{
Disconnected(this, e);
}
e.Completed -= Receive_Completed;
m_EventArgsPool.Push(e);
}
/// <summary>
/// Marks a specific connection for graceful shutdown. The next receive or send to be posted
/// will fail and close the connection.
/// </summary>
/// <param name="connection"></param>
public void Disconnect(Connection connection)
{
try
{
connection.Socket.Shutdown(SocketShutdown.Both);
}
catch (Exception)
{
}
}
/// <summary>
/// Telnet command codes.
/// </summary>
internal enum TelnetCommand
{
SE = 240,
NOP = 241,
DM = 242,
BRK = 243,
IP = 244,
AO = 245,
AYT = 246,
EC = 247,
EL = 248,
GA = 249,
SB = 250,
WILL = 251,
WONT = 252,
DO = 253,
DONT = 254,
IAC = 255
}
/// <summary>
/// Telnet command options.
/// </summary>
internal enum TelnetOption
{
Echo = 1,
SuppressGoAhead = 3,
Status = 5,
TimingMark = 6,
TerminalType = 24,
WindowSize = 31,
TerminalSpeed = 32,
RemoteFlowControl = 33,
LineMode = 34,
EnvironmentVariables = 36
}
}
There used to be a really good discussion of scalable TCP/IP using .NET written by Chris Mullins of Coversant. Unfortunately, it appears his blog has disappeared from its prior location, so I will try to piece together his advice from memory (some useful comments of his appear in this thread: C++ vs. C#: Developing a highly scalable IOCP server)
First and foremost, note that both using Begin/End and the Async methods on the Socket class make use of I/O completion ports (IOCP) to provide scalability. This makes a much bigger difference (when used correctly; see below) to scalability than which of the two methods you actually pick to implement your solution.
Chris Mullins' posts were based on using Begin/End, which is the one I personally have experience with. Note that Chris put together a solution based on this that scaled up to 10,000s of concurrent client connections on a 32-bit machine with 2 GB of memory, and well into 100,000s on a 64-bit platform with sufficient memory. From my own experience with this technique (although nowhere near this kind of load) I have no reason to doubt these indicative figures.
IOCP versus thread-per-connection or 'select' primitives
The reason you want to use a mechanism that uses IOCP under the hood is that it uses a very low-level Windows thread pool that does not wake up any threads until there is actual data on the I/O channel that you are trying to read from (note that IOCP can be used for file I/O as well). The benefit of this is that Windows does not have to switch to a thread only to find that there is no data yet anyway, so this reduces the number of context switches your server will have to make to the bare minimum required.
Context switches is what will definitely kill the 'thread-per-connection' mechanism, although this is a viable solution if you are only dealing with a few dozen connections. This mechanism is however by no stretch of the imagination 'scalable'.
Important considerations when using IOCP
Memory
First and foremost it is critical to understand that IOCP can easily result in memory issues under .NET if your implementation is too naive. Every IOCP BeginReceive call will result in "pinning" of the buffer you are reading into. For a good explanation of why this is a problem, see: Yun Jin's Weblog: OutOfMemoryException and Pinning.
Luckily this problem can be avoided, but it requires a bit of a trade-off. The suggested solution is to allocate a big byte[] buffer at application start-up (or close thereto), of at least 90 KB or-so (as of .NET 2, required size may be larger in later versions). The reason to do this is that large memory allocations automatically end up in a non-compacting memory segment (the large object heap) that is effectively automatically pinned. By allocating one large buffer at start-up you make sure that this block of unmovable memory is at a relatively 'low address' where it will not get in the way and cause fragmentation.
You then can use offsets to segment this one big buffer into separate areas for each connection that needs to read some data. This is where a trade-off comes into play; since this buffer needs to be pre-allocated, you will have to decide how much buffer space you need per connection, and what upper limit you want to set on the number of connections you want to scale to (or, you can implement an abstraction that can allocate additional pinned buffers once you need them).
The simplest solution would be to assign every connection a single byte at a unique offset within this buffer. Then you can make a BeginReceive call for a single byte to be read, and perform the rest of the reading as a result of the callback you get.
Processing
When you get the callback from the Begin call you made, it is very important to realise that the code in the callback will execute on the low-level IOCP thread. It is absolutely essential that you avoid lengthy operations in this callback. Using these threads for complex processing will kill your scalability just as effectively as using 'thread-per-connection'.
The suggested solution is to use the callback only to queue up a work item to process the incoming data, that will be executed on some other thread. Avoid any potentially blocking operations inside the callback so that the IOCP thread can return to its pool as quickly as possible. In .NET 4.0 I'd suggest the easiest solution is to spawn a Task, giving it a reference to the client socket and a copy of the first byte that was already read by the BeginReceive call. This task is then responsible for reading all data from the socket that represent the request you are processing, executing it, and then making a new BeginReceive call to queue the socket for IOCP once more. Pre .NET 4.0, you can use the ThreadPool, or create your own threaded work-queue implementation.
Summary
Basically, I'd suggest using Kevin's sample code for this solution, with the following added warnings:
Make sure the buffer you pass to BeginReceive is already 'pinned'
Make sure the callback you pass to BeginReceive does nothing more than queue up a task to handle the actual processing of the incoming data
When you do that, I have no doubt you could replicate Chris' results in scaling up to potentially hundreds of thousands of simultaneous clients (given the right hardware and an efficient implementation of your own processing code of course ;)
You already got the most part of the answer via the code samples above. Using asynchronous I/O operations is absolutely the way to go here. Async I/O is the way the Win32 is designed internally to scale. The best possible performance you can get is achieved using completion ports, binding your sockets to completion ports and have a thread pool waiting for completion port completion. The common wisdom is to have 2-4 threads per CPU (core) waiting for completion. I highly recommend to go over these three articles by Rick Vicik from the Windows Performance team:
Designing Applications for Performance - Part 1
Designing Applications for Performance - Part 2
Designing Applications for Performance - Part 3
The said articles cover mostly the native Windows API, but they are a must-read for anyone trying to get a grasp at scalability and performance. They do have some briefs on the managed side of things too.
The second thing you'll need to do is make sure you go over the Improving .NET Application Performance and Scalability book, that is available online. You will find pertinent and valid advice around the use of threads, asynchronous calls and locks in Chapter 5. But the real gems are in Chapter 17 where you'll find such goodies as practical guidance on tuning your thread pool. My applications had some serious problems until I adjusted the maxIothreads/maxWorkerThreads as per the recommendations in this chapter.
You say that you want to do a pure TCP server, so my next point is spurious. However, if you find yourself cornered and use the WebRequest class and its derivatives, be warned that there is a dragon guarding that door: the ServicePointManager. This is a configuration class that has one purpose in life: to ruin your performance. Make sure you free your server from the artificial imposed ServicePoint.ConnectionLimit or your application will never scale (I let you discover yourself what the default value is...). You may also reconsider the default policy of sending an Expect100Continue header in the HTTP requests.
Now about the core socket managed API, things are fairly easy on the Send side, but they are significantly more complex on the Receive side. In order to achieve high throughput and scale, you must ensure that the socket is not flow controlled, because you do not have a buffer posted for receive. Ideally for high performance you should post ahead 3-4 buffers and post new buffers as soon as you get one back (before you process the one got back), so you ensure that the socket always has somewhere to deposit the data coming from the network. You'll see why you probably won't be able to achieve this shortly.
After you're done playing with the BeginRead/BeginWrite API and start the serious work, you'll realize that you need security on your traffic, i.e., NTLM/Kerberos authentication and traffic encryption, or at least traffic tampering protection. The way you do this is you use the built in System.Net.Security.NegotiateStream (or SslStream if you need to go cross disparate domains). This means that instead of relying on straight socket asynchronous operations you will rely on the AuthenticatedStream asynchronous operations. As soon as you obtain a socket (either from connect on client or from accept on server) you create a stream on the socket and submit it for authentication, by calling either BeginAuthenticateAsClient or BeginAuthenticateAsServer. After the authentication completes (at least your safe from the native InitiateSecurityContext/AcceptSecurityContext madness...) you will do your authorization by checking the RemoteIdentity property of your Authenticated stream and doing whatever ACL verification your product must support.
After that you will send messages using the BeginWrite and you'll be receiving them with BeginRead. This is the problem I was talking before that you won't be able to post multiple receive buffers, because the AuthenticateStream classes don't support this. The BeginRead operation manages all the I/O internally until you have received an entire frame. Otherwise, it could not handle the message authentication (decrypt frame and validate signature on frame). Though in my experience the job done by the AuthenticatedStream classes is fairly good and shouldn't have any problem with it. I.e., you should be able to saturate a 1 Gbit/s network with only 4-5% CPU. The AuthenticatedStream classes will also impose the protocol-specific frame size limitations on you (16k for SSL, 12k for Kerberos).
This should get you started on the right track. I'm not going to post code here, and there is a perfectly good example on MSDN. I've done many projects like this and I was able to scale to about 1000 users connected without problems. Above that, you'll need to modify registry keys to allow the kernel for more socket handles. And make sure you deploy on a server OS, that is, Windows Server 2003, not Windows XP or Windows Vista (i.e., client OS), it makes a big difference.
BTW, make sure, if you have databases operations on the server or file I/O, you also use the async flavor for them, or you'll drain the thread pool in no time. For SQL Server connections, make sure you add the 'Asyncronous Processing=true' to the connection string.
I've got such a server running in some of my solutions. Here is a very detailed explanation of the different ways to do it in .NET: Get Closer to the Wire with High-Performance Sockets in .NET
Lately I've been looking for ways to improve our code and will be looking into this: "Socket Performance Enhancements in Version 3.5" that was included specifically "for use by applications that use asynchronous network I/O to achieve the highest performance".
"The main feature of these enhancements is the avoidance of the repeated allocation and synchronization of objects during high-volume asynchronous socket I/O. The Begin/End design pattern currently implemented by the Socket class for asynchronous socket I/O requires a System.IAsyncResult object be allocated for each asynchronous socket operation."
You can keep reading if you follow the link. I personally will be testing their sample code tomorrow to benchmark it against what I've got.
Here you can find working code for both client and server using the new 3.5 SocketAsyncEventArgs so you can test it within a couple minutes and go through the code. It is a simple approach, but it is the basis for starting a much larger implementation. Also this article from almost two years ago in MSDN Magazine was a interesting read.
Consider just using a WCF net TCP binding and a publish/subscribe pattern. WCF would allow you to focus (mostly) on your domain instead of plumbing...
There are lots of WCF samples and even a publish/subscribe framework available on IDesign's download section which may be useful: http://www.idesign.net
I am wondering about one thing:
I definitely do not want to start a
thread for each connection.
Why is that? Windows could handle hundreds of threads in an application since at least Windows 2000. I've done it, it's really easy to work with if the threads don't need to be synchronized. Especially given that you're doing a lot of I/O (so you're not CPU-bound, and a lot of threads would be blocked on either disk or network communication), I don't understand this restriction.
Have you tested the multi-threaded way and found it lacking in something? Do you intend to also have a database connection for each thread (that would kill the database server, so it's a bad idea, but it's easily solved with a 3-tier design). Are you worried that you'll have thousands of clients instead of hundreds, and then you'll really have problems? (Though I'd try a thousand threads or even ten thousand if I had 32+ GB of RAM - again, given that you're not CPU bound, thread switch time should be absolutely irrelevant.)
Here is the code - to see how this looks running, go to http://mdpopescu.blogspot.com/2009/05/multi-threaded-server.html and click on the picture.
Server class:
public class Server
{
private static readonly TcpListener listener = new TcpListener(IPAddress.Any, 9999);
public Server()
{
listener.Start();
Console.WriteLine("Started.");
while (true)
{
Console.WriteLine("Waiting for connection...");
var client = listener.AcceptTcpClient();
Console.WriteLine("Connected!");
// each connection has its own thread
new Thread(ServeData).Start(client);
}
}
private static void ServeData(object clientSocket)
{
Console.WriteLine("Started thread " + Thread.CurrentThread.ManagedThreadId);
var rnd = new Random();
try
{
var client = (TcpClient) clientSocket;
var stream = client.GetStream();
while (true)
{
if (rnd.NextDouble() < 0.1)
{
var msg = Encoding.ASCII.GetBytes("Status update from thread " + Thread.CurrentThread.ManagedThreadId);
stream.Write(msg, 0, msg.Length);
Console.WriteLine("Status update from thread " + Thread.CurrentThread.ManagedThreadId);
}
// wait until the next update - I made the wait time so small 'cause I was bored :)
Thread.Sleep(new TimeSpan(0, 0, rnd.Next(1, 5)));
}
}
catch (SocketException e)
{
Console.WriteLine("Socket exception in thread {0}: {1}", Thread.CurrentThread.ManagedThreadId, e);
}
}
}
Server main program:
namespace ManyThreadsServer
{
internal class Program
{
private static void Main(string[] args)
{
new Server();
}
}
}
Client class:
public class Client
{
public Client()
{
var client = new TcpClient();
client.Connect(IPAddress.Loopback, 9999);
var msg = new byte[1024];
var stream = client.GetStream();
try
{
while (true)
{
int i;
while ((i = stream.Read(msg, 0, msg.Length)) != 0)
{
var data = Encoding.ASCII.GetString(msg, 0, i);
Console.WriteLine("Received: {0}", data);
}
}
}
catch (SocketException e)
{
Console.WriteLine("Socket exception in thread {0}: {1}", Thread.CurrentThread.ManagedThreadId, e);
}
}
}
Client main program:
using System;
using System.Threading;
namespace ManyThreadsClient
{
internal class Program
{
private static void Main(string[] args)
{
// first argument is the number of threads
for (var i = 0; i < Int32.Parse(args[0]); i++)
new Thread(RunClient).Start();
}
private static void RunClient()
{
new Client();
}
}
}
Using .NET's integrated Async I/O (BeginRead, etc.) is a good idea if you can get all the details right. When you properly set up your socket/file handles it will use the OS's underlying IOCP implementation, allowing your operations to complete without using any threads (or, in the worst case, using a thread that I believe comes from the kernel's I/O thread pool instead of .NET's thread pool, which helps alleviate threadpool congestion.)
The main gotcha is to make sure that you open your sockets/files in non-blocking mode. Most of the default convenience functions (like File.OpenRead) don't do this, so you'll need to write your own.
One of the other main concerns is error handling - properly handling errors when writing asynchronous I/O code is much, much harder than doing it in synchronous code. It's also very easy to end up with race conditions and deadlocks even though you may not be using threads directly, so you need to be aware of this.
If possible, you should try and use a convenience library to ease the process of doing scalable asynchronous I/O.
Microsoft's Concurrency Coordination Runtime is one example of a .NET library designed to ease the difficulty of doing this kind of programming. It looks great, but as I haven't used it, I can't comment on how well it would scale.
For my personal projects that need to do asynchronous network or disk I/O, I use a set of .NET concurrency / I/O tools that I've built over the past year, called Squared.Task. It's inspired by libraries like imvu.task and twisted, and I've included some working examples in the repository that do network I/O. I also have used it in a few applications I've written - the largest publicly released one being NDexer (which uses it for threadless disk I/O). The library was written based on my experience with imvu.task and has a set of fairly comprehensive unit tests, so I strongly encourage you to try it out. If you have any issues with it, I'd be glad to offer you some assistance.
In my opinion, based on my experience using asynchronous/threadless I/O instead of threads is a worthwhile endeavor on the .NET platform, as long as you're ready to deal with the learning curve. It allows you to avoid the scalability hassles imposed by the cost of Thread objects, and in many cases, you can completely avoid the use of locks and mutexes by making careful use of concurrency primitives like futures and promises.
I used Kevin's solution, but he says that solution lacks code for reassembly of messages. Developers can use this code for reassembly of messages:
private static void ReceiveCallback(IAsyncResult asyncResult )
{
ClientInfo cInfo = (ClientInfo)asyncResult.AsyncState;
cInfo.BytesReceived += cInfo.Soket.EndReceive(asyncResult);
if (cInfo.RcvBuffer == null)
{
// First 2 byte is lenght
if (cInfo.BytesReceived >= 2)
{
//this calculation depends on format which your client use for lenght info
byte[] len = new byte[ 2 ] ;
len[0] = cInfo.LengthBuffer[1];
len[1] = cInfo.LengthBuffer[0];
UInt16 length = BitConverter.ToUInt16( len , 0);
// buffering and nulling is very important
cInfo.RcvBuffer = new byte[length];
cInfo.BytesReceived = 0;
}
}
else
{
if (cInfo.BytesReceived == cInfo.RcvBuffer.Length)
{
//Put your code here, use bytes comes from "cInfo.RcvBuffer"
//Send Response but don't use async send , otherwise your code will not work ( RcvBuffer will be null prematurely and it will ruin your code)
int sendLenghts = cInfo.Soket.Send( sendBack, sendBack.Length, SocketFlags.None);
// buffering and nulling is very important
//Important , set RcvBuffer to null because code will decide to get data or 2 bte lenght according to RcvBuffer's value(null or initialized)
cInfo.RcvBuffer = null;
cInfo.BytesReceived = 0;
}
}
ContinueReading(cInfo);
}
private static void ContinueReading(ClientInfo cInfo)
{
try
{
if (cInfo.RcvBuffer != null)
{
cInfo.Soket.BeginReceive(cInfo.RcvBuffer, cInfo.BytesReceived, cInfo.RcvBuffer.Length - cInfo.BytesReceived, SocketFlags.None, ReceiveCallback, cInfo);
}
else
{
cInfo.Soket.BeginReceive(cInfo.LengthBuffer, cInfo.BytesReceived, cInfo.LengthBuffer.Length - cInfo.BytesReceived, SocketFlags.None, ReceiveCallback, cInfo);
}
}
catch (SocketException se)
{
//Handle exception and Close socket here, use your own code
return;
}
catch (Exception ex)
{
//Handle exception and Close socket here, use your own code
return;
}
}
class ClientInfo
{
private const int BUFSIZE = 1024 ; // Max size of buffer , depends on solution
private const int BUFLENSIZE = 2; // lenght of lenght , depends on solution
public int BytesReceived = 0 ;
public byte[] RcvBuffer { get; set; }
public byte[] LengthBuffer { get; set; }
public Socket Soket { get; set; }
public ClientInfo(Socket clntSock)
{
Soket = clntSock;
RcvBuffer = null;
LengthBuffer = new byte[ BUFLENSIZE ];
}
}
public static void AcceptCallback(IAsyncResult asyncResult)
{
Socket servSock = (Socket)asyncResult.AsyncState;
Socket clntSock = null;
try
{
clntSock = servSock.EndAccept(asyncResult);
ClientInfo cInfo = new ClientInfo(clntSock);
Receive( cInfo );
}
catch (SocketException se)
{
clntSock.Close();
}
}
private static void Receive(ClientInfo cInfo )
{
try
{
if (cInfo.RcvBuffer == null)
{
cInfo.Soket.BeginReceive(cInfo.LengthBuffer, 0, 2, SocketFlags.None, ReceiveCallback, cInfo);
}
else
{
cInfo.Soket.BeginReceive(cInfo.RcvBuffer, 0, cInfo.BytesReceived, SocketFlags.None, ReceiveCallback, cInfo);
}
}
catch (SocketException se)
{
return;
}
catch (Exception ex)
{
return;
}
}
You can find a nice overview of techniques at the C10k problem page.
Well, .NET sockets seem to provide select() - that's best for handling input. For output I'd have a pool of socket-writer threads listening on a work queue, accepting socket descriptor/object as part of the work item, so you don't need a thread per socket.
You could try using a framework called Adaptive Communications Environment (ACE) which is a generic C++ framework for network servers. It's a very solid, mature product and is designed to support high-reliability, high-volume applications up to telco-grade.
The framework deals with quite a wide range of concurrency models and probably has one suitable for your application out of the box. This should make the system easier to debug as most of the nasty concurrency issues have already been sorted out. The trade-off here is that the framework is written in C++ and is not the most warm and fluffy of code bases. On the other hand, you get tested, industrial grade network infrastructure and a highly scalable architecture out of the box.
I would use SEDA or a lightweight threading library (Erlang or newer Linux. See NTPL scalability on the server side). Async coding is very cumbersome if your communication isn't :)
I would use the AcceptAsync/ConnectAsync/ReceiveAsync/SendAsync methods that were added in .NET 3.5. I have done a benchmark and they are approximately 35% faster (response time and bitrate) with 100 users constantly sending and receiving data.
To people copy pasting the accepted answer, you can rewrite the acceptCallback method, removing all calls of
_serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
and put it in a finally{} clause, this way:
private void acceptCallback(IAsyncResult result)
{
xConnection conn = new xConnection();
try
{
//Finish accepting the connection
System.Net.Sockets.Socket s = (System.Net.Sockets.Socket)result.AsyncState;
conn = new xConnection();
conn.socket = s.EndAccept(result);
conn.buffer = new byte[_bufferSize];
lock (_sockets)
{
_sockets.Add(conn);
}
//Queue recieving of data from the connection
conn.socket.BeginReceive(conn.buffer, 0, conn.buffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), conn);
}
catch (SocketException e)
{
if (conn.socket != null)
{
conn.socket.Close();
lock (_sockets)
{
_sockets.Remove(conn);
}
}
}
catch (Exception e)
{
if (conn.socket != null)
{
conn.socket.Close();
lock (_sockets)
{
_sockets.Remove(conn);
}
}
}
finally
{
//Queue the next accept, think this should be here, stop attacks based on killing the waiting listeners
_serverSocket.BeginAccept(new AsyncCallback(acceptCallback), _serverSocket);
}
}
You could even remove the first catch since its content is the same, but it's a template method and you should use a typed exception to better handle the exceptions and understand what caused the error, so just implement those catches with some useful code.
I would recommend to read these books on ACE,
C++ Network Programming: Mastering Complexity Using ACE and Patterns
C++ Network Programming: Systematic Reuse with ACE and Frameworks
to get ideas about patterns allowing you to create an efficient server.
Although ACE is implemented in C++, the books cover a lot of useful patterns that can be used in any programming language.
To be clear, I'm looking for .NET based solutions (C# if possible, but any .NET language will work)
You are not going to get the highest level of scalability if you go purely with .NET. GC pauses can hamper the latency.
I'm going to need to start at least one thread for the service. I am considering using the Asynch API (BeginReceive, etc..) since I don't know how many clients I will have connected at any given time (possibly hundreds). I definitely do not want to start a thread for each connection.
Overlapped I/O is generally considered to be Windows' fastest API for network communication. I don't know if this the same as your Asynch API. Do not use select as each call needs to check every socket that is open instead of having callbacks on active sockets.
You can use the Push Framework open source framework for high-performance server development. It is built on IOCP and is suitable for push scenarios and message broadcast.
Related
I have built application that connects with help of TCP Sockets to 4 devices.
For that i created an TCP class with asynchronous methods to send and receive data.
public delegate void dataRec(string recStr);
public event dataRec dataReceiveEvent;
public Socket socket;
public void Connect(string IpAdress, int portNum)
{
socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
IPEndPoint epServer = new IPEndPoint(IPAddress.Parse(IpAdress), portNum);
socket.Blocking = false;
AsyncCallback onconnect = new AsyncCallback(OnConnect);
m_sock.BeginConnect(epServer, onconnect, socket);
}
public void SetupRecieveCallback(Socket sock)
{
try
{
AsyncCallback recieveData = new AsyncCallback(OnRecievedData);
sock.BeginReceive(m_byBuff, 0, m_byBuff.Length, SocketFlags.None, recieveData, sock);
}
catch (Exception ex)
{
//nevermind
}
}
public void OnRecievedData(IAsyncResult ar)
{
// Socket was the passed in object
Socket sock = (Socket)ar.AsyncState;
try
{
int nBytesRec = sock.EndReceive(ar);
if (nBytesRec > 0)
{
string sRecieved = Encoding.ASCII.GetString(m_byBuff, 0, nBytesRec);
OnAddMessage(sRecieved);
SetupRecieveCallback(sock);
}
else
{
sock.Shutdown(SocketShutdown.Both);
sock.Close();
}
}
catch (Exception ex)
{
//nevermind
}
}
public void OnAddMessage(string sMessage)
{
if (mainProgram.InvokeRequired)
{
{
scanEventCallback d = new scanEventCallback(OnAddMessage);
mainProgram.BeginInvoke(d, sMessage);
}
}
else
{
dataReceiveEvent(sMessage);
}
}
I have 4 devices with 4 different IP's and Port's that i send data, and from which i receive data.
So i created 4 different instances of a class mentioned.
When i receive data i call callback functions to do their job with the data i received (OnAddMessage event).
The connection with devices is really good, latency is like: 1-2ms~ (its in internal network).
Functions i call by callbacks are preety fast, each function is not more than 100ms.
The problem is it is working really slow, and its not caused by callback functions.
For each data i send to device, i receive one message from it.
When i start sending them, and stop after like 1 minute of working, the program keep receiving data for like 4-5 sec, even when i turn off devices- its like some kind of lag, that i receive data, that should be delivered a lot earlier.
It looks like something is working really slow.
Im getting like 1 message per second from each device, so it shouldnt be a big deal.
Any ideas what else i should do or set, or what actually could slow me down ?
You haven't posted all the relevant code, but here are some things to pay attention to:
With a network sniffer, like Wireshark or tcpdump, you can see what is actually going on.
Latency it not the only relevant factor for "connection speed". Look also at throughput, packet loss, re-transmissions, etc..
Try to send and receive in large chunks. Sending and receive only single bytes is slow because it has a lot of overhead.
The receiver should read data faster than the sender can send it, or else internal buffers (OS, network) will fill up.
Try to avoid a "chatty" protocol, basically synchronous request/reply, if possible.
If you have a chatty protocol, you can get better performance by disabling the Nagle algorithm. The option to disable this algorithm is often called "TCP no delay" or similar.
Don't close/reopen the connection for each message. TCP connection setup and teardown has quite some overhead.
If you have long standing open TCP connections, close the connection when the connection is idle for some time, for example several minutes.
I have a small game server I'm making that will have dozens of connections sending player data constantly. While I've finally accomplished some basics and now have data sending/receiving, I now face a problem of flooding the server and the client with too much data. I've tried to throttle it back but even then I am hitting 90-100% cpu simply because of receiving and processing the data received running up the CPU.
The method below is a bare version of receiving data from the server. The server sends a List of data to be received by the player, then it goes through that list. I've thought perhaps instead just using a dictionary with a key based on type rather than for looping but I don't think that will significantly improve it, the problem is that it is processing data non-stop because player positions are constantly being updated, sent to the server, then send to other players.
The code below shows receive for the client, the server receive looks very similar. How might I begin to overcome this issue? Please be nice, I am still new to network programming.
private void Receive(System.Object client)
{
MemoryStream memStream = null;
TcpClient thisClient = (TcpClient)client;
List<System.Object> objects = new List<System.Object>();
while (thisClient.Connected && playerConnected == true)
{
try
{
do
{
//when receiving data, first comes length then comes the data
byte[] buffer = GetStreamByteBuffer(netStream, 4); //blocks while waiting for data
int msgLenth = BitConverter.ToInt32(buffer, 0);
if (msgLenth <= 0)
{
playerConnected = false;
thisClient.Close();
break;
}
if (msgLenth > 0)
{
buffer = GetStreamByteBuffer(netStream, msgLenth);
memStream = new MemoryStream(buffer);
}
} while (netStream.DataAvailable);
if (memStream != null)
{
BinaryFormatter formatter = new BinaryFormatter();
memStream.Position = 0;
objects = new List<System.Object>((List<System.Object>)formatter.Deserialize(memStream));
}
}
catch (Exception ex)
{
Console.WriteLine("Exception: " + ex.ToString());
if (thisClient.Connected == false)
{
playerConnected = false;
netStream.Close();
thisClient.Close();
break;
}
}
try
{
if (objects != null)
{
for (int i = 0; i < objects.Count; i++)
{
if(objects[i] != null)
{
if (objects[i].GetType() == typeof(GameObject))
{
GameObject p = (GameObject)objects[i];
GameObject item;
if (mapGameObjects.TryGetValue(p.objectID, out item))
{
mapGameObjects[p.objectID] = p;;
}
else
{
mapGameObjects.Add(p.objectID, p);
}
}
}
}
}
}
catch (Exception ex)
{
Console.WriteLine("Exception " + ex.ToString());
if (thisClient.Connected == false)
{
playerConnected = false;
netStream.Close();
break;
}
}
}
Console.WriteLine("Receive thread closed for client.");
}
public static byte[] GetStreamByteBuffer(NetworkStream stream, int n)
{
byte[] buffer = new byte[n];
int bytesRead = 0;
int chunk = 0;
while (bytesRead < n)
{
chunk = stream.Read(buffer, (int)bytesRead, buffer.Length - (int)bytesRead);
if (chunk == 0)
{
break;
}
bytesRead += chunk;
}
return buffer;
}
Based on the code shown, I can't say why the CPU utilization is high. The loop will wait for data, and the wait should not consume CPU. That said, it still polls the connection in checking the DataAvailable property, which is inefficient and can cause you to ignore received data (in the implementation shown...that's not an inherent problem with DataAvailable).
I'll go one further than the other answer and state that you should simply rewrite the code. Polling the socket is just no way to handle network I/O. This would be true in any scenario, but it is especially problematic if you are trying to write a game server, because you're going to use up a lot of your CPU bandwidth needlessly, taking it away from game logic.
The two biggest changes you should make here are:
Don't use the DataAvailable property. Ever. Instead, use one of the asynchronous APIs for dealing with network I/O. My favorite approach with the latest .NET is to wrap the Socket in a NetworkStream (or get the NetworkStream from a TcpClient as you do in your code) and then use the Stream.ReadAsync() along with async and await. But the older asynchronous APIs for Sockets work well also.
Separate your network I/O code from the game logic code. The Receive() method you show here has both the I/O and the actual processing of the data relative to the game state in the same method. This two pieces of functionality really belong in two separate classes. Keep both classes, and especially the interface between them, very simple and the code will be a lot easier to write and to maintain.
If you decide to ignore all of the above, you should at least be aware that your GetStreamByteBuffer() method has a bug in it: if you reach the end of the stream before reading as many bytes were requested, you still return a buffer as large as was requested, with no way for the caller to know the buffer is incomplete.
And finally, IMHO you should be more careful about how you shutdown and close the connection. Read about "graceful closure" for the TCP protocol. It's important that each end signal that they are done sending, and that each end receive the other end's signal, before either end actually closes the connection. This will allow the underlying networking protocol to release resources as efficiently and as quickly as possible. Note that TcpClient exposes the socket as the Client property, which you can use to call Shutdown().
Polling is rarely a good approach to communication, unless you're programming 16-bit microcontrollers (and even then, probably not the best solution).
What you need to do is to switch to a producer-consumer pattern, where your input port (a serial port, an input file, or a TCP socket) will act as a producer filling a FIFO buffer (a queue of bytes), and some other part of your program will be able to asynchronously consume the enqueued data.
In C#, there are several ways to do it: you can simply write a couple of methods using a ConcurrentQueue<byte>, or a BlockingCollection, or you can try a library like the TPL Dataflow Library which IMO doesn't add too much value over existing structures in .NET 4. Prior to .NET 4, you would simply use a Queue<byte>, a lock, and a AutoResetEvent to do the same job.
So the general idea is:
When your input port fires a "data received" event, enqueue all received data into the FIFO buffer and set a synchronization event to notify the consumer,
In your consumer thread, wait for the synchonization event. When the signal is received, check if there is enough data in the queue. If yes, process it, if not, continue waiting for the next signal.
For robustness, use an additional watchdog timer (or simply "time since last received data") to be able to fail on timeout.
You want to use the Task-based Asynchronous Pattern. Probably making liberal use of the async function modifier and the await keyword.
You'd be best replacing GetStreamByteBuffer with a direct call to ReadAsync.
For instance you could asynchronously read from a stream like this.
private static async Task<T> ReadAsync<T>(
Stream source,
CancellationToken token)
{
int requestLength;
{
var initialBuffer = new byte[sizeof(int)];
var readCount = await source.ReadAsync(
initialBuffer,
0,
sizeof(int),
token);
if (readCount != sizeof(int))
{
throw new InvalidOperationException(
"Not enough bytes in stream to read request length.");
}
requestLength = BitConvertor.ToInt32(initialBuffer, 0);
}
var requestBuffer = new byte[requestLength];
var bytesRead = await source.ReadAsync(
requestBuffer,
0,
requestLength,
token);
if (bytesRead != requestLength)
{
throw new InvalidDataException(
string.Format(
"Not enough bytes in stream to match request length." +
" Expected:{0}, Actual:{1}",
requestLength,
bytesRead));
}
var serializer = new BinaryFormatter();
using (var requestData = new MemoryStream(requestBuffer))
{
return (T)serializer.Deserialize(requestData);
}
}
Like your code this reads an int from the stream to get the length, then reads that number of bytes and uses the BinaryFormatter to deserialize the data to the specified generic type.
Using this generic function you can simplify your logic,
private Task Receive(
TcpClient thisClient,
CancellationToken token)
{
IList<object> objects;
while (thisClient.Connected && playerConnected == true)
{
try
{
objects = ReadAsync<List<object>>(netStream, token);
}
catch (Exception ex)
{
Console.WriteLine("Exception: " + ex.ToString());
if (thisClient.Connected == false)
{
playerConnected = false;
netStream.Close();
thisClient.Close();
break;
}
}
try
{
foreach (var p in objects.OfType<GameObject>())
{
if (p != null)
{
mapGameObjects[p.objectID] = p;
}
}
}
catch (Exception ex)
{
Console.WriteLine("Exception " + ex.ToString());
if (thisClient.Connected == false)
{
playerConnected = false;
netStream.Close();
break;
}
}
}
Console.WriteLine("Receive thread closed for client.");
}
You need to put a Thread.Sleep(10) in your while loop. This is also a very fragile way to receive tcp data because it assumes the other side has sent all data before you call this receive. If the other side has only sent half of the data this method fails. This can be countered by either sending fixed sized packages or sending the length of a package first.
Your player position update is similar to the framebuffer update in the VNC protocol where the client request a screen frame & server responds to it with the updated screen data. But there is one exception, VNC server doesn't blindly send the new screen it only sends the changes if there is one. So you need to change the logic from sending all the requested list of objects to only to the objects which are changed after the last sent. Also in addition to it, you should send entire object only once after that send only the changed properties, this will greatly reduce the size of data sent & processed both at clients & server.
I'm using the asynchronous methos BeginSend and I need some sort of a timeout mechanism. What I've implemented works fine for connect and receive timeouts but I have a problem with the BeginSend callback. Even a timeout of 25 seconds is often not enough and gets exceeded. This seems very strange to me and points towards a different cause.
public void Send(String data)
{
if (client.Connected)
{
// Convert the string data to byte data using ASCII encoding.
byte[] byteData = Encoding.ASCII.GetBytes(data);
client.NoDelay = true;
// Begin sending the data to the remote device.
IAsyncResult res = client.BeginSend(byteData, 0, byteData.Length, 0,
new AsyncCallback(SendCallback), client);
if (!res.IsCompleted)
{
sendTimer = new System.Threading.Timer(SendTimeoutCallback, null, 10000, Timeout.Infinite);
}
}
else MessageBox.Show("No connection to target! Send");
}
private void SendCallback(IAsyncResult ar)
{
if (Interlocked.CompareExchange(ref sendTimeoutflag, 1, 0) != 0)
{
// the flag was set elsewhere, so return immediately.
return;
}
sendTimeoutflag = 0; //needs to be reset back to 0 for next reception
// we set the flag to 1, indicating it was completed.
if (sendTimer != null)
{
// stop the timer from firing.
sendTimer.Dispose();
}
try
{
// Retrieve the socket from the state object.
Socket client = (Socket)ar.AsyncState;
// Complete sending the data to the remote device.
int bytesSent = client.EndSend(ar);
ef.updateUI("Sent " + bytesSent.ToString() + " bytes to server." + "\n");
}
catch (Exception e)
{
MessageBox.Show(e.ToString());
}
}
private void SendTimeoutCallback(object obj)
{
if (Interlocked.CompareExchange(ref sendTimeoutflag, 2, 0) != 0)
{
// the flag was set elsewhere, so return immediately.
return;
}
// we set the flag to 2, indicating a timeout was hit.
sendTimer.Dispose();
client.Close(); // closing the Socket cancels the async operation.
MessageBox.Show("Connection to the target has been lost! SendTimeoutCallback");
}
I've tested timeout values up to 30 seconds. The value of 30 seconds has proved to be the only one never to time out. But that just seems like an overkill and I believe there's a different underlying cause.Any ideas as to why this could be happening?
Unfortunately, there's not enough code to completely diagnose this. You don't even show the declaration of sendTimeoutflag. The example isn't self-contained, so there's no way to test it. And you're not clear about exactly what happens (e.g. do you just get the timeout, do you complete a send and still get a timeout, does something else happen?).
That said, I see at least one serious bug in the code, which is your use of the sendTimeoutflag. The SendCallback() method sets this flag to 1, but it immediately sets it back to 0 again (this time without the protection of Interlocked.CompareExchange()). Only after it's set the value to 0 does it dispose the timer.
This means that even when you successfully complete the callback, the timeout timer is nearly guaranteed to have no idea and to close the client object anyway.
You can fix this specific issue by moving the assignment sendTimeoutflag = 0; to a point after you've actually completed the send operation, e.g. at the end of the callback method. And even then only if you take steps to ensure that the timer callback cannot execute past that point (e.g. wait for the timer's dispose to complete).
Note that even having fixed that specific issue, you may still have other bugs. Frankly, it's not clear why you want a timeout in the first place. Nor is it clear why you want to use lock-free code to implement your timeout logic. More conventional locking (i.e. Monitor-based with the lock statement) would be easier to implement correctly and would likely not impose a noticeable performance penalty.
And I agree with the suggestion that you would be better-served by using the async/await pattern instead of explicitly dealing with callback methods (but of course that would mean using a higher-level I/O object, since Socket doesn't suppose async/await).
I have a client/server infrastructure. At present they use a TcpClient and TcpListener to send a receive data between all the clients and server.
What I currently do is when data is received (on it's own thread), it is put in a queue for another thread to process in order to free the socket so it is ready and open to receive new data.
// Enter the listening loop.
while (true)
{
Debug.WriteLine("Waiting for a connection... ");
// Perform a blocking call to accept requests.
using (client = server.AcceptTcpClient())
{
data = new List<byte>();
// Get a stream object for reading and writing
using (NetworkStream stream = client.GetStream())
{
// Loop to receive all the data sent by the client.
int length;
while ((length = stream.Read(bytes, 0, bytes.Length)) != 0)
{
var copy = new byte[length];
Array.Copy(bytes, 0, copy, 0, length);
data.AddRange(copy);
}
}
}
receivedQueue.Add(data);
}
However I wanted to find out if there is a better way to do this. For example if there are 10 clients and they all want to send data to the server at the same time, one will get through while all the others will fail.Or if one client has a slow connection and hogs the socket all other communication will halt.
Is there not some way to be able to receive data from all clients at the same time and add the received data in the queue for processing when it has finished downloading?
So here is an answer that will get you started - which is more beginner level than my blog post.
.Net has an async pattern that revolves around a Begin* and End* call. For instance - BeginReceive and EndReceive. They nearly always have their non-async counterpart (in this case Receive); and achieve the exact same goal.
The most important thing to remember is that the socket ones do more than just make the call async - they expose something called IOCP (IO Completion Ports, Linux/Mono has these two but I forget the name) which is extremely important to use on a server; the crux of what IOCP does is that your application doesn't consume a thread while it waits for data.
How to Use The Begin/End Pattern
Every Begin* method will have exactly 2 more arguments in comparisson to it's non-async counterpart. The first is an AsyncCallback, the second is an object. What these two mean is, "here is a method to call when you are done" and "here is some data I need inside that method." The method that gets called always has the same signature, inside this method you call the End* counterpart to get what would have been the result if you had done it synchronously. So for example:
private void BeginReceiveBuffer()
{
_socket.BeginReceive(buffer, 0, buffer.Length, BufferEndReceive, buffer);
}
private void EndReceiveBuffer(IAsyncResult state)
{
var buffer = (byte[])state.AsyncState; // This is the last parameter.
var length = _socket.EndReceive(state); // This is the return value of the method call.
DataReceived(buffer, 0, length); // Do something with the data.
}
What happens here is .Net starts waiting for data from the socket, as soon as it gets data it calls EndReceiveBuffer and passes through the 'custom data' (in this case buffer) to it via state.AsyncResult. When you call EndReceive it will give you back the length of the data that was received (or throw an exception if something failed).
Better Pattern for Sockets
This form will give you central error handling - it can be used anywhere where the async pattern wraps a stream-like 'thing' (e.g. TCP arrives in the order it was sent, so it could be seen as a Stream object).
private Socket _socket;
private ArraySegment<byte> _buffer;
public void StartReceive()
{
ReceiveAsyncLoop(null);
}
// Note that this method is not guaranteed (in fact
// unlikely) to remain on a single thread across
// async invocations.
private void ReceiveAsyncLoop(IAsyncResult result)
{
try
{
// This only gets called once - via StartReceive()
if (result != null)
{
int numberOfBytesRead = _socket.EndReceive(result);
if(numberOfBytesRead == 0)
{
OnDisconnected(null); // 'null' being the exception. The client disconnected normally in this case.
return;
}
var newSegment = new ArraySegment<byte>(_buffer.Array, _buffer.Offset, numberOfBytesRead);
// This method needs its own error handling. Don't let it throw exceptions unless you
// want to disconnect the client.
OnDataReceived(newSegment);
}
// Because of this method call, it's as though we are creating a 'while' loop.
// However this is called an async loop, but you can see it the same way.
_socket.BeginReceive(_buffer.Array, _buffer.Offset, _buffer.Count, SocketFlags.None, ReceiveAsyncLoop, null);
}
catch (Exception ex)
{
// Socket error handling here.
}
}
Accepting Multiple Connections
What you generally do is write a class that contains your socket etc. (as well as your async loop) and create one for each client. So for instance:
public class InboundConnection
{
private Socket _socket;
private ArraySegment<byte> _buffer;
public InboundConnection(Socket clientSocket)
{
_socket = clientSocket;
_buffer = new ArraySegment<byte>(new byte[4096], 0, 4096);
StartReceive(); // Start the read async loop.
}
private void StartReceive() ...
private void ReceiveAsyncLoop() ...
private void OnDataReceived() ...
}
Each client connection should be tracked by your server class (so that you can disconnect them cleanly when the server shuts down, as well as search/look them up).
You should use asynchronous socket programming to achieve this. Take a look at the example provided by MSDN.
You should use asynchronous method of reading the data, an example is:
// Enter the listening loop.
while (true)
{
Debug.WriteLine("Waiting for a connection... ");
client = server.AcceptTcpClient();
ThreadPool.QueueUserWorkItem(new WaitCallback(HandleTcp), client);
}
private void HandleTcp(object tcpClientObject)
{
TcpClient client = (TcpClient)tcpClientObject;
// Perform a blocking call to accept requests.
data = new List<byte>();
// Get a stream object for reading and writing
using (NetworkStream stream = client.GetStream())
{
// Loop to receive all the data sent by the client.
int length;
while ((length = stream.Read(bytes, 0, bytes.Length)) != 0)
{
var copy = new byte[length];
Array.Copy(bytes, 0, copy, 0, length);
data.AddRange(copy);
}
}
receivedQueue.Add(data);
}
Also you should consider using AutoResetEvent or ManualResetEvent to be notified when new data is added to the collection so the thread that handle the data will know when data is received, and if you are using 4.0 you better switch off to using BlockingCollection instead of Queue.
What I do usually is using a thread pool with several threads.
Upon each new connection I'm running the connection handling (in your case - everything you do in the using clause) in one of the threads from the pool.
By that you achieve both performance since you're allowing several simultaneously accepted connection and you also limiting the number of resources (threads, etc') you allocate for handling incoming connections.
You have a nice example here
Good Luck
I'm struggling a bit with socket programming (something I'm not at all familiar with) and I can't find anything which helps from google or MSDN (awful). Apologies for the length of this.
Basically I have an existing service which recieves and responds to requests over UDP. I can't change this at all.
I also have a client within my webapp which dispatches and listens for responses to that service. The existing client I've been given is a singleton which creates a socket and an array of response slots, and then creates a background thread with an infinite looping method that makes "sock.Receive()" calls and pushes the data received into the slot array. All kinds of things about this seem wrong to me and the infinite thread breaks my unit testing so I'm trying to replace this service with one which makes it's it's send/receives asynchronously instead.
Point 1: Is this the right approach? I want a non-blocking, scalable, thread-safe service.
My first attempt is roughly like this, which sort of worked but the data I got back was always shorter than expected (i.e. the buffer did not have the number of bytes requested) and seemed to throw exceptions when processed.
private Socket MyPreConfiguredSocket;
public object Query()
{
//build a request
this.MyPreConfiguredSocket.SendTo(MYREQUEST, packet.Length, SocketFlags.Multicast, this._target);
IAsyncResult h = this._sock.BeginReceiveFrom(response, 0, BUFFER_SIZE, SocketFlags.None, ref this._target, new AsyncCallback(ARecieve), this._sock);
if (!h.AsyncWaitHandle.WaitOne(TIMEOUT)) { throw new Exception("Timed out"); }
//process response data (always shortened)
}
private void ARecieve (IAsyncResult result)
{
int bytesreceived = (result as Socket).EndReceiveFrom(result, ref this._target);
}
My second attempt was based on more google trawling and this recursive pattern I frequently saw, but this version always times out! It never gets to ARecieve.
public object Query()
{
//build a request
this.MyPreConfiguredSocket.SendTo(MYREQUEST, packet.Length, SocketFlags.Multicast, this._target);
State s = new State(this.MyPreConfiguredSocket);
this.MyPreConfiguredSocket.BeginReceiveFrom(s.Buffer, 0, BUFFER_SIZE, SocketFlags.None, ref this._target, new AsyncCallback(ARecieve), s);
if (!s.Flag.WaitOne(10000)) { throw new Exception("Timed out"); } //always thrown
//process response data
}
private void ARecieve (IAsyncResult result)
{
//never gets here!
State s = (result as State);
int bytesreceived = s.Sock.EndReceiveFrom(result, ref this._target);
if (bytesreceived > 0)
{
s.Received += bytesreceived;
this._sock.BeginReceiveFrom(s.Buffer, s.Received, BUFFER_SIZE, SocketFlags.None, ref this._target, new AsyncCallback(ARecieve), s);
}
else
{
s.Flag.Set();
}
}
private class State
{
public State(Socket sock)
{
this._sock = sock;
this._buffer = new byte[BUFFER_SIZE];
this._buffer.Initialize();
}
public Socket Sock;
public byte[] Buffer;
public ManualResetEvent Flag = new ManualResetEvent(false);
public int Received = 0;
}
Point 2: So clearly I'm getting something quite wrong.
Point 3: I'm not sure if I'm going about this right. How does the data coming from the remote service even get to the right listening thread? Do I need to create a socket per request?
Out of my comfort zone here. Need help.
Not the solution for you, just a suggestion - come up with the simplest code that works peeling off all the threading/events/etc. From there start adding needed, and only needed, complexity. My experience always was that in the process I'd find what I was doing wrong.
So is your program SUDO outline as follows?
Socket MySocket;
Socket ResponceSocket;
byte[] Request;
byte[] Responce;
public byte[] GetUDPResponce()
{
this.MySocket.Send(Request).To(ResponceSocket);
this.MySocket.Receive(Responce).From(ResponceSocket);
return Responce;
}
ill try help!
The second code post is the one we can work with and the way forward.
But you are right! the documentation is not the best.
Do you know for sure that you get a response to the message you send? Remove the asynchronous behavior from the socket and just try to send and receive synchronously (even though this may block your thread for now). Once you know this behavior is working, edit your question and post that code, and I'll help you with the threading model. Once networking portion, i.e., the send/receive, is working, the threading model is pretty straightforward.
One POSSIBLE issue is if your send operation goes to the server, and it responds before windows sets up the asynchronous listener. If you arent listening the data wont be accepted on your side (unlike TCP)
Try calling beginread before the send operation.