Develop Reference

How can I investigate application/network bottlenecks in my TCP application server and the environment?

I'm trying to write a high-performance TCP server (a LDAP server) using this tutorial by David Fowler as a base part of the MyServerListener.cs to handle incoming connections.
This is a simple .net 7 console app (with little changes) that I borrowed from David, it just accepts incoming clients, process the requests and writes hello to the response :
internal class Program
{
const int PORT = 389; // injecting from config
const int BACKLOG_LENGTH = 200; // max backlog size in windows server
static async Task Main(string[] args)
{
var listenSocket = new Socket(SocketType.Stream, ProtocolType.Tcp);
listenSocket.Bind(new IPEndPoint(IPAddress.Any, port));
Console.WriteLine("Listening on port " + port);
listenSocket.Listen(BACKLOG_LENGTH);
while (true)
{
var socket = await listenSocket.AcceptAsync();
_ = ProcessLinesAsync(socket);
}
}
private static async Task ProcessLinesAsync(Socket socket)
{
#if DEBUG
Console.WriteLine($"[{socket.RemoteEndPoint}]: connected");
#endif
// Create a PipeReader over the network stream
var stream = new NetworkStream(socket);
var reader = PipeReader.Create(stream);
var writer = PipeWriter.Create(stream);
while (true)
{
ReadResult result = await reader.ReadAsync();
ReadOnlySequence<byte> buffer = result.Buffer;
while (TryReadLine(ref buffer, out ReadOnlySequence<byte> line))
{
// Process the line.
ProcessLine(line);
try
{
// writing a sample message to the response
var helloBytes = Encoding.ASCII.GetBytes("hello\n");
await writer.WriteAsync(helloBytes);
}
catch (Exception ex)
{
throw;
}
}
// Tell the PipeReader how much of the buffer has been consumed.
reader.AdvanceTo(buffer.Start, buffer.End);
// Stop reading if there's no more data coming.
if (result.IsCompleted)
{
break;
}
}
// Mark the PipeReader as complete.
await reader.CompleteAsync();
#if DEBUG
Console.WriteLine($"[{socket.RemoteEndPoint}]: disconnected");
#endif
}
private static bool TryReadLine(ref ReadOnlySequence<byte> buffer, out ReadOnlySequence<byte> line)
{
// Look for a EOL in the buffer.
SequencePosition? position = buffer.PositionOf((byte)'\n');
if (position == null)
{
line = default;
return false;
}
// Skip the line + the \n.
line = buffer.Slice(0, position.Value);
buffer = buffer.Slice(buffer.GetPosition(1, position.Value));
return true;
}
private static void ProcessLine(in ReadOnlySequence<byte> buffer)
{
foreach (var segment in buffer)
{
// Doing some tasks
#if DEBUG
Console.Write(Encoding.UTF8.GetString(segment.Span));
Console.WriteLine();
#endif
}
}
}
This server listens on a port (389), processes the incoming request, doing some jobs and then writes a message to the response using PipeReader and PipeWriter.
I'm trying to do my best to a less memory/heap allocation code (using span<>, memory<>, ...) as I can, to keep my codebase so fast and optimize. But for now, I'm trying to test the production environment with the above code to examine the throughput; I mean: the server resources, my TCP server application itself, clients and the network;
I'm using Apache JMeter to test (load/stress test).
In some scenarios (sending more than 5000 request/sec) I get Connection refused error messages in JMeter logs, but I don't have any high pressure in the server or client's (JMeter[s]) resources (CPU/Memory).
I tried to optimize the server's configuration and changed some TCP related parameters (I googled about them) like MaxUserPort: 65534, TcpTimedWaitDelay: 30 or different backlog size, but no improvements.
So I'm almost sure that there is sth related to the network (packet dropping/rejecting or sth like this).
I also turned off firewall in the testing clients and the server, But I don't have any access to the network configurations (and I don't know what are they) like firewalls, ISA, TMG, etc.
_____________
Update 1:
I already increased our clients ephemeral ports to the maximum range using this PS script:
netsh int ipv4 set dynamic tcp start=5000 num=65535
and now we have this :
netsh int ipv4 show dynamicport tcp
Start Port : 1024
Number of Ports : 64511
And we also checked JMeter logs to see any error indicating this situation (Ephemeral ports exhaustion), at first we saw this message :
Non HTTP response code: java.net.BindException,Non HTTP response
message: Address already in use
But now, it's gone and we don't have large number of TIME_WAIT ports to worry about.
And we are also testing our scenario with SO_LINGER:0 and monitoring real times TIME_WAIT ports (using some tools), and we are sure that this isn't our concern right now.
_____________
So my question is, how can I find out why I can't send more traffic (threads/requests per seconds in JMeter clients) to the server to testing my TCP server application performance? Because for now, the server CPU doesn't increase more than ~10%.
At this point, is this a network related problem? How can I be sure about that? e.g: can I use some network analyzers (e.g: PRTG network monitor) to find out any dropped TCP packets? Or any other tips welcomed

Most probably TCP ports are not recycled fast enough, there is a network parameter which controls the time which connection can stay in TIME_WAIT state so you might also want to reduce TcpTimedWaitDelay
Also it might be a good idea to increase maximum number of TCP connections via TcpNumConnections parameter
And last but not the least it might be the case JMeter is not capable of sending the requests fast enough so you might need to play the same trick on the load generator side. In addition make sure to follow JMeter Best Practices and monitor CPU/RAM/Network/Disk/Swap usage on JMeter side as it might be the case you will need to switch to Distributed Testing if one machine is not capable of giving more than 5k requests per second.

Modbus communication over tcp doesn't send data to all devices

I have a gateway connected to a computer which runs a C# program. I have a Modbus slave-master relation between computer and the gateway. Here are the gateway configurations (model EKI-1221-BE):
The set up I have is the same as in this picture but I have 6 devices connected instead of 2 and will later have 12:
I can send a Modbus command to any of the 6 devices without any problems but the issue happens when I try to send multiple commands in quick interval. In the following code, only the device 1,2 and 6 receive the command.
PumpsComm.SendMessage(1, 3099, 2000);
PumpsComm.ClientSocket.Receive(TempBuffer);
PumpsComm.SendMessage(2, 3099, 3000);
PumpsComm.ClientSocket.Receive(TempBuffer);
PumpsComm.SendMessage(3, 3099, 4000);
PumpsComm.ClientSocket.Receive(TempBuffer);
PumpsComm.SendMessage(4, 3099, 6000);
PumpsComm.ClientSocket.Receive(TempBuffer);
PumpsComm.SendMessage(5, 3099, 8000);
PumpsComm.ClientSocket.Receive(TempBuffer);
PumpsComm.SendMessage(6, 3099, 10000);
PumpsComm.ClientSocket.Receive(TempBuffer);
The function SendMessage is as follows, where ClientSocket is a TCP Socket that is connected to the gateway / Modbus slave.
public void SendMessage(int pumpID, int register, int command)
{
byte[] message = new byte[12];
message[0] = 0;
message[1] = 0; //Message number
message[2] = 0;
message[3] = 0;
message[4] = 0;
message[5] = 6; //Message length
message[6] = (byte)ToHexInDec(pumpID, 'L'); //Pump ID
message[7] = 6;
message[8] = (byte)ToHexInDec(register, 'H'); //Register high
message[9] = (byte)ToHexInDec(register, 'L'); //Register low
message[10] = (byte)ToHexInDec(command, 'H'); //Command high
message[11] = (byte)ToHexInDec(command, 'L'); //Command low
try
{
ClientSocket.Send(message);
}
catch
{
return;
}
}
If I make the Thread sleep for 30ms between each send, every command works but this is not an option in my case as I need to be able to update every device in a very short amount of time.
I looked at the packets that I was sending and receiving to and from the gateway with Wireshark and I think I found where the
This is what the output of Wireshark was (the Ip that ends with 200 is the computer and the one that ends with 107 is the gateway):
It looks as though the communication for the device 3,4,5 and 6 where put in the same TCP packet and for some reason, the only Modbus communication that was read from that packet is the last one.
I tried adding MBAP transaction numbers for every communication in the MBAP header of the Modbus communication but it didn't change anything. I then tried to add a blocking receive that makes the program wait for the Modbus response before sending the next communication but I am not receiving anything (even though we can see that the gateway is sending a Modbus response back when the communication was successfull)
I don't know what I can do anymore and was looking to see if anyone with more experience with TCP modbus communications could help. I can provide more details if necessary. Sorry if this is not clear enough, english is not my first language.
Thanks!

As per the spec
Several MODBUS transactions can be activated simultaneously on the same TCP
Connection.
Remark: If this is done then the MODBUS transaction identifier must be used to
uniquely identify the matching requests and responses
However you are sending messages with the transaction identifier set to 0000:
message[0] = 0;
message[1] = 0; //Message number
(I'm guessing that the gateway is detecting the duplication and assigning a new identifier because of this).
A better approach would be to send all of your requests (each with a different identifier; it does not matter what this is as long as it's unique) and then wait for the responses to come in (matching the response to the request using the transaction identifier in each response; responses may be in a different order to the requests).
Note that you should not assume that each response will come in a separate TCP packet (or that it will be in a single packet); the protocol takes this into account:
When MODBUS is carried over TCP, additional length information is
carried in the MBAP header to allow the recipient to recognize message
boundaries even if the message has been split into multiple packets for
transmission. The existence of explicit and implicit length rules, and use of a CRC-32 error check code (on Ethernet) results in an infinitesimal chance of undetected corruption to a request or response message.

How can a TCP server application handle reconnects in response to stale connections?

The main loop of basic server looks like this currently:
ConcurrentDictionary<EndPoint,Connection> connections = new ConcurrentDictionary<EndPoint, Monitor>();
tcpListener = new TcpListener(IPAddress.Any, config.Port);
tcpListener.Start();
while (true)
{
var client = await tcpListener.AcceptTcpClientAsync();
HandleNewConnection(client);
}
private void HandleNewConnection(TcpClient client)
{
var key = client.Client.RemoteEndPoint;
var connection = new Connection(client);
connections[key] = connection;
//run the connection client/server stuff, basically just Reads in a loop
_ = connection.MonitorAsync();
}
There are two specific cases I want to handle:
The connection is lost. This seems fairly easy, doing something like _ = connection.MonitorAsync().ContinueWith( () => connections.Remove(key) ?
A new connection comes in from an existing client. TCP connections are not great at detecting faults unless you write to them, so it's quite plausible a client could try to send data, detect a problem, drop the connection, and reconnect - and my server thinks the connection is still valid. The first it knows is a new connection comes in from a client it thinks is already connected.
In 2. it is conceptually quite simple that we look-up the existing connection from the same client endpoint, terminate it and store a Connection into connections.
But now if I'm not careful my continuation from 1. will kick in after 2, and remove the new element from the dictionary.
There must be a neat pattern but I keep tying myself in knots trying to figure it out. Servers must handle this situation all the time and need to be accurate not to accrue a huge pile of dead connection objects. What would be a normal way to handle this, other than a more stateless approach where Connections doesn't exist?

Simple One Thread Socket - TCP Server

First, I don't know if Stackoverflow is the best site to post this kind of message, but I don't know another sites like this.
In oder to understand properly tcp programmation in C#, I decided to do all possible ways from scratch. Here is what I want to know (not in the right order:
- Simple One Thread Socket Server (this article)
- Simple Multiple Threads Socket Server (I don't know how, cause threads are complicated)
- Simple Thread Socket Server (put the client management in another thread)
- Multiple Threads Socket Server
- Using tcpListener
- Using async / Await
- Using tasks
The ultimate objective is to know how to do the best tcp server, without just copy/paste some parts of come, but understand properly all things.
So, this is my first part : a single thread tcp server.
There is my code, but I don't think anybody will correct something, because it's quite a copy from MSDN : http://msdn.microsoft.com/en-us/library/6y0e13d3(v=vs.110).aspx
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Sockets;
using System.Text;
namespace SimpleOneThreadSocket
{
public class ServerSocket
{
private int _iPport = -1;
private static int BUFFER_SIZE = 1024;
private Socket _listener = null;
public ServerSocket(int iPort)
{
// Create a TCP/IP socket.
this._listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
// Save the port
this._iPport = iPort;
}
public void Start()
{
byte[] buffer = null;
String sDatasReceived = null;
// Bind the socket to loopback address
try
{
this._listener.Bind(new System.Net.IPEndPoint(System.Net.IPAddress.Loopback, _iPport));
this._listener.Listen(2);
}
catch (Exception e)
{
System.Console.WriteLine(e.ToString());
}
// Listening
try
{
Console.WriteLine("Server listening on 127.0.0.1:" + _iPport);
while (true)
{
Socket client = this._listener.Accept();
Console.WriteLine("Incoming connection from : " + IPAddress.Parse(((IPEndPoint)client.RemoteEndPoint).Address.ToString()) + ":" + ((IPEndPoint)client.RemoteEndPoint).Port.ToString());
// An incoming connection needs to be processed.
while (true)
{
buffer = new byte[BUFFER_SIZE];
int bytesRec = client.Receive(buffer);
sDatasReceived += Encoding.ASCII.GetString(buffer, 0, bytesRec);
if (sDatasReceived.IndexOf("<EOF>") > -1)
{
// Show the data on the console.
Console.WriteLine("Text received : {0}", sDatasReceived);
// Echo the data back to the client.
byte[] msg = Encoding.ASCII.GetBytes(sDatasReceived);
client.Send(msg);
sDatasReceived = "";
buffer = null;
}
else if (sDatasReceived.IndexOf("exit") > -1)
{
client.Shutdown(SocketShutdown.Both);
client.Close();
sDatasReceived = "";
buffer = null;
break;
}
}
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
}
}
But I have some questions about that :
Listen Method from Socket have a parameter : backlog. According to MSDN, backlog is the number of available connection. I don't know why, when I put 0, I can connect to my server with multiple Telnet sessions. EDIT : 0 & 1 both allow 2 connections (1 current, 1 pending), 2 allow 3 connections (1 current, 2 pending), etc... So I didn't understand well the meaning of MSDN.
Can you confirm that Accept Method will take each connection one after one, that's why I see text from differents Telnet session in my server ?
Can you confirm (my server is a C# library) I can't kill my server (with this kind of code) without killing the process ? It could be possible with threads but it will come later.
If something is wrong in my code, please help me :)
I will come back soon with a simple multiple thread socket server, but I don't know how (I think one step is available before using threads or async/await).

First off, do your best not to even learn this. If you can possibly use a SignalR server, then do so. There is no such thing as a "simple" socket server at the TCP/IP level.
If you insist on the painful route (i.e., learning proper TCP/IP server design), then there's a lot to learn. First, the MSDN examples are notoriously bad starting points; they barely work and tend to not handle any kind of error conditions, which is absolutely necessary in the real world when working at the TCP/IP level. Think of them as examples of how to call the methods, not examples of socket clients or servers.
I have a TCP/IP FAQ that may help you, including a description of the backlog parameter. This is how many connections the OS will accept on your behalf before your code gets around to accepting them, and it's only a hint anyway.
To answer your other questions: A single call to Accept will accept a single new socket connection. The code as-written has an infinite loop, so it will work like any other infinite loop; it will continue executing until it encounters an exception or its thread is aborted (which happens on process shutdown).
If something is wrong in my code, please help me
Oh, yes. There are lots of things wrong with this code. It's an MSDN socket example, after all. :) Off the top of my head:
The buffer size is an arbitrary value, rather low. I would start at 8K myself, so it's possible to get a full Ethernet packet in a single read.
The Bind explicitly uses the loopback address. OK for playing around, I guess, but remember to set this to IPAddress.Any in the real world.
backlog parameter is OK for testing, but should be int.MaxValue on a true server to enable the dynamic backlog in modern server OSes.
Code will fall through the first catch and attempt to Accept after a Bind/Listen failed.
If any exception occurs (e.g., from Listen or Receive), then the entire server shuts down. Note that a client socket being terminated will result in an exception that should be logged/ignored, but it would stop this server.
The read buffer is re-allocated on each time through the loop, even though the old buffer is never used again.
ASCII is a lossy encoding.
If a client cleanly shuts down without sending <EOF>, then the server enters an infinite busy loop.
Received data is not properly separated into messages; it is possible that the echoed message contains all of one message and part of another. In this particular example it doesn't matter (since it's just an echo server and it's using ASCII instead of a real encoding), but this example hides the fact that you need to handle message framing properly in any real-world application.
The decoding should be done after the message framing. This isn't necessary for ASCII (a lossy encoding), but it's required for any real encodings like UTF8.
Since the server is only either receiving or sending at any time (and never both), it cannot detect or recover from a half-open socket situation. A half-open socket will cause this server to hang.
The server is only capable of a single connection at a time.
That was just after a brief readthrough. There could easily be more.

Chat server with a lot of clients

I read some C# chat source code & I see that: on chat server with a lot of connected clients, server listener will run in a separated thread & each connected client will also run in a separated thread.
Code examples:
Start server & begin listening in a separated thread:
public void StartListening()
{
// Get the IP of the first network device, however this can prove unreliable on certain configurations
IPAddress ipaLocal = ipAddress;
// Create the TCP listener object using the IP of the server and the specified port
tlsClient = new TcpListener(1986);
// Start the TCP listener and listen for connections
tlsClient.Start();
// The while loop will check for true in this before checking for connections
ServRunning = true;
// Start the new tread that hosts the listener
thrListener = new Thread(KeepListening);
thrListener.Start();
}
private void KeepListening()
{
// While the server is running
while (ServRunning == true)
{
// Accept a pending connection
tcpClient = tlsClient.AcceptTcpClient();
// Create a new instance of Connection
Connection newConnection = new Connection(tcpClient);
}
}
And a connection will also run in a separated thread:
public Connection(TcpClient tcpCon)
{
tcpClient = tcpCon;
// The thread that accepts the client and awaits messages
thrSender = new Thread(AcceptClient);
// The thread calls the AcceptClient() method
thrSender.Start();
}
So, if a chat server with 10000 connected clients, the chat server application will have 10002 threads (one main thread, one server thread & 10000 client threads). I think the chat server will be overhead with a big number of threads. Please help me a solution. Thanks.
UPDATE:
I believe chat examples are only for learning networking & they are not suitable in real-world model. Please give me a real-world solution. Thanks.

If you use .Net framework 2.0 SP2 or higher, than you may use new asyncrhronous sockets model based on IO Completion ports. In this case you shouldn't create your own threads, because IO Completion ports do all job for you.
Here some examples:
tcpServer = new System.Net.Sockets.TcpListener(IPAddress.Any, port);
tcpServer.Start();
tcpServer.BeginAcceptSocket(EndAcceptSocket, tcpServer);
private void EndAcceptSocket(IAsyncResult asyncResult)
{
TcpListener lister = (TcpListener)asyncResult.AsyncState;
Socket sock = lister.EndAcceptSocket(asyncResult);
//handle socket connection (you may add socket to you internal storage or something)
//start accepting another sockets
lister.BeginAcceptSocket(EndAcceptSocket, lister);
SocketAsyncEventArgs e = new SocketAsyncEventArgs();
e.Completed += ReceiveCompleted;
e.SetBuffer(new byte[socketBufferSize], 0, socketBufferSize);
sock.ReceiveAsync(e);
}
void ReceiveCompleted(object sender, SocketAsyncEventArgs e)
{
var sock = (Socket)sender;
if (!sock.Connected)
{
//handle socket disconnection
}
var buf = new byte[size];
Array.Copy(e.Buffer, buf, size);
//handle received data
//start reading new data
sock.ReceiveAsync(e);
}

A standard mechanism to ease the burden is known as selection, which can multiplex multiple Socket instances to watch for the ones that are ready to be read or written to. See this document: http://codeidol.com/csharp/csharp-network/Csharp-Network-Programming-Classes/Csharp-Socket-Programming/ and scroll down to the section on select().

1) You'll NEVER want that many threads running - even if you could get them to run on your box (which you can't - each thread has a stack associated with it that takes real RAM and as you start more and more and more you'll run out of physical resources in your box and watch it blow up).
2) You'll want to look into thread pooling - using a smaller amount of threads to tackle a larger amount of work - typically reading from a queue of work that you try to get through as quickly as possible.
3) You'll want to look into io completion ports - a means of having a callback when io (likek a disk read or a network io) is waiting for you to take action - think of a thread (or pool of threads) dedicated to getting io notifications and then shoving the action to take for that io into a queue and then another pool of threads that take care of the actual messaging/logging/etc.
4) What happens when you scale beyond one machine? Which you hope to do if you're successful right? :-) Typically people dedicate a set of N machines to chat - then they hash based on a identifier for the user (think a GUID that represented the user - or a UserID/bigint depending on what corresponds to some internal authentication token that is consistent from login to login) which allows them to deterministically route the user's status/state information to a specific machine in that set of N boxes dedicated to messaging. So if a user that hashes to server N[2] needs to check if theri friends ar logged in it is easy to know for each of their friends exactly which machine their friend's status should be in because the backend consistently hashes those friends to the IM machine that corresponds to each userid hash. (i.e. you know just from the userid what server in the farm should be handling the IM status for that user.
Just dont' think you're gonna spin up a bunch of threads and that will save the day. It's sloppy and works only in very small numbers.

To make the matter worse you would also have to communicate between some arbitrary number of threads (it's a chat server, people want to talk to each other, not themselves.) I would suggest looking into UDP - can be done with a single thread on the server and fits the network activity well - people rarely write more then couple of sentences at a time in chat exchanges, which is very convenient for size-limited UDP datagrams.
There are other approaches of course, but one sure thing though is that you will never be able to do thread per socket at that scale.

I suggest you to read this great article on MSDN Magazine.
Describing:
Threaded Server
Select-Based Server
Asynchronous Server
codes in C# & VB.Net