I need some guidance on a project we are developing. When triggered, my program needs to contact 1,000 devices by TCP and exchange about 200 bytes of information. All the clients are wireless on a private network. The majority of the time the program will be sitting idle, but then needs to send these messages as quickly as possible. I have come up with two possible methods:
Method 1
Use thread pooling to establish a number of worker threads and have these threads process their way through the 1,000 conversations. One thread handles one conversation until completion. The number of threads in the thread pool would then be tuned for best use of resources.
Method 2
A number of threads would be used to handle multiple conversations per thread. For example a thread process would open 10 socket connections start the conversation and then use asynchronous methods to wait for responses. As a communication is completed, a new device would be contacted.
Method 2 looks like it would be more effective in that operations wouldn’t have to wait with the server device responded. It would also save on the overhead of starting the stopping all those threads.
Am I headed in the right direction here? What am I missing or not considering?
There is a well-established way to deal with this problem. Simply use async IO. There is no need to maintain any threads at all. Async IO uses no threads while the IO is in progress.
Thanks to await doing this is quite easy.
The select/poll model is obsolete in .NET.
Related
I have a problem with scalability and processing and I want to get the opinion of the stack overflow community.
I basically have XML data coming down a socket and I want to process that data. For each XML line sent processing can include writing to a text file, opening a socket to another server and using various database queries; all of which take time.
At the minute my solution involves the following threads:
Thread 1
Accepts incoming sockets and thus generates child threads that handle each socket (there will only be a couple of incoming sockets from clients). When an XML line comes through (ReadLine() method on StreamReader) I basically put this line into a Queue, which is accessible via a static method on a class. This static method contains locking logic to ensure that the program is threadsafe (I could use Concurrent Queue for this of course instead of manual locking).
Threads 2-5
Constantly take XML lines from the queue and processes them one at a time (database queries, file writes etc).
This method seems to be working but I was curious if there is a better way of doing things because this seems very crude. If I take the processing that threads 2-5 do into thread 1 this results in extremely slow performance, which I expected, so I created my worker threads (2-5).
I appreciate I could replace threads 2-5 with a thread pool but the thread pool would still be reading from the same Queue of XML lines so I wandered if there is a more efficient way of processing these events instead of using the Queue?
A queue1 is the right approach. But I would certainly move from manual thread control to the thread pool (and thus I don't need to do thread management) and let it manage the number of threads.2
But in the end there is only so much processing a single computer (however expensive) can do. At some point one of memory size, CPU-memory bandwidth, storage IO, network IO, … is going to be saturated. At that point using an external queuing system (MSMQ, WebSphere*MQ, Rabbit-MQ, …) with each task being a separate message allows many workers on many computers to process the data ("competing consumers" pattern).
1 I would move immediately to ConcurrentQueue: getting locking right is hard, the more you don't need to do it yourself the better.
2 At some point you might find you need more control than the thread pool providers, that is the time to switch to a custom thread pool. But prototype and test: it is quite possible your implementation will actually be worse: see paragraph 2.
We have a server receiving data from between 500-1500 GPS devices. Each device sends a packet containing around 1-4 GPS coordinates every 10-30 seconds. The server is designed asynchronously with a listener handling connections using Begin- EndAccept, and communication using Begin- EndReceive. Once a packet is received the data is processed and stored in a database.
With few devices (500-700 devices) this takes barely 50 ms, and we have less than 50 concurrent threads running, and a realistic CPU usage (20-40%). However when the server is pressured with connections (1000+) the number of threads explodes to 500-600 and the CPU usage also drops to a few %. The processing time is also increased to several minutes.
Is the asynchronous design bad for this particular scenario with many small packets being sent at this rate, or might the be a problem in the code?
We have currently had to distribute the load across three servers to accomodate all devices, and they are all VMs with 6 CPUs and 4GB memory hosted on a Hyper-V server.
SOLUTION:
The solution I found from the answers from people, was to immediately schedule it as a task using the .Net parallel library, as this is much smarter when scheduling threads across multiple cores:
void EndReceive(IAsyncResult res)
{
Task.Factory.StartNew((object o) => { HandleReceive(o as IAsyncResult); }, res, TaskCreationOptions.PreferFairness);
}
Now the threads rarely exceed 50.
It sounds like somewhere in your application you're using non-asynchronous IO in which you're blocking on the results of the operation. You may be using proper asynchrony in many places, such as the primary connection with the client from the server, but perhaps you're not when connecting to a database or something like that. This mixing of async and non-async is likely why you're having so many threads being created.
By ensuring you have no blocking IO it should ensure you don't have lots of thread pool threads sitting around doing nothing, which appears to be the situation you're in.
What kind of operations are you doing on the server?
If they are CPU-bound it's useless to have more threads than cores and adding more may clutter your server with a bunch of threads fighting like dogs ;)
In this case you should be more lucky with simple processing loops, one per core.
I have never worked on such many requests at the same time but what you could try is creating as many threads as you have cores on your cpu and then implements a queueing system. Your threads would be consumming the queue one device's coordinate at a time. This way I guess your CPU would be used at full throttle...
When you're dealing with socket IO using BeginReceive/EndReceive, the callback is invoked by an IOCP thread.
Once you're done receiving you need to process the data.
Should you do it on the callback's calling thread?
Or should you run the task using the ThreadPool.QueueUserWorkItem
Normally samples do the work on the callback thread, which is a little bit confusing.
If you're dealing with several hundred active connections, running the processing on the IOCP thread ends up with a process with hundreds of threads. Would ThreadPool help limiting the number of concurrent threads?
I don't know that there's a generic 'right answer' for everyone - it will depend on your individual use case.
These are some things to consider if you do consider going down the ThreadPool path.
Is Out of Order / Concurrent message processing something you can support?
If Socket A receives messages 1, 2, and 3 in rapid succession - they may end up being processed concurrently, or out of order.
.NET has per-CPU thread pools, if one CPU runs out of work, it may 'steal' tasks from other CPUs. This may mean that your messages might be executed in some random order.
Don't forget to consider what out-of-order processing might do to the client - eg if they send three messages which require responses, sending the responses out of order may be a problem.
If you must process each message in order, then you'll probably have to implement your own thread pooling.
Do you need to be concerned about potential Denial of Service?
If one socket suddenly sends a swarm of data, accepting all of that for processing may clog up internal queues and memory. You may have to limit the amount of un-processed data you accept.
IOCP threads has meant for serving IO operations. if your work executed by iocp callback can be long running or incures locks or other facilities that my block execution, you'd better hand it of to a worker thread.
I have a TCP/IP server written in C# .net which can have 10,000 connections at once easy. However, when a callback is received from a socket, it is dealt with by a new thread in the application thread pool. This means that the real concurrent communication limitation is down to the number of threads within the thread pool. For example if those 10,000 connections all attempt to send data at the same time, the majority will have to wait whilst the thread pool runs through as fast as it can. Can anyone share their experience with high performance socket services and advise how a large corporation would go about ensuring the 10,000 connections can not only be connected at the same time, but can also communicate at the same time? Thanks
Don't process the packets inline in the callback. Do the absolute minimum work there, and then hand them off to a separate worker thread pool via a producer-consumer queue that (ideally) never blocks the producer threads, which are your socket listeners. BlockingCollection<T> may be useful here.
You have to be careful that the queue does not grow unbounded - if your consumers are a lot slower than producers, and the queue grows under normal load, you have a problem to which throttling the network receives is the obvious solution, despite its undesirability.
YOu make a thought mistake here. Regardless how many thread you have, data always has to wait unless you have one CPU CORE PER CONNECTION. Scalability is not having unlimited paralellism, but being ab le to handle a lot of conenctions and keep the cpu at full power.
The thread pool is perfectly sized for that. Once the CPU reaches full utilization, you can not do anything else anyway.
and advise how a large corporation would go about ensuring the 10,000 connections can not only be
connected at the same time, but can also communicate at the same time?
MANY computers that have like a total of 500 processor cores. The trick is: what latency is acceptable. You dont need instant communication. You try to sovle that from the wrong end.
I'm creating a server-type application at the moment which will do the usual listening for connections from external clients and, when they connect, handle requests, etc.
At the moment, my implementation creates a pair of threads every time a client connects. One thread simply reads requests from the socket and adds them to a queue, and the second reads the requests from the queue and processes them.
I'm basically looking for opinions on whether or not you think having all of these threads is overkill, and importantly whether this approach is going to cause me problems.
It is important to note that most of the time these threads will be idle - I use wait handles (ManualResetEvent) in both threads. The Reader thread waits until a message is available and if so, reads it and dumps it in a queue for the Process thread. The Process thread waits until the reader signals that a message is in the queue (again, using a wait handle). Unless a particular client is really hammering the server, these threads will be sat waiting. Is this costly?
I'm done a bit of testing - had 1,000 clients connected continually nagging - the server (so, 2,000+ threads) and it seemed to cope quite well.
I think your implementation is flawed. This kind of design doesn't scale because creating threads is expensive and there is a limit on how many threads can be created.
That is the reason that most implementations of this type use a thread pool. That makes it easy to put a cap on the maximum amount of threads while easily managing new connections and reusing the threads when the work is finished.
If all you are doing with your thread is putting items in a queue, then use the
ThreadPool.QueueUserWorkItem method to use the default .NET thread pool.
You haven't given enough information in your question to specify for definite but perhaps you now only need one other thread, constantly running clearing down the queue, you can use a wait handle to signal when something has been added.
Just make sure to synchronise access to your queue or things will go horribly wrong.
I advice to use following patter. First you need thread pool - build in or custom. Have a thread that checks is there something available to read, if yes it picks Reader thread. Then reading thread puts into queue and then thread from pool of processing threads will pick it. it will minimize number of threads and minimize time spend in waiting state