Develop Reference

Deadlocked when both endpoints do Socket.SendAsync

So I have written this client/server socket application that uses the SocketAsyncEventArgs "method" for doing async sockets.
Using the same library I have used for many other applications, I now for the first time experience a situation that I never anticipated.
Our new client/server application when started, starts to send lot's of data in both directions.
When done in unit-tests using mock-objects (without delays) to mimic normal socket operations, it all works well.
But in real situations using real sockets, we get a sort of deadlock where both endpoints are stuck in a Socket.SendAsync() operation (yes it returned true, was not synchronously handled)
My idea is that the receive buffer of both parties are full, and the tcp stack is not acknowleding any frames anymore. (connected to 127.0.0.1)
So I made the receivebuffer twice as large as the sendbuffer, but unfortunately it is not that simple due to the nature of our "protocol", and how we determine to send or receive.
I now have to re-think the method that determines when to start sending and when to start receiving.
A complicating factor is, that the purpose of this connection is to mutliplex multiple bi-directional general purpose communication channels over this socket connection. That means that there is no pre-determined sequence of communication, all channels may have their own protocols.
Of course, there is a tls initiation, handshake and authentication, which all work well, but when the connection becomes operational, and the channels start their own communications, the only sure thing is that received data has a size and channelnumber as a header.
After each operation, I check to see if there is any waiting data in the receivebuffer, or by checking Socket.Available.
This combined with measuring how much data was received since last sent operation, and how full the transmitbuffer is getting, I decide to receive more or start sending, or do nothing, and poll again in xx ms.
I now realize that this is wronge.
Am I trying to accomplish something that is simply not possible using only one socket connection?
Anyone every tried to accomplish something simular, or know a good way of accomplish a safe way that does not introduce these odd lock-ups.
Thanks,
Theo.

.Net Socket (UDP) Sending, Receiving and Scheduling

I am currently on a personal project for learning purposes. I want to make a connection over UDP, for application such as games. Each datagram sent has a specific header that indicates which "logic" channel it belongs to - for example, channel 0 is just like UDP with the extra header overhead, and channel 1 uses more headers to bring some extra reliability. The channels objective is to "automatically" separate messages into logic groups, up to a specific amount.
In my current code, there is a simple loop in a separate thread that handles sending and receiving:
// This is pseudo code
public void Tick() {
if(Socket.Poll) {
do {
ReadMessage();
} while(Socket.Available > 0)
}
SendQueuedOutgoingMessages();
}
Though this works on an ideal world, I have this feeling that this logic fails when there are too many incoming or outgoing messages. Is it possible to use the same socket to simultaneously send and receive messages (i.e send and receive are asynchronous or in different threads)? Even if it is possible, would it be better if I simply used two or more UDP sockets (or mix TCP and UDP sockets, if I need reliability), having specially maintainability in mind?
The most direct alternative I can think around this would be to use a scheduling algorithm to control how many messages to read and send, by means of queue sizes or other factors, but this feels poor and inflexible in this situation.
Edit: Adding more information about the code.
The Tick() method is set to be called a specific amount of times per second if it immediately returns. For example, 30 times per second if no new in or out messages exists, and less if it needs some time to send or receive data. I've used blocking ReceiveFrom and SendTo methods as to avoid busy waiting or calls such as Sleep(0).
Though I immediately treat incoming messages, I use an outgoing messages queue to help with the channels idea - each channel has its own priority down to 'no priority', affecting its bandwidth share over time in smooth and busy moments.

Whether or not using 2 sockets for receiving and sending separately, or just a single one depends on the situation. If you are going to send a lot of messages, and even if you are on a dedicated thread, the socket might block if the outgoing queue that is used by the socket becomes full.
There are several solutions to this problem. Using 2 sockets and 2 distinct threads is one of them, using select in combination with asynchronous sockets is another. The point is that you don't want to stop receiving just because a send might block.
Each of these possible solutions have their own complexity.
The select api is meant to check if for a certain socket there is something to receive, but you can also use it to detect if a socket becomes writable again. You need a socket option to put the socket in non blocking state and you need to check for E_WOULDBLOCK return codes with each send. If so the send has failed and you have to queue the message yourself.
You don't really send and receive at the same time, it is sequentially. You use select to check if a socket is writable and readible by using 2 bitmasks manipulated with the fd_set api. You can use select even on multiple sockets at once. Then if select, which is a blocking call, returns, you can check each individual bit to check what actions needs to be performed.
The reason why a send can block, if a socket is not put in nonblocking state is that the output queue of the socket can be full. If the socket is blocking, it would simply wait for the queue to become ready again. But during that wait, you cannot receive anything on that same socket. This is the very reason why you need non blocking sockets and the select api, in combination with some kind of queueing mechanism yourself.

Why not simply the standard read loop setup?
while (true)
ReadMessage();
There is no scheduling or throttling necessary. It is not necessary to know whether a packet is ready or not.
You can read and write simultaneously on the same UDP socket.
There is no need for an outgoing queue, either. Just send.

Receive streaming of UDP packets sent at fixed, clocked rate

I have the requirement of receiving a stream of UDP packets at a fixed rate of 1000 packets per second. They are composed of 28 bytes of payload, where the first four bytes (Uint32) are the packet sequence number. The sender is an embedded device on the same local network, and addresses and ports are mutually known. I am not expected to receive every packet, but the host should be designed so that it doesn't add to the intrinsic UDP protocol limitations.
I have only limited previous experience with UDP and Sockets in general (just casual streaming from sensor app on android phone to PC, for a small demo project).
I am not sure what is the most sensible way to receive all the packets at a fast rate. I imagine some sort of loop, or else some sort of self-retriggering receive operation. I have studied the difference between synchronous and asynchronous receives, and also the Task-based Asynchronous Pattern (TAP), but I don't feel very confident yet. It looks like a simple task, but its understanding is still complicated to me.
So the questions are:
Is UdpClient class suitable to a scenario like this, or am I better off going with the Socket class? (or another one, by the way)
Should I use synchronous or asynchronous receiving? If I use synchronous, I am afraid of losing packets, and if I use asynchronous, how should I clock/throttle the receive operations so that I don't start them at a rate that's too large?
I thought about a tight loop testing for UdpClient.Available() like below. Would it be a good design choice?
while (running)
{
if (udpSocket.Available() > 0)
{
// do something (what?)
}
}

Just read from your UdpClient as fast as you can (or as fast as is convenient for your application) using UdpClient.Receive(). The Receive() operation will block (wait) if there is no incoming packet, so calling Receive() three times in a row will always return you three UDP packets.
Generally, when receiving UDP packets you do not have to worry about the sending rate. Only the sender needs to worry about the sending rate. So the opposite advice for the sender is: Do not send a lot of UDP packets as fast as you can.
So my answers would be:
Yes, use UdpClient.
Use synchronous receiving. Call UdpClient.Receive() in a loop running in your receive thread, of in the main loop of your program, what ever you like.
You do not need to check for available data. UdpClient.Receive() will block until there is data available.
Do not be afraid of loosing packets when receiving UDP: You will almost never lose UDP packets on the receiving host because you do something wrong in receiving. UDP packets mostly get dropped by network components like routers somewhere on the network path, which cannot send (forward) UDP packets as fast as they receive them.
Once the packets arrive at your machine, your OS does a fair amount of buffering (for example 128k on Linux), so even if your application is unresponsive for say 1 second it will not lose any UDP packets.

C# Server - TCP/IP Socket Efficiency

Good day all!
I am working on a open source server for a game that is closed source - the game operates using TCP/IP sockets (instead of UDP, doh...) So being a connection based protocal, I am contrained to work with this.
My current program structure (dumbed down):
Core Thread
Receive new connection and create a new client object.
Client Object
IOloop (runs on its own thread)
Get data from socket, process packets. (one packet at a time)
Send data buffered from other threads (one packet at a time)
The client will send data immediately (no delay) when it is it's own thread.
I am noticing a huge flaw with the program, mainly it sends data very slow. Because I send data packet by packet, synchronously. (Socket.Send(byte[] buffer))
I would like to send data almost immediately, without delay - asynchronously.
I have tried creating a new thread every time I wanted to send a packet (so each packet sends on it's own managed thread) but this was a huge mess.
My current system uses a synchronous sending with nagle algorithm disabled - but this has the flaw of bottlenecking - send one packet, send operation blocks until TCP to confirms, then send the next... I can issue easily 10 packets every 100ms, and if the packets take 400ms to send, this backs up and breaks. Of course on my Local host I don't get this issue.
So, my question: What would be the best way to send multiple small packets of data? I am thinking of merging the data to send at the end of every IO thread loop to one large byte buffer, to reduce the back and forth delay - but the obvious problem here is this undermines the nagle algorithm avoidance which I had hoped would stop the delays.
How does a synchronous send work? Does it block until the data is confirmed correctly received by the recipient as I believe? Is there a way I can do this without waiting for confirmation? I do understand the packets must all go in order and correctly (as per the protocol specification).

I have done some research and I'm now using the current system:
Async send/receive operations with no threading, using AsyncCallbacks.
So I have the server start a read operation, then callback until done, when a packet is received, it is processed, then a new read operation will begin...
This method drastically reduces system overheads - thread pools, memory etc.
I have used some methods from the following and customised to my liking: https://msdn.microsoft.com/en-us/library/bew39x2a(v=vs.110).aspx
Very efficient, highly recommend this approach. For any TCP/IP network application low lag is very important and async callbacks is the best way to do it.

Async Sockets example which shows passing an object?

I need to create a server process which can push high frequency data (1000 updates per second) to around 50 client. I'm thinking the best way you do this is using async sockets with the SocketAsyncEventArgs type.
The client -> server connections will be long running at least several days to indefinite. I plan to have a server process listening and the clients connect and the server starts pushing the data to the clients.
Can someone point me to or show me an example of how to do this? I can't find any example showing a server process pushing an object to a client.
EDIT: This is over a gigibit LAN. Using windows server with 16 cores and 24gb ram
thanks

First, some more requirements from your side is required. You have server with lots of muscle, but it will fail miserably if you don't do what has to be done.
can the client live without some of the data? I mean, does the stream of the data need to reach other side in proper order, without any drops?
how big is 'the data'? few bytes or?
fact: scheduling interval on windows is 10 msec.
fact: no matter WHEN you send, clients will receive it depending on lots of stuff - network config, number of routers in-between, client processor load, and so on. so you need some kind of timestamping here
Depending on all this, you could design a priority queue with one thread servicing it and sending out UDP datagrams for each client. Also, since (4) is in effect, you can 'clump' some of your data together and have 10 updates per second of 100 data.
If you want to achieve something else, then LAN will be required here with lots of quality network equipment.

If you want to use .NET Sockets to create this server-client project, then this is a good outline of what's needed:
Since the server will be transferring data to several clients simultaneously, you'll need to use the asynchronous Socket.Beginxxx methods or the SocketAsyncEventArgs class.
You'll have clients connect to your server. The server will accept those connections and then add the newly connected client to an internal clients list.
You'll have a thread running within the server, that periodically sends notifications to all sockets in the clients list. If any exceptions/errors occurs while sending data to a socket, then that client is removed from the list.
You'll have to make sure that access to the clients list is synchronized since the server is a multithreaded application.
You don't need to worry about buffering your send data since the TCP stack takes care of that. If you do not want to buffer your data at all (i.e. have the socket send data immediately), then set Socket.NoDelay to true.
It doesn't seem like you need any data from your clients, but if you do, you'd have to make sure your server has a Socket.BeginReceive loop if using Socket.BeginXXX pattern or Socket.ReceiveAsync method if using SocketAsyncEventArgs.
Once you have the connection and transmission of data between server and client going, you then need to worry about serialization and deserialization of objects between client and server.
Serialization which occurs on the server is easy, since you can use the BinaryFormatter or other encoders to encode your object and dump the data onto the socket.
Deserialization on the other hand, which occurs on the client, can be pretty complex because an object can span multiple packets and you can have multiple objects in one packet. You essentially need a way to identify the beginning and end of an object within the stream, so that you can pluck out the object data and deserialize it.
One way to do this is to embed your data in a well known protocol, like HTTP, and send it using that format. Unfortunately, this also means you'd have to write a HTTP parser at the client. Not an easy task.
Another way is to leverage an existing encoding scheme like Google's protocol buffers. This approach would require learning how to use the protocol buffers stack.
You can also embed the data in an XML structure and then have a stream-to-XML decoder on your client side. This is probably the easiest approach but the least efficient.
As you can see, this is not an easy project, but you can get started with the Socket.BeginSend examples here and here, and the SocketAsyncEventArgs example here
Other Tips:
You can improve the reliability of your communication by having the client maintain two connections to the server for redundancy purposes. The reason being that TCP connections take a while to establish, so if one fails, you can still receive data from the other one while the client attempts to reconnect the failed connection.
You can look into using TCPClient class for the client implementation, since it's mostly reading a stream from a network connection.

What about rendezvous or 29 west? It would save reinventing the wheel. Dunno about amqp or zeromq they might work fine too....