Multi-threaded TcpClient send timeout after long open connection - c#

I'm having a problem with a TcpClient closing with a send timeout in a multi-threaded application after the connection has been open for a long period of time (several hours or overnight). The NetworkStream is being used by two threads, a UI thread and a background thread. There is a StreamReader used for reading incoming data, and a StreamWriter used for outgoing data. The StreamReader is only ever accessed one thread (the background one), but the StreamWriter is accessed by both the UI thread and the background thread.
What happens is that if I open a connection and connect to a remote server, I can immediately send and receive data without any problems. I do not get any send timeouts and data is correctly sent and received. However, if I then walk away and do not send any data for several hours and then return and start sending data (this is a chat application if that helps make it make sense), the socket will timeout on the send. During the time that I walk away there is no problem at all receiving data. Additionally, the remote server polls for an active connection and my client must respond to that, and since the connection is open for several hours it must be correctly sending a response. This polling response is only sent on the background thread, though. Data I enter is sent from the UI thread, and that's where the timeout occurs.
I'm guessing it's something to do with concurrent access, but I can't figure out what's causing it and why I can initially send data from the UI without a problem and only have it timeout after being idle for several hours.
Below is the relevant code. The variables at the top are declared in the class. Address and Port are properties in the class. WriteLine is the only method anywhere in the application that sends data with the StreamWriter. I put a lock around the call to StreamWriter.WriteLine hoping that would correct any synchronization issues. WriteLine is called from the background thread inside ParseMessage, and elsewhere from the UI.
If I increase TcpClient.SendTimeout to something larger, that doesn't fix anything. It just takes longer for the socket to timeout. I can't have the background thread both read and write because the background thread is blocking on ReadLine, so nothing would ever get written.
private TcpClient _connection;
private StreamWriter _output;
private Thread _parsingThread;
private object _outputLock = new object();
public void Connect(string address, int port)
{
Address = address;
Port = port;
_parsingThread = new Thread(new ThreadStart(Run));
_parsingThread.IsBackground = true;
_parsingThread.Start();
}
private void Run()
{
try
{
using (_connection = new TcpClient())
{
_connection.Connect(Address, Port);
_connection.ReceiveTimeout = 180000;
_connection.SendTimeout = 60000;
StreamReader input = new StreamReader(_connection.GetStream());
_output = new StreamWriter(_connection.GetStream());
string line;
do
{
line = input.ReadLine();
if (!string.IsNullOrEmpty(line))
{
ParseMessage(line);
}
}
while (line != null);
}
}
catch (Exception ex)
{
//not actually catching exception, just compressing example
}
finally
{
//stuff
}
}
protected void WriteLine(string line)
{
lock (_outputLock)
{
_output.WriteLine(line);
_output.Flush();
}
}

The blocking methods (Read and Write) of the NetworkStream class are not designed to be used concurrently from multiple threads. From MSDN:
Use the Write and Read methods for simple single thread synchronous blocking I/O. If you want to process your I/O using separate threads, consider using the BeginWrite and EndWrite methods, or the BeginRead and EndRead methods for communication.
My assumption is that, when you call WriteLine (and, internally, NetworkStream.Write) from your UI thread, it would block until the concurrent ReadLine (internally, NetworkStream.Read) operation completes in the background thread. If the latter does not do so within the SendTimeout, then the Write would time out.
To work around your issue, you should convert your implementation to use non-blocking methods. However, as a quick hack to first test whether this is really the issue, try introducing a DataAvailable poll before your ReadLine:
NetworkStream stream = _connection.GetStream();
StreamReader input = new StreamReader(stream);
_output = new StreamWriter(stream);
string line;
do
{
// Poll for data availability.
while (!stream.DataAvailable)
Thread.Sleep(300);
line = input.ReadLine();
if (!string.IsNullOrEmpty(line))
{
ParseMessage(line);
}
}
while (line != null);
line = input.ReadLine();

Related

.NET sockets: beginReceive vs dedicated thread with receive

From what I've read, beginReceive is considered superior to receive in almost all cases (or all?). This is because beginReceive is asynchronous and waits for data to arrive on a seperate thread, thereby allowing the program to complete other tasks on the main thread while waiting.
But with beginReceive, the callback function still runs on the main thread. And so there is overhead with switching back and forth between the worker thread and the main thread each time data is received. I know the overhead is small, but why not avoid it by simply using a separate thread to host a continuous loop of receive calls?
Can someone explain to me what is inferior about programming with the following style:
static void Main()
{
double current_temperature = 0; //variable to hold data received from server
Thread t = new Thread (UpdateData);
t.start();
// other code down here that will occasionally do something with current_temperature
// e.g. send to a GUI when button pressed
... more code here
}
static void UpdateData()
{
Socket my_socket = new Socket(AddressFamily.InterNetwork,SocketType.Stream,ProtocolType.Tcp);
my_socket.Connect (server_endpoint);
byte [] buffer = new byte[1024];
while (true)
my_socket.Receive (buffer); //will receive data 50-100 times per second
// code here to take the data in buffer
// and use it to update current_temperature
... more code here
end
}

Semaphores and Web Sockets

I am trying to understand and fix the exception I am receiving:
There is already one outstanding 'SendAsync' call for this WebSocket instance. ReceiveAsync and SendAsync can be called simultaneously, but at most one outstanding operation for each of them is allowed at the same time.
So I have multiple threads going to a handler which wants to send specific clients specific information.
When a client connects a mapping is created from that client's specific connection to the data that he or she wants to have streamed to them via a web socket connection.
My code looks like:
foreach (KeyValuePair<socketInfo, data> pair in mapping)
{
//Get the unique sendAsync per connection
WebSocketSendAsync sendAsync = pair.Key.sendAsync;
//Get the data the specific connection wnats
dynamic info = fillData(pair.Value);
//Convert the data to send to bytes
string sendInfo = Newtonsoft.Json.JsonConvert.SerializeObject(attributeInfo);
byte[] infoBytes = System.Text.Encoding.UTF8.GetBytes(sendInfo);
//Send the data
Semaphore send = new Semaphore(1, 1);
send.WaitOne();
await sendAsync(new ArraySegment<byte>(infoBytes), 1, false, pair.Key.callCancelled);
send.Release(1);
}
I understand their can be only one sendAsync going at a time (even if multiple threads are trying to do it?), so I figured a semaphore would be the proper way to go about this. I want only one thread to be able to use the await sendAsync method at a time and have the other threads wait till the previous one is finished.
This is my first time using semaphores so I am not sure why it is not working, any help?
The problem seems that your instance of Semaphore is created on each loop. It should instead be created once and then you can use this instance to protect your code from being accessed by more than one thread at the same time.
I suggest that you use the SemaphoreSlim instead of Semaphore since you are using async/await in your code. This class has a WaitAsync method which is an awaitable method.
public class MyClass
{
SempahoreSlim _semaphore = new SemaphoreSlim(1, 1);
public void Foo(/*...*/)
{
foreach(/*...*/)
{
/*...*/
await _semaphore.WaitAsync();
try
{
await sendAsync(/*...*/);
}
finally
{
_semaphore.Release();
}
}
}
}

Constantly read from NetworkStream async

I am a farily new .NET-developer and I'm currently reading up on async/await. I need to work on a framework used for testing devices that are controlled by remotely accessing servers using TCP and reading/writing data from/to these servers. This will be used for unit tests.
There is no application-layer protocol and the server may send data based on external events. Therefore I must be able to continuously capture any data coming from the server and write it to a buffer, which can be read from a different context.
My idea goes somewhere along the lines of the following snippet:
// ...
private MemoryStream m_dataBuffer;
private NetworkStream m_stream;
// ...
public async void Listen()
{
while (Connected)
{
try
{
int bytesReadable = m_dataBuffer.Capacity - (int)m_dataBuffer.Position;
// (...) resize m_dataBuffer if necessary (...)
m_stream.ReadTimeout = Timeout;
lock (m_dataBuffer)
{
int bytesRead = await m_stream.ReadAsync(m_dataBuffer.GetBuffer(),
(int)m_dataBuffer.Position, bytesReadable);
m_stream.Position += bytesRead;
}
}
catch (IOException ex)
{
// handle read timeout.
}
catch (Exception)
{
throw new TerminalException("ReadWhileConnectedAsync() exception");
}
}
}
This seems to have the following disadvantages:
If calling and awaiting the Listen function, the caller hangs, even though the caller must be able to continue (as the network stream should be read as long as the connection is open).
If declaring it async void and not awaiting it, the application crashes when exceptions occur in the Task.
If declaring it async Task and not awaiting it, I assume the same happens (plus I get a warning)?
The following questions ensue:
Can I catch exceptions thrown in Listen if I don't await it?
Is there a better way to constantly read from a network stream using async/await?
Is it actually sane to try to continuously read from a network stream using async/await or is a thread a better option?
async void should at the very least be async Task with the return value thrown away. That makes the method adhere to sane standards and pushes the responsibility into the caller which is better equipped to make decisions about waiting and error handling.
But you don't have to throw away the return value. You can attach a logging continuation:
async Task Log(Task t) {
try { await t; }
catch ...
}
And use it like this:
Log(Listen());
Throw away the task returned by Log (or, await it if you wish to logically wait).
Or, simply wrap everything in Listen in a try-catch. This appears to be the case already.
Can I catch exceptions thrown in Listen if I don't await it?
You can find out about exceptions using any way that attaches a continuation or waits synchronously (the latter is not your strategy).
Is there a better way to constantly read from a network stream using async/await?
No, this is the way it's supposed to be done. At any given time there should be one read IO outstanding. (Or zero for a brief period of time.)
Is it actually sane to try to continuously read from a network stream using async/await or is a thread a better option?
Both will work correctly. There is a trade-off to be made. Synchronous code can be simpler, easier to debug and even less CPU intensive. Asynchronous code saved on thread stack memory and context switches. In UI apps await has significant benefits.
I would do something like this:
const int MaxBufferSize = ... ;
Queue<byte> m_buffer = new Queue<byte>(MaxBufferSize);
NetworkStream m_stream = ... ;
...
// this will create a thread that reads bytes from
// the network stream and writes them into the buffer
Task.Run(() => ReadNetworkStream());
private static void ReadNetworkStream()
{
while (true)
{
var next = m_stream.ReadByte();
if (next < 0) break; // no more data
while (m_buffer.Count >= maxBufferSize)
m_buffer.Dequeue(); // drop front
m_buffer.Enqueue((byte)next);
}
}

Trying to interrupt a thread stuck in reader.readLine()

I have a connection with an IRC server over TCP. I read the data with an independent task, so far, so good. However, if I want to quit the program, I can't quit the thread because it is stuck in the reader.ReadLine() command (threadShouldRun has no impact). Using Interrupt() or Abort() doesn't appear to change anything either.
Either I need a way to determine when there are more lines to read, or I need to forcefully kill the thread (even though that's bad).
private System.Threading.Thread myThread;
private bool threadShouldRun = true;
private StreamReader reader;
private void readStream()
{
while(threadShouldRun)
{
string line = reader.ReadLine();
if (line != null)
{
newLineEvent(this, new NewLineEventArgs(line));
}
}
}
Use asynchronous calls like BeginRead as shown here and avoid the loop:
http://msdn.microsoft.com/en-us/library/system.io.stream.beginread.aspx
A solution would be to set a TimeOut on your receiving socket ( http://msdn.microsoft.com/it-it/library/system.net.sockets.socket.receivetimeout(v=vs.110).aspx ).
Once the time expires, a SocketException will be raised, so you can catch it and reiterate the control on your threadShouldRun.
In case you want to quit (and hence set threadShouldRun to false), there are two scenarios:
Data received, you will handle it, and on the next check the variable will be found false and thread will terminate;
No Data received, TimeOut happens, Exception is raised, you will handle it, and on the check the thread will terminate.

Difference between NetworkStream.Read() and NetworkStream.BeginRead()?

I need to read from NetworkStream which would send data randomly and the size of data packets also keep varying. I am implementing a multi-threaded application where each thread would have its own stream to read from. If there is no data on the stream, the application should keep waiting for the data to arrive. However, if the server is done sending data and has terminated the session, then it should exit out.
Initially I had utilised the Read method to obtain the data from the stream, but it used to block the thread and kept waiting until data appeared on the stream.
The documentation on MSDN suggests,
If no data is available for reading,
the Read method returns 0. If the
remote host shuts down the connection,
and all available data has been
received, the Read method completes
immediately and return zero bytes.
But in my case, I have never got the Read method to return 0 and exit gracefully. It just waits indefinitely.
In my further investigation, I came across BeginRead which watches the stream and invokes a callback method asynchronously, as soon as it receives the data. I have tried to look for various implementations using this approach as well, however, I was unable to identify when would using BeginRead be beneficial as opposed to Read.
As I look at it, BeginRead has just the advantage of having the async call, which would not block the current thread. But in my application, I already have a separate thread to read and process the data from stream, so that wouldn't make much difference for me.
Can anyone please help me understand the Wait and Exit mechanism for
BeginRead and how is it different from Read?
What would be the best way to implement the desired functionality?
I use BeginRead, but continue blocking the thread using a WaitHandle:
byte[] readBuffer = new byte[32];
var asyncReader = stream.BeginRead(readBuffer, 0, readBuffer.Length,
null, null);
WaitHandle handle = asyncReader.AsyncWaitHandle;
// Give the reader 2seconds to respond with a value
bool completed = handle.WaitOne(2000, false);
if (completed)
{
int bytesRead = stream.EndRead(asyncReader);
StringBuilder message = new StringBuilder();
message.Append(Encoding.ASCII.GetString(readBuffer, 0, bytesRead));
}
Basically it allows a timeout of the async reads using the WaitHandle and gives you a boolean value (completed) if the read was completed in the set time (2000 in this case).
Here's my full stream reading code copied and pasted from one of my Windows Mobile projects:
private static bool GetResponse(NetworkStream stream, out string response)
{
byte[] readBuffer = new byte[32];
var asyncReader = stream.BeginRead(readBuffer, 0, readBuffer.Length, null, null);
WaitHandle handle = asyncReader.AsyncWaitHandle;
// Give the reader 2seconds to respond with a value
bool completed = handle.WaitOne(2000, false);
if (completed)
{
int bytesRead = stream.EndRead(asyncReader);
StringBuilder message = new StringBuilder();
message.Append(Encoding.ASCII.GetString(readBuffer, 0, bytesRead));
if (bytesRead == readBuffer.Length)
{
// There's possibly more than 32 bytes to read, so get the next
// section of the response
string continuedResponse;
if (GetResponse(stream, out continuedResponse))
{
message.Append(continuedResponse);
}
}
response = message.ToString();
return true;
}
else
{
int bytesRead = stream.EndRead(asyncReader);
if (bytesRead == 0)
{
// 0 bytes were returned, so the read has finished
response = string.Empty;
return true;
}
else
{
throw new TimeoutException(
"The device failed to read in an appropriate amount of time.");
}
}
}
Async I/O can be used to achieve the same amount of I/O in less threads.
As you note, right now your app has one thread per Stream. This is OK with small numbers of connections, but what if you need to support 10000 at once? With async I/O, this is no longer necessary because the read completion callback allows context to be passed identifying the relevant stream. Your reads no longer block, so you don't need one thread per Stream.
Whether you use sync or async I/O, there is a way to detect and handle stream closedown on the relevant API return codes. BeginRead should fail with IOException if the socket has already been closed. A closedown while your async read is pending will trigger a callback, and EndRead will then tell you the state of play.
When your application calls BeginRead,
the system will wait until data is
received or an error occurs, and then
the system will use a separate thread
to execute the specified callback
method, and blocks on EndRead until
the provided NetworkStream reads data
or throws an exception.
Did you try server.ReceiveTimeout? You can set the time which Read() functon will wait for incomming data before returning zero. In your case, this property is probably set to infinite somewhere.
BeginRead is an async process which means your main thread will start execute Read in another process. So now we have 2 parallel processes. if u want to get the result, u have to call EndRead, which will gives the result.
some psudo
BeginRead()
//...do something in main object while result is fetching in another thread
var result = EndRead();
but if your main thread doesn't have anything else to do and u have to need the result, u should call Read.

Categories

Resources