HttpWebRequest timing out, but no packets leaving client - c#

I have a c# application with the following requirements.
Periodically(700ms) send data to a cgi using a GET.
The server will respond with only the GET string sent as confirmation.
Values sent are constantly changing and are not stored, ordered, or retried.
Only one connection made to the server at a time.
Timer events that find a previous connection still in progress should simply exit quietly and let the next event carry on.
The client machine is Windows Server 2012. The HTTP server is a black box appliance(believed to be running linux). Environment is low-latency production with extremely tight security(network/policies/etc).
The problem:
Periodically, the client will begin throwing exceptions
"The operation timed out"
and the drip feed stops. Simultaneously, Wireshark shows
no connections being made from the client to the server.
However, VisualStudio shows that
every 700ms the client is attempting to connect and is timing out.
When it's not throwing the above exception, it seems to work fine. At no time does the HTTP server appear to be rejecting connections. Sometimes the client will recover, most of the time it will continue to throw exceptions until restarted. AppDomain thread pool is stable. Memory consumption is fine.
To confuse matters, I have identical hardware/software/network without the security in a lab where everything runs flawlessly for days at a time. My code is being fingered as "malfunctory".
I have recently discovered that the HttpWebRequest.Timeout property bounds the entire life of the transaction, not just the response wait. I am increasing this and it's currently under test.
Anyone see anything glaring(or otherwise) that would cause this problem in the below code?
// timer call back at 700ms interval
void m_postTimer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
if (Interlocked.Read(ref this.m_isLocked) == 0)
{
if (Monitor.TryEnter(this.m_lock, 10))
{
Interlocked.Exchange(ref this.m_isLocked, 1);
try
{
// value1 & value2 set elsewhere
string url = String.Format("http://1.1.1.1/set.cgi?value1={0}&value2={1}", value1, value2);
this.m_post = (HttpWebRequest)WebRequest.Create(url);
this.m_post.Timeout = 500;
this.m_post.Credentials = new NetworkCredential(this.user, this.pass);
WebResponse response = this.m_post.GetResponse();
response.Close();
Monitor.Exit(this.m_lock);
Interlocked.Exchange(ref this.m_isLocked, 0);
}
catch (WebException ex)
{
// handle web exceptions
// if we've locked earlier and hit an exception,
// the unlock has been skipped. unlock
if (Interlocked.Read(ref this.m_isLocked) > 0)
{
Monitor.Exit(this.m_lock);
Interlocked.Exchange(ref this.m_isLocked, 0);
}
}
}
}
else
{
// indicate that a pre-existing connection is still in progress
}
}
catch (Exception ex)
{
// handle generic exception
// if we've locked earlier and hit an exception,
// the unlock has been skipped. unlock
if (Interlocked.Read(ref this.m_isLocked) > 0)
{
Monitor.Exit(this.m_lock);
Interlocked.Exchange(ref this.m_isLocked, 0);
}
}
}

Related

NetworkStream exceptions and return codes

I'm trying to educate myself on the intricacies of reading from a NetworkStream, and understanding the various ways in which problems can occur. I have the following code:
public async Task ReceiveAll()
{
var ns = this.tcp.GetStream();
var readBuffer = new byte[1000];
while (true)
{
int bytesRead;
try
{
bytesRead = await ns.ReadAsync(readBuffer, 0, readBuffer.Length);
if (bytesRead == 0)
{
// Remote disconnection A?
break;
}
}
catch (IOException)
{
// Remote disconnection B?
break;
}
catch (ObjectDisposedException)
{
// Local disconnection?
break;
}
/*Do something with readBuffer */
}
}
I've marked three points in the code where the program says 'something has gone awry, there is no point continuing'.
The 'Local disconnection' isn't exactly something wrong, it will happen when I locally close the socket which is the only way to exit the loop under normal circumstances. I don't think anything else can cause this, so I think I'm safe to just swallow the exception.
The two 'Remote disconnection' points are what I'm not sure about. I know ReadAsync will return 0 if the connection is terminated remotely (A), but the IOException also seems to fire in some circumstances. If my remote client is a C# console, then closing the socket seems to make 'A' happen, and closing the console window seems to be make 'B' happen. I'm not sure I understand what the difference is between these scenarios?
Finally, a bit of a general question, but is there anything glaringly wrong with this bit of code or my above assumptions?
Thanks.
EDIT: In response to my use of ObjectDisposedException to abort out of the loop:
This is what my 'Stop' method looks like (from the same class as above):
public void Stop()
{
this.tcp.Close();
}
This causes the pending 'ReadAsync' to except with ObjectDisposedException. AFAIK there isn't any other way to abort this. Changing this to:
public void Stop()
{
this.tcp.Client.Shutdown(SocketShutdown.Both);
}
Doesn't appear to actually do anything to the pending call, it just continues waiting.
When NetworkStream returns 0, this means that the socket has received a disconnect packet from the remote party. This is how network connections are supposed to end.
The correct way to shut down the connection (especially if you are in a full-duplex conversation) is to call socket.Shutdown(SocketShutdown.Send) and give the remote party some time to close their send channel. This ensures that you receive any pending data instead of slamming the connection shut. ObjectDisposedException should never be part of the normal application flow.
Any exceptions thrown indicate that something went wrong, and I think it's safe to say you can no longer rely on the current connection.
TL;DR
I don't see anything wrong with your code, but (especially in full-duplex communication) I'd shut down the send channel and wait for a 0-byte packet to prevent receiving ObjectDisposedExceptions by default:
use tcp.Shutdown(SocketShutdown.Send) to tell the remote party you want to disconnect
your loop may still receive data that the remote party was sending
your loop will, if everything went right, then receive a 0-byte packet, indicating that the remote party is disconnecting
loop terminates the right way
you may want to decide to dispose the socket after a certain amount of time, if you haven't received the 0-byte packet

Trap timeouts in Parse

The Parse docs states that: "By default, all connections have a timeout of 10 seconds, so tasks will not hang indefinitely."
I have this code (from Parse web site:
try
{
Task t=test.SaveAsync();
await t;
int j = t.Id;
}
catch (ParseException exc)
{
if (exc.Code == ParseException.ErrorCode.ObjectNotFound)
{
// Uh oh, we couldn't find the object!
}
else
{
//Some other error occurred
}
}
I run my app (on iPhone/Xamarin) with the network on, then I set 100% network loss, then I run the code above. I neither get an exception, nor does code reach the line after the await statement where I read the task ID. In other words, I do not know what happened.
My question is:
How do I trap Pasre timeouts in case there is no network? Is there an event or something I can use or I have to implement it all by myself, with timers and all?
Thanks, don escamilloATgmail.com

Is writing zero bytes to a network stream a reliable way to detect closed connections?

I'm working on an application where a client connects with a TCP connection which then triggers an amount of work that may potentially take a lot of time to complete. This work must be cancelled if the user drops the TCP connection.
Currently, what I'm doing is starting up a timer that periodically checks the networks streams connectivity by doing this:
// stream is a Stream instance
var abort = false;
using (new Timer(x => {
try
{
stream.Write(new byte[0], 0, 0);
}
catch (Exception)
{
abort = true;
}
}, null, 1000, 1000))
{
// Do expensive work here and check abort periodically
}
I would have liked to read the CanWrite, CanRead or Connected but they report the last status of the stream. Is writing zero bytes a reliable way of testing connectivity, or can this itself cause issues? I cannot write or read any real data on the stream since that would mess up the client.
Let's just say that I have known it to work, decades ago, but there is no intrinsic reason why it should. Any of the API layers between you and the TCP stack is entitled to suppress the call to the next layer down, and even if it gets all the way into the stack it will only return an error if:
It checks for network errors before checking for zero length, which is implementation-dependent, and
There already was a network error, caused by some previous operation, or an incoming RST.
If you're expecting it to magically probe the network all the way to the other end, it definitely won't.

Problems on Compact Framework when communication is lost

I have a problem on a hand held device that should be listening for messages from my server application. When the device loses connection to the network my server is on, this background worker (implemented from OpenNetCF) stops responding. I've placed messages in the ProgressChanged and RunWorkerCompleted events to see when they are raised, and in the following code after the RecieveFrom in the while loop, and after the loop termination, as well as in all of the exceptions. I don't see the messages from the exceptions, or after the while loop at all, and the messages stop after the connection is lost. All of the messages are shown by setting the text in a visible label, with the background colour of the label changing so that I can see if the loop is running. The loop seems to stop running, even after the connection is re-gained, and attempting to re-run the backgroundworker generates an "Already in use" exception. So, why would the worker stop responding while continuing to run?
private void backgroundWorker1_DoWork(object sender, DoWorkEventArgs e)
{
BackgroundWorker wWorker = (BackgroundWorker)sender;
byte[] wBytes = new Byte[4096];
string wsReceive;
EndPoint wRemoteEP = (EndPoint)new IPEndPoint(IPAddress.Any, 0);
try
{
while (true)
{
if (wWorker.CancellationPending)
{
break;
}
mSocket.ReceiveFrom(wBytes, ref wRemoteEP);
if (wBytes.Length < 1)
{
continue;
}
wsReceive = Encoding.ASCII.GetString(wBytes, 0, wBytes.Length);
wWorker.ReportProgress(0, wsReceive);
}
mSocket.Close();
}
catch (ThreadAbortException)
{
}
catch (ThreadStateException)
{
}
catch (Exception E)
{
MessageBox.Show(E.Message, "Communication Error");
}
}
Right from the docs on Socket.ReceiveFrom:
If no data is available for reading, the ReceiveFrom method will block until data is available
So when communication is lost, you're unable to receive and your ReceiveFrom call blocks indefinitely. It's going to stop reporting anything while it waits for data, but the thread is still alive and therefore the BackgroundWorker can't be re-run.
Just my 2 cents:
Whenever I implement a socket communication, I treat 0 bytes received as an indication that the remote server has closed gracefully. I think that you should also treat this to start a reconnection attempt, instead of just "continuing" the loop.

WSACancelBlockingCall exception

Ok, I have a strange exception thrown from my code that's been bothering me for ages.
System.Net.Sockets.SocketException: A blocking operation was interrupted by a call to WSACancelBlockingCall
at System.Net.Sockets.Socket.Accept()
at System.Net.Sockets.TcpListener.AcceptTcpClient()
MSDN isn't terribly helpful on this : http://msdn.microsoft.com/en-us/library/ms741547(VS.85).aspx and I don't even know how to begin troubleshooting this one. It's only thrown 4 or 5 times a day, and never in our test environment. Only in production sites, and on ALL production sites.
I've found plenty of posts asking about this exception, but no actual definitive answers on what is causing it, and how to handle or prevent it.
The code runs in a separate background thread, the method starts :
public virtual void Startup()
{
TcpListener serverSocket= new TcpListener(new IPEndPoint(bindAddress, port));
serverSocket.Start();
then I run a loop putting all new connections as jobs in a separate thread pool. It gets more complicated because of the app architecture, but basically:
while (( socket = serverSocket.AcceptTcpClient()) !=null) //Funny exception here
{
connectionHandler = new ConnectionHandler(socket, mappingStrategy);
pool.AddJob(connectionHandler);
}
}
From there, the pool has it's own threads that take care of each job in it's own thread, separately.
My understanding is that AcceptTcpClient() is a blocking call, and that somehow winsock is telling the thread to stop blocking and continue execution.. but why? And what am I supposed to do? Just catch the exception and ignore it?
Well, I do think some other thread is closing the socket, but it's certainly not from my code.
What I would like to know is: is this socket closed by the connecting client (on the other side of the socket) or is it closed by my server. Because as it is at this moment, whenever this exception occurs, it shutsdown my listening port, effectively closing my service. If this is done from a remote location, then it's a major problem.
Alternatively, could this be simply the IIS server shutting down my application, and thus cancelling all my background threads and blocking methods?
Is it possible that the serverSocket is being closed from another thread? That will cause this exception.
This is my example solution to avoid WSAcancelblablabla:
Define your thread as global then you can use invoke method like this:
private void closinginvoker(string dummy)
{
if (InvokeRequired)
{
this.Invoke(new Action<string>(closinginvoker), new object[] { dummy });
return;
}
t_listen.Abort();
client_flag = true;
c_idle.Close();
listener1.Stop();
}
After you invoke it, close the thread first then the forever loop flag so it block further waiting (if you have it), then close tcpclient then stop the listener.
This could happen on a serverSocket.Stop(). Which I called whenever Dispose was called.
Here is how my exception handling for the listen thread looked like:
try
{
//...
}
catch (SocketException socketEx)
{
if (_disposed)
ar.SetAsCompleted(null, false); //exception because listener stopped (disposed), ignore exception
else
ar.SetAsCompleted(socketEx, false);
}
Now what happened was, every so often the exception would occur before _disposed was set to true. So the solution for me was to make everything thread safe.
Same here!
But i figured out, that the ReceiveBuffer on 'server-side' was flooded from the clients!
(In my case a bunch of RFID-Scanners, who kept spamming the TagCode, instead of stop sending until next TagCode arrives)
It helped to raise the ReceiveBuffers and reconfigure the scanners...
More recently I saw this exception when using HttpWebRequest to PUT a large file and the Timeout period was passed.
Using the following code as long as your upload time > 3 seconds it will cause this error as far as I could see.
string path = "Reasonably large file.dat";
int bufferSize = 1024;
byte[] buffer = new byte[bufferSize];
System.Net.HttpWebRequest req = (HttpWebRequest)System.Net.HttpWebRequest.Create("Some URL");
req.Method = "PUT";
req.Timeout = 3000; //3 seconds, small timeout to demonstrate
long length = new System.IO.FileInfo(path).Length;
using (FileStream input = File.OpenRead(path))
{
using (Stream output = req.GetRequestStream())
{
long remaining = length;
int bytesRead = 0;
while ((bytesRead = input.Read(buffer, 0, (int)Math.Min(remaining, (decimal)bufferSize))) > 0)
{
output.Write(buffer, 0, bytesRead);
remaining -= bytesRead;
}
output.Close();
}
input.Close();
}

Categories

Resources