I have a server that is doing work instructed by an Azure queue. It is almost always on very high CPU doing multiple tasks in parallel and some of the tasks use Parallel.ForEach.
During the running of the tasks I write analytic events to another Azure queue by calling CloudQueue.AddMessageAsync with await.
I noticed thousands of these analytic writings that fail with the following error:
WebException: The remote server returned an error: (500) Internal Server Error.
I checked Azure's storage event logs, and I have a nice bunch of PutMessage commands that take 80.000ms end to end, but they only take 1ms for Azure itself. The HTTP status code I get is 500 and Azure describes the reason as client timeout.
What I think is happening is that my code calls the AddMessageAsync and from that point my thread is released and the network driver is sending the request and waiting for a response. When getting a response, the network driver needs a thread to get the response and a task is scheduled to do that and calls my continuation. Because my server is constantly on high load, the task takes a long time to get a thread and by then the Azure server decides this is a client timeout.
The code calling azure:
await cloudQueue.AddMessageAsync(new CloudQueueMessage(aMessageContent));
The exception:
StorageException: The remote server returned an error: (500) Internal Server Error.
Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result):11
Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions+<>c__DisplayClass4.<CreateCallbackVoid>b__3(IAsyncResult ar):45
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
AzureCommon.Data.AsyncQueueDataContext+<AddMessage>d__d.MoveNext() in c:\BuildAgent\work\14078ab89161833\Azure\AzureCommon\Data\Async\AsyncQueueDataContext.cs:60
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):82
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):41
AzureCommon.Storage.AzureEvent+<DispatchAsync>d__1.MoveNext() in c:\BuildAgent\work\14078ab89161833\Azure\AzureCommon\Events\AzureEvent.cs:354
WebException: The remote server returned an error: (500) Internal Server Error.
System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult):41
Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndGetResponse[T](IAsyncResult getResponseResult):44
Am I right about why this is happening? If so, would using a single-threaded synchronization context for this call be better for me?
A row from Azure storage log. You can find details about what each property means here.
<request-start-time> <operation-type> <request-status> <http-status-code> <end-to-end-latency-in-ms> <server-latency-in-ms>
2014-07-29T14:55:20.0794198Z PutMessage ClientTimeoutError 500 86929 1
Thanks.
The error 500 means that the server has received a bad request or it has crashed for various other reasons. I don't believe that it has to do with the high load of your threads. Please consider taking the following actions:
Check the name of the queue you are using. The name needs to be lowercase, starting with a character. This is a common issue that causes error 500 with no enlighting error message from the server.
Set up the retry policy of the Azure Storage SDK client, preferably with an Exponential retry policy.
Make sure you are using the latest Azure Storage SDK, as the underlying protocol has recently changed to a more efficient one.
'Bad Request' is a 400 error, not a 500 error. A 500 Error indicates any kind of server error, so it's perfectly reasonable to get that response, and many client-side libraries will use a 500 error code for similar types of unexpected issues.
Normally a 'client timeout' response would never make it to the client (because it timed out!). The only situation I can think of where a client timeout response could make it to the client would be if the request was more than a single network packet and the client was too slow in sending packets after the first one. This could easily be caused by CPU contention on the client device. I would recommend using a higher priority thread for listening to network responses but then immediately pass off the processing of the response to a normal priority thread. Overloaded CPU will cause all sorts of timeout issues because the code can't tell the difference between a network response not coming in soon enough and the CPU not scheduling the listener in time to receive the response (or even to send the request). Even local disk I/O and locking can timeout in these situations, depending on the underlying implementation.
Related
I've an application (on production environment) that makes a lot of concurrent (multithreading) calls to WCF services (.Net Framework 4.0, SOAP BasicHttpBinding). Sometimes, the last request throws a TimeoutException:
System.TimeoutException: The request channel timed out while waiting for a reply after...
I can't reproduce it on a local envrionment, so it's difficult to apply changes to try. Anyway I've increased the timeout but the exception is throwing equally (later evidently). I've traced the server searching for inter-threading locks or Oracle locks, but I couldn't find anything.
I've activated internal code traces and I've deduced that the request didn't reach my server code, so, watching the IIS traces I've found an HTTP error ~2' after the request:
sc-status sc-substatus sc-win32-status time-taken
500 0 64 118265
But the timeout exceptions is thrown later depending on the wcf binding configuration. So I have two questions:
Why WCF is not catching that http error and remains waiting for response? I've searched for the error but I have high values on servicethrottling parameters.
Why the server is throwing that error without getting the server code not even an on an IDispatchMessageInspector I've implemented to log some data on request and response?
I have a .Net application up and running.
We have had a fluctuating connection yesterday. While testing in such scenarios we had received multiple server time out exception emails like below.
Server Time Out
Type : System.Web.HttpException, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
Message : Request timed out.
Source :
Help link :
WebEventCode : 3001
ErrorCode : -2147467259
Data : System.Collections.ListDictionaryInternal
TargetSite :
HResult : -2147467259
Stack Trace : The stack trace is unavailable.
Additional Info:
IMPORTANT: Above exception occurred while doing a ajax post by a button placed with in update panel.
My question here is why would a slow internet on client side raise such server time out exception?
Isn't server timeout exception is related to such cases where server cannot execute the request in underlying time mention in HttpRuntime setting? May be due to some lengthy operation or some long database execution which takes longer than the time mentioned in setting under HttpRuntime.
If server is not able to connect to the client due to clients fluctuating internet, then Client Disconnected exception would be raised which we did yesterday. But I am not able to conclude the reason for this server timeout exception.
I already know that increasing the execution timeout will fix the issue, but I have to provide technical explanation for the reason as to why such exception of Server Timeout raised.
My best guess here is that the ajax request would be doing some continuous communication with server for executing of single request server and would raise timeout exception if it does not receives some required further communication messages due to client's bad internet. I have search over internet for the same to support my guess but in vain.
Also to provide environmental details, there is a load balancer serving the request.
Any help would be highly appreciated.
It is because (as you write) the connection of client to server is slow, so if the server (or client) sending data to this server, connection can´t handle it, so you get timeout error, because the data can´t been transfered in defined time.
You also write, that this is caused by sending Ajax request, so maybe try to increase execusion timeout in web configuration file (web.config):
<httpRuntime executionTimeout = "number(in seconds)"/>
More about executionTimeout here and here about Ajax requests.
Firstly, I think the cause of this error is because of execution time required by your application request connecting with the remote server is exceeding the currently set ASP.NET request execution timeout value. As per the MSDN Exception Document, default value is set to 110 seconds, in that it is remarked like:
The ExecutionTimeout property indicates the maximum number of seconds
a request is allowed to execute before being automatically shut down
by ASP.NET.
So based on error detail with event code 3001 occurs because no response was received during the time-out period for a request. You can use IIS troubleshooting failed request mechanism to figure it out exact issue like any poor performance/deadlocks when making calls from your ASP.NET application.
Secondly, it is not related to user's internet connectivity issue otherwise you get exception with status like connection-closed or keep alive failure. See this article for detail. The browser is going to wait for a 60 minutes(which is very long period of time that server isn't going to answer any request)for server to response.
And at any case when the browser abandons any request, it is going to close the socket and you'll get an error page from the browser. You don't get anything related to sever-end.
I have an issue with WCF timing out. The strange thing is that my method is actually being called on the server, but the client call ton the object returned from CreateChannel() is timing out with an exception.
The entire error messsage:
This request operation sent to net.pipe://localhost/AndonServer did not receive a reply within the configured timeout (00:01:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.
I could just decrease the timeout setting to 5 seconds, say, but that's a bit dirty. Anyone have any ideas why this might be happening?
Mark
It means you elapsed the timeout period waiting for a reply from the server. By default, all calls in WCF have both a request and a reply, even void methods. The server needs to complete the call promptly so WCF will send a reply. Another is option is to use a one-way call if the client does not require a reply from the server.
I want to monitor a WCF server, and send email notification if the server is down. To accomplish that, I am writing a console app to periodically send dummy request to the server, and check if response is sent back. When the console app received exception the server has issues, including the server being down.
However, the problem is that I received different exception on different status of the server. Below is the exceptions returned from the server when it is on different status. However, all seem belong to server down category. Any idea??:
When IIS is turned off
System.ServiceModel.EndpointNotFoundException,
Message:
There was no endpoint listening at http://localhost/service.svc that could accept the message. This is often caused by an incorrect
address or SOAP action. See InnerException, if present, for more
details.
Inner Exception Message:The remote server returned an error: (404) Not Found
When a Web.config file is deliberately changed to a wrong name:
System.ServiceModel.ServiceActivationException
Link:
http://localhost/service.svc
Message:
The requested service, 'http://localhost/service.svc' could not be activated. See the server's diagnostic trace logs for more
information.
For other unknown reason
System.ServiceModel.ServerTooBusyException
Message:
The HTTP service located at http://localhost/service.svc' is too busy.
Message:
The remote server returned an error: (503) Server Unavailable.
Update 1
The exception does NOT always return http status code.
Update 2
Apart from using WCF proxy to call the service, I have to use WebRequest too, as below:
try
{
WebRequest webRequest = WebRequest.Create(uri);
webRequest.Method = "GET";
HttpWebResponse httpWebResponse = (HttpWebResponse)webRequest.GetResponse();
}
catch () //what excpetion will tell me server is down??
{
...
}
The actual content of the error shouldn't really be of consequence - unless you're monitoring individual operations on the service (i.e. should a POST with some data to a particular URL return a specific response) - realistically, then, you're just going to be looking at the status code itself; and for that you want to look through all the HTTP Status Codes and see those which look like errors as far as you're concerned.
As a good starting point - you might want to consider nearly all of the 5xx codes; as they are all connected with server errors.
You might also want to consider some of the 4xx codes (although these are usually connected with clients, so be ruthless). In particular:
400 - Bad Request - so long as you can be sure that the server should be able to understand the request
404 - Not Found - if you're sure that the given URL should be present
405 - Method Not Allowed - if you're sure that the given HTTP verb should be supported (e.g. a POST or DELETE)
For some of the narrower 4xx codes, e.g. 413 Request Entity Too Large or 414 Request-URI Too Long; these could conceivably happen after days or months of normal operation due to things like security updates. In which case you're not necessarily identifying that the service is down as such, but you might be anticipating it being unable to perform it's intended function.
Any HTTP status result code in the 400 or 500 series is a problem that will prevent you're request from processing. All of these errors derive from System.ServiceModel.CommunicationException so check for that.
I am using the C# UdpClient class to to UDP networking. There is one UdpClient object, bound to a fixed local port, but not to any remote endpoint, because it needs to be able to send/receive to/from multiple different endpoints.
I have two threads: One for sending, one for receiving. Now, when I send data to an endpoint that exists, but doesn't listen on that port, I expect a SocketException. And I do get one. Unfortunately, it is not my Send call that returns the exception, but the Receive call. So on my sending thread, I send data to an "invalid" endpoint, and my receiving thread gets the exception. Unfortunately, at that point, I have of course no idea what endpoint caused that exception to happen.
Storing the endpoint before sending, then accessing that in the receiving thread is just a race condition error waiting to happen.
Unfortunately, the SocketException does not give me the endpoint that caused the error.
Any ideas? Is it somehow possible to make the exception be thrown on the sending thread?
Help is greatly appreciated.
When you send() a UDP packet, it goes out on the wire and effectively disappears. You should not assume that you will get any feedback at all.
Sometimes, if there is no listener at the destination, the destination might be kind enough to send back an ICMP_UNREACH_PORT message. The routers in between then might be kind enough to deliver that message to your operating system. If that happens, it will be long after your original send() call returned. For ICMP_UNREACH_PORT, the OS typically caches it and reports an error the next time yo do a send() to the same destination. Other ICMP messages (you didn't mention which exception you are getting) could affect other calls.
So the bottom line is that there's no telling when, or if, UDP errors will be reported. It depends on a lot of variables. So be prepared to handle exceptions on any call, and be prepared for packets to just disappear without any error reported.
I think this is expected behavior for UDP. A UDP send() is not a blocking operation, so it won't wait for a potential error. (Not to mention the fact that you can't rely on the error messages being reliably received when sending to an active host with a closed port - it could be firewalled, rate-limited or otherwise dropped due to congestion, etc.)
You can connect() the UDP socket to a specific remote endpoint, which would allocate a unique port number and allow the OS to [most likely] distinguish errors from that specific endpoint from any other random host. But again, you should not rely on the ability to handle these errors.
It's too bad there isn't more information in the exception. This seems like an oversight in the way .NET handles UDP sockets. According to the documentation, you need to check the exception's ErrorCode and handle the error appropriately. (which, in your case, could likely mean ignoring the error.)