We're having an issue in production that is really difficult to diagnose. The symptom is, when using SignalR one or more threads on the server periodically hangs. When it happens the other threads seem to continue merrily, unless one of them also experience this problem.
No error is logged, but creating a dump reveals the threads experiencing an ObjectDisposedException at:
System.Net.Sockets.SafeCloseSocket.GetOrAllocateThreadPoolBoundHandle(Boolean)
System.Net.Sockets.Socket.GetOrAllocateThreadPoolBoundHandleSlow()
System.Net.Sockets.SocketAsyncEventArgs.DoOperationSendSingleBuffer(System.Net.Sockets.SafeCloseSocket)
System.Net.Sockets.Socket.SendAsync(System.Net.Sockets.SocketAsyncEventArgs)
If we disable SignalR on the client side, there is no problem.
We've scaled back the websocket functionality to a point where we only create an empty hub, and the problem still persists.
We initialize SignalR with:
services.AddSignalR();
app.UseSignalR(routes =>
{
routes.MapHub<LiveHub>("/live");
});
The contents of LiveHub.cs is:
public class LiveHub : Hub<ILiveHub>
{
public Task Ping()
{
return Clients.Caller.SendMessage("pong");
}
}
We are running on AWS behind an Application Loadbalancer, but only on one server currently.
Anyone else have had this problem?
Related
I'm using RestSharp to communicate with a .Net Core Web API. Both Client and Server are written by me.
I have a suite of Services all inheriting a Base class which contains an async method to perform the request via a RestClient. Here is the method within the base class that creates the RestClient.
private async Task<ServiceResponse> RequestAsync(ServiceRequest request)
{
try
{
var result = await new RestClient(_digiCore.Config.GetApiBaseUrl()).ExecuteTaskAsync(request.Request, request.CancellationTokenSource.Token);
switch (result.StatusCode)
{
case HttpStatusCode.OK:
case HttpStatusCode.Created:
case HttpStatusCode.NoContent:
return new ServiceResponse
{
Code = ServiceResponseCode.Success,
Content = result.Content
};
// User wasn't authenticated for this one - better luck next time!
case HttpStatusCode.Unauthorized:
Logger.LogError($"Unauthorized {request.Method.ToString()}/{request.Path}");
default:
Logger.LogError($"An error occurred {request.Method.ToString()}/{request.Path}");
}
}
catch (Exception e)
{
Logger.LogError($"A Rest Client error occurred {request.Method.ToString()}/{request.Path}");
}
}
My understanding is that the RestClient (unlike HttpClient) is thread safe and OK to create a new instance each time.
When Load testing my application, one I reach a certain point, I find that I occasionally receive the following response back from the API. Refreshing the page might bring back the correct result, or I might get another error. When the Load Test has finished everything goes back to normal. The numbers of the load test are nothing crazy, so its started to get me worried about real-life performance...
Only one usage of each socket address (protocol/network address/port) is normally permitted
I believe this is down to port exhaustion. Both API and Client are running in Azure as App services.
I have tried making the RestClient part of a singleton service, and checked that it is only created once - but this did not alleviate the issue.
I have also looked into setting the DefaultConnectionLimit in the Startup function of my startup class from the default (2) to 12, but I did not see any improvements here either.
I'm really struggling to see how I can improve this...
Found out the reason for this...
It seems there is currently an issue with RestSharp whereby socket connections are not closed down immediately and instead are left in TIME_WAIT state.
This issue is currently logged with the RestSharp guys here ->
https://github.com/restsharp/RestSharp/issues/1322
I have chosen to switch over to using HttpClient - and running the exact same load test I had zero problems.
Here is an screenshot taken from Azure showing the difference between the two load tests.
Was with my app using RestSharp
Was with my app using Http Client
Hope this helps somebody. It's a shame I had to ditch RestSharp as it was a nice little library - unfortunately with an issue like this it's just too risky to run in a Production environment.
As we have moved from NSB5 to NSB6 we also looked into removing NServiceBus.Host and instead use Topshelf. When we did, our service no longer shows that it has stopped when we receive a critical failure.
As an example, when we have trouble to reach the database for any reason I want the service to end and in Services Manager it should indicate not running. Though, it still says running but service is actually stopped. Therefore no recovery is run either.
This was working as we were using NServiceBus.Host.
I was looking in the wrong direction, towards Topshelf. The answer lies in how to configure NServiceBus to take care of critical errors.
EndpointConfiguration.DefineCriticalErrorAction(OnCriticalError);
and
private async Task OnCriticalError(ICriticalErrorContext context)
{
await context.Stop().ConfigureAwait(false);
}
This worked for me.
I am developing a web-api that takes data from client, and saves it for later use. Now i have an external system that needs to know of all events, so i want to setup a notification component in my web-api.
What i do is, after data is saved, i execute a SendNotification(message) method in my new component. Meanwhile i don't want my client to wait or even know that we're sending notifications, so i want to return a 201 Created / 200 OK response as fast as possible to my clients.
Yes this is a fire-and-forget scenario. I want the notification component to handle all exception cases (if notification fails, the client of the api doesn't really care at all).
I have tried using async/await, but this does not work in the web-api, since when the request-thread terminates, the async operation does so aswell.
So i took a look at Task.Run().
My controller looks like so:
public IHttpActionResult PostData([FromBody] Data data) {
_dataService.saveData(data);
//This could fail, and retry strategy takes time.
Task.Run(() => _notificationHandler.SendNotification(new Message(data)));
return CreatedAtRoute<object>(...);
}
And the method in my NotificationHandler
public void SendNotification(Message message) {
//..send stuff to a notification server somewhere, syncronously.
}
I am relatively new in the C# world, and i don't know if there is a more elegant(or proper) way of doing this. Are there any pitfalls with using this method?
It really depends how long. Have you looked into the possibility of QueueBackgroundWorkItem as detailed here. If you want to implement a very fast fire and forget you also might want to consider a queue to pop these messages onto so you can return from the controller immediately. You'd then have to have something which polls the queue and sends out the notifications i.e. Scheduled Task, Windows service etc. IIRC, if IIS recycles during a task, the process is killed whereas with QueueBackgroundWorkItem there is a grace period for which ASP.Net will let the work item finish it's job.
I would take a look on Hangfire. It is fairly easy to setup, it should be able to run within your ASP.NET process and is easy to migrate to a standalone process in case your IIS load suddenly increases.
I experimented with Hangfire a while ago but in standalone mode. It has enough docs and easy to understand API.
I have a REST service in a self hosted ASP.Net WebApi application (Console).
Some clients poll the server in specific intervals to fetch new data. In general all is working fine.
The problem is, that the server stops responding to requests after some random duration (~30mins - 2.5 hours). All client requests start to time out.
The weird thing is, the server doesn't seem to receive the requests anymore as no controller method is invoked anymore). Server didn't throw any exceptions and the console app is still responsive. So I can only suppose there is a problem, before the request reaches the API controller.
In the debugger everything seems fine.
How can I diagnose such an issue?
What else can I try to fix the described behavior?
Notes:
Tested on multiple systems
.Net 4.5.1
Asp.Net WebApi 5.1.2
I have found the issue, the reason this is happening is because of connection leaks. If you are sending requests and aren't closing them correctly, either after the request is finished, or within an exception, the amount of open connections will eventuelly reach it's max value. Either you change the max amount of open connections in the connectionstring or(the prefered way) make sure your code is handling the closing part:
SqlConnection myConnection = new SqlConnection(ConnectionString);
try
{
conn.Open();
someCall (myConnection);
}
finally
{
myConnection.Close();
}
Credit goes to How can I solve a connection pool problem between ASP.NET and SQL Server? Where you can read more about this.
In my case, the issue was caused by never ending tasks. Due a misusage of the ReactiveExtensions Api, I randomly created never ending tasks. It seems, at some point the task scheduler simply couldn't handle them anymore, although I'm not completely sure about that.
Thing learned: It seems, by doing bad things in your app code (too many tasks, SQL connections ...) you can kill the WebApi infrastructure, so that it doesn't handle requests - at any level - anymore.
The project I'm working on is a client-server application with all services written in WCF and the client in WPF. There are cases where the server needs to push information to the client. I initially though about using WCF Duplex Services, but after doing some research online, I figured a lot of people are avoiding it for many reasons.
The next thing I thought about was having the client create a host connection, so that the server could use that to make a service call to the client. The problem however, is that the application is deployed over the internet, so that approach requires configuring the firewall to allow incoming traffic and since most of the users are regular users, that might also require configuring the router to allow port forwarding, which again is a hassle for the user.
My third option is that in the client, spawns a background thread which makes a call to the GetNotifications() method on server. This method on the server side then, blocks until an actual notification is created, then the thread is notified (using an AutoResetEvent object maybe?) and the information gets sent to the client. The idea is something like this:
Client
private void InitializeListener()
{
Task.Factory.StartNew(() =>
{
while (true)
{
var notification = server.GetNotifications();
// Display the notification.
}
}, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
}
Server
public NotificationObject GetNotifications()
{
while (true)
{
notificationEvent.WaitOne();
return someNotificationObject;
}
}
private void NotificationCreated()
{
// Inform the client of this event.
notificationEvent.Set();
}
In this case, NotificationCreated() is a callback method called when the server needs to send information to the client.
What do you think about this approach? Is this scalable at all?
For each client you are going to hold a thread on the server. If you have a few hundred clients and the server wouldn't use the memory anyway, that may be fine. If there can be more clients, or you do not wish to burn 1MB of stack per client, you should make some changes:
Use an async WCF action method. They allow you to unblock the request thread while the method is waiting.
Change the event model to an async once. SemaphoreSlim has async support. You can also use TaskCompletionSource.
That way you can scale up to many connections.