Develop Reference

Different threads try to handle the same TCP socket: can this cause instability?

I'm working on a C# Docker application, where I am creating a microservice in order to handle TCP socket communication. This works, but it seems to be very unstable (some packets pass, some not). I have added a log entry, which might explain something (it's about sending a message over a TCP socket):
Source code (shown on multiple lines, but it's a oneliner):
Debug.WriteLine(
$"T[{System.Threading.Thread.CurrentThread.ManagedThreadId}],
{DateTime.UtcNow}: Handle().
Trying to send [{Message}] to [{ConnectionName}]");
Results:
T[14], 12/14/2022 15:26:08: Handle(). Trying to send [abc] to [XL_Test]
T[19], 12/14/2022 15:26:32: Handle(). Trying to send [abc] to [XL_Test]
As you can see, apparently my application always uses another thread to handle the requests. So, I'm left with a very simple question: is multithreaded programming allowed when working with TCP sockets?
For your information: I have already worked with multithreaded applications on TCP sockets before, where one thread was used for regular checking the connection and another for sending the messages, but here I have one thread which regularly checks the connection, while for sending messages, always another thread gets opened.
Edit: I don't know if this is helpful, but the name of the thread, handling the message, is .Net ThreadPool Worker.
Edit2: this is the way this Send() method is called:
await _mediator.Send(command);
... where _mediator is an IMediator from the MediatR library.
The first comment refers to another StackOverflow post which is using locks, but while trying this, I got a compilation error:
object lockobject = new object();
lock(lockobject)
{
await _mediator.Send(command);
}
The compiler message is CS1996: Cannot await in the body of a lock statement. (Pardon my ignorance, but I'm very new at this)

WCF request returns wrong response

I have a c# application that the client uses wcf to talk to the server. In the background every X seconds the client calls a Ping method to the server (through WCF). The following error has reproduced a couple of times (for different method calls):
System.ServiceModel.ProtocolException: A reply message was received for operation 'MyMethodToServer' with action 'http://tempuri.org/IMyInterface/PingServerResponse'. However, your client code requires action 'http://tempuri.org/IMyInterface/MyMethodToServerResponse'.
MyMethodToServer is not consistent and it falls on different methods.
How can this happen that a request receives a different response?

I think you have a pretty mess problem with async communication, main suggestion (as your question isn't clear very well), is try to identify every request, catch the calls and waiting for them, do asyncronic communication and getting a several work with threading.
As you present it, is a typical architecture problem.
If you present more code, can I suggest some code fixing in my answer and I'll be glad to update my answer.

If this occurs randomly and not you consistently, you might be running in a load-balanced setup, and deployed an update to only one of the servers?

Wild guess: your client uses same connection to do two requests in parallel. So what happens is:
Thread 1 sends request ARequest
Thread 2 sends request BRequest
Server sends reply BReply
Thread 1 receives reply BReply while expecting AReply
If you have request logs on the server, it'll be easy to confirm - you'll likely see two requests coming with short delay from the client host experiencing the issue
I think MaxConcurrentCall and ConcurrencyMode may be relevant here (although I did not touch WCF for a long while)

DownloadFileAsync throw Get TimeOut exception

I have put Flurl in high load using DownloadFileAsync method to download files in private network from one server to another and after several hours the method starts to throw exceptions "Get TimeOut". The only solution to solve that is restart application.
downloadUrl.DownloadFileAsync(Helper.CreateTempFolder()).Result;
I have added second method as failover using HTTPClient and its download files fine after flurl fails, so it is not server problem.
private void DownloadFile(string fileUri, string locationToStoreTo)
{
using (var client = new HttpClient())
using (var response = client.GetAsync(new Uri(fileUri)).Result)
{
response.EnsureSuccessStatusCode();
var stream = response.Content.ReadAsStreamAsync().Result;
using (var fileStream = File.Create(locationToStoreTo))
{
stream.CopyTo(fileStream);
}
}
}
Do you have any idea why Get TimeOut error starts popup on high load using the method?
public static Task<string> DownloadFileAsync(this string url, string localFolderPath, string localFileName = null, int bufferSize = 4096, CancellationToken cancellationToken = default(CancellationToken));
The two download code differ only that Flurl re-use HttpClient instance for all request and my code destroy and create new HttpClient object for every new request. I know that creating and destroying HttpClient is time and resource consuming I rather would use Flurl if it would work.

As others point out, you're trying to use Flurl synchronously by calling .Result. This is not supported, and under highly concurrent workloads you're most likely witnessing deadlocks.
The HttpClient solution is using a new instance for every call, and since instances aren't shared at all it's probably less prone to deadlocks. But it's inviting a whole new problem: port exhaustion.
In short, if you want to continue using Flurl then go ahead and do so, especially since you're getting smart HttpClient reuse "for free". Just use it asynchronously (with async/await) as intended. See the docs for more information and examples.

I can think of two or three possibilities (I'm sure there are others that I can't think of as well)
Server IP address has changed.
You wrote that Flurl reuses a HttpClient. I've never used, or even heard of Flurl, so I have no idea how it works. But an HttpClient re-uses a pool of connections, which is why it's efficient to reuse a single instance and why it's critical to do so in a high-volume microservice application, otherwise you're likely to exhaust all ports, but that gives a different error message, not a time out, so I know you haven't hit that case. However, while it's important to re-use an HttpClient in the short term, HttpClient will cache DNS results, which means it's important to dispose and create new HttpClients periodically. In short-lived processes, you can use a static or singleton instance. But in long running processes, you should create a new instance periodically. If you only use it to access one server, that server's DNS TTL is a good value to use.
So, what might be happening is the server changed IP addresses a few hours after your program started, and because Flurl keep reusing the same HttpClient, it doesn't get the new IP address from the DNS entry. One way to check if this is the problem is write the server's IP address to a log at the beginning of the process and when you encounter the problem, check if the IP address is the same or not.
If this is the problem, you can look into ASP.NET Core 2.1's HttpClientFactory. It's a bit awkward to use outside of ASP.NET, but I did once. It gives you re-use of HttpClients, to avoid the TCP port exhaustion problem of using more than 32k HttpClients in 120 seconds, but also avoid DNS caching issues. My memory is that it creates a new HttpClient every 5 minutes by default.
Reaching the maximum connections per server
ServicepointManager.DefaultConnectionLimit sets the maximum number of HTTP connections that a client will open to a server. If your code tries to use more than this simultaneously, the requests that exceed the limit will wait for an existing HTTP client to finish its request, then it will use the newly available connection. However, in the past when I was looking into this, the HTTP timeout started from when the HttpClient's method was called, not when the HttpClient sends the request to the server over a connection. This means that if your limit is 2 and both are used for longer than the timeout period (for example if downloading 2 large files), other requests to download from the same server will time out, even though no http request was ever sent to the server.
So, depending on your application and server, you may be able to use a higher connection limit, otherwise you need to implement request queuing in your app.
Thread pool exhaustion
Async code is awesome for performance when used correctly in highly concurrent, IO bound workloads. I sometimes think it's a bad idea to use anywhere else because it such huge potential for causing weird problems when used incorrectly. Like Crowcoder wrote in a comment on the question, you shouldn't use .Result, or any code that blocks a running thread, when in an async context. Although the code sample you provided says public void DownloadFile(... , if it's actually public async Task DownloadFile(..., or if DownloadFile is called from an async method, then there's real risk of issues. If DownloadFile is not called from an async method, but is called on the thread pool, there's the same risk of errors.
Understanding async is a huge topic, unfortunately with a lot of misinformation on the internet as well, so I can't possibly cover it in detail here. A key thing to note is that async tasks run on the thread pool. So, if you call ThreadPool.QueueUserWorkItem and block the thread that your code runs on, or if you have async tasks that you block on (for example by calling .Result), what could happen is that you block every thread in the thread pool, and when an HTTP response comes back from the network, the .NET run time has no threads available to complete the task. The problem with this idea is that there are also no threads available to signal the timeout, so I don't believe you're exhausting the thread pool (if you were, I would expect a deadlock), but I don't know how timeouts are implemented. If timeouts/timers use a dedicated thread it could be possible for a cancellation token (the thing that signals a timeout) to be set by the timer's thread, and then any code on a blocking wait for either the HTTP response or the cancellation token could be triggered. But thread pool exhaustion generally causes deadlocks, so if you're getting an error back, it's probably not this.
To check if you're having threadpool exhaustion issues, when your program starts getting the timeout errors, get a memory dump of your app (for example using Task Manager). If you have the Enterprise or Ultimate SKU of Visual Studio, you can open/debug the memory dump in VS. Otherwise you'll need to learn how to use windbg (or find another tool). When debugging the memory dump, check the number of threads. If there's a very large number of threads, that's a hint you might be on the right track. Check where the thread was at the time of the memory dump. If they're all in blocking calls like WaitForObject, or something similar, then there's a real risk you've exhausted the thread pool. I've never debugged an async task deadlock/thread pool exhaustion issue before, so I'm not sure if there's a way to get a list of tasks and see from their runstate if they're likely to be deadlocked or not. If you ever see more tasks in the running state than you have cores on your CPU, you almost certainly have blocking in an async task, however.
In summary, you haven't given us enough details to give you an answer that will work with 100% certainty. You need to keep investigating to understand the problem until you can either solve it yourself, or provide us with more information. I've given you some of the most likely causes, but it could very easily be something else completely.

webapi 2 - how to properly invoke long running method async/in new thread, and return response to client

I am developing a web-api that takes data from client, and saves it for later use. Now i have an external system that needs to know of all events, so i want to setup a notification component in my web-api.
What i do is, after data is saved, i execute a SendNotification(message) method in my new component. Meanwhile i don't want my client to wait or even know that we're sending notifications, so i want to return a 201 Created / 200 OK response as fast as possible to my clients.
Yes this is a fire-and-forget scenario. I want the notification component to handle all exception cases (if notification fails, the client of the api doesn't really care at all).
I have tried using async/await, but this does not work in the web-api, since when the request-thread terminates, the async operation does so aswell.
So i took a look at Task.Run().
My controller looks like so:
public IHttpActionResult PostData([FromBody] Data data) {
_dataService.saveData(data);
//This could fail, and retry strategy takes time.
Task.Run(() => _notificationHandler.SendNotification(new Message(data)));
return CreatedAtRoute<object>(...);
}
And the method in my NotificationHandler
public void SendNotification(Message message) {
//..send stuff to a notification server somewhere, syncronously.
}
I am relatively new in the C# world, and i don't know if there is a more elegant(or proper) way of doing this. Are there any pitfalls with using this method?

It really depends how long. Have you looked into the possibility of QueueBackgroundWorkItem as detailed here. If you want to implement a very fast fire and forget you also might want to consider a queue to pop these messages onto so you can return from the controller immediately. You'd then have to have something which polls the queue and sends out the notifications i.e. Scheduled Task, Windows service etc. IIRC, if IIS recycles during a task, the process is killed whereas with QueueBackgroundWorkItem there is a grace period for which ASP.Net will let the work item finish it's job.

I would take a look on Hangfire. It is fairly easy to setup, it should be able to run within your ASP.NET process and is easy to migrate to a standalone process in case your IIS load suddenly increases.
I experimented with Hangfire a while ago but in standalone mode. It has enough docs and easy to understand API.

Run task on background but return response to client in ASP MVC web application

I'm trying to implement a functionality where there is a stored procedure on SQL Server that has to be called from ASP MVC application and processed on the background (it might take long since it calls another stored procedure remotely to process an excel file stored on a server). But the response of the last HTTP request should be returned back to client so the UI will not be hanging waiting for processing.
I have tried so many different ways but UI is still not responding right away.
I tried BackgroundWorker but it's not allowing the main thread to response back to client until its done processing,
also I tried:
Thread.QueueUserWorkItem(delegate { //method which performs stored procedure calls//
});
It still not returning response and HttpContext.Current not available in background thread.
Maybe there is a way to start background processing, pause it for letting main thread to return response to browser and then resume background thread to make all processing with stored procedure calls?
Am I missing something?
Could someone please give an idea how I can solve this problem? Would be much appreciated.

What I ended up with and it works fine in my case. I'm not sure about efficiency, but it perfectly works. So, the code which makes calls to a stored procedure I put on a separate thread, so that the main thread is finished while the processing off background calls is happening on a separate thread and finishes successfully after some period of time. At the same time UI is available so user can make another request which will also be processed the same way. I tested three requests. One after another overlapping, meaning that while first request was being processed on the background I submitted another one and yet another one. UI was responding immediately and all the work was done.
// ...main thread is working here
//put a call to stored procedure on a separate thread
Thread t = new Thread(()=> {
//call stored procedure which will run longer time since it calls another remote stored procedure and
//waits until it's done processing
});
t.Start();
// ...main thread continue to work here and finishes the request so it looks for user as the response is coming right away, all other stuff is being processed on that new thread so user even doesn't suspect

I shall quote Stephan Clearys great article:
When you use async on the server side (e.g., with ApiController), then you can treat each web request as an asynchronous operation. But when you yield, you only yield to the web server thread pool, not to the client. HTTP only allows a single response, so the response can only be sent when the request is fully complete.
Basically, this doesn't adhere to the HTTP protocol, where each request has only one response.
This can be achieved using multiple calls to the ASP.NET service, where a request returns a unique ID immediately, which the client can query multiple times for progress. You may look into SignalR for help with such an implementation:
What is SignalR and "real-time web" functionality? It's the ability to have your server-side code push content to the connected clients as it happens, in real-time.

There is part1 and part2 article by Dino Esposito outlines a way to achieve your polling using a client side timer and controller actions. You would basically serialize access to a progress worker controller method that returns task status and completion data. However, it may be a little chatty if you are only going to be performing one or two long running processes.

If the response to the client does not depend on the result of the background process (i.e. run the process in the background and don't have the UI waiting for it), then you could use Revalee (an open-source tool) to perform the background task.
The UI will request this route...
public class ForegroundController : Controller
{
public ActionResult InitiateBackgroundTask()
{
// The absolute URL that will be requested on the callback.
var callbackUri = new Uri("http://localhost/BackgroundTask/Callback");
// The information that will be needed to initiate the background task.
object state = "Any object";
this.CallbackNowAsync(callbackUri, state)
// ... your controller will now return a response to the browser
return View();
}
}
Background work will be executed in this controller...
public class BackgroundTaskController : Controller
{
[AllowAnonymous]
[HttpPost]
[CallbackAction]
public ActionResult Callback(Guid callbackId, object state)
{
// Perform the background work
// ... insert your background task code here ...
// Return a status code back to the Revalee Service.
return new HttpStatusCodeResult(HttpStatusCode.OK);
}
}
Revalee Project Site