Develop Reference

Net 6 ConsoleApp multiple BlockingCollection<T> huge CPU consumption

I have a Net 6 Console app where I use several BlockingCollections to process files that are dropped in a folder. I watch the folder using Net's FileWatcher().
In the Created event, I use a Channel to handle the processing, which is done in two phases, and after each phase the result item is moved to a BlockingCollection, that will then be consumed by the next phase.
Program.cs
public static async Task Main(string[] args)
{
BlockingCollection<FileMetadata> _fileMetaDataQueue = new BlockingCollection<FileMetadata>()
var channel = Channel.CreateUnbounded<FileSystemEventArgs>();
// Start a task to monitor the channel and process notifications
var notificationProcessor = Task.Run(() => ProcessNotifications(channel, _fileMetaDataQueue));
Task fileCopyingTask = Task.Run(() => fileCopyThread.Start()); //injected using DI
Task processMovedFile = Task.Run(() => ProcessDestinationThread.Start()); //injected using DI
Task retryOnErrorTask = Task.Run(() => RetryOnErrorThread.Start()); //injected using DI
using var watcher = new FileSystemWatcher(sourceFolder); //C:\temp
// other fw related config
watcher.Created += (sender, e) => channel.Writer.WriteAsync(e);
}
private async Task ProcessNotifications(Channel<FileSystemEventArgs> channel, BlockingCollection<FileMetadata> queue)
{
await foreach (var e in channel.Reader.ReadAllAsync())
{
Thread.Sleep(300); // So the file is released after it is dropped
try
{
// Process the file and add its name and extension to the queue
FileMetaData fileInfo = ExtractFileMetadata(e.FullPath); //processing method
queue.Add(fileInfo);
}
try
{
// logging etc
}
}
}
The BlockingCollection queue is then consumed in the FileCopyThread class, with the Start() method exposed (and called)
FileCopyThread.cs
BlockingCollection<FileMetadata> resultQueue = new();
BlockingCollection<FileMetadata> retryQueue = new();
public async Task Start()
{
await Task.Run(() => {
ProcessQueue();
});
}
private void ProcessQueue()
{
// Since IsCompleted is never set, it will always run
while (!fileMetadataQueue.IsCompleted)
{
// Try to remove an item from the queue
if (fileMetadataQueue.TryTake(out FileMetadata result))
{
// Copy the file to a new location
var newFileLocation = processorOps.MoveFile(result); // move file to other path
// Add the new file location to the result queue
if (newFileLocation != String.Empty)
{
result.newFileLocation = newFileLocation;
resultQueue.Add(result);
}
else {
retryQueue.Add(result);
}
}
}
}
The ProcessDestinationThread and RetryOnErrorThread work in exactly the same way, but do some different processing, and consume the resultQueue and the retryQueue, respectively.
Now when I run this app, it works fine, everything gets processed as expected, but my CPU and power usage is between 85% and 95%, which is huge, IMO, and does so even when it is not processing anything, just sitting idle. I figured this is because all the infinite loops, but how can I remedy this?
Birds eye view: What I would like is that in case the filewatcher.created event is not firing (ie no files are dropped) then the all the queues after it can be running in idle, so to speak. No need for constant checking, then.
I thought about calling CompleteAdding() on the BlockingCollection<T>, but it seems that I cannot reverse that. And the app is supposed to run indefinitely: So if the drop folder is empty, it might be receiving new files at any time.
Is there a way how I can reduce the CPU usage of my application?
Ps. I am aware that this code is not a fully working example. The real code is far more complex than this, and I had to remove a lot of stuff that is distracting. If you think any pieces of relevant code are missing, I can provide them. I hope this code will at least make clear what I am trying to achieve.

private void ProcessQueue()
{
while (!fileMetadataQueue.IsCompleted)
{
if (fileMetadataQueue.TryTake(out FileMetadata result))
{
//...
}
}
}
This pattern for consuming a BlockingCollection<T> is incorrect. It causes a tight loop that burns unproductively a CPU core. The correct pattern is to use the GetConsumingEnumerable method:
private void ProcessQueue()
{
foreach (FileMetadata result in fileMetadataQueue.GetConsumingEnumerable())
{
//...
}
}

CPU is maxed when running a while loop inside a Long Running task

I am running a constant poll to a messaging server. When a message arrives on the server I grab the message and process it. Unfortunately this is using 80-100% CPU for a simple task. UPDATE:I have reduced it to the while loop itself. The while loop inside 10 tasks are causing the CPU to max out at 100% easy.
int i = 0;
while(true){
//At the start of every minute
if (DateTime.Now.Seconds == 0)
{
i++;
}
}
Is there a way that I can throttle the loop or better yet a better way that I can write this code so that it does not use 100% CPU? I have tried to add a Task.Delay of 1 second but that doesn't help much at all.
Any help or advice you can provide I would greatly appreciate it.
// First I poll 10 different locations for messages and each task polls its own queue
foreach(Queue queue in queueList)
{
task.Add(Task.Run(() => PollIndividualQueue(queue));
}
t= Task.WhenAll(task.ToArray()).WithAggregatedExceptions();
t.Wait();
//Then in the PollIndividualQueue method I implement the while loop that constantly polls a message queue for the next hour
private async Task<string> PollIndividualQueue(Queue queue)
{
var cancellationToken = new CancellationTokenSource(Timespan.FromMinutes(60)).Token;
while(!cancellationToken.IsCancellationRequested)
{
//Poll the queue and if there is a message grab it and process it
if(!await GetMessage())
{
//I call a stored procedure that inserts this message into the database.
using(var conn = new SqlConnection(...)
{
using(var cmd = new SqlCommand(MyStoredProc, conn)
{
cmd.CommandType = Command.StoredProcedure;
cmd.Parameters.Add(new SQlParameter(...InputMessage));
await conn.OpenAsync();
await cmd.ExecuteNonQueryAsync();
}
}
}
else
{
await Task.Delay(1000);
}
}
}
private Task<bool> getMessage(Queue queue)
{
object myLock = new object();
try
{
Monitor.Enter(myLock);
queue.Get(message);
}
catch(MQException ex)
{
if(ex.ReasonCode == 2033)
{
return false;
}
}
finally
{
Monitor.Exit(myLock);
}
}
EDIT: Thank you all for the comments but it seems to be getting off track to the actual question. The question is there a way to use a percentage of the total CPU?
I can literally take all of this code out of the While loop and the CPU is still at 100%. The while loop is what is driving the CPU load it seems.

Ok. Clearly, you are a newbie, so I'll attempt to be kind with my comments.
Your while loop at the top of your questions is extremely bad programming. You wonder why it is maxing out your CPU? If you are running on an Intel i9-9900K then that loop will be executed many billions of times per second. I'm not kidding!!
If you want to do something once a second then you need to do a simple time calculation then sleep. To quiet the CPU down, your code should be:
int i = 0;
while (true) {
//At the start of every minute
Thread.Sleep((int)(60000 - ((DateTime.Now.Ticks / TimeSpan.TicksPerMillisecond) % 60000)));
i++;
}
I really don't understand you are using locks in the getMessage method. The Queue's Get method blocks when it attempts to retrieve a message.
If you are going to use threading then why don't you have each thread separately connect to the queue manager as there is no reason to share a connection between threads.

How to cancel SendPingAsync

I have this code to check if Server is available or not:
public static async Task<bool> PingServer()
{
System.Net.NetworkInformation.Ping p1 = new System.Net.NetworkInformation.Ping();
System.Net.NetworkInformation.PingReply PR = await p1.SendPingAsync("pc2");
// check when the ping is not success
if (PR.Status != System.Net.NetworkInformation.IPStatus.Success)
{
return false;
}
return true;
}
Now, my problem is that SendPingAsync does not support CancellationTokenSource so how can I cancel the last ping to perform a new one to prevent a lot of pings?
When the server is lost it takes seconds to PingServer() return a false value.
This is how I call PingServer():
var task = await PingServer();
if (task == true)
{
//do some stuff like retrieve a table data from SQL server.
}
Why I use ping? because when I want to get a table from SQL server if the server is disconnected my app enter a breakpoint.
public async static Task<System.Data.Linq.Table<Equipment>> GetEquipmentTable()
{
try
{
DataClassesDataContext dc = new DataClassesDataContext();
// note : i tried so many ways to get table synchronize but it still freeze my UI !
return await Task.Run(() => dc.GetTable<Equipment>());
}
catch
{
return null;
}
}
EDIT : I used ping to reduce chance of entering my app in brake mode by getting a table from database. is there any other way to prevent break mode ? is it best way to ping server before calling dc.GetTable() ?

Have you considered using the timeout parameter?
From the documentation:
This overload allows you to specify a time-out value for the
operation.
If that doesn't suffice and your problem is that the ping call is blocking, you could perform it on a background thread providing this is tightly controlled.

Consider
public static async Task<bool> PingServer() {
using (var ping = new System.Net.NetworkInformation.Ping()) {
try {
var maxDelay = TimeSpan.FromSeconds(2); //Adjust as needed
var tokenSource = new CancellationTokenSource(maxDelay);
System.Net.NetworkInformation.PingReply PR = await Task.Run(() =>
ping.SendPingAsync("pc2"), tokenSource.Token);
// check when the ping is not success
if (PR.Status != System.Net.NetworkInformation.IPStatus.Success) {
return false;
}
return true;
} catch {
return false;
}
}
}
Where the ping is done with a cancellation token using Task.Run; If the ping result returns before the allotted time then all is well.
wrapped the component in a using to dispose of it when exiting function.

The system you depend on might fail, or the connection can go down right after your ping, when you're executing your code.
This is why a retry strategy is a much more robust approach then simply pinging a system before calling it.
Here is how you can implement a retry Cleanest way to write retry logic?
I would go with a retry approach, but if you still want to stay with your design you could
Schedule a periodic task to ping the system in question Is there a Task based replacement for System.Threading.Timer?
Make sure you schedule this periodic task in one central place (application startup or the like)
Invoke your PingServer from this periodic task and make sure you call Ping.SendAsync with PingOptions.Timeout being set, see this overload
From your PingServer set some kind of shared state, it could be a static variable, or an implementation of the Registry pattern
Make sure your shared state is thread-safe
The rest of your code can call this shared state, to find out if a system is online and available
As you can see, this approach is more complex, but you will prevent "lots of pings" to the system you depend on

I need a thread keeping running forever. which method is best?

I need to a background long running thread from the asp.net application.
BTW,the method fetches something from external data source period and maybe exception happens, I provide some ways to fullfill this task, please advice which way is best, please advice if there is better way.
Way1=Looping when exception happens.
static void LongRunningMethod()
{
do
{
try
{
//fetch something from external period and maybe exception happens.
Thread.sleep(100000);
}
catch(Exception ex)
{
//log exception..
}
} while (true);
}
Way2=Running the following method period by something like timer, and open a new thread when exception happens.
static void LongRunningMethod()
{
try
{
//fetch something from external period and maybe exception happens.
Thread.sleep(100000);
}
catch(Exception ex)
{
//log exception..
Thread T2 = new Thread(LongRunningMethod);
T2.Start();
}
}
Way3=Call itself when exception happens.
static void LongRunningMethod()
{
try
{
//fetch something from external period and maybe exception happens.
Thread.sleep(100000);
}
catch(Exception ex)
{
//log exception..
LongRunningMethod();
}
}

I will use none of the three. In fact, I seldom build interval tasks base on ASP.NET because:
1. The start of the task is out of control
Based on ASP.NET the task has to be started/registered on Application_Start method, which is triggered by the first request, that means your tasks is started when the first request comes. Theoretically it can be long time after your application is deployed.
2. The end of the task is out of control
The web servers (thinking IIS) can be configured as recycle-on-demand (AFAIK this is the default behavior). That is, your thread may be killed when executing. Under most circumstances you need to deal with the persistence of your tasks and states, and add retrying codes and etc... That's a nightmare!
For me it's a better idea to use a windows service, or the task scheduler function of windows, the latter is even easier because you don't need to write a timer or interval call codes. It's more stable, less code, and friendly logs in the event viewer! That's really make everything easier.

I like to use System.ComponentModel.BackgroundWorker:
static BackgroundWorker looper = new BackgroundWorker();
static bool isRunning = true;//you can set this to false when closing
public static void initialize(){
looper.DoWork+= doLoop;
looper.RunRunWorkerAsync();
}
private static void doLoop(object sender, DoWorkEventArgs e){
while(isRunning){
//do looping code
System.Threading.Thread.Sleep(5000);
if(!isRunning)
break;
}
}
PS - sorry for weird spacing - I edited this directly in the code window.

Proxy or FiddlerCore

I'm trying to write a program that catches the HTTP get requests.
I have found Fiddler-core a genius library that should do exactly what I want.
The thing is, I'm trying to execute a big piece code-work inside the void FiddlerApplication_BeforeRequest(Session oSession) and it seems to block all the request and damage my surfing speed a great deal.
I have tried to use threads/tasks with no avail.
What am I doing wrong?
This is my code:
public event RequestCapture RequestCaptured;
private CancellationTokenSource cancelTokenSource = new CancellationTokenSource();
//...stat public function:
public void RunWatch() {
Fiddler.FiddlerApplication.BeforeRequest += FiddlerApplication_BeforeRequest;
Fiddler.FiddlerApplication.Startup(0, FiddlerCoreStartupFlags.Default);
}
void FiddlerApplication_BeforeRequest(Session oSession)
{
if (RequestCaptured != null)
{
CancellationToken ct = cancelTokenSource.Token;
Task.Factory.StartNew(() =>RequestCaptured(oSession.fullUrl), ct);
//Handle the event in a new thread, so the Listener will continue to listen
}
}
//close public function:
public void Close() {
try
{
FiddlerApplication.Shutdown();
cancelTokenSource.Cancel();
}
catch { }
}
now i have i different class that do that:
public Form1()
{
Listiner = new HttpWatcher.Listner();
Listiner.RequestCaptured += RequestCaptured;
Listiner.RunWatch();
}
void RequestCaptured(string url)
{
System.Threading.Thread.Sleep(10000);
}
edit
The question is: Is there a better way using fiddler-core? or am i to build a simple proxy for that? using something else? Thanks!
edit2
I have edited the code, so it would fill the missing parts.

Just to be clear here, FiddlerCore processes each Session on a threadpool thread. If you need blocking behavior, there's no need to spin up an additional thread or anything like that. If you don't need to process things in a blocking manner, then feel free to queue the data on a background queue and use tasks or another asynchronous mechanism to perform processing.
You should explain exactly what you mean when you say damage my surfing speed a great deal, and whether or not you see different behavior when using Fiddler rather than your application.