Worker Threads Blocking When ConcurrentQueue has too many items - c#

This is a weird one, I have a Thread[] of worker threads which each process items in a ConcurrentQueue<string> until the queue is empty, at which point the rest of the program continues.
This works until about ~1500 items at which point all threads stay blocked in the WaitSleepJoin state and never process any of the items in the queue.
I've tried stepping through my code and it appears that the threads are still created, still started and are alive but get blocked immediately and never run their relevant function.
I'm completely flummoxed so any help would be appreciated!
The relevant sections of code are below:
Main Thread Segment:
ConcurrentQueue<string> convertionQueue = new ConcurrentQueue<string>();
List<Thread> converterThreads = new List<Thread>();
Directory.GetFiles(_folderOne, "*.fdf", SearchOption.AllDirectories).ToList().ForEach(file => convertionQueue.Enqueue(file));
Directory.GetFiles(_folderTwo, "*.fdf", SearchOption.AllDirectories).ToList().ForEach(file => convertionQueue.Enqueue(file));
int filesDone = 0;
int totalFiles = convertionQueue.Count;
progressBar.Maximum = totalFiles;
panel1.Visible = true;
for (int i = 0; i < Environment.ProcessorCount; i++)
{
converterThreads.Add(new Thread(() => ConvThreadWorker(convertionQueue, ref filesDone)));
}
converterThreads.ForEach(thread => thread.Start());
DateTime lastTick = DateTime.Now;
int lastFilesDone = 0;
int[] valuesSpeed = { 1, 1, 1, 1, 1 };
int[] valuesTime = { 1, 1, 1, 1, 1 };
int counter = 0;
while (converterThreads.Any(thread => thread.IsAlive))
{
TimeSpan t = DateTime.Now - lastTick;
int deltaFiles = filesDone - lastFilesDone;
double speed = (float)t.TotalMilliseconds <= 0.0 ? 0.0 : deltaFiles / (float)t.TotalMilliseconds;
double tMinus = speed <= 0 ? 0.0 : (totalFiles - filesDone) / speed;
int currentSpeed = (int)(speed * 1000);
int currentTime = (int)(tMinus / 1000);
valuesSpeed[counter] = currentSpeed;
valuesTime[counter] = currentTime;
lblFilesLeft.Text = string.Format("{0}/{1}", filesDone, totalFiles);
lblSpeed.Text = valuesSpeed.Sum() / 5 + " /s";
lblTime.Text = valuesTime.Sum() / 5 + " s";
lblFilesLeft.Update();
lblSpeed.Update();
lblTime.Update();
progressBar.Value = filesDone;
progressBar.Update();
lastTick = DateTime.Now;
lastFilesDone = filesDone;
counter = ++counter % 5;
Thread.Sleep(500);
}
Worker Function:
private void ConvThreadWorker(ConcurrentQueue<string> queue, ref int fileCounter)
{
while (!queue.IsEmpty)
{
string file;
if (queue.TryDequeue(out file))
{
ConvToG(file);
fileCounter++;
}
}
}
Convertion Function:
private void ConvToG(string file)
{
MessageBox.Show("Entering Convertion Function");
if (!_fileCreationDictionary.ContainsKey(file))
{
DateTime lastTimeModified = File.GetLastWriteTime(file);
_fileCreationDictionary.AddOrUpdate(file, lastTimeModified, (key,oldvalue)=>lastTimeModified);
}
ProcessStartInfo procStart = new ProcessStartInfo
{
Arguments = file,
UseShellExecute = true,
FileName = Fdfg,
WindowStyle = ProcessWindowStyle.Hidden
};
Process process = new Process {StartInfo = procStart};
MessageBox.Show("Starting convertion process");
process.Start();
process.WaitForExit();
MessageBox.Show("Finished");
}
The confusing part appears to be how this all revolves around the number of items in the queue, yet there appears to be no overflow.
UPDATE: Adding the mbox's shows that it freezes on the process.Start() section of code, with no errors and will not proceed past that point.
UPDATE 2: If UseShellExecute = false the code works. Which is very confusing to say the least.

I have done something similar with threads spawning processes to collate data. I had issues around the actual process starting and hanging. What I did to get my program working was something like this:
using (Process process = Process.Start(startInfo)) {
if(process.WaitForExit(timeOutMilliseconds)) {
MessageBox.Show("Process exited ok");
//...snip
} else {
MessageBox.Show("Process did not exit in time!");
//...snip
process.Kill();
}
}
There is a bit more going on in the background, regarding limiting the number of running process, etc, but I found that occasionally, for an unknown reason, that I would see several process in the task manager just hanging around forever.
Hope that helps?

Related

Get the realtime CPU speed in C#?

The existing question suggests CurrentClockSpeed, but in my system, it just returns the same value as MaxClockSpeed. The code below prints out the same two values over and over again.
Task.Run(() =>
{
ManagementObject Mo = new ManagementObject("Win32_Processor.DeviceID='CPU0'");
while (true)
{
Debug.WriteLine("Max=" + Mo["MaxClockSpeed"] + ", Current=" + Mo["CurrentClockSpeed"]);
System.Threading.Thread.Sleep(1000);
}
Mo.Dispose(); //return and such later in the code
});
But all other applications like Task Manager, CPU-Z, Hardware Info, etc, show variable clock speed. That is, if I run a process that uses 100% of the CPU, the speed goes up, and if I terminate that process, it goes down. How can I get THAT value?
I mean, for example, the value in the "Speed" section of the screenshot I found in Google Search. Not the "Maximum speed" value that never changes.
If you mean CPU current usage processes
use this function in seperate thread :
private void get_cpuUsage()
{
try
{
string processname = System.Reflection.Assembly.GetExecutingAssembly().GetName().Name;
var perfCounter = new PerformanceCounter("Process", "% Processor Time", processname);
int coreCount = 0;
foreach (var item in new System.Management.ManagementObjectSearcher("Select * from Win32_Processor").Get())
{
coreCount += int.Parse(item["NumberOfCores"].ToString());
}
while (true)
{
Thread.Sleep(500);
double perfVal = perfCounter.NextValue() / Environment.ProcessorCount;
int cpu = (int)Math.Round(perfVal, 0);// /
double cpuvalue = Math.Round(perfVal, 1);
Invoke((MethodInvoker)delegate
{
cpu_bar.Text = cpuvalue.ToString(); // diaplay current % processes
});
}
}
catch(Exception ex)
{
messagebox.show(ex.message);
}
}

Trying to implement timer and perform tasks every x seconds but the tasks seem to get delayed

I'm writing an app where a user specifies a length of time, length of an interval and a length of time in between intervals. I want to have a timer label showing the user the total time but then I also want to have a label showing the work status (recording if in the interval, break if between interval time and break end).
Heres an Example: Total time = 2 min, Interval = 20 seconds, Break = 10 seconds
In this example there will be 4 intervals. So from 0:00-0:19 I want to display "Recording" and then from 0:20-0:29 I want to display break and then from 0:30-0:49 I display "Recording" and 0:50-0:59 I display "Break" and so on. All while the timer counts the time.
So I thought this would be pretty straightforward but what seems to happen is the timer increments properly but after the 1st interval the label doesnt switch from break to recording until 0:31 or 0:32 so it looks a little delayed.
Here is the code I am using currently (Note obs is an object Im passing in that has data from user input).
int TotalInterval = obs.Interval + obs.Break;
int WorkingInterval = obs.Interval;
int NumberOfIntervals = (obs.Duration*60) / TotalInterval;
DateTime ObservationEnd = obs.DateCreated.AddMinutes(obs.Duration);
Timer.Text = "Starting Timer";
int minutes = 0;
int seconds = 0;
int InIntervalCounter = 0;
Device.StartTimer(TimeSpan.FromSeconds(1), () =>
{
// called every 1 second
Timer.Text = "Started";
if (ObservationEnd < DateTime.UtcNow)
{
Timer.Text = "Time Over";
Results.IsVisible = true;
return false;
}
else
{
seconds++;
InIntervalCounter++;
if (InIntervalCounter > WorkingInterval)
IntervalOrBreak.Text = "Break";
if (InIntervalCounter > TotalInterval)
{
IntervalOrBreak.Text = "Recording";
InIntervalCounter = 0;
}
Timer.Text = "Time: " + minutes + ":" + seconds.ToString("D2");
return true;
}
});
I'm pretty new to app development/xamarin so any help is greatly appreciated.
Try using simple Threads with Thead.sleep() like this:
final long delay_millis = 100;
Thread thread_something;
thread_something = new Thread(new Runnable() {
#Override
public void run() {
while (true) {
try {
long start_time = SystemClock.elapsedRealtime();
// Do some task
long time_need_for_delay = (start_time + delay_millis) - SystemClock.elapsedRealtime();
if(time_need_for_delay > 0)
thread_something.sleep(time_need_for_delay);
} catch (Exception e) { }
}
}
});
thread_something.start();
after the 1st interval the label doesnt switch from break to recording
until 0:31 or 0:32 so it looks a little delayed.
If you want to display break from 0:20-0:29 and display "Recording" from 0:30-0:49, I think the if statement should change to InIntervalCounter >= WorkingInterval and InIntervalCounter >= TotalInterval, InIntervalCounter > WorkingInterval may cause the 1 second delay.

Thread start passed integer changes itself beyond what it should be [duplicate]

This question already has an answer here:
Thread alters passed Int, if start() is called separately
(1 answer)
Closed 7 years ago.
I have no idea what is going on in this. I'm trying to test thread safety of a class by spawning 100 threads to access it constantly, but it seems my anonymous method parameters are changing themselves to values they should never be and I'm confused as to why. As I have no idea what's going on, I'll just post all the functions involved in testing. Somehow I'm ending up with "Thread 98" getting the parameter "num = 100"... That shouldn't be possible and I have no idea what kind of hokey pokey is going on that is changing the integer. (in method "ThreadWriting(int num)" you'll see the point where I check for "num" to equal 100, where I put a break point to catch the offending thread in the act, and it breaks every time. Otherwise it throws an "IndexOutofRangeException" on the array "counts". I'm just trying to see if my threads are generally getting equal access to the class they're all trying to use at once.
public delegate void TempDel();
public TempDel InvokeTest;
public void TRTest3(Form1 sender)
{
InvokeTest = new TempDel(UpdateInvoke);
Thread t = new Thread(() => ConsoleUpdateTest(sender));
t.IsBackground = true;
t.Start();
POConsole.Instance.MaxLines = 20;
for(int i = 0; i < 100; i++)
{
Thread t2 = new Thread(() => ThreadWriting(i));
t2.IsBackground = true;
t2.Name = String.Format("Thread {0}", i);
t2.Start();
}
}
public ulong[] counts = new ulong[100];
public void ThreadWriting(int num)
{
if(num == 100)
{
bool stop = true;
}
while (true)
{
POConsole.Instance.WriteLine("Hello from Thread " + num);
counts[num]++;
}
}
public void ConsoleUpdateTest(Form1 sender)
{
while(true)
{
sender.Invoke(InvokeTest);
Thread.Sleep(5);
}
}
public void UpdateInvoke()
{
QuickTestBox.Text = POConsole.Instance.FullFeed;
}
All my threads are named, as you can see, and none of them receives the name "Thread 100" so I have no idea how one of the other threads could get passed a parameter of 100 or the parameter could be corrupted in some way.
Apparently my thread-safety checking isn't thread safe in some way?
This is a simple closure issue, you should not be using the for loop counter as a threading parameter issue, issue happens out here, for loop and thread execution do not run at same speed, so value of i can change for multiple threads:
for(int i = 0; i < 100; i++)
{
Thread t2 = new Thread(() => ThreadWriting(i));
t2.IsBackground = true;
t2.Name = String.Format("Thread {0}", i);
t2.Start();
}
Use following modification, create a local variable from loop counter
for(int i = 0; i < 100; i++)
{
int j = i;
Thread t2 = new Thread(() => ThreadWriting(j));
t2.IsBackground = true;
t2.Name = String.Format("Thread {0}", j);
t2.Start();
}

Infinite loop in task - cpu usage

I'm making service for watch on some controller data and if it changing then I write it to DB. Seems simple. previosly I realized the same with Delphi, but now I am on C# (.Net 4.5). Now service works good with 100 tasks, but eats about 7-8% of CPU time. My Delphi service eats about 0%.
How can I reduce time which service eat from CPU?
P.S.: each task has own nstance of class to connect and insert into DB and work with local copy of data.
int TagCnt = DataCtrl.TagList.Count;
stopExec = false;
if (TagCnt != 0)
{
tasks = new Task[TagCnt];
for (int i = 0; i <= TagCnt - 1; i++)
{
int TempID = i;
tasks[TempID] = Task.Run(async () => // make threads for parallel read-write tasks // async
{
Random rand = new Random();
TimeSpan delay = TimeSpan.FromMilliseconds(rand.Next(1000, 1500))
try
{
while (!stopExec)
{
cToken.ThrowIfCancellationRequested();
//do basic job here
await Task.Delay(delay, cToken);
}//while end
}
catch (...)
{
...
}
}, cToken);
}
Recently I've been facing a similar conundrum and managed to solve the erratic CPU usage by using a set of dedicated long-running tasks to carry out the asynchronous work in my app like so:
Dim NumThreads As Integer = 10
Dim CanTokSrc As New CancellationTokenSource
Dim LongRunningTasks As Task() = New Task(NumThreads) {}
Dim i As Integer
Do Until i = LongRunningTasks.Count
LongRunningTasks(i) = Task.Factory.StartNew(Sub()
Do Until CanTokSrc.IsCancellationRequested
'DO WORK HERE
Loop
End Sub, CanTokSrc.Token, TaskCreationOptions.LongRunning)
i = i + 1
Loop
This image shows the difference it made in CPU usage for the same workload (shown after 9am).
So I think bypassing the thread pool by using dedicated/ long running tasks like above could improve CPU utilization in some cases. It certainly did in mine :-)
I moved to timer instructions because it's a windows service. Every event on timer load is about 7-10% and between is 0%. I tried to apply tasks, ThreadSchedule - they seems more heavy.
private void OnReadTimer(object source, ElapsedEventArgs e) //check states on timer
{
int TagCnt = DataCtrl.TagList.Count;
po.MaxDegreeOfParallelism = DataCtrl.TagList.Count;
// string ss = "tags=" + TagCnt;
//int TempID;
Random rand = new Random();
try
{
if (TagCnt != 0)
{
ParallelLoopResult loopResult = Parallel.For(0, TagCnt - 1, po, (i, loopState) =>
{
po.CancellationToken.ThrowIfCancellationRequested();
int TempID = i;
Thread.Sleep(rand.Next(100, 200));
int ID = 0;
bool State = false;
long WT = 0;
int ParID = 0;
bool Save = false;
ReadStates(TempID, out ID, out State, out WT, out ParID, out Save);
lock (locker)
{
if (Save) WriteState(ID, State, WT, ParID);
}
});
}
}
catch (TaskCanceledException)
{
}
catch (System.NullReferenceException eNullRef)
{
AddLog("Error:" + eNullRef);
}
catch (System.ArgumentOutOfRangeException e0)
{
AddLog("Error:" + e0);
}
catch (Exception e1)
{
//AddLog("Error while processing data: " + e1);
}
}
I moved to basic threads with infinite loops inside. It gets endless threads for my needs. No heavy recreating/restarting and so on. Now it works nice like Delphi service, but more comfortable job with data and DB. I starts threads with this procedure from lambda new thread()=>:
void RWDeviceState(int i)
{
try
{
int TempID = i;
long StartTime;
long NextTime;
long Period = 3000;
int ID = 0;
bool State = false;
long WT = 0;
int ParID = 0;
bool Save = false;
while (ExecutionAllowed)
{
Save = false;
ReadStates(TempID, out ID, out State, out WT, out ParID, out Save);
lock (locker)
{
if (Save) WriteState(ID, State, WT, ParID);
}
StartTime = DateTime.Now.Ticks / TimeSpan.TicksPerMillisecond;
NextTime = StartTime + Period;
while (DateTime.Now.Ticks / TimeSpan.TicksPerMillisecond < NextTime && ExecutionAllowed)
{
Thread.Sleep(40);
}
}
There are two particular techniques that will help reduce CPU usage in long loop waits. One, is to use the threading sleep method. This is good for example in standalone applications, less in windows services.
In a service, for the second, you should be using timers. These fire at regular intervals, so in between the intervals the CPU is not solicited.

How to use multi threading in a For loop

I want to achieve the below requirement; please suggest some solution.
string[] filenames = Directory.GetFiles("C:\Temp"); //10 files
for (int i = 0; i < filenames.count; i++)
{
ProcessFile(filenames[i]); //it takes time to execute
}
I wanted to implement multi-threading. e.g There are 10 files. I wanted to process 3 files at a time (configurable, say maxthreadcount). So 3 files will be processed in 3 threads from the for loop and if any thread completes the execution, it should pick the next item from the for loop. Also wanted to ensure all the files are processed before it exits the for loop.
Please suggest best approach.
Try
Parallel.For(0, filenames.Length, i => {
ProcessFile(filenames[i]);
});
MSDN
It's only available since .Net 4. Hope that acceptable.
This will do the job in .net 2.0:
class Program
{
static int workingCounter = 0;
static int workingLimit = 10;
static int processedCounter = 0;
static void Main(string[] args)
{
string[] files = Directory.GetFiles("C:\\Temp");
int checkCount = files.Length;
foreach (string file in files)
{
//wait for free limit...
while (workingCounter >= workingLimit)
{
Thread.Sleep(100);
}
workingCounter += 1;
ParameterizedThreadStart pts = new ParameterizedThreadStart(ProcessFile);
Thread th = new Thread(pts);
th.Start(file);
}
//wait for all threads to complete...
while (processedCounter< checkCount)
{
Thread.Sleep(100);
}
Console.WriteLine("Work completed!");
}
static void ProcessFile(object file)
{
try
{
Console.WriteLine(DateTime.Now.ToString() + " recieved: " + file + " thread count is: " + workingCounter.ToString());
//make some sleep for demo...
Thread.Sleep(2000);
}
catch (Exception ex)
{
//handle your exception...
string exMsg = ex.Message;
}
finally
{
Interlocked.Decrement(ref workingCounter);
Interlocked.Increment(ref processedCounter);
}
}
}
Take a look at the Producer/Consumer Queue example by Joe Albahari. It should provide a good starting point for what you're trying to accomplish.
You could use the ThreadPool.
Example:
ThreadPool.SetMaxThreads(3, 3);
for (int i = 0; i < filenames.count; i++)
{
ThreadPool.QueueUserWorkItem(new WaitCallback(ProcessFile), filenames[i]);
}
static void ProcessFile(object fileNameObj)
{
var fileName = (string)fileNameObj;
// do your processing here.
}
If you are using the ThreadPool elsewhere in your application then this would not be a good solution since it is shared across your app.
You could also grab a different thread pool implementation, for example SmartThreadPool
Rather than starting a thread for each file name, put the file names into a queue and then start up three threads to process them. Or, since the main thread is now free, start up two threads and let the main thread work on it, too:
Queue<string> MyQueue;
void MyProc()
{
string[] filenames = Directory.GetFiles(...);
MyQueue = new Queue(filenames);
// start two threads
Thread t1 = new Thread((ThreadStart)ProcessQueue);
Thread t2 = new Thread((ThreadStart)ProcessQueue);
t1.Start();
t2.Start();
// main thread processes the queue, too!
ProcessQueue();
// wait for threads to complete
t1.Join();
t2.Join();
}
private object queueLock = new object();
void ProcessQueue()
{
while (true)
{
string s;
lock (queueLock)
{
if (MyQueue.Count == 0)
{
// queue is empty
return;
}
s = MyQueue.Dequeue();
}
ProcessFile(s);
}
}
Another option is to use a semaphore to control how many threads are working:
Semaphore MySem = new Semaphore(3, 3);
void MyProc()
{
string[] filenames = Directory.GetFiles(...);
foreach (string s in filenames)
{
mySem.WaitOne();
ThreadPool.QueueUserWorkItem(ProcessFile, s);
}
// wait for all threads to finish
int count = 0;
while (count < 3)
{
mySem.WaitOne();
++count;
}
}
void ProcessFile(object state)
{
string fname = (string)state;
// do whatever
mySem.Release(); // release so another thread can start
}
The first will perform somewhat better because you don't have the overhead of starting and stopping a thread for each file name processed. The second is much shorter and cleaner, though, and takes full advantage of the thread pool. Likely you won't notice the performance difference.
Can set max threads unsing ParallelOptions
Parallel.For Method (Int32, Int32, ParallelOptions, Action)
ParallelOptions.MaxDegreeOfParallelism
var results = filenames.ToArray().AsParallel().Select(filename=>ProcessFile(filename)).ToArray();
bool ProcessFile(object fileNameObj)
{
var fileName = (string)fileNameObj;
// do your processing here.
return true;
}

Categories

Resources