Develop Reference

Closing active threads once an error threshold has been exceeded [duplicate]

This question already has answers here:
Question about terminating a thread cleanly in .NET
(8 answers)
Closed 6 years ago.
I have an application which will spin up a List of threads then kick them off one after another until a predefined thread limit has been hit, once this happens it will wait until a thread has finished before starting another. I want to implement a way to exit all threads that are currently running if a threshold of consecutive errors is reached. I have got as far as clearing the list of threads to process, so no more will be started but I'm not sure how to go about closing the remaining threads that are running once the thread limit has been reached, I recognise one thread can't talk to another and that's where Im getting hung up..I've provided a cut down version of main method below.
public static void Run(string distinguishedName, string hrExtractFile, string sanctionsFileLocation, string exemptionsFileLocation, string url, string soapAction, int threadLimit)
{
UsersList Employees = new UsersList(distinguishedName, hrExtractFile); //imports users from HR file
int errorcount = 0;
ManualResetEvent resetEventThreadComplete = new ManualResetEvent(false);
ManualResetEvent resetEventNoMoreThreads = new ManualResetEvent(false);
List<Thread> threads = new List<Thread>();
int toProcess = Employees.Count;
for (int i = 0; i < Employees.Count; i++)
{
int current = i;
threads.Add(new Thread(delegate ()
{
User u = Employees[current];
User drUser = new User();
try
{
drUser = fetchUser(u, url, soapAction);
bool userExists = false;
if (drUser != null)
{
userExists = true;
}
//removes a user if they're in the deleted users OU as well as being in system
if (u.IsDeleted)
{
if (userExists == true)
{
Console.WriteLine("Removing " + u.AccountName);
Log.writeToLogs("activitylog.txt", "Removing " + u.AccountName + ":", u, drUser);
DeleteUser(u, url, soapAction);
}
}
errorcount = 0;
}
}
catch (Exception e)
{
if (errorcount <= 5)
{
Log.writeToLogs("syslog.txt", e.ToString());
Log.writeToLogs("activitylog.txt", u.AccountName + " - Failed to true up!");
Console.WriteLine("Failed on " + u.AccountName + ": An error occured, check logs for details");
errorcount++;
}
else
{
lock (_syncObject)
{
//removes threads from list of threads that are pending
if (threads.Count > 0)
{
threads.Clear();
}
}
}
}
resetEventThreadComplete.Set();
if (Interlocked.Decrement(ref toProcess) == 0)
resetEventNoMoreThreads.Set();
}));
}
/*
* Kicks off the first x number of threads (where x = threadLimit) and removes them from list of pending threads
*/
for (int i = 0; i < threadLimit; i++)
{
if (threads.Count < 1)
break;
//runningThreads++;
threads[0].Start();
threads.RemoveAt(0);
}
/*
*Controls the initiation of thread exection; When one thread finishes, another will be started and removed from the list of pending threads
*/
while (threads.Count > 0)
{
resetEventThreadComplete.WaitOne();
resetEventThreadComplete.Reset();
threads[0].Start();
threads.RemoveAt(0);
}
if (toProcess > 0) resetEventNoMoreThreads.WaitOne();
//Log.sendLogs();
}

Assuming you don't mind killing your threads (not usually a great idea), you could use some code like this, just before clearing the list:
if (threads.Count > 0)
{
threads.ForEach(t => t.Abort()); //Kill all threads
threads.Clear();
}
That being said, you should probably not use Thread.Abort(), if you believe this poster:
Did you read the "remark" section on the MSDN? NEVER USE ABORT. IT IS INTENDED AS LAST RESORT. IF YOU USE THREAD.ABORT, SKY MAY FALL DOWN, AND KITTEN WILL BE KILLED
Whatever method you choose, you can put it into that ForEach expression, e.g. if you prefer to use Interrupt:
threads.ForEach(t => t.Interrupt());
Another (far simpler) approach is to modify your threads to check a global flag variable and exit cleanly when it is set. In your main program, when you wish to kill the threads, just set the flag. Be sure to use a volatile variable, or better yet, a WaitHandle.

Worker Threads Blocking When ConcurrentQueue has too many items

This is a weird one, I have a Thread[] of worker threads which each process items in a ConcurrentQueue<string> until the queue is empty, at which point the rest of the program continues.
This works until about ~1500 items at which point all threads stay blocked in the WaitSleepJoin state and never process any of the items in the queue.
I've tried stepping through my code and it appears that the threads are still created, still started and are alive but get blocked immediately and never run their relevant function.
I'm completely flummoxed so any help would be appreciated!
The relevant sections of code are below:
Main Thread Segment:
ConcurrentQueue<string> convertionQueue = new ConcurrentQueue<string>();
List<Thread> converterThreads = new List<Thread>();
Directory.GetFiles(_folderOne, "*.fdf", SearchOption.AllDirectories).ToList().ForEach(file => convertionQueue.Enqueue(file));
Directory.GetFiles(_folderTwo, "*.fdf", SearchOption.AllDirectories).ToList().ForEach(file => convertionQueue.Enqueue(file));
int filesDone = 0;
int totalFiles = convertionQueue.Count;
progressBar.Maximum = totalFiles;
panel1.Visible = true;
for (int i = 0; i < Environment.ProcessorCount; i++)
{
converterThreads.Add(new Thread(() => ConvThreadWorker(convertionQueue, ref filesDone)));
}
converterThreads.ForEach(thread => thread.Start());
DateTime lastTick = DateTime.Now;
int lastFilesDone = 0;
int[] valuesSpeed = { 1, 1, 1, 1, 1 };
int[] valuesTime = { 1, 1, 1, 1, 1 };
int counter = 0;
while (converterThreads.Any(thread => thread.IsAlive))
{
TimeSpan t = DateTime.Now - lastTick;
int deltaFiles = filesDone - lastFilesDone;
double speed = (float)t.TotalMilliseconds <= 0.0 ? 0.0 : deltaFiles / (float)t.TotalMilliseconds;
double tMinus = speed <= 0 ? 0.0 : (totalFiles - filesDone) / speed;
int currentSpeed = (int)(speed * 1000);
int currentTime = (int)(tMinus / 1000);
valuesSpeed[counter] = currentSpeed;
valuesTime[counter] = currentTime;
lblFilesLeft.Text = string.Format("{0}/{1}", filesDone, totalFiles);
lblSpeed.Text = valuesSpeed.Sum() / 5 + " /s";
lblTime.Text = valuesTime.Sum() / 5 + " s";
lblFilesLeft.Update();
lblSpeed.Update();
lblTime.Update();
progressBar.Value = filesDone;
progressBar.Update();
lastTick = DateTime.Now;
lastFilesDone = filesDone;
counter = ++counter % 5;
Thread.Sleep(500);
}
Worker Function:
private void ConvThreadWorker(ConcurrentQueue<string> queue, ref int fileCounter)
{
while (!queue.IsEmpty)
{
string file;
if (queue.TryDequeue(out file))
{
ConvToG(file);
fileCounter++;
}
}
}
Convertion Function:
private void ConvToG(string file)
{
MessageBox.Show("Entering Convertion Function");
if (!_fileCreationDictionary.ContainsKey(file))
{
DateTime lastTimeModified = File.GetLastWriteTime(file);
_fileCreationDictionary.AddOrUpdate(file, lastTimeModified, (key,oldvalue)=>lastTimeModified);
}
ProcessStartInfo procStart = new ProcessStartInfo
{
Arguments = file,
UseShellExecute = true,
FileName = Fdfg,
WindowStyle = ProcessWindowStyle.Hidden
};
Process process = new Process {StartInfo = procStart};
MessageBox.Show("Starting convertion process");
process.Start();
process.WaitForExit();
MessageBox.Show("Finished");
}
The confusing part appears to be how this all revolves around the number of items in the queue, yet there appears to be no overflow.
UPDATE: Adding the mbox's shows that it freezes on the process.Start() section of code, with no errors and will not proceed past that point.
UPDATE 2: If UseShellExecute = false the code works. Which is very confusing to say the least.

I have done something similar with threads spawning processes to collate data. I had issues around the actual process starting and hanging. What I did to get my program working was something like this:
using (Process process = Process.Start(startInfo)) {
if(process.WaitForExit(timeOutMilliseconds)) {
MessageBox.Show("Process exited ok");
//...snip
} else {
MessageBox.Show("Process did not exit in time!");
//...snip
process.Kill();
}
}
There is a bit more going on in the background, regarding limiting the number of running process, etc, but I found that occasionally, for an unknown reason, that I would see several process in the task manager just hanging around forever.
Hope that helps?

Threadpool issue Using C#

I'm working on my university project. One of main requirement is to use multithreading (user can choose threads numbers).
I'm new in C# and based on internet research. I choose ThreadPool.
I spent a lot of time observing how the threads act using parallel watch in VS and i have no idea how this thing works. For example threadNumber = 10 but parallel watch shows only 4 activated threads.
Here is my code:
public void calculateBeta()
{
var finished = new CountdownEvent(1);
for (int i = 0; i < threadNumber; i++)
{
finished.AddCount();
ThreadPool.QueueUserWorkItem(
(state) =>
{
try
{
doSth();
}
finally
{
finished.Signal();
}
});
}
finished.Signal();
finished.Wait();
}
What am I doing wrong? I tried to test this code with many different values of threads number and it didn't work as i looked for.
EDIT:
private void myTask(object index)
{
int z = (int)index;
double[] result = countBeta(createTableB(z), createTableDiagonalA(z));
int counter = 0;
if ((rest != 0) && (z == threadNumber - 1))
{
for (int j = z * numbersInRow; j < (z + 1) * numbersInRow + rest; j++)
{
N[j] = result[counter];
counter++;
}
}
else
{
for (int j = z * numbersInRow; j < (z + 1) * numbersInRow; j++)
{
N[j] = result[counter];
counter++;
}
}
threads[z] = true;
}
public void calculateBeta()
{
N = new double[num];
setThreadNumber(2);
checkThreadNumber();
setNumberInRow();
setRest();
threads = new bool[threadNumber];
for (int i = 0; i < threadNumber; i++)
{
Thread thread = new Thread(this.myTask);
thread.IsBackground = true;
thread.Start(i);
}
while (!checkThreads())
{
}
}
private bool checkThread()
{
bool result = true;
for (int i = 0; i < threads.Length; i++)
{
if (!threads[i])
result = false;
}
return result;
}
static void Main(string[] args)
{
Jacobi jacobi = new Jacobi();
Console.WriteLine("Metoda Jacobiego");
Console.WriteLine("Rozwiazywanie ukladu n-rownan z n-niewiadomymi Ax=b");
jacobi.getNum();
jacobi.getA();
jacobi.getB();
jacobi.calculateBeta();
jacobi.calculateM();
jacobi.calculateX();
jacobi.countNorms();
Console.ReadLine();
}
I need results from calculateBeta to further calculations. Sometimes threads are not finished yet but the program moves forward without data that need to be provided by threads. I'm using bool variable now but this solution is not an elegant way to deal with it(Creating bool table, checking if all thread are fnished) How can i manage with that in a different way?

This is because you're using ThreadPool to manage your threads. It will create a certain number of threads based on many factors. You can tweak some of the settings but by and large when you commit to using ThreadPool to managing your threads you commit to a black box. Check out GetMaxThreads and GetMinThreads and their setter counterparts for some of your options.
Check out this ThreadPool Architecture article on MSDN. It gives good background to the hows and whys of the class. But in the introductory paragraph you will see this sentence, which is key to your conundrum:
The thread pool is primarily used to reduce the number of application
threads and provide management of the worker threads.
If you want to have the kind of control where you launch 10 threads in quick succession you should avoid ThreadPool and just manage the threads yourself. Here is a simple, absolutely minimal example of launching ten threads and also passing different data to each, in this case an index:
void ButtonClickHandlerOrSomeOtherMethod()
{
for (int i=1; i<=10; i++) // using a 1-based index
{
new Thread(ThreadTask).Start(i);
}
}
void ThreadTask(object i)
{
Console.WriteLine("Thread " + i + " ID: " + Thread.CurrentThread.ManagedThreadId);
}
And some sample output:
Thread 1 ID: 19
Thread 2 ID: 34
Thread 3 ID: 26
Thread 4 ID: 5
Thread 5 ID: 36
Thread 6 ID: 18
Thread 7 ID: 9
Thread 8 ID: 38
Thread 9 ID: 39
Thread 10 ID: 40
Follow-up code demonstrating synching with threads and "waiting" until they are all finished:
void ButtonClickHandlerOrSomeOtherMethod()
{
// need a collection of threads to call Join after Start(s)
var threads = new List<Thread>();
// create threads, add to List and start them
for (int i=1; i<=10; i++) {
var thread = new Thread(ThreadTask);
threads.Add(thread);
// a background thread will allow main app to exit even
// if the thread is still running
thread.IsBackground = true;
thread.Start(i);
}
// call Join on each thread which makes this thread wait on
// all 10 other threads
foreach (var thread in threads)
thread.Join();
// this message will not show until all threads are finished
Console.WriteLine("All threads finished.");
}
void ThreadTask(object i)
{
Console.WriteLine("Thread " + i + " ID: " + Thread.CurrentThread.ManagedThreadId);
// introducing some randomness to how long a task "works on something"
Thread.Sleep(100 * new Random().Next(0, 10));
Console.WriteLine("Thread " + i + " finished.");
}

The whole design of the thread pool is that it doesn't have to create a new actual thread every time a new item is queued up. If the pool notices that it has items pending in the queue for an extended period of time it will eventually start spinning up new threads, over time. If you're continually saturating the thread pool with operations, you'll see the number of actual threads rise. It will also only add new threads up to a limit; based on what it feels is going to have the best throughput. For example, it will avoid creating a lot more threads than cores assuming all of the threads are actively running CPU bound work.
The idea of using the thread pool is if you don't care how many actual threads there are, but rather just want to have efficient throughput of the operations that you have, allowing the framework lots of freedom on how to best optimize that work. If you have very specific requirements as to how many threads you have, you'll need to create threads manually rather than using a pool.

// Array of threads launched.
// This array is useful to trace threads status.
Thread[] threads;
private void myTask(object index)
{
Console.Write("myTask {0} started\n", index);
Console.Write("myTask {0} finisced\n", index);
}
public void calculateBeta(UInt16 threadNumber)
{
// Allocate a new array with size of requested number of threads
threads = new Thread[threadNumber];
// For each thread
for (int i = 0; i < threadNumber; i++)
{
// Thread creation
threads[i] = new Thread(this.myTask);
// IsBackground set to true grants that the allication can be "killed" without wait for all threads termination
// This is useful in debug to be sure that an error in task doesn't freeze the app.
// Leave it to false in release
#if DEBUG
threads[i].IsBackground = true;
#endif
// Start the thread
threads[i].Start(i);
}
// Waits until all threads complete.
while (!checkThreads());
}
private bool checkThreads()
{
bool result = true;
for (int i = 0; i < threads.Length; i++)
{
// If the thread wasn't disposed
if (threads[i] != null)
{
// Check if the thead is alive (means is working)
if (threads[i].IsAlive == true)
{
result = false;
}
else // The thread is not working
{
// Dispose the thread
threads[i].Join();
// Set pointer to null to signal that the task was
threads[i] = null;
}
}
}
return result;
}
private void Button_Click(object sender, RoutedEventArgs e)
{
Console.Write("Starting tasks!!\n");
calculateBeta(10);
Console.Write("All tasks finished!!\n");
}

Create multiple threads and wait for them all to complete, calling again for the complete

I have a Web Service, to make the load of the database server for a local database, making 100 requests for records.
Since the process is slow, I want to create ten threads, not to use too much memory, making Web Service calls, and when one of the threads, finished, over 100 call records. How do part of the thread?
Example:
Create thread 1
Create thread 2
Create thread 3
Create thread 4
thread 1 complete change Web Service again
Edit
My code not working. Variable sendalways gets the value 10 and not 0,1,2,3,4 and etc.
Int32 page = 0;
do
{
for (int iterator=0; iterator < 10; iterator++)
{
listTask[iterator] = Task.Factory.StartNew(() =>
{
Int32 send = iterator + page * 10;
DoStatus("Page: " + send.ToString());
Processamento(parametros, filial, send);
});
}
Task.WaitAll(listTask);
page++;
}
while (true); // Test only

You're closing over the loop variable. You need to remember that lambdas close over variables not over values. Your tasks will each read the value of iterator at the time that the lambda executes iterator + page * 10. By the time that that happens the main thread has already incremented it to 10.
This is simple enough to resolve. Make a copy of the loop variable inside of your for loop so that the closure closes over that variable, which never changes.
for (int iterator=0; iterator < 10; iterator++)
{
int i = iterator;
listTask[iterator] = Task.Factory.StartNew(() =>
{
Int32 send = i + page * 10;
DoStatus("Page: " + send.ToString());
Processamento(parametros, filial, send);
});
}

If I understand your question, you want to create 10 threads, wait for all, then recreate 10 threads, etc. Each thread load 100 results.
In this answer, results are String but that can be changed.
private void Load()
{
Boolean loading = true;
List<String> listResult = new List<String>();
Int32 boucle = 0;
Task[] listTask = new Task[10];
do
{
// create 10 threads (=1000 results)
for (int iterator=0; iterator < 10; iterator++)
{
// [0-99] [100-199] [200-299] ...
Int32 start = 100 * iterator + 1000 * boucle;
Int32 end = start + 99;
listTask[iterator] = Task<List<String>>.Factory.StartNew(() =>
{
List<String> data = LoadData(start, end);
return data;
});
}
// wait for 10 threads to finish
Task.WaitAll(listTask);
// collapse results
for (int i=0; i < 10; i++)
{
listResult.AddRange((listTask[i] as Task<List<String>>).Result);
}
// check if there is 100 results in last thread
loading = (listTask[9] as Task<List<String>>).Result.Count == 100;
// ready for another iteration (next 1000 results)
boucle++;
}
while (loading);
}
private List<string> LoadData(int p1, int p2)
{
// TODO : load data from p1 to p2
throw new NotImplementedException();
}

How to use multi threading in a For loop

I want to achieve the below requirement; please suggest some solution.
string[] filenames = Directory.GetFiles("C:\Temp"); //10 files
for (int i = 0; i < filenames.count; i++)
{
ProcessFile(filenames[i]); //it takes time to execute
}
I wanted to implement multi-threading. e.g There are 10 files. I wanted to process 3 files at a time (configurable, say maxthreadcount). So 3 files will be processed in 3 threads from the for loop and if any thread completes the execution, it should pick the next item from the for loop. Also wanted to ensure all the files are processed before it exits the for loop.
Please suggest best approach.

Try
Parallel.For(0, filenames.Length, i => {
ProcessFile(filenames[i]);
});
MSDN
It's only available since .Net 4. Hope that acceptable.

This will do the job in .net 2.0:
class Program
{
static int workingCounter = 0;
static int workingLimit = 10;
static int processedCounter = 0;
static void Main(string[] args)
{
string[] files = Directory.GetFiles("C:\\Temp");
int checkCount = files.Length;
foreach (string file in files)
{
//wait for free limit...
while (workingCounter >= workingLimit)
{
Thread.Sleep(100);
}
workingCounter += 1;
ParameterizedThreadStart pts = new ParameterizedThreadStart(ProcessFile);
Thread th = new Thread(pts);
th.Start(file);
}
//wait for all threads to complete...
while (processedCounter< checkCount)
{
Thread.Sleep(100);
}
Console.WriteLine("Work completed!");
}
static void ProcessFile(object file)
{
try
{
Console.WriteLine(DateTime.Now.ToString() + " recieved: " + file + " thread count is: " + workingCounter.ToString());
//make some sleep for demo...
Thread.Sleep(2000);
}
catch (Exception ex)
{
//handle your exception...
string exMsg = ex.Message;
}
finally
{
Interlocked.Decrement(ref workingCounter);
Interlocked.Increment(ref processedCounter);
}
}
}

Take a look at the Producer/Consumer Queue example by Joe Albahari. It should provide a good starting point for what you're trying to accomplish.

You could use the ThreadPool.
Example:
ThreadPool.SetMaxThreads(3, 3);
for (int i = 0; i < filenames.count; i++)
{
ThreadPool.QueueUserWorkItem(new WaitCallback(ProcessFile), filenames[i]);
}
static void ProcessFile(object fileNameObj)
{
var fileName = (string)fileNameObj;
// do your processing here.
}
If you are using the ThreadPool elsewhere in your application then this would not be a good solution since it is shared across your app.
You could also grab a different thread pool implementation, for example SmartThreadPool

Rather than starting a thread for each file name, put the file names into a queue and then start up three threads to process them. Or, since the main thread is now free, start up two threads and let the main thread work on it, too:
Queue<string> MyQueue;
void MyProc()
{
string[] filenames = Directory.GetFiles(...);
MyQueue = new Queue(filenames);
// start two threads
Thread t1 = new Thread((ThreadStart)ProcessQueue);
Thread t2 = new Thread((ThreadStart)ProcessQueue);
t1.Start();
t2.Start();
// main thread processes the queue, too!
ProcessQueue();
// wait for threads to complete
t1.Join();
t2.Join();
}
private object queueLock = new object();
void ProcessQueue()
{
while (true)
{
string s;
lock (queueLock)
{
if (MyQueue.Count == 0)
{
// queue is empty
return;
}
s = MyQueue.Dequeue();
}
ProcessFile(s);
}
}
Another option is to use a semaphore to control how many threads are working:
Semaphore MySem = new Semaphore(3, 3);
void MyProc()
{
string[] filenames = Directory.GetFiles(...);
foreach (string s in filenames)
{
mySem.WaitOne();
ThreadPool.QueueUserWorkItem(ProcessFile, s);
}
// wait for all threads to finish
int count = 0;
while (count < 3)
{
mySem.WaitOne();
++count;
}
}
void ProcessFile(object state)
{
string fname = (string)state;
// do whatever
mySem.Release(); // release so another thread can start
}
The first will perform somewhat better because you don't have the overhead of starting and stopping a thread for each file name processed. The second is much shorter and cleaner, though, and takes full advantage of the thread pool. Likely you won't notice the performance difference.

Can set max threads unsing ParallelOptions
Parallel.For Method (Int32, Int32, ParallelOptions, Action)
ParallelOptions.MaxDegreeOfParallelism

var results = filenames.ToArray().AsParallel().Select(filename=>ProcessFile(filename)).ToArray();
bool ProcessFile(object fileNameObj)
{
var fileName = (string)fileNameObj;
// do your processing here.
return true;
}