How to know when all my Producer - Consumer jobs have finished - c#

I have Winforms application that read several network folders and search for files inside this folders, this function receive List<stirng> folders:
private decimal _numberOfFiles;
private static List<string> _folders;
public delegate void OnFileAddDelegate(List<string> files);
public event OnFileAddDelegate OnFileAddEventHandler;
public delegate void OnFinishSearchDelegate();
public event OnFinishSearchDelegate OnFinishSearchEventHandler;
public void SearchFiles()
{
foreach (string folder in _folders)
{
if (Directory.Exists(folder))
{
var files = Directory.EnumerateFiles(folder, "*.doc", SearchOption.TopDirectoryOnly)
.OrderByDescending(x => new FileInfo(x).CreationTime).Take((int)_numberOfFiles).ToList<string>();
if (OnFileAddEventHandler != null)
OnFileAddEventHandler(files);
}
}
if (OnFinishSearchEventHandler != null)
OnFinishSearchEventHandler();
}
After OnFileAddEventHandler(files) event is fired my ProducerConsumer class start to check this List of files that found and do the work (if the file is OK fired up event to my main UI to add this files into my ListView):
public class ProducerConsumer
{
public delegate void OnFileAddDelegate(PcapFileDetails pcapFileDetails);
public event OnFileAddDelegate OnFileAddEventHandler;
public delegate void AllFilesProcessedDelegate();
public event AllFilesProcessedDelegate AllFilesProcessedEventHandler;
private readonly Queue<string> _queue;
private int counter;
public ProducerConsumer(int workerCount, IEnumerable<string> list)
{
_isSearchFinished = true;
_queue = new Queue<string>(list); // fill the queue
counter = _queue.Count; // set up counter
for (int i = 0; i < workerCount; i++)
Task.Factory.StartNew(Consumer);
}
private void Consumer()
{
FileChecker fileChecker = new FileChecker();
for (; ; )
{
string file;
lock (_queue)
{
// synchronize on the queue
if (_queue.Count == 0) return; // we are done
file = _queue.Dequeue(); // get file name to process
} // release the lock to allow other consumers to access the queue
// do the job
string result = fileChecker.Check(file); // Check my file
if (OnFileAddEventHandler != null && result ) // In case my file OK, fired up event to my main UI
OnFileAddEventHandler(file);
// decrement the counter
if (Interlocked.Decrement(ref counter) != 0)
continue; // not the last
// all done - we were the last
if (AllFilesProcessedEventHandler != null)
AllFilesProcessedEventHandler();
return;
}
}
}
Now while this search is in process my UI is locked to prevent unnecessary clicks and i want to know when all my folders finish to search to unlock.
But my problem is because i am search several folders the event AllFilesProcessedEventHandler() fired up several times and i want to know when all my searches finished.

Here is a recursive sample with QuickIO.Net
using System;
using System.Collections.Concurrent;
using System.Threading;
using System.Threading.Tasks;
using SchwabenCode.QuickIO;
namespace ConsoleApplication3
{
internal class Program
{
private static readonly BlockingCollection<QuickIOFileInfo> fileInfos = new BlockingCollection<QuickIOFileInfo>();
private static void Main(string[] args)
{
var task = Task.Factory.StartNew(() =>
{
Int32 totalSize = 0;
Parallel.ForEach(fileInfos.GetConsumingEnumerable(), fi =>
{
Interlocked.Add(ref totalSize, (int)fi.Bytes);
});
Console.WriteLine("All docs bytes amount to {0}", totalSize);
});
ProcessDirectory("C:\\");
fileInfos.CompleteAdding();
Task.WaitAll(task);
}
private static void ProcessDirectory(string path)
{
Parallel.ForEach(QuickIODirectory.EnumerateDirectories(path), dir =>
{
try
{
Parallel.ForEach(QuickIODirectory.EnumerateFiles(dir), file =>
{
if (file.AsFileInfo().Extension.Equals(".docx"))
fileInfos.Add(file);
});
ProcessDirectory(dir.FullName);
}
catch (Exception e)
{
Console.WriteLine("Unable to access directory {0}", dir.FullName);
}
});
}
}
}
Blocking collection will automatically signal to Parallel ForEach when all elements have been added, by calling CompleteAdding().
To scan a 256GB SSD, with 74GB left and total of 738k+ files took 16.8s.

Related

Design solution for working with multiple instantiations at the same time

I don't know if my title is correct. But here's what I want to know.
I have a Download class that returns certain events and has a couple of methods. Each instance of Download class can download a single file. And all those events and methods are related to the file being downloaded.
As it's a multi file downloader, multiple instantiations are required when more than a single file needs to be downloaded.
Each download has a download id, but that is not supplied to the Download class, to keep it independent from the other classes.
Now getting all the info from each instance of the file download and being able to control a single download, is the problem. How do I know which download is which?
Any solutions? Or design patterns you could recommend? I've hit a roadblock.
Download class:
using System;
using System.Diagnostics;
using System.IO;
using System.Net;
using System.Threading;
namespace Mackerel_Download_Manager
{
public class Download
{
public event EventHandler<DownloadStatusChangedEventArgs> ResumablityChanged;
public event EventHandler<DownloadProgressChangedEventArgs> ProgressChanged;
public event EventHandler Completed;
public bool stop = true; // by default stop is true
public bool paused = false;
SemaphoreSlim pauseLock = new SemaphoreSlim(1);
string filename;
public void DownloadFile(Uri DownloadLink, string Path)
{
filename = System.IO.Path.GetFileName(Path);
stop = false; // always set this bool to false, everytime this method is called
var fileInfo = new FileInfo(Path);
long existingLength = 0;
if (fileInfo.Exists)
existingLength = fileInfo.Length;
var request = (HttpWebRequest)HttpWebRequest.Create(DownloadLink);
request.Proxy = null;
request.AddRange(existingLength);
try
{
using (var response = (HttpWebResponse)request.GetResponse())
{
long fileSize = existingLength + response.ContentLength; //response.ContentLength gives me the size that is remaining to be downloaded
bool downloadResumable; // need it for not sending any progress
if ((int)response.StatusCode == 206) //same as: response.StatusCode == HttpStatusCode.PartialContent
{
//Console.WriteLine("Resumable");
downloadResumable = true;
}
else // sometimes a server that supports partial content will lose its ability to send partial content(weird behavior) and thus the download will lose its resumability
{
if (existingLength > 0)
{
if (ResumeUnsupportedWarning() == false) // warn and ask for confirmation to continue if the half downloaded file is unresumable
{
return;
}
}
existingLength = 0;
downloadResumable = false;
}
OnResumabilityChanged(new DownloadStatusChangedEventArgs(downloadResumable));
using (var saveFileStream = fileInfo.Open(downloadResumable ? FileMode.Append : FileMode.Create, FileAccess.Write))
using (var stream = response.GetResponseStream())
{
byte[] downBuffer = new byte[4096];
int byteSize = 0;
long totalReceived = byteSize + existingLength;
var sw = Stopwatch.StartNew();
while (!stop && (byteSize = stream.Read(downBuffer, 0, downBuffer.Length)) > 0)
{
saveFileStream.Write(downBuffer, 0, byteSize);
totalReceived += byteSize;
float currentSpeed = totalReceived / (float)sw.Elapsed.TotalSeconds;
OnProgressChanged(new DownloadProgressChangedEventArgs(totalReceived, fileSize, (long)currentSpeed));
pauseLock.Wait();
pauseLock.Release();
}
sw.Stop();
}
}
if (!stop) OnCompleted(EventArgs.Empty);
}
catch (WebException e)
{
System.Windows.MessageBox.Show(e.Message, filename);
}
}
public void pause()
{
if (!paused)
{
paused = true;
// Note this cannot block for more than a moment
// since the download thread doesn't keep the lock held
pauseLock.Wait();
}
}
public void unpause()
{
if (paused)
{
paused = false;
pauseLock.Release();
}
}
public void StopDownload()
{
stop = true;
this.unpause(); // stop waiting on lock if needed
}
public bool ResumeUnsupportedWarning()
{
var messageBoxResult = System.Windows.MessageBox.Show("When trying to resume the download , Mackerel got a response from the server that it doesn't support resuming the download. It's possible that it's a temporary error of the server, and you will be able to resume the file at a later time, but at this time Mackerel can download this file from the beginning.\n\nDo you want to download this file from the beginning?", filename, System.Windows.MessageBoxButton.YesNo);
if (messageBoxResult == System.Windows.MessageBoxResult.Yes)
{
return true;
}
else
{
return false;
}
}
protected virtual void OnResumabilityChanged(DownloadStatusChangedEventArgs e)
{
var handler = ResumablityChanged;
if (handler != null)
{
handler(this, e);
}
}
protected virtual void OnProgressChanged(DownloadProgressChangedEventArgs e)
{
var handler = ProgressChanged;
if (handler != null)
{
handler(this, e);
}
}
protected virtual void OnCompleted(EventArgs e)
{
var handler = Completed;
if (handler != null)
{
handler(this, e);
}
}
}
public class DownloadStatusChangedEventArgs : EventArgs
{
public DownloadStatusChangedEventArgs(bool canResume)
{
ResumeSupported = canResume;
}
public bool ResumeSupported { get; private set; }
}
public class DownloadProgressChangedEventArgs : EventArgs
{
public DownloadProgressChangedEventArgs(long totalReceived, long fileSize, long currentSpeed)
{
BytesReceived = totalReceived;
TotalBytesToReceive = fileSize;
CurrentSpeed = currentSpeed;
}
public long BytesReceived { get; private set; }
public long TotalBytesToReceive { get; private set; }
public float ProgressPercentage
{
get
{
return ((float)BytesReceived / (float)TotalBytesToReceive) * 100;
}
}
public float CurrentSpeed { get; private set; } // in bytes
public TimeSpan TimeLeft
{
get
{
var bytesRemainingtoBeReceived = TotalBytesToReceive - BytesReceived;
return TimeSpan.FromSeconds(bytesRemainingtoBeReceived / CurrentSpeed);
}
}
}
}
Download class is instantiated inside a Mackerel class, that starts the download for the given downloads.
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Windows;
namespace Mackerel_Download_Manager
{
public static class Mackerel
{
//Main Menu functions
public static void ResumeDownload(string[] DownloadIDs)
{
foreach (var DownloadID in DownloadIDs)
{
var itemToResume = Downloads.DownloadEntries.Where(download => download.DownloadID == DownloadID).FirstOrDefault();
if (itemToResume.Running == false)
{
itemToResume.Running = true;
var download = new Download();
download.DownloadFile(itemToResume.DownloadLink, itemToResume.SaveTo);
var window = new Dialogs.DownloadProgress(itemToResume);
window.Show();
double progress = 0;
itemToResume.Status = string.Format("{0:0.00}%", progress);
Downloads.DownloadEntries.CollectionChanged += delegate
{
if (!itemToResume.Running) window.Close();
};
}
}
}
public static void StopDownload(string[] DownloadIDs)
{
foreach (var DownloadID in DownloadIDs)
{
var itemToStop = Downloads.DownloadEntries.Where(download => download.DownloadID == DownloadID).FirstOrDefault();
if (itemToStop.Running == true)
itemToStop.Running = false;
}
}
public static void StopAllDownloads()
{
foreach (var itemToStop in Downloads.DownloadEntries.Where(download => download.Running == true))
itemToStop.Running = false;
}
public static void RemoveDownload(string[] DownloadIDs) // this method is able to delete multiple downloads
{
foreach (var DownloadID in DownloadIDs)
{
// delete from the download list
var selectedDownload = Downloads.DownloadEntries.Where(download => download.DownloadID == DownloadID).FirstOrDefault();
var selectedDownloadIndex = Downloads.DownloadEntries.IndexOf(selectedDownload);
Downloads.DownloadEntries.RemoveAt(selectedDownloadIndex);
//delete from the harddrive
if (File.Exists(selectedDownload.SaveTo))
File.Delete(selectedDownload.SaveTo);
}
Downloads.Serialize(); // save current state of object
}
public static void RemoveCompletedDownloads() // this method just removes all completed downloads from Mackerel's download list (it doesn't delete them from the hard drive)
{
foreach (var itemToRemove in Downloads.DownloadEntries.Where(download => download.Status == "Complete").ToList())
{
Downloads.DownloadEntries.Remove(itemToRemove);
}
}
// Context Menu
public static void OpenDownloadProperties(string DownloadID) // Open "Download Properties" for the given download ID
{
var DownloadProperties = new Dialogs.Context_Menu.DownloadProperties(DownloadID);
DownloadProperties.Owner = Application.Current.MainWindow; // so that this dialog centers to its parent window, as its window is set to WindowStartupLocation="CenterOwner"
DownloadProperties.ShowDialog();
}
}
}
Full source code is here: https://github.com/Expenzor/mackerel-download-manager
Sounds like your making the download itself an object but are using "downloader" as the name instead. I would maybe suggest an array of downloaded objects or similar. The downloader class can have a method which creates a new object - perhaps a download object. Using an array or linked list etc will give you the opportunity to address the object and call them independently - know which is which.
Posting your code would help as well.

BlockingCollection multiple thread Producer Consumer

i want to implement multiple thread Producer Consumer, my purpose is insert strings into my Queue, each string represent DOC file that need to be check (simple search inside the document) and in this check is OK the doc file can be add into my ListView.
public class ProducerConsumer
{
public delegate void OnFileAddDelegate(string file);
public event OnFileAddDelegate OnFileAddEventHandler;
private BlockingCollection<string> queue = new BlockingCollection<string>();
public void Start(int workerCount)
{
int count = 0;
while (count < workerCount)
{
Thread thread = new Thread(StartConsuming);
thread.IsBackground = true;
thread.Start();
count++;
}
}
public void Produce(string item)
{
queue.Add(item);
}
private void StartConsuming()
{
while (queue.Count > 0)
{
string item = queue.Take();
// Check my file
FileChecker fileChecker = new FileChecker();
string result = fileChecker.Check(item);
// If the file is OK fire up an event to my main form to add this file
if (result != null && OnFileAddEventHandler != null)
OnFileAddEventHandler(result);
}
}
}
Now i want add to my class the option to add files in multiple thread.
Any suggestions how to do that ?

Randomly Timeout(TimeSpan) doesn't work in Rx

The earlier post seems not very clear, so after some testing, I reopened this post with much more simplified words, hope somebody could help.
My singleton observable was turned from multiple source of I/O events, means they're concurrently raised up in underlying, based on testing (to prove Rx is not thread safe) and RX design guideline, I made it serialized, see that lock(...):
public class EventFireCenter
{
public static event EventHandler<GTCommandTerminalEventArg> OnTerminalEventArrived;
private static object syncObject = new object();
public static void TestFireDummyEventWithId(int id)
{
lock (syncObject)
{
var safe = OnTerminalEventArrived;
if (safe != null)
{
safe(null, new GTCommandTerminalEventArg(id));
}
}
}
}
This is the singleton Observable:
public class UnsolicitedEventCenter
{
private readonly static IObservable<int> publisher;
static UnsolicitedEventCenter()
{
publisher = Observable.FromEventPattern<GTCommandTerminalEventArg>(typeof(EventFireCenter), "OnTerminalEventArrived")
.Select(s => s.EventArgs.Id);
}
private UnsolicitedEventCenter() { }
/// <summary>
/// Gets the Publisher property to start observe an observable sequence.
/// </summary>
public static IObservable<int> Publisher { get { return publisher; } }
}
The scenario of Subscribe(...) can be described by following code, you can see the Subscribe(...) could be called concurrently in different threads:
for (var i = 0; i < concurrentCount; i++)
{
var safe = i;
Scheduler.Default.Schedule(() =>
{
IDisposable dsp = null;
dsp = UnsolicitedEventCenter.Publisher
.Timeout(TimeSpan.FromMilliseconds(8000))
.Where(incomingValue => incomingValue == safe)
.ObserveOn(Scheduler.Default)
//.Take(1)
.Subscribe((incomingEvent) =>
{
Interlocked.Increment(ref onNextCalledTimes);
dsp.Dispose();
}
, ex =>
{
Interlocked.Increment(ref timeoutExceptionOccurredTimes);
lock (timedOutEventIds)
{
// mark this id has been timed out, only for unit testing result check.
timedOutEventIds.Add(safe);
}
dsp.Dispose();
});
Interlocked.Increment(ref threadPoolQueuedTaskCount);
});
}
As pointed out times by experienced people, call Dispose() in OnNext(...) is not recommended, but let's ignore it here since the code was from production.
Now the problem is randomly that .Timeout(TimeSpan.FromMilliseconds(8000)) is not working, the ex was never called, anyone could see any abnormal in the code?
for testing, I setup the stress testing, but so far, I didn't reproduced it, while in production, it appeared several times per day. Just in case, I pasted all the testing code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reactive.Concurrency;
using System.Reactive.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace Rx
{
class Program
{
static void Main(string[] args)
{
// avoid thread creation delay in thread pool.
ThreadPool.SetMinThreads(200, 50);
// let the test run for 100 times
for (int t = 0; t < 100; t++)
{
Console.WriteLine("");
Console.WriteLine("======Current running times: " + t);
// at meantime, 150 XXX.Subscribe(...) will be called.
const int concurrentCount = 150;
// how many fake event will be fire to santisfy that 150 XXX.Subscribe(...).
const int fireFakeEventCount = 40;
int timeoutExceptionOccurredTimes = 0;
var timedOutEventIds = new List<int>();
int onNextCalledTimes = 0;
int threadPoolQueuedTaskCount = 0;
for (var i = 0; i < concurrentCount; i++)
{
var safe = i;
Scheduler.Default.Schedule(() =>
{
IDisposable dsp = null;
dsp = UnsolicitedEventCenter.Publisher
.Timeout(TimeSpan.FromMilliseconds(8000))
.Where(incomingValue => incomingValue == safe)
.ObserveOn(Scheduler.Default)
//.Take(1)
.Subscribe((incomingEvent) =>
{
Interlocked.Increment(ref onNextCalledTimes);
dsp.Dispose();
}
, ex =>
{
Interlocked.Increment(ref timeoutExceptionOccurredTimes);
lock (timedOutEventIds)
{
// mark this id has been timed out, only for unit testing result check.
timedOutEventIds.Add(safe);
}
dsp.Dispose();
});
Interlocked.Increment(ref threadPoolQueuedTaskCount);
});
}
Console.WriteLine("Starting fire event: " + DateTime.Now.ToString("HH:mm:ss.ffff"));
int threadPoolQueuedTaskCount1 = 0;
// simulate a concurrent event fire
for (int i = 0; i < fireFakeEventCount; i++)
{
var safe = i;
Scheduler.Default.Schedule(() =>
{
EventFireCenter.TestFireDummyEventWithId(safe);
Interlocked.Increment(ref threadPoolQueuedTaskCount1);
});
}
// make sure all proceeding task has been done in threadPool.
while (threadPoolQueuedTaskCount < concurrentCount)
{
Thread.Sleep(1000);
}
// make sure all proceeding task has been done in threadPool.
while (threadPoolQueuedTaskCount1 < fireFakeEventCount)
{
Thread.Sleep(100);
}
Console.WriteLine("Finished fire event: " + DateTime.Now.ToString("HH:mm:ss.ffff"));
// sleep a time which >3000ms.
Thread.Sleep(8000);
Console.WriteLine("timeoutExceptionOccurredTimes: " + timeoutExceptionOccurredTimes);
Console.WriteLine("onNextCalledTimes: " + onNextCalledTimes);
if ((concurrentCount - fireFakeEventCount) != timeoutExceptionOccurredTimes)
{
try
{
Console.WriteLine("Non timeout fired for these ids: " +
Enumerable.Range(0, concurrentCount)
.Except(timedOutEventIds).Except(Enumerable.Range(0, fireFakeEventCount)).Select(i => i.ToString())
.Aggregate((acc, n) => acc + "," + n));
}
catch (Exception ex) { Console.WriteLine("faild to output timedout ids..."); }
break;
}
if (fireFakeEventCount != onNextCalledTimes)
{
Console.WriteLine("onNextOccurredTimes assert failed");
break;
}
if ((concurrentCount - fireFakeEventCount) != timeoutExceptionOccurredTimes)
{
Console.WriteLine("timeoutExceptionOccurredTimes assert failed");
break;
}
}
Console.WriteLine("");
Console.WriteLine("");
Console.WriteLine("DONE!");
Console.ReadLine();
}
}
public class EventFireCenter
{
public static event EventHandler<GTCommandTerminalEventArg> OnTerminalEventArrived;
private static object syncObject = new object();
public static void TestFireDummyEventWithId(int id)
{
lock (syncObject)
{
var safe = OnTerminalEventArrived;
if (safe != null)
{
safe(null, new GTCommandTerminalEventArg(id));
}
}
}
}
public class UnsolicitedEventCenter
{
private readonly static IObservable<int> publisher;
static UnsolicitedEventCenter()
{
publisher = Observable.FromEventPattern<GTCommandTerminalEventArg>(typeof(EventFireCenter), "OnTerminalEventArrived")
.Select(s => s.EventArgs.Id);
}
private UnsolicitedEventCenter() { }
/// <summary>
/// Gets the Publisher property to start observe an observable sequence.
/// </summary>
public static IObservable<int> Publisher { get { return publisher; } }
}
public class GTCommandTerminalEventArg : System.EventArgs
{
public GTCommandTerminalEventArg(int id)
{
this.Id = id;
}
public int Id { get; private set; }
}
}
Most likely the Timeout is not triggering because you have it before the Where filter. This means that all events are flowing through and resetting the timer, and then most of the events get filtered by the Where clause. To your subscribing observer, it will seem like it never gets a result and the timeout never triggers. Move the Timeout to be after the Where and you should now have a system that times out individual observers if they do not get their expected event on time.

Producer Consumer Pattern How do you notify completion?

Working on .net 2.0 I need to implement the some threading and I was looking at a dummy examples but cannot find anything which implement eventnotification.
Need to know when all is done and also some sort of progress bar if you like.
I am been playing with following code by cannot see to get the event notification correctly.
How do I detect that I have finished the processing and possible updating the ui with what I have been doing?
Example code
class Program
{
static void Main(string[] args)
{
using (PCQueue q = new PCQueue(2))
{
q.TaskCompleted += new EventHandler(OnTaskCompleted);
q.PercentageCompleted += new EventHandler(OnPercentageCompleted);
for (int i = 1; i < 100; i++)
{
string itemNumber = i.ToString(); // To avoid the captured variable trap
q.EnqueueItem(itemNumber);
}
Console.WriteLine("Waiting for items to complete...");
Console.Read();
}
}
private static void OnPercentageCompleted(object sender, EventArgs e)
{
}
static void OnTaskCompleted(object sender, EventArgs e)
{
}
}
public class PCQueue : IDisposable
{
readonly object locker = new object();
Thread[] _workers;
Queue<string> _itemQ = new Queue<string>();
public PCQueue(int workerCount)
{
_workers = new Thread[workerCount];
// Create and start a separate thread for each worker
for (int i = 0; i < workerCount; i++)
{
(_workers[i] = new Thread(Consume)).Start();
}
}
public void EnqueueItem(string item)
{
lock (locker)
{
_itemQ.Enqueue(item); // We must pulse because we're
Monitor.Pulse(locker); // changing a blocking condition.
}
}
void Consume()
{
while (true) // Keep consuming until
{ // told otherwise.
string item;
lock (locker)
{
while (_itemQ.Count == 0) Monitor.Wait(locker);
item = _itemQ.Dequeue();
}
if (item == null) return; // This signals our exit.
DoSomething(item); // Execute item.
}
}
private void DoSomething(string item)
{
Console.WriteLine(item);
}
public void Dispose()
{
// Enqueue one null item per worker to make each exit.
foreach (Thread worker in _workers)
{
EnqueueItem(null);
}
}
//where/how can I fire this event???
public event EventHandler TaskCompleted;
protected void OnCompleted(EventArgs e)
{
if (this.TaskCompleted != null)
{
this.TaskCompleted(this, e);
}
}
//where/how can I fire this event???
public event EventHandler PercentageCompleted;
protected void OnPercentageCompleted(EventArgs e)
{
if (this.PercentageCompleted != null)
{
this.PercentageCompleted(this, e);
}
}
}
Any suggestions?
You can't raise the progress event inside your queue for the simple reason that the queue does not know the total number items which are supposed to be processed. So it can't calculate a percentage. You just stick something in and it gets processed.
What you could do is to raise a ItemProcessed event and subscribe to that. Then in your main program you can do the logic of counting how many items were processed so far in relation to how many are supposed to be processed.
You can raise the complete event just before you are returning from your Consume function. However you need to keep track of how many threads are still active as Brian said in his answer. I modified the code to reflect that.
So something along these lines:
...
private int _ActiveThreads;
public PCQueue(int workerCount)
{
_ActiveThreads = workerCount;
_workers = new Thread[workerCount];
// Create and start a separate thread for each worker
for (int i = 0; i < workerCount; i++)
{
(_workers[i] = new Thread(Consume)).Start();
}
}
void Consume()
{
while (true) // Keep consuming until
{ // told otherwise.
string item;
lock (locker)
{
while (_itemQ.Count == 0) Monitor.Wait(locker);
item = _itemQ.Dequeue();
}
if (item == null) // This signals our exit.
{
if (Interlocked.Decrement(ref _ActiveThreads) == 0)
{
OnCompleted(EventArgs.Empty);
}
return;
}
DoSomething(item); // Execute item.
OnItemProcessed();
}
}
public event EventHandler ItemProcessed;
protected void OnItemProcessed()
{
var handler = ItemProcessed;
if (handler != null)
{
handler(this, EventArgs.Empty);
}
}
...
Of course you might want to create some meaningfull event args and actually pass the item which was processed to the event.
Then in main:
...
static void Main(string[] args)
{
using (PCQueue q = new PCQueue(2))
{
q.ItemProcessed += ItemProcessed;
q.TaskCompleted += OnTaskCompleted;
for (int i = 1; i <= totalNumberOfItems; i++)
{
string itemNumber = i.ToString(); // To avoid the captured variable trap
q.EnqueueItem(itemNumber);
}
Console.WriteLine("Waiting for items to complete...");
Console.Read();
}
}
private static int currentProcessCount = 0;
private static int totalNumberOfItems = 100;
private static void ItemProcessed(object sender, EventArgs e)
{
currentProcessCount++;
Console.WriteLine("Progress: {0}%", ((double)currentProcessCount / (double)totalNumberOfItems) * 100.0);
}
static void OnTaskCompleted(object sender, EventArgs e)
{
Console.WriteLine("Done");
}
...
Needless to say that all that static stuff should go. This is just based on your example.
One more remark:
Your PCQueue currently requires that you enqueue as many null values as you have worker threads otherwise only one thread will quit and the others will wait until your process quits. You can change that by looking at the first item and only removing it when it is not null - thus leaving the marker there for all threads. So Consume would change to this:
void Consume()
{
while (true) // Keep consuming until
{ // told otherwise.
string item;
lock (locker)
{
while (_itemQ.Count == 0) Monitor.Wait(locker);
item = _itemQ.Peek();
if (item != null) _itemQ.Dequeue();
else Monitor.PulseAll(); // if the head of the queue is null then make sure all other threads are also woken up so they can quit
}
if (item == null) // This signals our exit.
{
if (Interlocked.Decrement(ref _ActiveThreads) == 0)
{
OnCompleted(EventArgs.Empty);
}
return;
}
DoSomething(item); // Execute item.
OnItemProcessed();
}
}
In your PCQueue class you will need to keep track of how many worker threads are still active and raise TaskCompleted only after all threads have been instructed to end.
void Consume()
{
while (true)
{
string item;
lock (locker)
{
while (_itemQ.Count == 0) Monitor.Wait(locker);
item = _itemQ.Dequeue();
}
if (item == null)
{
// activeThreads is set to the number of workers in the constructor.
if (Interlocked.Decrement(ref activeThreads) == 0)
{
// Take a snapshot of the event so that a null check + invocation is safe.
// This works because delegates are immutable.
var copy = TaskCompleted;
if (copy != null)
{
copy(this, new EventArgs());
}
}
return;
}
DoSomething(item); // Execute item.
}
}
A couple of other points:
Kudos for getting the blocking queue implemented correctly. Most people get it wrong.
Remember to marshal the TaskCompleted event handler back onto the UI thread before touching any UI controls.
You could raise PercentCompleted from DoSomething, but without a clear indication of how many items the queue is suppose to hold the value will not make sense. I second Chris' recommendation on this point.

speeding up windows file search with C#

i made a program that search logical drives to find a specific file .if user type file name
an click search button , searching begins , but i don't know how to stop searching in the middle of process.can you help me ?
You should perform the search on a background thread so that it doesn't block the UI. This article has a good introduction and walkthrough of the changes that you'll need to make to your app.
You need to use Backgroundworker class in .net. It executes on separate thread and it has inbuilt methods/properties for cancellation, report progress and lot more...
Have a look at following article to get started with it:
http://msdn.microsoft.com/en-us/library/system.componentmodel.backgroundworker.aspx
You need to run the search in a background thread (Using BackgroundWorker is the most convenient way to do this) then you can still handle input to cancel it.
When you are handling the cancel request you may need to use Thread.Abort on the executing thread or BackgroundWorker.CancelAsync() on the BackgroundWorker.
Alternatively you can have the executing thread check a variable while in the processing loop or at the start of a recursive function - to cancel you simple need to set this variable when handling the cancel request.
As others have mentioned, your solution may be to use BackgroundWorker with CancelAsync method.
Here is some working code which you could use with minor modifications:
class Program
{
static void Main(string[] args)
{
var search = new FileSearcher().FindFile(#"d:\users", "autofac.dll", f => Console.WriteLine(f.FullName), () => Console.WriteLine("Finished"));
Console.WriteLine("C - cancel, else - finish");
for (; ; )
{
var command = Console.ReadLine();
switch (command)
{
case "C":
search.Cancel();
break;
default:
return;
}
}
}
}
public class FileSearcher
{
public FileSearch FindFile(string searchPath, string fileName, Action<FileInfo> onFileFound, Action onSearchFinished)
{
var search = new FileSearch(new DirectoryInfo(searchPath), fileName);
search.FileFound += onFileFound;
search.Finished += onSearchFinished;
search.Run();
return search;
}
}
public class FileSearch
{
readonly BackgroundWorker _worker = new BackgroundWorker();
readonly DirectoryInfo _searchPath;
readonly string _template;
public FileSearch(DirectoryInfo searchPath, string template)
{
_searchPath = searchPath;
_template = template;
_worker.DoWork += _worker_DoWork;
_worker.RunWorkerCompleted += _worker_RunWorkerCompleted;
_worker.WorkerSupportsCancellation = true;
}
void _worker_DoWork(object sender, DoWorkEventArgs e)
{
foreach (var directory in GetPartiallyFlatDirectories(_searchPath, 4))
{
if (_worker.CancellationPending)
break;
foreach (var file in directory.GetFiles(_template, SearchOption.AllDirectories))
FileFound.Raise(file);
}
}
static IEnumerable<DirectoryInfo> GetPartiallyFlatDirectories(DirectoryInfo directory, int flatDepth)
{
if (flatDepth == 0)
{
yield return directory;
yield break;
}
foreach (var subDir in directory.GetDirectories())
{
var flattened = GetPartiallyFlatDirectories(subDir, flatDepth - 1);
if (!flattened.Any())
yield return subDir;
else
foreach (var flatDirectory in flattened)
yield return flatDirectory;
}
}
void _worker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
{
Finished.Raise();
}
public void Cancel()
{
_worker.CancelAsync();
}
public event Action<FileInfo> FileFound;
public event Action Finished;
public void Run()
{
_worker.RunWorkerAsync();
}
}
public static class DelegateExtensions
{
public static void Raise<T>(this Action<T> action, T obj)
{
if (action != null)
action(obj);
}
public static void Raise(this Action action)
{
if (action != null)
action();
}
}

Categories

Resources