I am writing a multithreaded application it is windows service. I have 20 folders. I create 15 threads onstart method. I want to achieve that; 15 threads go to folders 1,2,3,...,15 sequentially. When one thread finished, it creates another thread. This created thread must go 16.th folder. It must not go to working folders. How can I do this? That is, how can I be sure that two threads do not go the same folder?
Could you not just have a static variable that would be a counter for the folder name?
Something like:
private static int _folderNameCounter = 0;
private static readonly object _padlock = new object();
public static int GetFolderCounter()
{
lock(_padlock)
{
_folderNameCounter++;
return _folderNameCounter;
}
}
public static void Main()
{
for(int i = 0; i < 20; i++)
{
Task.Factory.StartNew(() =>
{
var path = #"c:\temp\" + GetFolderCounter();
Directory.CreateDirectory(path);
// add your own code for the thread here
});
}
}
Note: I've used the TPL instead of using Threads directly since I think that the TPL is a better solution. You can of course have specific requirements which can mean that Threads is the better solution for
your case.
Use a BlockingCollection<T> and fill the collection with the folder numbers. Each task handles an item of the collection, and the collection itself handles the multi-threading aspect so that each item is only handled by one consumer.
// Define the blocking collection with a maximum size of 15.
const int maxSize = 15;
var data = new BlockingCollection<int>(maxSize);
// Add the data to the collection.
// Do this in a separate task since BlockingCollection<T>.Add()
// blocks when the specified capacity is reached.
var addingTask = new Task(() => {
for (int i = 1; i <= 20; i++) {
data.Add(i);
}
).Start();
// Define a signal-to-stop bool
var stop = false;
// Create 15 handle tasks.
// You can change this to threads if necessary, but the general idea is that
// each consumer continues to consume until the stop-boolean is set.
// The Take method returns only when an item is/becomes available.
for (int t = 0; t < maxSize; t++) {
new Task(() => {
while (!stop) {
int item = data.Take();
// Note: the Take method will block until an item comes available.
HandleThisItem(item);
}
}).Start();
};
// Wait until you need to stop. When you do, set stop true
stop = true;
Related
By using the below code firstly some of the calls are not getting made lets say out of 250 , 238 calls are made and rest doesn't.Secondly I am not sure if the calls are made at the rate of 20 calls per 10 seconds.
public List<ShowData> GetAllShowAndTheirCast()
{
ShowResponse allShows = GetAllShows();
ShowCasts showCast = new ShowCasts();
showCast.showCastList = new List<ShowData>();
using (Semaphore pool = new Semaphore(20, 20))
{
for (int i = 0; i < allShows.Shows.Length; i++)
{
pool.WaitOne();
Thread t = new Thread(new ParameterizedThreadStart((taskId) =>
{
showCast.showCastList.Add(MapResponse(allShows.Shows[i]));
}));
pool.Release();
t.Start(i);
}
}
//for (int i = 0; i < allShows.Shows.Length; i++)
//{
// showCast.showCastList.Add(MapResponse(allShows.Shows[i]));
//}
return showCast.showCastList;
}
public ShowData MapResponse(Show s)
{
CastResponse castres = new CastResponse();
castres.CastlistResponse = (GetShowCast(s.id)).CastlistResponse;
ShowData sd = new ShowData();
sd.id = s.id;
sd.name = s.name;
if (castres.CastlistResponse != null && castres.CastlistResponse.Any())
{
sd.cast = new List<CastData>();
foreach (var item in castres.CastlistResponse)
{
CastData cd = new CastData();
cd.birthday = item.person.birthday;
cd.id = item.person.id;
cd.name = item.person.name;
sd.cast.Add(cd);
}
}
return sd;
}
public ShowResponse GetAllShows()
{
ShowResponse response = new ShowResponse();
string showUrl = ClientAPIUtils.apiUrl + "shows";
response.Shows = JsonConvert.DeserializeObject<Show[]>(ClientAPIUtils.GetDataFromUrl(showUrl));
return response;
}
public CastResponse GetShowCast(int showid)
{
CastResponse res = new CastResponse();
string castUrl = ClientAPIUtils.apiUrl + "shows/" + showid + "/cast";
res.CastlistResponse = JsonConvert.DeserializeObject<List<Cast>>(ClientAPIUtils.GetDataFromUrl(castUrl));
return res;
}
All the Calls should be made , but I am not sure where they are getting aborted and even please let me know how to check the rate of calls being made.
I'm assuming that your goal is to process all data about shows but no more than 20 at once.
For that kind of task you should probably use ThreadPool and limit maximum number of concurrent threads using SetMaxThreads.
https://learn.microsoft.com/en-us/dotnet/api/system.threading.threadpool?view=netframework-4.7.2
You have to make sure that collection that you are using to store your results is thread-safe.
showCast.showCastList = new List<ShowData>();
I don't think that standard List is thread-safe. Thread-safe collection is ConcurrentBag (there are others as well). You can make standard list thread-safe but it requires more code. After you are done processing and need to have results in list or array you can convert collection to desired type.
https://learn.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentbag-1?view=netframework-4.7.2
Now to usage of semaphore. What your semaphore is doing is ensuring that maximum 20 threads can be created at once. Assuming that this loop runs in your app main thread your semaphore has no purpose. To make it work you need to release semaphore once thread is completed; but you are calling thread Start() after calling Release(). That results in thread being executed outside "critical area".
using (Semaphore pool = new Semaphore(20, 20)) {
for (int i = 0; i < allShows.Shows.Length; i++) {
pool.WaitOne();
Thread t = new Thread(new ParameterizedThreadStart((taskId) =>
{
showCast.showCastList.Add(MapResponse(allShows.Shows[i]));
pool.Release();
}));
t.Start(i);
}
}
I did not test this solution; additional problems might arise.
Another issue with this program is that it does not wait for all threads to complete. Once all threads are started; program will end. It is possible (and in your case I'm sure) that not all threads completed its operation; this is why ~240 data packets are done when program finishes.
thread.Join();
But if called right after Start() it will stop main thread until it is completed so to keep program concurrent you need to create a list of threads and Join() them at the end of program. It is not the best solution. How to wait on all threads that program can add to ThreadPool
Wait until all threads finished their work in ThreadPool
As final note you cannot access loop counter like that. Final value of loop counter is evaluated later and with test I ran; code has tendency to process odd records twice and skip even. This is happening because loop increases counter before previous thread is executed and causes to access elements outside bounds of array.
Possible solution to that is to create method that will create thread. Having it in separate method will evaluate allShows.Shows[i] to show before next loop pass.
public void CreateAndStartThread(Show show, Semaphore pool, ShowCasts showCast)
{
pool.WaitOne();
Thread t = new Thread(new ParameterizedThreadStart((s) => {
showCast.showCastList.Add(MapResponse((Show)s));
pool.Release();
}));
t.Start(show);
}
Concurrent programming is tricky and I would highly recommend to do some exercises with examples on common pitfalls. Books on C# programming are sure to have a chapter or two on the topic. There are plenty of online courses and tutorials on this topic to learn from.
Edit:
Working solution. Still might have some issues.
public ShowCasts GetAllShowAndTheirCast()
{
ShowResponse allShows = GetAllShows();
ConcurrentBag<ShowData> result = new ConcurrentBag<ShowData>();
using (var countdownEvent = new CountdownEvent(allShows.Shows.Length))
{
using (Semaphore pool = new Semaphore(20, 20))
{
for (int i = 0; i < allShows.Shows.Length; i++)
{
CreateAndStartThread(allShows.Shows[i], pool, result, countdownEvent);
}
countdownEvent.Wait();
}
}
return new ShowCasts() { showCastList = result.ToList() };
}
public void CreateAndStartThread(Show show, Semaphore pool, ConcurrentBag<ShowData> result, CountdownEvent countdownEvent)
{
pool.WaitOne();
Thread t = new Thread(new ParameterizedThreadStart((s) =>
{
result.Add(MapResponse((Show)s));
pool.Release();
countdownEvent.Signal();
}));
t.Start(show);
}
I'm looking for some advice on writing unit tests for multi-threading in C#. Specifically, I want to check that an object is being locked correctly. However, in order to test this I need to assert against that object, which may have changed before the assert(s) are implemented (with the lock being released, another thread may change the object).
Using AutoResetEvent I have been able to control the flow in the unit test side, allowing me to effectively emulate the lock in the tested object. The issue with this is that I no longer need the lock for the test to pass.
What I'd like is to have a test that passes with the lock in and fails with it out.
Obviously, this is a simplified example. It's also .Net 4, so there is no async and await option (although if that would help, changing could be an option).
Suggestions welcome. Thanks.
Below is example code:
public class BasicClass
{
public int Val
{
get { lock (lockingObject) { return val; } }
private set { lock (lockingObject) { val = value; } }
}
private int val;
public BasicClass(int val = -1)
{
Val = val;
}
public void SetValue(int val)
{
Val = val;
}
private object lockingObject = new object();
}
This is the (NUnit) unit test:
[Test]
public void BasicClassTest()
{
for (int repeat = 0; repeat < 1000; repeat++) // Purely for dev testing and can get away with as no SetUp/TearDown
{
BasicClass b = new BasicClass();
int taskCount = 10;
Task[] tasks = new Task[taskCount];
var taskControl = new AutoResetEvent(false);
var resultControl = new AutoResetEvent(false);
int expected = -1;
for (int i = 0; i < taskCount; i++)
{
int temp = i;
tasks[temp] = new Task(() =>
{
taskControl.WaitOne(); // Hold there here until set
b.SetValue(temp);
expected = temp;
resultControl.Set(); // Allows asserts to be processed.
});
}
// Start each task
foreach (var t in tasks)
t.Start();
// Assert results as tasks finish.
for (int i = 0; i < taskCount; i++)
{
taskControl.Set(); // Unblock, allow one thread to proceed.
resultControl.WaitOne(); // Wait for a task to set a expected value
Assert.That(b.Val, Is.EqualTo(expected));
Console.WriteLine("b.Val = {0}, expected = {1}", b.Val, expected); // Output values to ensure they are changing
}
// Wait for all tasks to finish, but not forever.
Task.WaitAll(tasks, 1000);
}
}
As for other system functions like DateTime.Now, I prefer to abstract threading functions like sleep, mutex, signals and so on (yes, I know there are libraries for DateTime.Now and other system functions, but I think to abstract it is a better way).
So you end up with a kind of IThreadind interface with methods to Sleep and so on. The disadvantage is, that you can't use the handy lock statement in this case. You could have a method Lock(object) that returns you an IDisposable that you can use with the "using" statement, to get nearly the same comfort.
using(threading.Lock(lockObject))
{
...
}
Now you can Create a real implementation with the real functions and a Mock for your unit tests which is injected. So you could for example for your tests shortcut any sleep call to e few ms in order to speed up your tests. And you can verify that all functions where called that you expected.
Sounds like a lot of work? Think over, how many time you will spend to debug some nasty threading issue which from time to time crashes your production system with your customer running amok.
I am working through a Parallel Programming example on Race conditions
In the example they are demonstrating the isolation patter to deal with race conditions
Why is it in this example following that a race condition does not occur when creating the task and the stateObject is passed as part of the task creation
I understand that we use isolatedBalance to do the updateing ...but at the point where we assign the isolatedbalance = (int)stateObject could not another tasks finished balance be there i.e not 0 but 100 ???
So if there where enough tasks and that the task scheduler started an early task and it finished at a point when a later task is being created and assinged the account.Balance value would be 100 etc for when 1 of the tasks had finshed for a taks that was starting
class BankAccount
{
public int Balance { get; set; }
}
class Program
{
static void Main(string[] args)
{
var account = new BankAccount();
var tasks = new Task<int>[1000];
for (int i = 0; i < 1000; i++)
{
tasks[i] = new Task<int>((stateObject)=>
{
int isobalance = (int) stateObject;
for (int j = 0; j < 1000; j++)
{
isobalance ++;
}
return isobalance;
}, account.Balance);
tasks[i].Start();
}
Task.WaitAll(tasks);
for (int i = 0; i < 1000; i++)
{
account.Balance += tasks[i].Result;
}
Console.WriteLine("Epectecd valeu {0}, Counter value {1}",1000000,account.Balance);
// wait for input before exiting
Console.WriteLine("Press enter to finish");
Console.ReadLine();
}
}
The method that you have passed to the Task constructor does not update account.Balance it only uses the initial value of account.Balance. It does not update it. int is pass by value. From MSDN:
A value-type variable contains its data directly as opposed to a reference-type variable, which contains a reference to its data. Therefore, passing a value-type variable to a method means passing a copy of the variable to the method. Any changes to the parameter that take place inside the method have no affect on the original data stored in the variable. If you want the called method to change the value of the parameter, you have to pass it by reference, using the ref or out keyword. For simplicity, the following examples use ref.
Therefore account.Balance is not updated until after Task.WaitAll(tasks); is called. Task.WaitAll() causes the code to stop there until all tasks have finished. Only after that, once all the results have been computed. will account.Balance be updated with the values returned from tasks[i].Result.
It does not cause a race condition because you only copy the current value of account.Balance and assign it to a local variable inside the thread. Upon creation of each thread, they simply copy the current value of account.Balance on their stack and then to a local variable, but no thread actually changes it, they all work on their local copy. Imagine this to be like a method call. When you pass an int to a method, it is copied by value and then even if you modify it inside the method you will not see any changes outside.
Having said this, my favorite example to illustrate what you are asking is the very common "assign a unique id to each thread" problem. Consider these two cases:
Not thread-safe:
for(int i = 0; i < n; i++)
{
Thread t = new Thread(
o =>
{
int index = i;
// do whatever
});
t.Start();
}
This is not thread-safe because the main thread continues to loop over i while the threads are using it inside their code. When each thread t actually starts, i may have already reached n.
Thread-safe:
for(int i = 0; i < n; i++)
{
Thread t = new Thread(
o =>
{
int index = (int)o;
// do whatever
});
t.Start(i);
}
This is thread-safe given my initial explanation. Each thread receives the current value of i upon creation and copies it inside a local variable, such that the threads will correctly have ids 0, 1, ..., n-1. I hope this example makes it more clear.
I am trying to bypass the the wait64 handle limit that .net 3.5 imposes
I have seen this thread : Workaround for the WaitHandle.WaitAll 64 handle limit?
So I understand the general idea but I am having difficulty because I am not using a delegate but rather
I am basically working of this example :
http://msdn.microsoft.com/en-us/library/3dasc8as%28VS.80%29.aspx
This link http://www.switchonthecode.com/tutorials/csharp-tutorial-using-the-threadpool
is similar but again the int variable keeping track of the tasks is a member variable.
Where in the above example would I pass the threadCount integer?
Do I pass it in the callback method as an object? I think I am having trouble with the callback method and passing by reference.
Thanks Stephen,
That link is not entirely clear to me.
Let me post my code to help myself clarify:
for (int flows = 0; flows < NumFlows; flows++)
{
ResetEvents[flows] = new ManualResetEvent(false);
ICalculator calculator = new NewtonRaphson(Perturbations);
Calculators[flows] = calculator;
ThreadPool.QueueUserWorkItem(calculator.ThreadPoolCallback, flows);
}
resetEvent.WaitOne();
Where would I pass in my threadCount variable. I assume it needs to be decremented in calculator.ThreadPoolCallback?
You should not be using multiple wait handles to wait for the completion of multiple work items in the ThreadPool. Not only is it not scalable you will eventually bump into the 64 handle limit imposed by the WaitHandle.WaitAll method (as you have done already). The correct pattern to use in this situation is a counting wait handle. There is one available in the Reactive Extensions download for .NET 3.5 via the CountdownEvent class.
var finished = new CountdownEvent(1);
for (int flows = 0; flows < NumFlows; flows++)
{
finished.AddCount();
ICalculator calculator = new NewtonRaphson(Perturbations);
Calculators[flows] = calculator;
ThreadPool.QueueUserWorkItem(
(state) =>
{
try
{
calculator.ThreadPoolCallback(state);
}
finally
{
finished.Signal();
}
}, flows);
}
finished.Signal();
finished.Wait();
An anonymous method might be easiest:
int threadCount = 0;
for (int flows = 0; flows < NumFlows; flows++)
{
ICalculator calculator = new NewtonRaphson(Perturbations);
Calculators[flows] = calculator;
// We're about to queue a new piece of work:
// make a note of the fact a new work item is starting
Interlocked.Increment(ref threadCount);
ThreadPool.QueueUserWorkItem(
delegate
{
calculator.ThreadPoolCallback(flows);
// We've finished this piece of work...
if (Interlocked.Decrement(ref threadCount) == 0)
{
// ...and we're the last one.
// Signal back to the main thread.
resetEvent.Set();
}
}, null);
}
resetEvent.WaitOne();
I have an application that has many cases. Each case has many multipage tif files. I need to covert the tf files to pdf file. Since there are so many file, I thought I could thread the conversion process. I'm currently limiting the process to ten conversions at a time (i.e ten treads). When one conversion completes, another should start.
This is the current setup I'm using.
private void ConvertFiles()
{
List<AutoResetEvent> semaphores = new List<AutoResetEvet>();
foreach(String fileName in filesToConvert)
{
String file = fileName;
if(semaphores.Count >= 10)
{
WaitHandle.WaitAny(semaphores.ToArray());
}
AutoResetEvent semaphore = new AutoResetEvent(false);
semaphores.Add(semaphore);
ThreadPool.QueueUserWorkItem(
delegate
{
Convert(file);
semaphore.Set();
semaphores.Remove(semaphore);
}, null);
}
if(semaphores.Count > 0)
{
WaitHandle.WaitAll(semaphores.ToArray());
}
}
Using this, sometimes results in an exception stating the WaitHandle.WaitAll() or WaitHandle.WaitAny() array parameters must not exceed a length of 65. What am I doing wrong in this approach and how can I correct it?
There are a few problems with what you have written.
1st, it isn't thread safe. You have multiple threads adding, removing and waiting on the array of AutoResetEvents. The individual elements of the List can be accessed on separate threads, but anything that adds, removes, or checks all elements (like the WaitAny call), need to do so inside of a lock.
2nd, there is no guarantee that your code will only process 10 files at a time. The code between when the size of the List is checked, and the point where a new item is added is open for multiple threads to get through.
3rd, there is potential for the threads started in the QueueUserWorkItem to convert the same file. Without capturing the fileName inside the loop, the thread that converts the file will use whatever value is in fileName when it executes, NOT whatever was in fileName when you called QueueUserWorkItem.
This codeproject article should point you in the right direction for what you are trying to do: http://www.codeproject.com/KB/threads/SchedulingEngine.aspx
EDIT:
var semaphores = new List<AutoResetEvent>();
foreach (String fileName in filesToConvert)
{
String file = fileName;
AutoResetEvent[] array;
lock (semaphores)
{
array = semaphores.ToArray();
}
if (array.Count() >= 10)
{
WaitHandle.WaitAny(array);
}
var semaphore = new AutoResetEvent(false);
lock (semaphores)
{
semaphores.Add(semaphore);
}
ThreadPool.QueueUserWorkItem(
delegate
{
Convert(file);
lock (semaphores)
{
semaphores.Remove(semaphore);
}
semaphore.Set();
}, null);
}
Personally, I don't think I'd do it this way...but, working with the code you have, this should work.
Are you using a real semaphore (System.Threading)? When using semaphores, you typically allocate your max resources and it'll block for you automatically (as you add & release). You can go with the WaitAny approach, but I'm getting the feeling that you've chosen the more difficult route.
Looks like you need to remove the handle the triggered the WaitAny function to proceed
if(semaphores.Count >= 10)
{
int index = WaitHandle.WaitAny(semaphores.ToArray());
semaphores.RemoveAt(index);
}
So basically I would remove the:
semaphores.Remove(semaphore);
call from the thread and use the above to remove the signaled event and see if that works.
Maybe you shouldn't create so many events?
// input
var filesToConvert = new List<string>();
Action<string> Convert = Console.WriteLine;
// limit
const int MaxThreadsCount = 10;
var fileConverted = new AutoResetEvent(false);
long threadsCount = 0;
// start
foreach (var file in filesToConvert) {
if (threadsCount++ > MaxThreadsCount) // reached max threads count
fileConverted.WaitOne(); // wait for one of started threads
Interlocked.Increment(ref threadsCount);
ThreadPool.QueueUserWorkItem(
delegate {
Convert(file);
Interlocked.Decrement(ref threadsCount);
fileConverted.Set();
});
}
// wait
while (Interlocked.Read(ref threadsCount) > 0) // paranoia?
fileConverted.WaitOne();