I'm attempting to write a function which starts a thread for each "contact" and then queries (over the network) for results from that contact. I want my waiting function to wait at most 1.5 second for responses, and after that, simply terminate any remaining threads.
The issue I'm having is that the function is returning before all the threads are done, even though according to the logic, this should not be possible. The while loop in the main function should be waiting until all threads have fully completed, yet I get the following output:
FAIL: Storage test 1 exists 0 times in the DHT.
: Storage test 2 exists 0 times in the DHT.
Added storage test 1 to the entries.
Added storage test 2 to the entries.
(The FAIL lines comes from the main testing program seeing how many results were returned by Get().)
According to what I can see, this shouldn't be possible. Does anyone know where the race condition might be occurring (or any other assumptions I've made that are not correct)?
The function definition is as such:
public IList<Entry> Get(ID key)
{
ConcurrentBag<Entry> entries = new ConcurrentBag<Entry>();
List<Thread> threads = new List<Thread>();
foreach (Contact c in this.p_Contacts)
{
Thread t = new Thread(delegate()
{
try
{
FetchMessage fm = new FetchMessage(this, c, key);
fm.Send();
int ticks = 0;
// Wait until we receive data, or timeout.
while (!fm.Received && ticks < 1500)
{
Thread.Sleep(100);
ticks += 100;
}
if (fm.Received)
{
foreach (Entry e in fm.Values)
{
Console.WriteLine("Added " + e.Value + " to the entries.");
entries.Add(e);
}
if (entries.Count == 0)
Console.WriteLine("There were no entries to add.");
}
else
Console.WriteLine("The node did not return in time.");
}
catch (Exception e)
{
Console.WriteLine(e);
}
}
);
t.IsBackground = false;
t.Start();
}
while (true)
{
bool stopped = true;
foreach (Thread t in threads)
{
if (t.ThreadState != ThreadState.Stopped)
stopped = false;
}
if (stopped)
break;
Thread.Sleep(100);
}
return new List<Entry>(entries.ToArray());
}
Looks like you're never putting the Thread (t) in the List<Thread> (threads). Your foreach loop never executes.
The main thread just waits 100ms and continues.
#Toby has the correct answer, but if I may introduce some other things to improve the code. Essentially, you are manually managing your own ThreadPool and timeouts--and this is something that .Net provides for you out of the box. See: http://msdn.microsoft.com/en-us/library/system.threading.threadpool(v=VS.100).aspx
If you combine your ThreadPool with a .Net 4 Barrier, you can simplify the code a lot. Essentially, a Barrier will block all threads until they sync up. When you pass in the same barrier to your threads and sync up at the end, you can pause your main thread until all the worker threads are done. The refactored code would look like this:
// For the number of threads + 1 for the main thread
Barrier barrier = new Barrier(this.p_Contacts.count + 1);
ConcurrentBag<Entry> entries = new ConcurrentBag<Entry>();
foreach (Contact c in this.p_Contacts)
{
ThreadPool.RegisterWaitForSingleObject(
new EventWaitHandle(false, EventResetMode.AutoReset),
(stateInfo,timedOut) => {
try
{
FetchMessage fm = new FetchMessage(this, c, key);
fm.Send();
while(!fm.Received || !timedOut)
{
Thread.Sleep(100);
}
if(fm.Received)
{
foreach (Entry e in fm.Values)
{
entries.Add(e);
Console.WriteLine("Added " + e.Value + " to the entries.");
}
// avoid counting other thread's work
if (fm.Values.count == 0)
{
Console.WriteLine("There were no entries to add.");
}
}
else
{
Console.WriteLine("The node did not return in time.");
}
barrier.SignalAndWait();
}
catch(Exception e)
{
Console.WriteLine(e);
}
}, null, TimeSpan.FromSeconds(1.5), true);
);
}
// This limits total time waited to only 1.5 seconds
barrier.SignalAndWait(TimeSpan.FromSeconds(1.5));
return new List<Entry>(entries.ToArray());
Instead of manually managing the spin locks like you were doing, let .Net do it for you.
The Threads don't get added to your list so the while loop will break right away?
The solution to this problem was to use a ConcurrentDictionary to keep track of the contacts that had their threads finished:
public IList<Entry> Get(ID key)
{
ConcurrentBag<Entry> entries = new ConcurrentBag<Entry>();
ConcurrentDictionary<Contact, bool> done = new ConcurrentDictionary<Contact, bool>();
List<Thread> threads = new List<Thread>();
foreach (Contact c in this.p_Contacts)
{
Thread t;
ThreadStart ts = delegate()
{
try
{
FetchMessage fm = new FetchMessage(this, c, key);
fm.Send();
int ticks = 0;
// Wait until we receive data, or timeout.
while (!fm.Received && ticks < 1500)
{
Thread.Sleep(100);
ticks += 100;
}
if (fm.Received)
{
foreach (Entry e in fm.Values)
{
Console.WriteLine("Added " + e.Value + " to the entries.");
entries.Add(e);
}
if (entries.Count == 0)
Console.WriteLine("There were no entries to add.");
}
else
Console.WriteLine("The node did not return in time.");
Thread.MemoryBarrier();
done[c] = true;
}
catch (Exception e)
{
Console.WriteLine(e);
}
};
t = new Thread(ts);
done[c] = false;
t.IsBackground = true;
t.Start();
}
while (true)
{
bool stopped = true;
foreach (Contact c in this.p_Contacts)
{
if (!done[c])
stopped = false;
}
if (stopped)
break;
Thread.Sleep(100);
}
return new List<Entry>(entries.ToArray());
}
Related
I am trying to splice the work on multiple threads using ThreadPooling. I wanna use every available thread to assign (and calculate in a different part of the program down the line) the output of an array member to another array members.
It does work, but it is much slower than just adding them on a single thread. Is my usage wrong or is this operation too simple for multithreading?
arrayI and arrayX variables are in classScope, i couldnt pass them as QueueUserWorkItem argument without converting them in setNeuronInput.
if (layerType != 0)
{
for (arrayI = 0; arrayI < layerSize -1 ; arrayI++)
{
for (arrayX = 0; arrayX < network.Layers[layerIndex - 1].layerSize - 1; arrayX++)
{
ThreadPool.QueueUserWorkItem(new WaitCallback(setNeuronInput), null);
//Neurons[i].input[x] = _network.Layers[layerIndex - 1].Neurons[x].output;
}
}
}
//ThreadPool.GetAvailableThreads(out availableThreads, out placeHolder);
//while (availableThreads != maxThreads)
//{
// ThreadPool.GetAvailableThreads(out availableThreads, out placeHolder);
//}
//return;
}
public void setNeuronInput(object o)
{
try
{
//Console.WriteLine("Thread is working");
Neurons[arrayI].input[arrayX] = network.Layers[layerIndex - 1].Neurons[arrayX].output;
}
catch(Exception e)
{
Console.WriteLine(e);
Console.WriteLine("ArrayI is : " + arrayI);
Console.WriteLine("ArrayX is : " + arrayX);
Console.ReadLine();
}
}
I'm trying to build a Windows Forms tool that runs queries asynchronously.
The app has a datagridview with 30 possible queries to run. The user checks the queries he wants to execute, say 10 queries, and hits a button.
The app has a variable called maxthreads = 3 (for the sake of discussion) that indicates how many threads can be used to async run the queries. The queries run on a production environment and we don't want to overload the system with too many threads running in the same time. Each query runs for an average of 30 sec. (some 5 min., others 2 sec.)
In the datagridview there is an image column containing an icon that depicts the status of each query (0- Available to be run, 1-Selected for running, 2- Running, 3- Successfully completed, -1 Error)
I need to be able to communicate with the UI every time a query starts and finishes. Once a query finishes, the results are being displayed in a datagridview contained in a Tabcontrol (one tab per query)
The approach: I was thinking to create a number of maxthread backgroundworkers and let them run the queries. As a backgroundworker finishes it communicates to the UI and is assigned to a new query and so on until all queries have been run.
I tried using an assignmentWorker that would dispatch the work to the background workers but don't know how to wait for all threads to finish. Once a bgw finishes it reports progress on the RunWorkerCompleted event to the assignmentWorker, but that one has already finished.
In the UI thread I call the assignment worker with all the queries that need to be run:
private void btnRunQueries_Click(object sender, EventArgs e)
{
if (AnyQueriesSelected())
{
tcResult.TabPages.Clear();
foreach (DataGridViewRow dgr in dgvQueries.Rows)
{
if (Convert.ToBoolean(dgr.Cells["chk"].Value))
{
Query q = new Query(dgr.Cells["ID"].Value.ToString(),
dgr.Cells["Name"].Value.ToString(),
dgr.Cells["FileName"].Value.ToString(),
dgr.Cells["ShortDescription"].Value.ToString(),
dgr.Cells["LongDescription"].Value.ToString(),
dgr.Cells["Level"].Value.ToString(),
dgr.Cells["Task"].Value.ToString(),
dgr.Cells["Importance"].Value.ToString(),
dgr.Cells["SkillSet"].Value.ToString(),
false,
new Dictionary<string, string>()
{ { "#ClntNb#", txtClntNum.Text }, { "#Staff#", "100300" } });
qryList.Add(q);
}
}
assignmentWorker.RunWorkerAsync(qryList);
}
else
{
MessageBox.Show("Please select at least one query.",
"Warning",
MessageBoxButtons.OK,
MessageBoxIcon.Information);
}
}
Here is the AssignmentWorker:
private void assignmentWorker_DoWork(object sender, DoWorkEventArgs e)
{
foreach (Query q in (List<Query>)e.Argument)
{
while (!q.Processed)
{
for (int threadNum = 0; threadNum < maxThreads; threadNum++)
{
if (!threadArray[threadNum].IsBusy)
{
threadArray[threadNum].RunWorkerAsync(q);
q.Processed = true;
assignmentWorker.ReportProgress(1, q);
break;
}
}
//If all threads are being used, sleep awhile before checking again
if (!q.Processed)
{
Thread.Sleep(500);
}
}
}
}
All bgw run the same event:
private void backgroundWorkerFiles_DoWork(object sender, DoWorkEventArgs e)
{
try
{
Query qry = (Query)e.Argument;
DataTable dtNew = DataAccess.RunQuery(qry).dtResult;
if (dsQryResults.Tables.Contains(dtNew.TableName))
{
dsQryResults.Tables.Remove(dtNew.TableName);
}
dsQryResults.Tables.Add(dtNew);
e.Result = qry;
}
catch (Exception ex)
{
}
}
Once the Query has returned and the DataTable has been added to the dataset:
private void backgroundWorkerFiles_RunWorkerCompleted(object sender,
RunWorkerCompletedEventArgs e)
{
try
{
if (e.Error != null)
{
assignmentWorker.ReportProgress(-1, e.Result);
}
else
{
assignmentWorker.ReportProgress(2, e.Result);
}
}
catch (Exception ex)
{
int o = 0;
}
}
The problem I have is that the assignment worker finishes before the bgw finish and the call to assignmentWorker.ReportProgress go to hell (excuse my French).
How can I wait for all the launched bgw to finish before finishing the assignment worker?
Thank you!
As noted in the comment above, you have overcomplicated your design. If you have a specific maximum number of tasks (queries) that should be executing concurrently, you can and should simply create that number of workers, and have them consume tasks from your queue (or list) of tasks until that queue is empty.
Lacking a good Minimal, Complete, and Verifiable code example that concisely and clearly illustrates your specific scenario, it's not feasible to provide code that would directly address your question. But, here's an example using a List<T> as your original code does, which will work as I describe above:
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
namespace TestSO42101517WaitAsyncTasks
{
class Program
{
static void Main(string[] args)
{
Random random = new Random();
int maxTasks = 30,
maxActive = 3,
maxDelayMs = 1000,
currentDelay = -1;
List<TimeSpan> taskDelays = new List<TimeSpan>(maxTasks);
for (int i = 0; i < maxTasks; i++)
{
taskDelays.Add(TimeSpan.FromMilliseconds(random.Next(maxDelayMs)));
}
Task[] tasks = new Task[maxActive];
object o = new object();
for (int i = 0; i < maxActive; i++)
{
int workerIndex = i;
tasks[i] = Task.Run(() =>
{
DelayConsumer(ref currentDelay, taskDelays, o, workerIndex);
});
}
Console.WriteLine("Waiting for consumer tasks");
Task.WaitAll(tasks);
Console.WriteLine("All consumer tasks completed");
}
private static void DelayConsumer(ref int currentDelay, List<TimeSpan> taskDelays, object o, int workerIndex)
{
Console.WriteLine($"worker #{workerIndex} starting");
while (true)
{
TimeSpan delay;
int delayIndex;
lock (o)
{
delayIndex = ++currentDelay;
if (delayIndex < taskDelays.Count)
{
delay = taskDelays[delayIndex];
}
else
{
Console.WriteLine($"worker #{workerIndex} exiting");
return;
}
}
Console.WriteLine($"worker #{workerIndex} sleeping for {delay.TotalMilliseconds} ms, task #{delayIndex}");
System.Threading.Thread.Sleep(delay);
}
}
}
}
In your case, each worker would report progress to some global state. You don't show the ReportProgress handler for your "assignment" worker, so I can't say specifically what this would look like. But presumably it would involve passing either -1 or 2 to some method that knows what to do with those values (i.e. what would otherwise have been your ReportProgress handler).
Note that the code can simplified somewhat, particularly where the individual tasks are consumed, if you use an actual queue data structure for the tasks. That approach would look something like this:
using System;
using System.Collections.Concurrent;
using System.Threading.Tasks;
namespace TestSO42101517WaitAsyncTasks
{
class Program
{
static void Main(string[] args)
{
Random random = new Random();
int maxTasks = 30,
maxActive = 3,
maxDelayMs = 1000,
currentDelay = -1;
ConcurrentQueue<TimeSpan> taskDelays = new ConcurrentQueue<TimeSpan>();
for (int i = 0; i < maxTasks; i++)
{
taskDelays.Enqueue(TimeSpan.FromMilliseconds(random.Next(maxDelayMs)));
}
Task[] tasks = new Task[maxActive];
for (int i = 0; i < maxActive; i++)
{
int workerIndex = i;
tasks[i] = Task.Run(() =>
{
DelayConsumer(ref currentDelay, taskDelays, workerIndex);
});
}
Console.WriteLine("Waiting for consumer tasks");
Task.WaitAll(tasks);
Console.WriteLine("All consumer tasks completed");
}
private static void DelayConsumer(ref int currentDelayIndex, ConcurrentQueue<TimeSpan> taskDelays, int workerIndex)
{
Console.WriteLine($"worker #{workerIndex} starting");
while (true)
{
TimeSpan delay;
if (!taskDelays.TryDequeue(out delay))
{
Console.WriteLine($"worker #{workerIndex} exiting");
return;
}
int delayIndex = System.Threading.Interlocked.Increment(ref currentDelayIndex);
Console.WriteLine($"worker #{workerIndex} sleeping for {delay.TotalMilliseconds} ms, task #{delayIndex}");
System.Threading.Thread.Sleep(delay);
}
}
}
}
I try to synchronize three threads (named "1", "2" and "3") using Semaphore class. They have to print a string into console respecting consequence: 1->2->3. Here is my code:
class MyThread
{
public Thread Thrd;
static Semaphore sem = new Semaphore(1, 1);
static int flag = 1;
public MyThread(string name)
{
Thrd = new Thread(this.Run);
Thrd.Name = name;
Thrd.Start();
}
void Run()
{
sem.WaitOne();
if (Convert.ToInt32(Thrd.Name) == flag)
{
Console.WriteLine("Thread " + Thrd.Name);
flag++;
}
if (flag == 4)
flag = 1;
Thread.Sleep(300);
sem.Release();
}
}
class SemaphoreDemo
{
static void Main()
{
for (int i = 0; i < 10; i++)
{
MyThread mt1 = new MyThread("1");
MyThread mt2 = new MyThread("2");
MyThread mt3 = new MyThread("3");
mt1.Thrd.Join();
mt2.Thrd.Join();
mt3.Thrd.Join();
}
}
}
But sometimes strings from threads #2 and #3 are not seen. Where is my mistake and how can I fix this problem?
Thank you very much!
The problem is that sometimes a thread will acquire the semaphore out of order, and you don't have any retry logic. Take a look at your Run method.
void Run()
{
sem.WaitOne();
if (Convert.ToInt32(Thrd.Name) == flag)
{
Console.WriteLine("Thread " + Thrd.Name);
flag++;
}
if (flag == 4)
flag = 1;
Thread.Sleep(300);
sem.Release();
}
Now, what happens if the thread named "3" acquires the semaphore first? flag is equal to 1, so the conditional code won't be executed. The thread will just sleep for 300 milliseconds and then exit. If you want this to work, you have to make the thread retry:
void Run()
{
bool success = false;
while (!success)
{
sem.WaitOne();
if (Convert.ToInt32(Thrd.Name) == flag)
{
Console.WriteLine("Thread " + Thrd.Name);
flag++;
success = true;
}
sem.Release();
if (!success)
{
// let somebody else try
Thread.Sleep(300);
}
}
}
That will make your sample work as expected.
I suspect that this is just an exercise to see how threading and semaphores work. Note, however, that a Mutex is usually more appropriate than a Semaphore with a maximum count of 1.
Note also that there are other ways to make threads execute in sequence, although truthfully if you're going to sequence your threads then you probably don't need more than one thread. Unless those threads are doing other things and they only need to sequence once--or infrequently.
I have one problem. In my project i need for more then 100 threads, that's why i prefer to use ThreadPool. here is a part of code, but in this case i have got a lot of memory usage and my form is very laggy, cuz of A lot of BeginInvoke calls(i suppose).
Is there any solutions for this problem?
public void launch()
{
while (data.Count > 0)
{
string[] part;
if (data.Count> 1000)
{
part = data.Take(1000).ToArray();
data = data.Skip(1000).ToList();
}
else
{
part = data.Take(data.Count).ToArray(); data = data.Skip(1000).ToList();
}
foreach (var input in part)
{
try
{
char splitter = ':';
if (input.Contains(';')) splitter = ';';
string login = input.Split(splitter)[0];
string pass = input.Split(splitter)[1];
EncryptCore ec = new EncryptCore(new byte[15]);
PacketSend ps = new PacketSend(ec, "");
ps._login = login;
ps._password = pass;
ps.Brutted+=ps_Parsed;
ps.Failed+=ps_Failed;
ThreadPool.QueueUserWorkItem(ps.Parse);
}
catch { Interlocked.Increment(ref curr); }
}
Thread.Sleep(1000);
}
data.Clear();
}
private void ps_Brutted(User Account)
{
toGet.Add(Account);
Interlocked.Increment(ref good);
goodLabl.BeginInvoke(new Action(delegate()
{
goodLabl.Text = "Good: " + good;
}));
Update();
}
private void Update()
{
try
{
Interlocked.Increment(ref curr);
progLabel.BeginInvoke(new Action(delegate()
{
progLabel.Text = (double.Parse(curr.ToString()) / double.Parse(max.ToString())).ToString("#%");
}));
progressBar.BeginInvoke(new Action(delegate()
{
progressBar.Text = (double.Parse(curr.ToString()) / double.Parse(max.ToString()) * 100).ToString("#");
}));
checkedLabl.BeginInvoke(new Action(delegate()
{
checkedLabl.Text = "Checked: " + curr + " / " + max;
}));
}
catch { }
}
So you have thousands of tasks, and each time one of them completes you update the UI by scheduling 4 updates on the UI thread. What you might consider doing is having the UI update on a timer-based schedule (try 100ms or 200ms) based on the integers that are updating.
Aside from that, you've got some strange numeric operations going on. You can just cast an int to a double, you don't have to round-trip through a String.
In my multi threaded web app I invoke in the ThreadPool SomeMethod which can throw an exception. Suppose I want to make a few attempts if it causes an exception at first call. I decide to use System.Timers.Timer inside my action for attempts. Can I use the code below? Is it safely?
static void Caller()
{
ThreadPool.QueueUserWorkItem(action =>
{
try
{
SomeMethod();
Console.WriteLine("Done.");
}
catch
{
var t = new System.Timers.Timer(1000);
t.Start();
var count = 0;
t.Elapsed += new System.Timers.ElapsedEventHandler((o, a) =>
{
var timer = o as System.Timers.Timer;
count++;
var done = false;
Exception exception = null;
try
{
Console.WriteLine(count);
SomeMethod();
done = true;
}
catch (Exception ex)
{
exception = ex;
}
if (done || count == 10)
{
Console.WriteLine(String.Format("Stopped. done: {0}, count: {1}", done, count));
t.Stop();
if (!done) throw exception;
}
});
}
});
Thread.Sleep(100000);
}
static void SomeMethod()
{
var x = 1 / new Random().Next(0, 2);
}
You should Dispose each Timer after use, that's for sure. But, probably you could do something even simpler:
static void Main()
{
ThreadPool.QueueUserWorkItem(action =>
{
while (TrySomeMethod() == false)
Thread.Sleep(1000);
});
// wait here
Console.Read();
}
static bool TrySomeMethod()
{
try
{
SomeMethod();
return true;
}
catch
{
return false;
}
}
I do not think that using a timer in a thread pool thread is a safe approach. I may be wrong, but the timer will raise its elapsed event when the thread method has already been finished to execute. In this case, the exception will be thrown. Also, I do not see that you are not disposing the timer which leads to resource leaks. If you explain why you need the timer, I will try to find a safe solution...
I don't see the point of using a Timer within a ThreadPool queue, because the ThreadPool would spawn a new thread, and the Timer would spawn a new thread as well.
I would just have a loop within that delegate, because it would not block the main thread either way. Groo showed a good example of that.