Develop Reference

How to iterate dictionary in "await foreach"

Help with a small problem...
I have a method that returns a dictionary. I need to rewrite it in such a way that I can enumerate the result of this method using await foreach. Please help me, something is not working for me at all.
It's my method:
public IDictionary<long, long> TransformListInDictionary(IList<string> list)
{
var result = new Dictionary<long, long>();
foreach (var member in list)
{
var idx = member.IndexOf(':');
var key = member.Substring(0, idx);
var value = member.Substring(idx + 1);
result.Add(Convert.ToInt64(key), Convert.ToInt64(value));
}
return result;
}
To use a dictionary in an await foreach loop, this method requires a return value of IAsyncEnumerable<KeyValuePair<long,long>>.
Therefore, I have a question, how to rewrite the method published above is the return value.
Why do I need it.
I have some code which I am posting below. I'll try to describe my idea.
When the code enters the for loop, it does some work and instantiates the dictionary. Which is further processed in the foreach loop.
var logic = new AllLogic();
var variable = Convert.ToInt32(Console.ReadLine());
var randomSecundForPause = new Random();
for (int i = 0; i < variable; i++)
{
var list = new List<string>();
var dict = logic.TransformListInDictionary(list);
//some code
foreach (var item in dict)
{
try
{
//some code
Thread.Sleep(randomSecundForPause.Next(100000, 150000));
}
catch (Exception e)
{
//some code
Thread.Sleep(randomSecundForPause.Next(100000, 150000));
}
}
}
I would like this foreach loop to run in the background and the main code flow to go to a new iteration of the for loop.
As I understand it, I need to replace foreach with await foreach.

What you want is probably an asynchronous iterator:
#pragma warning disable CS1998
public async IAsyncEnumerable<KeyValuePair<long, long>> ToAsyncEnumerable(
#pragma warning restore CS1998
IList<string> list)
{
foreach (var member in list)
{
var idx = member.IndexOf(':');
var key = member.Substring(0, idx);
var value = member.Substring(idx + 1);
yield return KeyValuePair.Create(Convert.ToInt64(key), Convert.ToInt64(value));
}
}
The method must be async, it must have IAsyncEnumerable<X> as return type, and it must contain the yield contextual keyword.
The #pragma warning disable CS1998 is needed in order to suppress a warning about an async method that lacks the await keyword. Without it the program will still compile, but the C# compiler will emit a warning.

I would like this foreach loop to run in the background and the main
code flow to go to a new iteration of the for loop.
You don't need an await foreach. The for loop body would not continue to the next iteration once it hits an await of any kind. Instead, it would asynchronously wait for whatever it is to finish, and only when it does will the next iteration of the for loop start.
If you want to understand it better, try running this code:
for(int i = 0; i < 10; i++)
{
Console.WriteLine($"{i} Started");
await Task.Delay(2000);
Console.WriteLine($"{i} Finished");
}
If you want the foreach body to run in the background, you need to wrap the body inside a Task.Run(...) call. This call would return a Task, so store the Task in a collection and await Task.WhenAll(...) afterwards.
Would something like this:
var tasks = new List<Task>(variable);
for (int i = 0; i < variable; i++)
{
var list = new List<string>();
var dict = logic.TransformListInDictionary(list);
//some code
var task = Task.Run(async () =>
{
foreach (var item in dict)
{
try
{
//some code
await Task.Delay(randomSecundForPause.Next(100000, 150000)); // This is better because the thread is not blocked
}
catch (Exception e)
{
//some code
await Task.Delay(randomSecundForPause.Next(100000, 150000));
}
}
});
tasks.Add(task);
}
await Task.WhenAll(tasks);
Notice that the Dictionary class is thread-safe for multiple readers, but not for multiple writers, so depending on what //some code does you might want to consider a ConcurrentDictionary.

Run same code multiple times in parallel with different parameter

This very simple example:
int numLanes = 8;
var tasks = new List<Task>();
for (var i = 0; i < numLanes; ++i)
{
var t = new Task(() =>
{
Console.WriteLine($"Lane {i}");
});
tasks.Add(t);
}
tasks.ForEach((t) => t.Start());
Task.WaitAll(tasks.ToArray());
Produces:
Lane 8
Lane 8
Lane 8
Lane 8
Lane 8
Lane 8
Lane 8
Lane 8
Which is not as expected, the parameter i isn't passed correctly. I had thought to use Action<int> to wrap the code but couldn't see how I would. I do not want to write a dedicated method like Task CreateTask(int i) I'm interested how to do it using lambdas.
What is normal way to do this - spin up the same code a bunch of times in parallel with a different parameter value?

You've got a captured loop variable i, try to add temp variable inside a loop and pass it to the Task
for (var i = 0; i < numLanes; ++i)
{
var temp = i;
var t = new Task(() =>
{
Console.WriteLine($"Lane {temp}");
});
tasks.Add(t);
}
Further reading How to capture a variable in C# and not to shoot yourself in the foot. foreach loop has the same behavior before C# 5, but according to link above
with the release of the C# 5.0 standard this behavior was changed by
declaring the iterator variable inside every loop iteration, not
before it on the compilation stage, but for all other constructions
similar behavior remained without any changes
So, you may use foreach without temp variable

You need to capture the value inside the for loop otherwise all of the Tasks are still referring to the same object:
for (var i = 0; i < numLanes; ++i)
{
var innerI = I; // Copy the variable here
var t = new Task(() =>
{
Console.WriteLine($"Lane {innerI}");
});
tasks.Add(t);
}
See here for more info.

You could use LINQ to create a closure for each lambda you pass to the Task constructor:
var tasks = Enumerable.Range(0, numLanes - 1)
.Select(i => new Task(() => Console.WriteLine($"Lane {i}")));

Another approach (without introducing additional variable inside for loop) is to use constructor Task(Action<object>, object):
int numLanes = 8;
var tasks = new List<Task>();
for (int i = 0; i < numLanes; ++i)
{
// Variable "i" is passed as an argument into Task constructor.
var t = new Task(arg =>
{
Console.WriteLine("Lane {0}", arg);
}, i);
tasks.Add(t);
}
tasks.ForEach((t) => t.Start());
Task.WaitAll(tasks.ToArray());
In C# 5 and later foreach loop introduces a new variable on each iteration. Therefore in C# 5 and later it is possible to use foreach to create tasks where each task captures its own loop variable (also no need to introduce additional variable inside loop):
int numLanes = 8;
var tasks = new List<Task>();
foreach (int i in Enumerable.Range(0, numLanes))
{
// A new "i" variable is introduced on each iteration.
// Therefore each task captures its own variable.
var t = new Task(() =>
{
Console.WriteLine("Lane {0}", i);
});
tasks.Add(t);
}
tasks.ForEach((t) => t.Start());
Task.WaitAll(tasks.ToArray());

How to create working thread that start on signal and also rest

I want to create a working thread that starts on signal(task was added to shared task list), and will rest when done.
Requirements:
Other system threads can add task any time
the working thread should rest if it has nothing to-do
if more tasks are added while the working thread is active, it should complete them too.
working thread can rest for hours -> work arrive like raindrops(sometime we have a storm)
After thread adds new task to the workingList(shared list) it signals(AutoRestEvent.Set()) the working-thread to start working.
I have race condition between the Set() and the WaitOne() functions.
public static void AddWork(object obj)
{
Monitor.Enter(_syncO);
_workingList.Add(obj);
_signal.Set();
Monitor.Exit(_syncO);
}
static object _syncO = new object();
static AutoResetEvent _signal = new AutoResetEvent(false);
static List<object> _workingList = new List<object>();
static void DoWork()
{
Thread tradeThread1 = new Thread(() =>
{
Thread.CurrentThread.IsBackground = true;
string tradeMessage = string.Empty;
while (true)
{
Monitor.Enter(_syncO);
var arr = _workingList.ToArray();
Monitor.Exit(_syncO);
// race condition when the set happens just before the
// thread was locked
if (arr.Count() == 0)
_signal.WaitOne();
Monitor.Enter(_syncO);
arr = _workingList.ToArray();
Monitor.Exit(_syncO);
int count = 0;
var deleteList = new List<object>();
while (true)
{
foreach (var item in arr)
{
// the value is changing every iteration.
// this is why I need the Sleep at the end
bool b = Handle4(item);
if (b)
deleteList.Add(item);
}
if (count == 100)
break;
Thread.Sleep(100);
count++;
}
// remove done tasks from _workingList
RemoveItems(deleteList);
// we can't close, so we re-set footprints(notifications) on our price cable. -> execute on broker tick(the right tick)
// reHang trade on cable
foreach (var item in _workingList)
{
// re-use the undeleted tasks
}
}
});
tradeThread1.Start();
}
Based on the help of #John Wu I come up with the following solution:
The BlockingCollection will act as gate for the working thread. each iteration will copy the new task
private static void AddWork(object tap)
{
queue.Add(tap);
}
private static BlockingCollection<object> queue = new BlockingCollection<object>();
static void Work()
{
Thread tradeThread1 = new Thread(() =>
{
while (true)
{
var workingList = new List<object>();
var deleteList = new List<object>();
var reEvaluateList = new List<object>();
while (true)
{
if (workingList.Count() == 0)
{
// thread will wait until new work arrives -> it will start working again on the first task to come in.
workingList.Add(queue.Take());
}
foreach (var item in workingList)
{
bool b = Handle4(item);
if (b)
deleteList.Add(item);
else
item.ExitCounter++;
if (item.ExitCounter == 1000)
reEvaluateList.Add(item);
}
RemoveItems(deleteList, workingList);
// we can't close, so we re-set
// we reevaluate tasks that are working for X amount of time and didn't finish
foreach (var item in reEvaluateList)
ReEvaluate(item);
RemoveItems(reEvaluateList, workingList);
// wait.. the item change-over-time, so a wait is a type of calculation.
Thread.Sleep(100);
// we want to avoid locking if we still have task to process
if (queue.Count() == 0)
continue;
// add new work to local list
workingList.Add(queue.Take());
}
}
});
tradeThread1.Start();
}
It feels a-little-bit messy. Any ideas on how to make it better?

How to prevent threads using the same variables

I have a multi-line textbox and I want to process each line with multi threads.
The textbox could have a lot of lines (1000+), but not as many threads. I want to use custom amount of threads to read all those 1000+ lines without any duplicates (as in each thread reading UNIQUE lines only, if a line has been read by other thread, not to read it again).
What I have right now:
private void button5_Click(object sender, EventArgs e)
{
for (int i = 0; i < threadCount; i++)
{
new Thread(new ThreadStart(threadJob)).Start();
}
}
private void threadJob()
{
for (int i = 0; i < txtSearchTerms.Lines.Length; i++)
{
lock (threadLock)
{
Console.WriteLine(txtSearchTerms.Lines[i]);
}
}
}
It does start the correct amount of threads, but they all read the same variable multiple times.

Separate data collection and data processing and next possible steps after calculation. You can safely collect results calculated in parallel by using ConcurrentBag<T>, which is simply thread-safe collection.
Then you don't need to worry about "locking" objects and all lines will be "processed" only once.
1. Collect data
2. Execute collected data in parallel
3. Handle calculated result
private string Process(string line)
{
// Your logic for given line
}
private void Button_Click(object sender, EventArgs e)
{
var results = new ConcurrentBag<string>();
Parallel.ForEach(txtSearchTerms.Lines,
line =>
{
var result = Process(line);
results.Add(result);
});
foreach (var result in results)
{
Console.WriteLine(result);
}
}
By default Parallel.ForEach will use as much threads as underlying scheduler provides.
You can control amount of used threads by passing instance of ParallelOptions to the Parallel.ForEach method.
var options = new ParallelOptions
{
MaxDegreeOfParallelism = Environment.ProcessorCount
};
var results = new ConcurrentBag<string>();
Parallel.ForEach(values,
options,
value =>
{
var result = Process(value);
results.Add(result);
});

Consider using Parallel.ForEach to iterate over the Lines array. It is just like a normal foreach loop (i.e. each value will be processed only once), but the work is done in parallel - with multiple Tasks (threads).
var data = txtSearchTerms.Lines;
var threadCount = 4; // or whatever you want
Parallel.ForEach(data,
new ParallelOptions() { MaxDegreeOfParallelism = threadCount },
(val) =>
{
//Your code here
Console.WriteLine(val);
});
The above code will need this line to be added at the top of your file:
using System.Threading.Tasks;
Alternatively if you want to not just execute something, but also return / project something then instead try:
var results = data.AsParallel(new ParallelLinqOptions()
{
MaxDegreeOfParallelism = threadCount
}).Select(val =>
{
// Your code here, I just return the value but you could return whatever you want
return val;
}).ToList();
which still executes the code in parallel, but also returns a List (in this case with the same values in the original TextBox). And most importantly, the List will be in the same order as your input.

There many ways to do it what you want.
Take an extra class field:
private int _counter;
Use it instead of loop index. Increment it inside the lock:
private void threadJob()
{
while (true)
{
lock (threadLock)
{
if (_counter >= txtSearchTerms.Lines.Length)
return;
Console.WriteLine(txtSearchTerms.Lines[_counter]);
_counter++;
}
}
}
It works, but it very inefficient.
Lets consider another way. Each thread will handle its part of the dataset independently from the others.
public void button5_Click(object sender, EventArgs e)
{
for (int i = 0; i < threadCount; i++)
{
new Thread(new ParameterizedThreadStart(threadJob)).Start(i);
}
}
private void threadJob(object o)
{
int threadNumber = (int)o;
int count = txtSearchTerms.Lines.Length / threadCount;
int start = threadNumber * count;
int end = threadNumber != threadCount - 1 ? start + count : txtSearchTerms.Lines.Length;
for (int i = start; i < end; i++)
{
Console.WriteLine(txtSearchTerms.Lines[i]);
}
}
This is more efficient because threads do not wait on the lock. However, the array elements are processed not in a general manner.

Pass non reference value to Task

I often get strange resulst when passing int variables to tasks such as in this example:
List<List<object>> ListToProcess = new List<List<object>>();
// place some lists in list to process
foreach (var temp in Foo)
ListToProcess.Add(temp);
foreach (var tempArray in ListToProcess)
{
// initialize each list in ListToProcess
}
int numberOfChunks = ListToProcess.Count;
Task[] tasks = new Task[numberOfChunks];
for (int counter = 0; counter < numberOfChunks; counter++)
{
tasks[counter] = Task.Factory.StartNew(() =>
{
// counter is always = 5 why? <---------------------------
var t = ListToProcess[counter];
});
}
How can I solve this problem?

This is known as a closure. You are not using the value of the variable, you are using the variable itself. When the code executes, it uses the value at the time of execution, not the value when the Task was defined.
To fix this issue, I believe you would do something like this:
for (int counter = 0; counter < numberOfChunks; counter++)
{
int cur = counter;
tasks[counter] = Task.Factory.StartNew(() =>
{
// counter is always = 5 why? <---------------------------
var t = ListToProcess[cur];
});
}

There is no guarantee as to when the 'counter' variable in the Action block of StartNew will be accessed. What is likely to happen is that all 5 values are looped through, and the tasks are created, then the tasks are scheduled to run.
When they do run, the following is executed:
var t = ListToProcess[counter];
But at this stage count is already equal to 5.
Perhaps you should look at parallel collections?
ListToProcess.AsParallel().ForAll(list => dosomething(list));
There are many other options around this area.

for (int counter = 0; counter < numberOfChunks; counter++)
{
var referenceVariable = new{val=counter};
tasks[counter] = Task.Factory.StartNew(() =>
{
var t = ListToProcess[referenceVariable.val];
});
}

Since variables are captured, you can solve this by redeclaring a new variable in each loop.
for (int counter = 0; counter < numberOfChunks; counter++)
{
int localCounter = counter;
tasks[localCounter] = Task.Factory.StartNew(() =>
{
// counter is always = 5 why? <---------------------------
var t = ListToProcess[localCounter];
});
}