Develop Reference

A Better Way To Make My Parallel.ForEach Thread Safe?

I would like to make the following code thread-safe. Unfortunately, I have tried locking at various levels within this code with no success. The only instance I can seem to achieve thread-safety is to place a lock around the entire loop which effectively makes the Parallel.ForEach no faster (possibly even slower) than just using foreach. The code is relatively/almost safe with no locking. It only seems to show slight variations in the summation of the geneTokens.Value[-1] keys and gtCandidates.Value[-1] keys about once out of every 20 or so executions.
I realize that Dictionary is not thread-safe. However, I cannot change this particular object to a ConcurrentDictionary without taking a major performance hit downstream. I would rather run this part of the code with a regular foreach than change that particular object. However, I am using ConcurrentDictionary to hold the individual Dictionary objects. I have also tried making this change, and it does not solve my race issue.
Here are my Class level variables:
//Holds all tokens derived from each sequence chunk
public static ConcurrentBag<sequenceItem> tokenBag =
new ConcurrentBag<sequenceItem>();
public BlockingCollection<sequenceItem> sequenceTokens = new
BlockingCollection<sequenceItem>(tokenBag);
public ConcurrentDictionary<string, int> categories = new
ConcurrentDictionary<string, int>();
public ConcurrentDictionary<int, Dictionary<int, int>> gtStartingFrequencies = new
ConcurrentDictionary<int, Dictionary<int, int>>();
public ConcurrentDictionary<string, Dictionary<int, int>> gtCandidates = new
ConcurrentDictionary<string, Dictionary<int, int>>();
public ConcurrentDictionary<string, Dictionary<int, int>> geneTokens = new
ConcurrentDictionary<string, Dictionary<int, int>>();
Here is the Parallel.ForEach:
Parallel.ForEach(sequenceTokens.GetConsumingEnumerable(), seqToken =>
{
lock (locker)
{
//Check to see if the Sequence Token is a Gene Token
Dictionary<int, int> geneTokenFreqs;
if (geneTokens.TryGetValue(seqToken.text, out geneTokenFreqs))
{ //The Sequence Token is a Gene Token
*****************Race Issue Seems To Occur Here****************************
//Increment or create category frequencies for each category provided
int frequency;
foreach (int category in seqToken.categories)
{
if (geneTokenFreqs.TryGetValue(category, out frequency))
{ //increment the category frequency, if it already exists
frequency++;
geneTokenFreqs[category] = frequency;
}
else
{ //Create the category frequency, if it does not exist
geneTokenFreqs.Add(category, 1);
}
}
//Update the frequencies total [-1] by the total # of categories incremented.
geneTokenFreqs[-1] += seqToken.categories.Length;
******************************************************************************
}
else
{ //The Sequence Token is NOT yet a Gene Token
//Check to see if the Sequence Token is a Gene Token Candidate yet
Dictionary<int, int> candidateTokenFreqs;
if (gtCandidates.TryGetValue(seqToken.text, out candidateTokenFreqs))
{
*****************Race Issue Seems To Occur Here****************************
//Increment or create category frequencies for each category provided
int frequency;
foreach (int category in seqToken.categories)
{
if (candidateTokenFreqs.TryGetValue(category, out frequency))
{ //increment the category frequency, if it already exists
frequency++;
candidateTokenFreqs[category] = frequency;
}
else
{ //Create the category frequency, if it does not exist
candidateTokenFreqs.Add(category, 1);
}
}
//Update the frequencies total [-1] by the total # of categories incremented.
candidateTokenFreqs[-1] += seqToken.categories.Length;
*****************************************************************************
//Only update the candidate sequence count once per sequence
if (candidateTokenFreqs[-3] != seqToken.sequenceId)
{
candidateTokenFreqs[-3] = seqToken.sequenceId;
candidateTokenFreqs[-2]++;
//Promote the Token Candidate to a Gene Token, if it has been found >=
//the user defined candidateThreshold
if (candidateTokenFreqs[-2] >= candidateThreshold)
{
Dictionary<int, int> deletedCandidate;
gtCandidates.TryRemove(seqToken.text, out deletedCandidate);
geneTokens.TryAdd(seqToken.text, candidateTokenFreqs);
}
}
}
else
{
//create a new token candidate frequencies dictionary by making
//a copy of the default dictionary from
gtCandidates.TryAdd(seqToken.text, new
Dictionary<int, int>(gtStartingFrequencies[seqToken.sequenceId]));
}
}
}
});

Clearly one data race comes from the fact that some threads will be adding items here:
geneTokens.TryAdd(seqToken.text, candidateTokenFreqs);
and others will be reading here:
if (geneTokens.TryGetValue(seqToken.text, out geneTokenFreqs))

How i have used Concurrent Dictionary in my Project:
I m putting a flag in the dictionary and checking from another thread , that if flag is there or not. If flag present i do my task accordingly..
For this what i m doing is as :
1) Declaring Concurrent Dictionary
2)Adding the flag using TryADD method
3)trying to retrieve the flat using TryGet Methid.
1) Declaring
Dim cd As ConcurrentDictionary(Of Integer, [String]) = New ConcurrentDictionary(Of Integer, String)()
2) Adding
If cd.TryAdd(1, "uno") Then
Console.WriteLine("CD.TryAdd() succeeded when it should have failed")
numFailures += 1
End If
3) Retrieving
If cd.TryGetValue(1, "uno") Then
Console.WriteLine("CD.TryAdd() succeeded when it should have failed")
numFailures += 1
End If

Nested dictionary objects?

just messing around, trying to expand my bag o' tricks: I was just experimenting and want to do something like a Dictionary object with another inner Dictionary as the outside Dictionary's .Value
var dictionary = new Dictionary<ObjectType, Dictionary<string, string>>();
ObjectType is an enum
so...what then...either you're not suppose to do this or I just don't know how 'cause I started running into a wall when I was trying to figure out how to populate and retrieve data from it.
Purpose might help: I'm being passed an ObjectGUID and need to flip through a bunch of database tables to determine which table the object exists in. The method I've already written just queries each table and returns count (here are a couple examples)
// Domain Check
sql = string.Format(#"select count(domainguid) from domains where domainguid = ?ObjectGUID");
count = (int)MySQLHelper.ExecuteScalar(ConnectionStrings.ConnectionStrings.V4DB_READ, sql, pObjectGUID).ToString().Parse<int>();
if (count > 0)
return ObjectType.Domain;
// Group Check
sql = string.Format(#"select count(domaingroupguid) from domaingroups where domaingroupguid = ?ObjectGUID");
count = (int)MySQLHelper.ExecuteScalar(ConnectionStrings.ConnectionStrings.V4DB_READ, sql, pObjectGUID).ToString().Parse<int>();
if (count > 0)
return ObjectType.Group;
So, that's all done and works fine...but because the fieldname and table name are the only things that change for each check I started thinking about where I could re-use the repetitive code, I created a dictionary and a foreach loop that flips through and changes the sql line (shown below)...but, as you can see below, I need that ObjectType as kind of the key for each table/fieldname pair so I can return it without any further calculations
Dictionary<string, string> objects = new Dictionary<string,string>();
objects.Add("domains", "domainguid");
objects.Add("domaingroups", "domaingroupguid");
objects.Add("users", "userguid");
objects.Add("channels", "channelguid");
objects.Add("categorylists", "categorylistguid");
objects.Add("inboundschemas", "inboundschemaguid");
objects.Add("catalogs", "catalogguid");
foreach (var item in objects)
{
sql = string.Format(#"select count({0}) from {1} where {0} = ?ObjectGUID", item.Value, item.Key);
count = (int)MySQLHelper.ExecuteScalar(ConnectionStrings.ConnectionStrings.V4DB_READ, sql, pObjectGUID).ToString().Parse<int>();
if (count > 0)
return ?????
}
This isn't all that important since my original method works just fine but I thought you StackOverflow geeks might turn me on to some new clever ideas to research...I'm guessing someone is going to smack me in the head and tell me to use arrays... :)
EDIT # Jon Skeet ------------------------------------------
Heh, sweet, think I might have come upon the right way to do it...haven't run it yet but here's an example I wrote for you
var objectTypes = new Dictionary<string, string>();
objectTypes.Add("domainguid", "domains");
var dictionary = new Dictionary<ObjectType, Dictionary<string, string>>();
dictionary.Add(ObjectType.Domain, objectTypes);
foreach(var objectType in dictionary)
{
foreach(var item in objectType.Value)
{
sql = string.Format(#"select count({0}) from {1} where {0} = ?ObjectGUID", item.Key, item.Value);
count = (int)MySQLHelper.ExecuteScalar(ConnectionStrings.ConnectionStrings.V4DB_READ, sql, pObjectGUID).ToString().Parse<int>();
if (count > 0)
return objectType.Key;
}
}
This chunk should hit the domains table looking for domainguid and if count > 0 return ObjectType.Domain...look right? Only problem is, while it might seem somewhat clever, it's like 2 dictionary objects, a couple strings, some nested loops, harder to read and debug than my first version, and about 10 more lines per check hehe...fun to experiment though and if this looks like to you then I guess it's one more thing I can add to my brain :)
also found this how to fetch data from nested Dictionary in c#

You can definitely do it, although you're currently missing a closing angle bracket and parentheses. It should be:
var dictionary = new Dictionary<ObjectType, Dictionary<string, string>>().
To add a given value you probably want something like:
private void AddEntry(ObjectType type, string key, string value)
{
Dictionary<string, string> tmp;
// Assume "dictionary" is the field
if (!dictionary.TryGetValue(type, out tmp))
{
tmp = new Dictionary<string, string>();
dictionary[type] = tmp;
}
tmp.Add(key, value);
}
If that doesn't help, please show the code that you've tried and failed with - the database code in your question isn't really relevant as far as I can tell, as it doesn't try to use a nested dictionary.

Collection of strings to dictionary

Given an ordered collection of strings:
var strings = new string[] { "abc", "def", "def", "ghi", "ghi", "ghi", "klm" };
Use LINQ to create a dictionary of string to number of occurrences of that string in the collection:
IDictionary<string,int> stringToNumOccurrences = ...;
Preferably do this in a single pass over the strings collection...

var dico = strings.GroupBy(x => x).ToDictionary(x => x.Key, x => x.Count());

Timwi/Darin's suggestion will perform this in a single pass over the original collection, but it will create multiple buffers for the groupings. LINQ isn't really very good at doing this kind of counting, and a problem like this was my original motiviation for writing Push LINQ. You might like to read my blog post on it for more details about why LINQ isn't terribly efficient here.
Push LINQ and the rather more impressive implementation of the same idea - Reactive Extensions - can handle this more efficiently.
Of course, if you don't really care too much about the extra efficiency, go with the GroupBy answer :)
EDIT: I hadn't noticed that your strings were ordered. That means you can be much more efficient, because you know that once you've seen string x and then string y, if x and y are different, you'll never see x again. There's nothing in LINQ to make this particularly easier, but you can do it yourself quite easily:
public static IDictionary<string, int> CountEntries(IEnumerable<string> strings)
{
var dictionary = new Dictionary<string, int>();
using (var iterator = strings.GetEnumerator())
{
if (!iterator.MoveNext())
{
// No entries
return dictionary;
}
string current = iterator.Current;
int currentCount = 1;
while (iterator.MoveNext())
{
string next = iterator.Current;
if (next == current)
{
currentCount++;
}
else
{
dictionary[current] = currentCount;
current = next;
currentCount = 1;
}
}
// Write out the trailing result
dictionary[current] = currentCount;
}
return dictionary;
}
This is O(n), with no dictionary lookups involved other than when writing the values. An alternative implementation would use foreach and a current value starting off at null... but that ends up being pretty icky in a couple of other ways. (I've tried it :) When I need special-case handling for the first value, I generally go with the above pattern.
Actually you could do this with LINQ using Aggregate, but it would be pretty nasty.

The standard LINQ way is this:
stringToNumOccurrences = strings.GroupBy(s => s)
.ToDictionary(g => g.Key, g => g.Count());

If this is actual production code, I'd go with Timwi's response.
If this is indeed homework and you're expected to write your own implementation, it shouldn't be too tough. Here are just a couple of hints to point you in the right direction:
Dictionary<TKey, TValue> has a ContainsKey method.
The IDictionary<TKey, TValue> interface's this[TKey] property is settable; i.e., you can do dictionary[key] = 1 (which means you can also do dictionary[key] += 1).
From those clues I think you should be able to figure out how to do it "by hand."

If you are looking for a particularly efficient (fast) solution, then GroupBy is probably too slow for you. You could use a loop:
var strings = new string[] { "abc", "def", "def", "ghi", "ghi", "ghi", "klm" };
var stringToNumOccurrences = new Dictionary<string, int>();
foreach (var str in strings)
{
if (stringToNumOccurrences.ContainsKey(str))
stringToNumOccurrences[str]++;
else
stringToNumOccurrences[str] = 1;
}
return stringToNumOccurrences;

This is a foreach version like the one that Jon mentions that he finds "pretty icky" in his answer. I'm putting it in here, so there's something concrete to talk about.
I must admit that I find it simpler than Jon's version and can't really see what's icky about it. Jon? Anyone?
static Dictionary<string, int> CountOrderedSequence(IEnumerable<string> source)
{
var result = new Dictionary<string, int>();
string prev = null;
int count = 0;
foreach (var s in source)
{
if (prev != s && count > 0)
{
result.Add(prev, count);
count = 0;
}
prev = s;
++count;
}
if (count > 0)
{
result.Add(prev, count);
}
return result;
}
Updated to add a necessary check for empty source - I still think it's simpler than Jon's :-)

Assigning values from list to list excluding nulls

Sorry for the title but I don't know other way of asking.
EDITED FOR EASY EXPLANATION
public Dictionary<string, string>[] getValuesDB(DataSet ds)
{
Dictionary<string, string>[] info;
Dictionary<string, string>[] temp = new Dictionary<string, string>[ds.Tables[0].Rows.Count];
Dictionary<string, string> d;
string ciclo = "CICLO 02";
try
{
for (int c = 0; c < ds.Tables[0].Rows.Count; c++)
{
d = new Dictionary<string, string>();
OracleConn ora = OracleConn.getInstancia();
DataSet oraDs = ora.getClientByCycle(ds.Tables[0].Columns["IDCliente"].Table.Rows[c].ItemArray[1].ToString(), ciclo);
if (oraDs.Tables[0].Rows.Count > 0)
{
oraDs.Tables[0].Columns[5].Table.Rows[0].ToString();
d.Add("DomID", ds.Tables[0].Columns["DomID"].Table.Rows[c].ItemArray[0].ToString());
d.Add("ClientID", ds.Tables[0].Columns["ClientID"].Table.Rows[c].ItemArray[1].ToString());
d.Add("AccountType", ds.Tables[0].Columns["AccountType"].Table.Rows[c].ItemArray[2].ToString());
temp[c] = d;
}
}
}
catch (Exception eo)
{
}
int count = 0;
for (int i = 0; i < temp.Length; i++)
{
if (temp[i] != null)
count++;
}
info = new Dictionary<string, string>[count];
return info;
NOW I need to get all the non-null values from 'temp' and put it in info
Any idea

Try this
Dictionary<string, string> dictionaryWithoutNulls ;
dictionaryWithoutNulls = dictionaryWithNulls
.Where(d => d.Value != null)
.ToDictionary(s=>s.Key, s=>s.Value);

Simple Answer
A simple, direct, not-pretty answer that works with .NET 2.0 and disturbs as little code as possible:
// ...code
info = new Dictionary<string, string>[count];
// Beginning of new code
int j = 0;
foreach (Dictionary<string, string> fromTemp in temp)
{
if (fromTemp == null)
continue;
info[j] = fromTemp;
j++;
}
// End of new code
return info;
Better Answer
However, there are several problems with your code snippet that should be corrected. Please forgive me if you already know these things and just did not do them for the sake of brevity. If that's the case, hopefully someone else will be able to learn from these comments.
1. The return type: Dictionary<string, string>[]
An array of dictionaries isn't very self explanatory and doesn't seem very useful. Dictionaries are perfect for looking up a value by a key, but once you place a bunch of these dictionaries in an array and remove the nulls, it seems like it would be pretty difficult to figure out which dictionary you need.
Since I don't know what you plan to do with the result of this method, it's hard to recommend an alternative. Maybe try an IDictionary<string, IDictionary<string, string>>, using ds.Tables[0].Columns["IDCliente"].Table.Rows[c].ItemArray[1].ToString() for the key of the outer dictionary. That would make it easier to locate values in the result. Or you could try an IList<IDictionary<string, string>>. Either of those would be able to auto-resize as you add new elements, eliminating the need to count the non-nulls and allocate a new array. Auto-resizing hurts performance, so initialize the collection with an appropriate capacity if you can.
Edit
After reviewing your code again, it appears that each dictionary gets the same three keys. It would be more appropriate to use a class for this:
public class ClientInfo
{
private string _domID;
public string DomID
{
get { return _domID; }
set { _domID = value; }
}
// ... also add properties for ClientID and AccountType
}
Then the return value could be something like IList<ClientInfo>.
2. Querying the database inside a loop
I'm guessing that ora.getClientByCycle(...) is a database query. It's almost always possible to rewrite your query or batch queries together so that you only have to make one round-trip to the database. Even if it's a fast query it's still probably going to be the slowest line in this method. Query the database once rather than ds.Tables[0].Rows.Count times.
3. Swallowing System.Exception
// !!! Don't do this!!!
catch (Exception eo)
{ }
If the database query might fail in certain cases and you're OK with that, then catch (OracleException). Always catch the most specific exception you can. Otherwise the catch block could catch and ignore other exceptions that you weren't expecting. Then your program doesn't behave correctly and you have no helpful exception information to help you debug it. For more information, see http://www.codeproject.com/KB/architecture/exceptionbestpractices.aspx.

Looping Over Dictionary in C#

I realize that you cannot iterate over a Dictionary in C# and edit the underlying Dictionary as in the following example:
Dictionary<Resource, double> totalCost = new Dictionary<Resource, double>();
// Populate the Dictionary in here - (not showing code).
foreach (Resource resource in totalCost.Keys)
{
totalCost[resource] = 5;
}
One way I see to fix this is to make a List backed by the Dictionary's keys, like this:
Dictionary<Resource, double> totalCost = new Dictionary<Resource, double>();
// Populate the Dictionary in here - (not showing code).
foreach (Resource resource in new List(totalCost.Keys))
{
totalCost[resource] = 5;
}
Because I'm not editing the keys themselves, is there any reason that this should not be done or that it's bad to choose this as a solution. (I realize if I was editing those keys, this could cause a lot of problems.)
Thank you.
Edit: Fixed my code example. Sorry about that.

You can loop over dictionaries with using KeyValuePair class.
Dictionary<string, string> d1 = new Dictionary<string, string>();
foreach (KeyValuePair<string, string> val in d1)
{
...
}

in your examples it doesn't look to me like you're editing the dictionary values (or keys)?
In general your solution looks fine, you could do it with a bit less code like this:
List<double> total = new List<double>();
foreach (AKeyObject key in aDictionary.Keys.ToList())
{
for (int i = 0; i < aDictionary[key].Count; i++)
{
total[i] += aDictionary[key][i];
}
}

Your first piece of code looks fine to me - you're not editing the dictionary at all.

Here is another more LINQ-esque version:
totalCost = totalCost
.ToDictionary( kvp => kvp.Key, 5 );
Or, if 5 is not quite what you want :)
totalCost = totalCost
.ToDictionary( kvp => kvp.Key, CalculateSomething(kvp.Value) );
(Note: this doesn't edit the underlying dictionary, instead it replaces it with a new one)