Develop Reference

Return Linq.Where object (IEnumerable) from within a lock - is it thread safe?

Consider the following code block
public class Data
{
public bool Init { get; set; }
public string Value {get; set; }
}
public class Example
{
private Object myObject = new Object();
private List<Data> myList = new List<Data>
{
new Data { Init = true, Value = "abc" },
new Data { Init = false, Value = "def" },
};
public IEnumerable<string> Get()
{
lock(this.myObject)
{
return this.myList.Where(i => i.Init == true).Select(i => i.Value);
}
}
public void Set(string value)
{
lock (this.myObject)
{
this.myList.Add(new Data { Init = false, Value = value });
}
}
}
If multiple threads are calling Get() - will the method be thread-safe?
In addition - will invoking .ToList() at the linq query will make it thread-safe?
return this.myList.Where(i => i.Init == true).Select(i => i.Value).ToList()

Note that you do not lock here:
public void Set(string value)
{
this.myList.Add(new Data { Init = false, Value = value });
}
So it's not thread-safe in any case.
Assuming you just forgot to do that - it's still not safe because Get returns "lazy" IEnumerable. It holds a reference to myList and it will enumerate it only when returned IEnumerable itself will be enumerated. So you are leaking reference to myList you are trying to protect with lock, outside of lock statement to arbitrary code.
You can test it like this:
var example = new Example();
var original = example.Get();
example.Clear(); // new method which clears underlying myList
foreach (var x in original)
Console.WriteLine(x);
Here we call Get, then we clear myList and then we enumerate what we got from Get. Naively one may assume original will contain original 2 items we had, but it will not contain anything, because it's evaluated only when we enumerate original, and at that point in time - list has already been cleared and is empty.
If you use
public IList<string> Get()
{
lock(this.myObject)
{
return this.myList.Where(i => i.Init == true).Select(i => i.Value).ToList();
}
}
Then it will be "safe". Now you return not "lazy" IEnumerable but a new instance of List<> with copies of values you have in myList. Note that it's a good idea to change return type to IList here, otherwise caller might pay additional overhead (like calling ToArray or ToList which makes a copy) while it's not necessary in this case.

You have to be aware about the difference between the potential to enumerate a sequence (= the IEnumerable) and the enumerated sequence itself (the List, Array, etc after you enumerated the sequence.
So you have a class Example, which internally holds a List<Data> in member MyList. Every Data has at least a string property 'Value`.
Class Example has Methods to extract Value, and to add new elements to the MyList.
I'm not sure if it is wise to call them Set and Get, these names are quite confusing. Maybe you've simplified your example (which by the way made it more difficult to talk about it).
You have an object of class Example and two threads, that both have access to this object. You worry, that while one thread is enumerating the elements of the sequence, that the other thread is adding elements of the sequence.
Your Get method returns the "potential to enumerate". The sequence is not enumerated yet after you return from Get, and after the Lock is disposed.
This means, that when you start enumerating the sequence, the Data is not locked anymore. If you've ever returned an IEnumerable from data that you fetched from a database, you probably have seen the same problem: the connection to the database is disposed before you start enumerating.
Solution 1: return enumerated data: inefficient
You already mention one solution: enumerate the data in a List before you return. This way, property MyList will not be accesses after you return from Get, so the lock is not needed anymore:
public IEnumerable<string> GetInitializedValues()
{
lock(this.MyList)
{
return this.MyList
.Where(data => data.Init == true)
.Select(data => data.Value)
.ToList();
}
}
In words: Lock MyList, which is a sequence of Data. Keep only those Data in this sequence that have a true value for property Init. From every remaining Data, take the value of property Value and put them in a List. Dispose the lock and return the List.
This is not efficient if the caller doesn't need the complete list.
// Create an object of class Example which has a zillion Data in MyList:
Example example = CreateFilledClassExample();
// I only want the first element of MyList:
string onlyOneString = example.GetInitializedValues().FirstOrDefault();
GetInitializedValues creates a list of zillion elements, and returns it. The caller only takes the first initialized value and throws the rest of the list away. What a waste of processing power.
Solution 2: use yield return: only enumerate what must be enumerated
The keyword yield means something like: return the next element of the sequence. Keep everything alive, until the caller disposes the IEnumerator
public IEnumerable<string> GetInitializedValues()
{
lock(this.MyList)
{
IEnumerable<string> initializedValues = this.MyList
.Where(data => data.Init == true)
.Select(data => data.Value);
foreach (string initializedValue in initializedValues)
{
yield return initializedValue;
}
}
}
Because the yield is inside the lock, the lock remains active, until you dispose the enumerator:
List<string> someInitializedValues = GetInitializedValues()
.Take(3)
.ToList();
This one is save, and only enumerates the first three elements.
Deep inside it will do something like this:
List<string> someInitializedValues = new List<string>();
IEnumerable<string> enumerableInitializedValues = GetInitializedValues();
// MyList is not locked yet!
// Create the enumerator. This is Disposable, so use using statement:
using (IEnumerator<string> initializedValuesEnumerator = enumerableInitializedValues.GetEnumerator())
{
// start enumerating the first 3 elements (remember: we used Take(3)
while (initializedValuesEnumerator.MoveNext() && someInitializedValues.Count < 3)
{
// there is a next element, fetch it and add the fetched value to the list
string fetchedInitializedValue = initializedValuesEnumerator.Current;
someInitializedValues.Add(fetchedInitializedValue);
}
// the enumerator is not disposed yet, MyList is still locked.
}
// the enumerator is disposed. MyList is not locked anymore

List<T>.AddRange and the yield statement

I am aware that the yield keyword indicates that the method in which it appears is an iterator. I was just wondering how that works with something like List<T>.AddRange.
Let's use the below example:
static void Main()
{
foreach (int i in MyInts())
{
Console.Write(i);
}
}
public static IEnumerable<int> MyInts()
{
for (int i = 0; i < 255; i++)
{
yield return i;
}
}
So in the above example after each yield, a value is returned in the foreach loop in Main and is printed to the console.
If we change Main to this:
static void Main()
{
var myList = new List<int>();
myList.AddRange(MyInts());
}
how does that work? Does AddRange get called for each int returned by the yield statement or does it somehow wait for all 255 values before adding the entire range?

The implementation of AddRange will iterate over the IEnumerable input using the iterator's .MoveNext() method until all values have been produced by your yielding method. This can be seen here.
So myList.AddRange(MyInts()); is called once and its implementation forces MyInts to return all of it values before moving on.
AddRange exhausts all values of the iterator because of how is implemented, but the following hypothetic method would only evaluate the first value of the iterator:
public void AddFirst<T>(IEnumerable<T> collection)
{
Insert(collection.First());
}
An interesting experiment while you play around with this is to add a Console.WriteLine(i); line in your MyInts method to see when each number is generated.

Short answer: When you call AddRange, it will internally iterate every item in your IEnumerable and add to the list.
If you did something like this:
var myList = new List<int>();
myList.AddRange(MyInts());
foreach (int i in myList)
{
Console.Write(i);
}
Then your values would be iterated twice, from the start to the end:
Once when adding to your list
Then in your for loop
Playing a bit
Now, let's suppose you created your own extension method for AddRange like this:
public static IEnumerable<T> AddRangeLazily<T>(this ICollection<T> col, IEnumerable<T> values)
{
foreach (T i in values)
{
yield return i; // first we yield
col.Add(i); // then we add
}
}
Then you could use it like this:
foreach (int i in myList.AddRangeLazily(MyInts()))
{
Console.Write(i);
}
...and it would be iterated twice as well, without going from the start to the end both times. It would lazily add each value to the list/collection and at the same time allow you to do something else (like printing it to output) after every new item being added.
If you had some sort of logic to stop the adding to the list in the middle of the operation, this should be helpful somehow.
The downside if this AddRangeLazily is: values will only be added to the collection once you iterate over AddRangeLazily like my code sample. If you just do this:
var someList = new List<int>();
someList.AddRangeLazily(MyInts());
if (someList.Any())
// it wouldn't enter here...
...it won't add values at all. If you wanted that behaviour, you should use AddRange. Forcing the iterationg over AddRangeLazily method would work, though:
var someList = new List<int>();
someList.AddRangeLazily(MyInts());
if (someList.AddRangeLazily(MyInts()).Count())
// it would enter here...thus adding all values to the someList
...however, depending on how lazy is the method you calling, it wouldn't iterate everything. For example:
var someList = new List<int>();
someList.AddRangeLazily(MyInts());
if (someList.AddRangeLazily(MyInts()).Any())
// it would enter here, plus adding only the first value to someList
Since Any() is true as soon as any item exists, then Any() just needs one iterationg to return true, therefore it just needs the first item to be iterated over.
I actually don't remember having to do something like this, it was just to play around with yield.
Fiddle here!!!

Interesting question.
The behavior is different if the enumerable is for a class that implements ICollection, such as another list or an array, but let's say it doesn't since your example doesn't. AddRange() simply uses the enumerator to insert items into the list one at a time.
using(IEnumerator<T> en = collection.GetEnumerator()) {
while(en.MoveNext()) {
Insert(index++, en.Current);
If the type of the enumerator is ICollection then AddRange first expands the list and then does a block copy.
If you want to see the code yourself:
https://referencesource.microsoft.com/#mscorlib/system/collections/generic/list.cs,51decd510e5bfe6e

Remove and Return First Item of List

I was wondering if there was a build in method to remove and return the first item of a list with one method/command.
I used this, which was not pretty
Item currentItem = items.First();
items.RemoveAt(0);
So I could wrote an extension-method:
public static class ListExtensions
{
public static T RemoveAndReturnFirst<T>(this List<T> list)
{
T currentFirst = list.First();
list.RemoveAt(0);
return currentFirst;
}
}
//Example code
Item currentItem = items.RemoveAndReturnFirst();
Is this the best possibility or is there any built-in method?
The list is returned from a nHibernate-Query and therefore it should remain a List<T>.

Most suitable collection for this operation is Queue:
var queue = new Queue<int>();
queue.Enqueue(10); //add first
queue.Enqueue(20); //add to the end
var first = queue.Dequeue(); //removes first and returns it (10)
Queue makes Enqueue and Dequeue operations very fast. But, if you need to search inside queue, or get item by index - it's bad choice. Compare, how many different types of operations do you have and according to this choose the most suitable collection - queue, stack, list or simple array.
Also you can create a Queue from a List:
var list = new List<int>();
var queue = new Queue<int>(list);

There is no built-in method. Your code looks fine to me.
One small thing, I would use the indexer, not the First extension method:
T currentFirst = list[0];
And check your list if there is a Count > 0.
public static T RemoveAndReturnFirst<T>(this List<T> list)
{
if (list == null || list.Count == 0)
{
// Instead of returning the default,
// an exception might be more compliant to the method signature.
return default(T);
}
T currentFirst = list[0];
list.RemoveAt(0);
return currentFirst;
}
If you have to worry about concurrency, I would advice to use another collection type, since this one isn't thread-safe.

Remove item from List and get the item simultaneously

In C# I am trying to get an item from a list at a random index. When it has been retrieved I want it to be removed so that it can't be selected anymore. It seems as if I need a lot of operations to do this, isn't there a function where I can simply extract an item from the list? the RemoveAt(index) function is void. I would like one with a return value.
What I am doing:
List<int> numLst = new List<int>();
numLst.Add(1);
numLst.Add(2);
do
{
int index = rand.Next(numLst.Count);
int extracted = numLst[index];
// do something with extracted value...
numLst.removeAt(index);
}
while(numLst.Count > 0);
What I would like to do:
List<int> numLst = new List<int>();
numLst.Add(1);
numLst.Add(2);
do
{
int extracted = numLst.removeAndGetItem(rand.Next(numLst.Count));
// do something with this value...
}
while(numLst.Count > 0);
Does such a "removeAndGetItem" function exist?

No, as it's a breach of pure function etiquette, where a method either has a side effect, or returns a useful value (i.e. not just indicating an error state) - never both.
If you want the function to appear atomic, you can acquire a lock on the list, which will stop other threads from accessing the list while you are modifying it, provided they also use lock:
public static class Extensions
{
public static T RemoveAndGet<T>(this IList<T> list, int index)
{
lock(list)
{
T value = list[index];
list.RemoveAt(index);
return value;
}
}
}

public static class ListExtensions
{
public static T RemoveAndGetItem<T>(this IList<T> list, int iIndexToRemove}
{
var item = list[iIndexToRemove];
list.RemoveAt(iIndexToRemove);
return item;
}
}
These are called extension methods, call as new List<T>().RemoveAndGetItem(0).
Things to consider in the extension method
Exception handling with the index that you pass, check that the index is withing 0 and the count of the list before doing this.

Is there a way to know I am getting the last element in the foreach loop

I need to do special treatment for the last element in the collection. I am wondering if I can know I hit the last element when using foreach loop.

Only way I know of is to increment a counter and compare with length on exit, or when breaking out of loop set a boolean flag, loopExitedEarly.

There isn't a direct way. You'll have to keep buffering the next element.
IEnumerable<Foo> foos = ...
Foo prevFoo = default(Foo);
bool elementSeen = false;
foreach (Foo foo in foos)
{
if (elementSeen) // If prevFoo is not the last item...
ProcessNormalItem(prevFoo);
elementSeen = true;
prevFoo = foo;
}
if (elementSeen) // Required because foos might be empty.
ProcessLastItem(prevFoo);
Alternatively, you could use the underlying enumerator to do the same thing:
using (var erator = foos.GetEnumerator())
{
if (!erator.MoveNext())
return;
Foo current = erator.Current;
while (erator.MoveNext())
{
ProcessNormalItem(current);
current = erator.Current;
}
ProcessLastItem(current);
}
It's a lot easier when working with collections that reveal how many elements they have (typically the Count property from ICollection or ICollection<T>) - you can maintain a counter (alternatively, if the collection exposes an indexer, you could use a for-loop instead):
int numItemsSeen = 0;
foreach(Foo foo in foos)
{
if(++numItemsSeen == foos.Count)
ProcessLastItem(foo)
else ProcessNormalItem(foo);
}
If you can use MoreLinq, it's easy:
foreach (var entry in foos.AsSmartEnumerable())
{
if(entry.IsLast)
ProcessLastItem(entry.Value)
else ProcessNormalItem(entry.Value);
}
If efficiency isn't a concern, you could do:
Foo[] fooArray = foos.ToArray();
foreach(Foo foo in fooArray.Take(fooArray.Length - 1))
ProcessNormalItem(foo);
ProcessLastItem(fooArray.Last());

Unfortunately not, I would write it with a for loop like:
string[] names = { "John", "Mary", "Stephanie", "David" };
int iLast = names.Length - 1;
for (int i = 0; i <= iLast; i++) {
Debug.Write(names[i]);
Debug.Write(i < iLast ? ", " : Environment.NewLine);
}
And yes, I know about String.Join :).
I see others already posted similar ideas while I was typing mine, but I'll post it anyway:
void Enumerate<T>(IEnumerable<T> items, Action<T, bool> action) {
IEnumerator<T> enumerator = items.GetEnumerator();
if (!enumerator.MoveNext()) return;
bool foundNext;
do {
T item = enumerator.Current;
foundNext = enumerator.MoveNext();
action(item, !foundNext);
}
while (foundNext);
}
...
string[] names = { "John", "Mary", "Stephanie", "David" };
Enumerate(names, (name, isLast) => {
Debug.Write(name);
Debug.Write(!isLast ? ", " : Environment.NewLine);
})

Not without jumping through flaming hoops (see above). But you can just use the enumerator directly (slightly awkward because of C#'s enumerator design):
IEnumerator<string> it = foo.GetEnumerator();
for (bool hasNext = it.MoveNext(); hasNext; ) {
string element = it.Current;
hasNext = it.MoveNext();
if (hasNext) { // normal processing
Console.Out.WriteLine(element);
} else { // special case processing for last element
Console.Out.WriteLine("Last but not least, " + element);
}
}
Notes on the other approaches I see here: Mitch's approach requires having access to a container which exposes it's size. J.D.'s approach requires writing a method in advance, then doing your processing via a closure. Ani's approach spreads loop management all over the place. John K's approach involves creating numerous additional objects, or (second method) only allows additional post processing of the last element, rather than special case processing.
I don't understand why people don't use the Enumerator directly in a normal loop, as I've shown here. K.I.S.S.
This is cleaner with Java iterators, because their interface uses hasNext rather than MoveNext. You could easily write an extension method for IEnumerable that gave you Java-style iterators, but that's overkill unless you write this kind of loop a lot.

Is it Special treatment can be done only while processing on the foreach loop, Is it you can't do that while adding to the collection. If this is your case, have your own custom collection,
public class ListCollection : List<string>
{
string _lastitem;
public void Add(string item)
{
//TODO: Do special treatment on the new Item, new item should be last one.
//Not applicable for filter/sort
base.Add(item);
}
}

List<int> numbers = new ....;
int last = numbers.Last();
Stack<int> stack = new ...;
stack.Peek();
update
var numbers = new int[] { 1, 2,3,4,5 };
var enumerator = numbers.GetEnumerator();
object last = null;
bool hasElement = true;
do
{
hasElement = enumerator.MoveNext();
if (hasElement)
{
last = enumerator.Current;
Console.WriteLine(enumerator.Current);
}
else
Console.WriteLine("Last = {0}", last);
} while (hasElement);
Console.ReadKey();

Deferred Execution trick
Build a class that encapsulates the values to be processed and the processing function for deferred execution purpose. We will end up using one instance of it for each element processed in the loop.
// functor class
class Runner {
string ArgString {get;set;}
object ArgContext {get;set;}
// CTOR: encapsulate args and a context to run them in
public Runner(string str, object context) {
ArgString = str;
ArgContext = context;
}
// This is the item processor logic.
public void Process() {
// process ArgString normally in ArgContext
}
}
Use your functor in the foreach loop to effect deferred execution by one element:
// intended to track previous item in the loop
var recent = default(Runner); // see Runner class above
// normal foreach iteration
foreach(var str in listStrings) {
// is deferred because this executes recent item instead of current item
if (recent != null)
recent.Process(); // run recent processing (from previous iteration)
// store the current item for next iteration
recent = new Runner(str, context);
}
// now the final item remains unprocessed - you have a choice
if (want_to_process_normally)
recent.Process(); // just like the others
else
do_something_else_with(recent.ArgString, recent.ArgContext);
This functor approach uses memory more but prevents you from having to count the elements in advance. In some scenarios you might achieve a kind of efficiency.
OR
Shorter Workaround
If you want to apply special processing to the last element after processing them all in a regular way ....
// example using strings
var recentStr = default(string);
foreach(var str in listStrings) {
recentStr = str;
// process str normally
}
// now apply additional special processing to recentStr (last)
It's a potential workaround.