Develop Reference

IEnumerable<T> and .Where Linq method behaviour?

I thought I know everything about IEnumerable<T> but I just met a case that I cannot explain. When we call .Where linq method on a IEnumerable, the execution is deferred until the object is enumerated, isn't it?
So how to explain the sample below :
public class CTest
{
public CTest(int amount)
{
Amount = amount;
}
public int Amount { get; set; }
public override string ToString()
{
return $"Amount:{Amount}";
}
public static IEnumerable<CTest> GenerateEnumerableTest()
{
var tab = new List<int> { 2, 5, 10, 12 };
return tab.Select(t => new CTest(t));
}
}
Nothing bad so far!
But the following test gives me an unexpected result although my knowledge regarding IEnumerable<T> and .Where linq method :
[TestMethod]
public void TestCSharp()
{
var tab = CTest.GenerateEnumerableTest();
foreach (var item in tab.Where(i => i.Amount > 6))
{
item.Amount = item.Amount * 2;
}
foreach (var t in tab)
{
var s = t.ToString();
Debug.Print(s);
}
}
No item from tab will be multiplied by 2. The output will be :
Amount:2
Amount:5
Amount:10
Amount:12
Does anyone can explain why after enumerating tab, I get the original value.
Of course, everything work fine after calling .ToList() just after calling GenerateEnumerableTest() method.

var tab = CTest.GenerateEnumerableTest();
This tab is a LINQ query that generates CTest instances that are initialized from int-values which come from an integer array which will never change. So whenever you ask for this query you will get the "same" instances(with the original Amount).
If you want to "materialize" this query you could use ToList and then change them.
Otherwise you are modifying CTest instances that exist only in the first foreach loop. The second loop enumerates other CTest instances with the unmodified Amount.
So the query contains the informations how to get the items, you could also call the method directly:
foreach (var item in CTest.GenerateEnumerableTest().Where(i => i.Amount > 6))
{
item.Amount = item.Amount * 2;
}
foreach (var t in CTest.GenerateEnumerableTest())
{
// now you don't expect them to be changed, do you?
}

Like many LINQ operations, Select is lazy and use deferred execution so your lambda expression is never being executed, because you're calling Select but never using the results. This is why, everything work fine after calling .ToList() just after calling GenerateEnumerableTest() method:
var tab = CTest.GenerateEnumerableTest().ToList();

Is it the same to iterate over Linq expression result than to assign it first to a variable?

So, this is more difficult to explain in words, so i will put code examples.
let's suppose i already have a list of clients that i want to filter.
Basically i want to know if this:
foreach(var client in list.Where(c=>c.Age > 20))
{
//Do something
}
is the same as this:
var filteredClients = list.Where(c=>c.Age > 20);
foreach(var client in filteredClients)
{
//Do something
}
I've been told that the first approach executes the .Where() in every iteration.
I'm sorry if this is a duplicate, i couldn't find any related question.
Thanks in advance.

Yes, both those examples are functionally identical. One just stores the result from Enumerable.Where in a variable before accessing it while the other just accesses it directly.
To really see why this will not make a difference, you have to understand what a foreach loop essentially does. The code in your examples (both of them) is basically equivalent to this (I’ve assumed a known type Client here):
IEnumerable<Client> x = list.Where(c=>c.Age > 20);
// foreach loop
IEnumerator<Client> enumerator = x.GetEnumerator();
while (enumerator.MoveNext())
{
Client client = enumerator.Current;
// Do something
}
So what actually happens here is the IEnumerable result from the LINQ method is not consumed directly, but an enumerator of it is requested first. And then the foreach loop does nothing else than repeatedly asking for a new object from the enumerator and processing the current element in each loop body.
Looking at this, it doesn’t make sense whether the x in the above code is really an x (i.e. a previously stored variable), or whether it’s the list.Where() call itself. Only the enumerator object—which is created just once—is used in the loop.
Now to cover that SharePoint example which Colin posted. It looks like this:
SPList activeList = SPContext.Current.List;
for (int i=0; i < activeList.Items.Count; i++)
{
SPListItem listItem = activeList.Items[i];
// do stuff
}
This is a fundamentally different thing though. Since this is not using a foreach loop, we do not get that one enumerator object which we use to iterate through the list. Instead, we repeatedly access activeList.Items: Once in the loop body to get an item by index, and once in the continuation condition of the for loop where we get the collection’s Count property value.
Unfortunately, Microsoft does not follow its own guidelines all the time, so even if Items is a property on the SPList object, it actually is creating a new SPListItemCollection object every time. And that object is empty by default and will only lazily load the actual items when you first access an item from it. So above code will eventually create a large amount of SPListItemCollections which will each fetch the items from the database. This behavior is also mentioned in the remarks section of the property documentation.
This generally violates Microsoft’s own guidelines on choosing a property vs a method:
Do use a method, rather than a property, in the following situations.
The operation returns a different result each time it is called, even if the parameters do not change.
Note that if we used a foreach loop for that SharePoint example again, then everything would have been fine, since we would have again only requested a single SPListItemCollection and created a single enumerator for it:
foreach (SPListItem listItem in activeList.Items.Cast<SPListItem>())
{ … }

They are not quite the same:
Here is the original C# code:
static void ForWithVariable(IEnumerable<Person> clients)
{
var adults = clients.Where(x => x.Age > 20);
foreach (var client in adults)
{
Console.WriteLine(client.Age.ToString());
}
}
static void ForWithoutVariable(IEnumerable<Person> clients)
{
foreach (var client in clients.Where(x => x.Age > 20))
{
Console.WriteLine(client.Age.ToString());
}
}
Here is the decompiled Intermediate Language (IL) code this results in (according to ILSpy):
private static void ForWithVariable(IEnumerable<Person> clients)
{
Func<Person, bool> arg_21_1;
if ((arg_21_1 = Program.<>c.<>9__1_0) == null)
{
arg_21_1 = (Program.<>c.<>9__1_0 = new Func<Person, bool>(Program.<>c.<>9.<ForWithVariable>b__1_0));
}
IEnumerable<Person> enumerable = clients.Where(arg_21_1);
foreach (Person current in enumerable)
{
Console.WriteLine(current.Age.ToString());
}
}
private static void ForWithoutVariable(IEnumerable<Person> clients)
{
Func<Person, bool> arg_22_1;
if ((arg_22_1 = Program.<>c.<>9__2_0) == null)
{
arg_22_1 = (Program.<>c.<>9__2_0 = new Func<Person, bool>(Program.<>c.<>9.<ForWithoutVariable>b__2_0));
}
foreach (Person current in clients.Where(arg_22_1))
{
Console.WriteLine(current.Age.ToString());
}
}
As you can see, there is a key difference:
IEnumerable<Person> enumerable = clients.Where(arg_21_1);
A more practical question, however, is whether the differences hurt performance. I concocted a test to measure that.
class Program
{
public static void Main()
{
Measure(ForEachWithVariable);
Measure(ForEachWithoutVariable);
Console.ReadKey();
}
static void Measure(Action<List<Person>, List<Person>> action)
{
var clients = new[]
{
new Person { Age = 10 },
new Person { Age = 20 },
new Person { Age = 30 },
}.ToList();
var adultClients = new List<Person>();
var sw = new Stopwatch();
sw.Start();
for (var i = 0; i < 1E6; i++)
action(clients, adultClients);
sw.Stop();
Console.WriteLine(sw.ElapsedMilliseconds.ToString());
Console.WriteLine($"{adultClients.Count} adult clients found");
}
static void ForEachWithVariable(List<Person> clients, List<Person> adultClients)
{
var adults = clients.Where(x => x.Age > 20);
foreach (var client in adults)
adultClients.Add(client);
}
static void ForEachWithoutVariable(List<Person> clients, List<Person> adultClients)
{
foreach (var client in clients.Where(x => x.Age > 20))
adultClients.Add(client);
}
}
class Person
{
public int Age { get; set; }
}
After several runs of the program, I was not able to find any significant difference between ForEachWithVariable and ForEachWithoutVariable. They were always close in time, and neither was consistently faster than the other. Interestingly, if I change 1E6 to just 1000, the ForEachWithVariable is actually consistently slower, by about 1 millisecond.
So, I conclude that for LINQ to Objects, there is no practical difference. The same type of test could be run if your particular use case involves LINQ to Entities (or SharePoint).

Change a list's property using linq

How to make the following code shorter, perhaps using anonymous method or extensions and LINQ.
Since I have to repeat this code several times and I want to make it as succinct as possible.
var imagesToUnlock = App.ImageListVM.Items.Where(img => img.Category == key);
foreach (var image in imagesToUnlock)
{
image.IsLocked = false;
}

The other solutions here feel dirty because they mutate objects in a collection via the use of LINQ.
I would instead, put the code and the filter condition into an extension method and call that:
public static IEnumerable<Item> UnlockWhere(this IEnumerable<Item> list, Func<Item, bool> condition) {
foreach (var image in list)
if (condition(image)) {
image.IsLocked = false;
yield return image;
}
}
The keeps the immutability-concerns of LINQ intact and still produces the expected result.
The call becomes:
var unlockedItems = App.ImageListVM.Items.UnlockWhere(img => img.Category == key);
EDIT
Re-written to completely remove LINQ. Instead, this new method iterates only once and returns a new, mutated collection.

Not the most efficient way to do it, but I believe you can do
var imagesToUnlock = App.ImageListVM.Items.Where(img => img.Category == key).ToList().Foreach(f => f.IsLocked = false);
Check out the Foreach method on List<T> for more info.
I would also like to note (as some have pointed out in the comments) that this is not considered best practice by some people. You should take a look at this article by Eric Lippert, who explains the issue in better detail.

Here's a stab as an extension method
Code
public static IEnumerable<T> SetPropertyValues<T>(this IEnumerable<T> items, Action<T> action)
{
foreach (var item in items)
{
action(item);
yield return item;
}
}
Usage
private class Foo
{
public string Bar { get; set; }
}
[TestMethod]
public void SetPropertyValuesForMiscTests()
{
var foos = new[] { new Foo { Bar = "hi" }, new Foo { Bar = "hello" } };
var newList = foos.SetPropertyValues(f => f.Bar = "bye");
Assert.AreEqual("bye", newList.ElementAt(0).Bar);
Assert.AreEqual("bye", newList.ElementAt(1).Bar);
}
I tested it and it works fine.

Yeah you can do this. Adapted from this answer.
imagesToUnlock.Select(i => {i.IsLocked = false; return i;}).ToList();
Edit: A lot of people are saying this is bad practice. I agree with dasblinkenlight here.. Exploring the limits of LINQ and C# is our duty as programmers. It isn't unreasonable to change the objects type from the DTO to the view model or domain object, I know its not the best, but if encapsulated and commented it isn't the end of the world to use select to do this. But please be conscious of the best practices explained by Eric.

C# failing to set property inside IEnumerable

Seen a weird bit of behaviour in some C# code that I'm at a loss to explain. Could be I'm missing an important bit of understanding, so hoping someone out there can switch on the light for me.
Got a block of code that looks like this:
IEnumberable<myObject> objects = GetObjectsFromApiCall();
for (int i = 0; i < objects.Count(); i++)
{
if (String.IsNullOrEmpty(objects.ElementAt(i).SubObject.Title))
{
SubObject sub = GetSubObjectFromDatabase((long)objects.ElementAt(i).SubObject.Id);
if (sub != null)
{
objects.ElementAt(i).SubObject.Title = sub.Title;
}
}
}
When you step through it, everything about this code seems to work properly. The "objects" collection is populated as expected. "sub" is fetched as collected and has a full set of expected properties, including a populated Title property. No errors are thrown during execution.
But ... the SubObject.Title property (which just has standard get; set; code) that exists in each Object stubbornly remains empty.
I'm at a loss. Anyone explain what's going on?
EDIT: For those who suggested I shouldn't use a for loop and ElementAt, I started with a foreach loop but thought it might be the source of the problem because it was fetching a new SubObject each time round. Fixed now, thanks to your help, and the ForEach restored.
Cheers,
Matt

I would fix it this way:
var objects = GetObjectsFromApiCall().ToList();
Then you could keep the loop as is (it works), or optimize it a bit using foreach and some Linq as suggested by other answers, but it does not really matter: the problem was that you attempted to change an element on an IEnumerator<> as explained in this question pointed by #Ahmet Kakıcı.

Try this
List<myObject> objects = GetObjectsFromApiCall().ToList();
foreach(var obj in objects.Where(o => string.IsNullOrEmpty(objects.SubObject.Title)).ToList())
{
var subObject = GetSubObjectFromDatabase(obj.SubObject.Id);
if(subObject == null) continue;
obj.SubObject.Title = subObject.Title;
}

First of all, you should not use ElementAt() for this kind of code, use
foreach (var o in objects)
{
if (string.IsNullOrEmpty(o.SubObject.Title))
{
o.SubObject.Title = ...;
}
}
Also you should note that if your method returns a dynamic IEnumerable then every time you call objects.Something() the API is called again and a fresh copy is retrieved. If this is the case, you should copy the enumerable into a list using .ToList() method.
There is also a way of not putting a copy in the list - by creating a dynamic enumerator like this:
objects = objects.Select(o =>
{
if (string.IsNullOrEmpty(o.SubObject.Title))
{
o.SubObject.Title = ...;
}
return o;
});
As for the value not being set correctly (if previous things did not help) - try adding a throw new Exception(value) in the setter for Title property - see if that is being called with the correct value.

I guest the function GetObjectsFromApiCall looks like following:
public IEnumberable<myObject> GetObjectsFromApiCall(){
for(var i = 0; i < 10; i++)
{
yield return new myObject();
}
}
If I'm right, every time you call objects.ElementAt(i) function to get the object, you will get a new object by "yield return new myObject()".

But how do you check if the Title property is changed? Do you call GetObjectsFromApiCall() again? Or do you foreach through the same objects instance again?
An IEnumerable instance may create and yield new objects each time it is "enumerated". So here's a simple example for illustration. For the example, define:
class SomeObject
{
public string Title { get; set; }
}
Then we will consider two types of "source", first an array, and then an iterator block defined like this:
static IEnumerable<SomeObject> GetSomeSequence()
{
yield return new SomeObject { Title = "Alpha", };
yield return new SomeObject { Title = "Beta", };
yield return new SomeObject { Title = "Gamma", };
}
Then test it this way:
static void Main()
{
IEnumerable<SomeObject> thingsToModify;
// set source to an array
thingsToModify = new[] { new SomeObject { Title = "Alpha", }, new SomeObject { Title = "Beta", }, new SomeObject { Title = "Gamma", }, };
foreach (var t in thingsToModify)
Console.WriteLine(t.Title);
foreach (var t in thingsToModify)
t.Title = "Changed!";
foreach (var t in thingsToModify)
Console.WriteLine(t.Title); // OK, modified
// set source to something which yields new object each time a new GetEnumerator() call is made
thingsToModify = GetSomeSequence();
foreach (var t in thingsToModify)
Console.WriteLine(t.Title);
foreach (var t in thingsToModify)
t.Title = "Changed!"; // no-one keeps these modified objects
foreach (var t in thingsToModify)
Console.WriteLine(t.Title); // new objects, titles not modified
}
Conclusion: It's perfectly possible to modify the state of a mutable object which belongs to the source we're iterating over. But some types of IEnumerable sources yield new copies of the data each time they are called, and then it's useless to make modifications to the copy.

Is there a way to know I am getting the last element in the foreach loop

I need to do special treatment for the last element in the collection. I am wondering if I can know I hit the last element when using foreach loop.

Only way I know of is to increment a counter and compare with length on exit, or when breaking out of loop set a boolean flag, loopExitedEarly.

There isn't a direct way. You'll have to keep buffering the next element.
IEnumerable<Foo> foos = ...
Foo prevFoo = default(Foo);
bool elementSeen = false;
foreach (Foo foo in foos)
{
if (elementSeen) // If prevFoo is not the last item...
ProcessNormalItem(prevFoo);
elementSeen = true;
prevFoo = foo;
}
if (elementSeen) // Required because foos might be empty.
ProcessLastItem(prevFoo);
Alternatively, you could use the underlying enumerator to do the same thing:
using (var erator = foos.GetEnumerator())
{
if (!erator.MoveNext())
return;
Foo current = erator.Current;
while (erator.MoveNext())
{
ProcessNormalItem(current);
current = erator.Current;
}
ProcessLastItem(current);
}
It's a lot easier when working with collections that reveal how many elements they have (typically the Count property from ICollection or ICollection<T>) - you can maintain a counter (alternatively, if the collection exposes an indexer, you could use a for-loop instead):
int numItemsSeen = 0;
foreach(Foo foo in foos)
{
if(++numItemsSeen == foos.Count)
ProcessLastItem(foo)
else ProcessNormalItem(foo);
}
If you can use MoreLinq, it's easy:
foreach (var entry in foos.AsSmartEnumerable())
{
if(entry.IsLast)
ProcessLastItem(entry.Value)
else ProcessNormalItem(entry.Value);
}
If efficiency isn't a concern, you could do:
Foo[] fooArray = foos.ToArray();
foreach(Foo foo in fooArray.Take(fooArray.Length - 1))
ProcessNormalItem(foo);
ProcessLastItem(fooArray.Last());

Unfortunately not, I would write it with a for loop like:
string[] names = { "John", "Mary", "Stephanie", "David" };
int iLast = names.Length - 1;
for (int i = 0; i <= iLast; i++) {
Debug.Write(names[i]);
Debug.Write(i < iLast ? ", " : Environment.NewLine);
}
And yes, I know about String.Join :).
I see others already posted similar ideas while I was typing mine, but I'll post it anyway:
void Enumerate<T>(IEnumerable<T> items, Action<T, bool> action) {
IEnumerator<T> enumerator = items.GetEnumerator();
if (!enumerator.MoveNext()) return;
bool foundNext;
do {
T item = enumerator.Current;
foundNext = enumerator.MoveNext();
action(item, !foundNext);
}
while (foundNext);
}
...
string[] names = { "John", "Mary", "Stephanie", "David" };
Enumerate(names, (name, isLast) => {
Debug.Write(name);
Debug.Write(!isLast ? ", " : Environment.NewLine);
})

Not without jumping through flaming hoops (see above). But you can just use the enumerator directly (slightly awkward because of C#'s enumerator design):
IEnumerator<string> it = foo.GetEnumerator();
for (bool hasNext = it.MoveNext(); hasNext; ) {
string element = it.Current;
hasNext = it.MoveNext();
if (hasNext) { // normal processing
Console.Out.WriteLine(element);
} else { // special case processing for last element
Console.Out.WriteLine("Last but not least, " + element);
}
}
Notes on the other approaches I see here: Mitch's approach requires having access to a container which exposes it's size. J.D.'s approach requires writing a method in advance, then doing your processing via a closure. Ani's approach spreads loop management all over the place. John K's approach involves creating numerous additional objects, or (second method) only allows additional post processing of the last element, rather than special case processing.
I don't understand why people don't use the Enumerator directly in a normal loop, as I've shown here. K.I.S.S.
This is cleaner with Java iterators, because their interface uses hasNext rather than MoveNext. You could easily write an extension method for IEnumerable that gave you Java-style iterators, but that's overkill unless you write this kind of loop a lot.

Is it Special treatment can be done only while processing on the foreach loop, Is it you can't do that while adding to the collection. If this is your case, have your own custom collection,
public class ListCollection : List<string>
{
string _lastitem;
public void Add(string item)
{
//TODO: Do special treatment on the new Item, new item should be last one.
//Not applicable for filter/sort
base.Add(item);
}
}

List<int> numbers = new ....;
int last = numbers.Last();
Stack<int> stack = new ...;
stack.Peek();
update
var numbers = new int[] { 1, 2,3,4,5 };
var enumerator = numbers.GetEnumerator();
object last = null;
bool hasElement = true;
do
{
hasElement = enumerator.MoveNext();
if (hasElement)
{
last = enumerator.Current;
Console.WriteLine(enumerator.Current);
}
else
Console.WriteLine("Last = {0}", last);
} while (hasElement);
Console.ReadKey();

Deferred Execution trick
Build a class that encapsulates the values to be processed and the processing function for deferred execution purpose. We will end up using one instance of it for each element processed in the loop.
// functor class
class Runner {
string ArgString {get;set;}
object ArgContext {get;set;}
// CTOR: encapsulate args and a context to run them in
public Runner(string str, object context) {
ArgString = str;
ArgContext = context;
}
// This is the item processor logic.
public void Process() {
// process ArgString normally in ArgContext
}
}
Use your functor in the foreach loop to effect deferred execution by one element:
// intended to track previous item in the loop
var recent = default(Runner); // see Runner class above
// normal foreach iteration
foreach(var str in listStrings) {
// is deferred because this executes recent item instead of current item
if (recent != null)
recent.Process(); // run recent processing (from previous iteration)
// store the current item for next iteration
recent = new Runner(str, context);
}
// now the final item remains unprocessed - you have a choice
if (want_to_process_normally)
recent.Process(); // just like the others
else
do_something_else_with(recent.ArgString, recent.ArgContext);
This functor approach uses memory more but prevents you from having to count the elements in advance. In some scenarios you might achieve a kind of efficiency.
OR
Shorter Workaround
If you want to apply special processing to the last element after processing them all in a regular way ....
// example using strings
var recentStr = default(string);
foreach(var str in listStrings) {
recentStr = str;
// process str normally
}
// now apply additional special processing to recentStr (last)
It's a potential workaround.