Develop Reference

Expensive IEnumerable: Any way to prevent multiple enumerations without forcing an immediate enumeration? [duplicate]

This question already has answers here:
Is there an IEnumerable implementation that only iterates over it's source (e.g. LINQ) once?
(4 answers)
Closed 9 months ago.
I have a very large enumeration and am preparing an expensive deferred operation on it (e.g. sorting it). I'm then passing this into a function which may or may not consume the IEnumerable, depending on some logic of its own.
Here's an illustration:
IEnumerable<Order> expensiveEnumerable = fullCatalog.OrderBy(c => Prioritize(c));
MaybeFullFillSomeOrders(expensiveEnumerable);
// Elsewhere... (example use-case for multiple enumerations, not real code)
void MaybeFullFillSomeOrders(IEnumerable<Order> nextUpOrders){
if(notAGoodTime())
return;
foreach(var order in nextUpOrders)
collectSomeInfo(order);
processInfo();
foreach(var order in nextUpOrders) {
maybeFulfill(order);
if(atCapacity())
break;
}
}
I'm would like to prepare my input to the other function such that:
If they do not consume the enumerable, the performance price of sorting is not paid.
This already precludes calling e.g. ToList() or ToArray() on it
If they choose to enumerate multiple times (perhaps not realizing how expensive it would be in this case) I want some defence in place to prevent the multiple enumeration.
Ideally, the result is still an IEnumerable<T>
The best solution I've come up with is to use Lazy<>
var expensive = new Lazy<List<Order>>>(
() => fullCatalog.OrderBy(c => Prioritize(c)).ToList());
This appears to satisfy criteria 1 and 2, but has a couple of drawbacks:
I have to change the interface to all downstream usages to expect a Lazy.
The full list (which in this case was built up from a SelectMany() on serveral smaller partitions) would need to be allocated as a new single contiguous list in memory. I'm not sure there's an easy way around this if I want to "cache" the sort result, but if you know of one I'm all ears.
One idea I had to solve the first problem was to wrap Lazy<> in some custom class that either implements or can implicitly be converted to an IEnumerable<T>, but I'm hoping someone knows of a more elegant approach.

You certainly could write your own IEnumerable<T> implementation that wraps another one, remembering all the elements it's already seen (and whether it's exhausted or not). If you need it to be thread-safe that becomes trickier, and you'd need to remember that at any time there may be multiple iterators working against the same IEnumerable<T>.
Fundamentally I think it would come down to working out what to do when asked for the next element (which is somewhat-annoyingly split into MoveNext() and Current, but that can probably be handled...):
If you've already read the next element within another iterator, you can yield it from your buffer
If you've already discovered that there is no next element, you can return that immediately
Otherwise, you need to ask the original iterator for the next element, and remember if for all the other wrapped iterators.
The other aspect that's tricky is knowing when to dispose of the underlying IEnumerator<T> - if you don't need to do that, it makes things simpler.
As a very sketchy attempt that I haven't even attempted to compile, and which is definitely not thread-safe, you could try something like this:
public class LazyEnumerable<T> : IEnumerable<T>
{
private readonly IEnumerator<T> iterator;
private List<T> buffer;
private bool completed = false;
public LazyEnumerable(IEnumerable<T> original)
{
// TODO: You could be even lazier, only calling
// GetEnumerator when you first need an element
iterator = original.GetEnumerator();
}
IEnumerator GetEnumerator() => GetEnumerator();
public IEnumerator<T> GetEnumerator()
{
int index = 0;
while (true)
{
// If we already have the element, yield it
if (index < buffer.Count)
{
yield return buffer[index];
}
// If we've yielded everything in the buffer and some
// other iterator has come to the end of the original,
// we're done.
else if (completed)
{
yield break;
}
// Otherwise, see if there's anything left in the original
// iterator.
else
{
bool hasNext = iterator.MoveNext();
if (hasNext)
{
var current = iterator.Current;
buffer.Add(current);
yield return current;
}
else
{
completed = true;
yield break;
}
}
index++;
}
}
}

Regarding evaluation of Enumerable/List [duplicate]

This question already has answers here:
Is there an IEnumerable implementation that only iterates over it's source (e.g. LINQ) once?
(4 answers)
Closed 3 years ago.
I've been playing with Lists and Enumerables and I think I understand the basics:
Enumerable: The elements are evaluated each time they are consumed.
List: The elements are evaluated on definition and are not reevaluated at any point.
I've done some tests:
Enumerable. https://www.tutorialspoint.com/tpcg.php?p=bs75zCKL
List: https://www.tutorialspoint.com/tpcg.php?p=PpyY2iif
SingleEvaluationEnum: https://www.tutorialspoint.com/tpcg.php?p=209Ciiy7
Starting with the Enumerable example:
var myList = new List<int>() { 1, 2, 3, 4, 5, 6 };
var myEnumerable = myList.Where(p =>
{
Console.Write($"{p} ");
return p > 2;
}
);
Console.WriteLine("");
Console.WriteLine("Starting");
myEnumerable.First();
Console.WriteLine("");
myEnumerable.Skip(1).First();
The output is:
Starting
1 2 3
1 2 3 4
If we add .ToList() after the .Where(...) then the output is:
1 2 3 4 5 6
Starting
I also was able to have a bit of both worlds with this class:
class SingleEvaluationEnum<T>
{
private IEnumerable<T> Enumerable;
public SingleEvaluationEnum(IEnumerable<T> enumerable)
=> Enumerable = enumerable;
public IEnumerable<T> Get()
{
if (!(Enumerable is List<T>))
Enumerable = Enumerable.ToList().AsEnumerable();
return Enumerable;
}
}
You can see the output is:
Starting
1 2 3 4 5 6
This way the evaluation is deferred until the first consumption and is not re-evaluated in the next ones. But the whole list is evaluated.
My question is: Is there a way to get this output?
Starting
1 2 3
4
In other words: I want myEnumerable.First() to evaluate only the necesary elements, but no more. And I want myEnumerable.Skip(1).First() to reuse the already evaluated elements.
EDIT: Clarification: I want that any "query" over the Enumerable applies to all the elements in the list. That's why (AFAIK) an Enumerator doesn't work.
Thanks!

LINQ is fundamentally a functional approach to working with collections. One of the assumptions is that there are no side-effects to evaluating the functions. You're violating that assumption by calling Console.Write in the function.
There's no magic involved, just functions. IEnumerable has just one method - GetEnumerator. That's all that is needed for LINQ, and that's all that LINQ really does. For example, a naïve implementation of Where would look like this:
public static IEnumerable<T> Where<T>(this IEnumerable<T> #this, Func<T, bool> filter)
{
foreach (var item in #this)
{
if (filter(item)) yield return item;
}
}
A Skip might look like this:
public static IEnumerable<T> Skip<T>(this IEnumerable<T> #this, int skip)
{
foreach (var item in #this)
{
if (skip-- > 0) continue;
yield return item;
}
}
That's all there is to it. It doesn't have any information about what IEnumerable is or represents. In fact, that's the whole point - you're abstracting those details away. There are a few optimizations in those methods, but they don't do anything smart. In the end, the difference between the List and IEnumerable in your example isn't anything fundamental - it's that myEnumerable.Skip(1) has side-effects (because myEnumerable itself has side-effects) while myList.Skip(1) doesn't. But both do the exact same thing - evaluate the enumerable, item by item. There's no other method than GetEnumerator on an enumerable, and IEnumerator only has Current and MoveNext (of those that matter for us).
LINQ is immutable. That's one of the reasons why it's so useful. This allows you to do exactly what you're doing - query the same enumerable twice but getting the exact same result. But you're not happy with that. You want things to be mutable. Well, nothing is stopping you from making your own helper functions. LINQ is just a bunch of functions, after all - you can make your own.
One such simple extension could be a memoized enumerable. Wrap around the source enumerable, create a list internally, and when you iterate over the source enumerable, keep adding items to the list. The next time GetEnumerator is called, start iterating over your internal list. When you reach the end, continue with the original approach - iterate over the source enumerable and keep adding to the list.
This will allow you to use LINQ fully, just inserting Memoize() to your LINQ queries at the places where you want to avoid iterating over the source multiple times. In your example, this would be something like:
myEnumerable = myEnumerable.Memoize();
Console.WriteLine("");
Console.WriteLine("Starting");
myEnumerable.First();
Console.WriteLine("");
myEnumerable.Skip(1).First();
The first call to myEnumerable.First() will iterate through the first three items in myList, and the second will only work with the fourth.

Basically it sounds like you're looking for an Enumerator which you can get by calling GetEnumerator on an IEnumerable. An Enumerator keeps track of it's position.
var myList = new List<int>() { 1, 2, 3, 4, 5, 6 };
var myEnumerator = myList.Where(p =>
{
Console.Write($"{p} ");
return p > 2;
}
).GetEnumerator();
Console.WriteLine("Starting");
myEnumerator.MoveNext();
Console.WriteLine("");
myEnumerator.MoveNext();
This will get you the output:
Starting
1 2 3
4
Edit to respond to your comment:
First of all this sounds like an extremely bad idea. An enumerator represents something that can be enumerated. This is why you can pipe all those fancy LINQ queries on top of it. However all calls to First "visualize" this enumeration (which results in GetEnumerator being called to get an Enumerator and walking over that until we're done and then disposing it). You however ask for every visualization to change the IEnumerable it's visualizing (this is not good practice).
However since you said this is for learning I'll give you code that ends up with an IEnumerable that will give you your desired output. I would not recommend you ever use this in real code, this is not a good and solid way of doing things.
First we create a custom Enumerator that doesn't dispose, but just keeps enumerating some internal enumerator:
public class CustomEnumerator<T> : IEnumerator<T>
{
private readonly IEnumerator<T> _source;
public CustomEnumerator(IEnumerator<T> source)
{
_source = source;
}
public T Current => _source.Current;
object IEnumerator.Current => _source.Current;
public void Dispose()
{
}
public bool MoveNext()
{
return _source.MoveNext();
}
public void Reset()
{
throw new NotImplementedException();
}
}
Then we create a custom IEnumerable class that, instead of creating a new Enumerator everytime GetEnumerator() is called, but will secretly keep using the same enumerator:
public class CustomEnumerable<T> : IEnumerable<T>
{
public CustomEnumerable(IEnumerable<T> source)
{
_internalEnumerator = new CustomEnumerator<T>(source.GetEnumerator());
}
private IEnumerator<T> _internalEnumerator;
public IEnumerator<T> GetEnumerator()
{
return _internalEnumerator;
}
IEnumerator IEnumerable.GetEnumerator()
{
return _internalEnumerator;
}
}
And finally we create an IEnumerable extension method to convert an IEnumerable into our CustomEnumerable:
public static class IEnumerableExtensions
{
public static IEnumerable<T> ToTrackingEnumerable<T>(this IEnumerable<T> source) => new CustomEnumerable<T>(source);
}
Finally when we can now do this:
var myList = new List<int>() { 1, 2, 3, 4, 5, 6 };
var myEnumerable = myList.Where(p =>
{
Console.Write($"{p} ");
return p > 2;
}).ToTrackingEnumerable();
Console.WriteLine("Starting");
var first = myEnumerable.First();
Console.WriteLine("");
var second = myEnumerable.Where(p => p % 2 == 1).First();
Console.WriteLine("");
I changed the last part so show that we can still use LINQ on it. The output is now:
Starting
1 2 3
4 5

Replace items in a list -- in place

I'm facing an issue with some simple C# code which I would easily fix in C/C++.
I guess I'm missing something.
I want to do the following (modifying items in a list -- in place):
//pseudocode
void modify<T>(List<T> a) {
foreach(var item in a) {
if(condition(item)) {
item = somethingElse;
}
}
}
I understand that foreach loops on a collection viewed as immutable, so the code above can't work.
I therefore tried the following :
void modify<T>(List<T> a) {
using (var sequenceEnum = a.GetEnumerator())
{
while (sequenceEnum.MoveNext())
{
var m = sequenceEnum.Current;
if(condition(m)) {
sequenceEnum.Current = somethingElse;
}
}
}
}
Naively thinking that Enumerator was some kind of pointer to my Element. Apparently enumerators are also immutable.
In C++ I would write something like that:
template<typename T>
struct Node {
T* value;
Node* next;
}
being then able to modify *value without touching anything in Node and therefore in the parent collection:
Node<T>* current = a->head;
while(current != nullptr) {
if(condition(current->value))
current->value = ...
}
current = current->next;
}
Do I really have to to unsafe code?
Or am i stuck the awfulness of calling subscript inside the loop?

You could also use a simple for loop.
void modify<T>(List<T> a)
{
for (int i = 0; i < a.Count; i++)
{
if (condition(a[i]))
{
a[i] = default(T);
}
}
}

In short - do not modify lists. You can achieve desired effect with
a = a.Select(x => <some logic> ? x : default(T)).ToList()
In general lists in C# are immutable during iteration. You can hovewer use .RemoveAll or similar methods.

As described in documentation here, System.Collections.Generic.List<T> is a generic implementation of the System.Collections.ArrayList which has O(1) complexity in indexed accessor. Very much like C++ std::vector<>, complexity of insertion/addition of elements is unpredictible, but access is time constant (complexity-wise, with respect to caching, etc).
The equivalent of your C++ code snippet would be LinkedList
As for the immutability of your collection during iteration, it is clearly stated in the documentation of GetEnumerator method here. Indeed, during enumeration (within a foreach, or using IEnumerator.MoveNext directly):
Enumerators can be used to read the data in the collection, but they
cannot be used to modify the underlying collection.
Moreover, modifying the list will invalidate the enumerator and usually throw an exception:
An enumerator remains valid as long as the collection remains
unchanged. If changes are made to the collection, such as adding,
modifying, or deleting elements, the enumerator is irrecoverably
invalidated and its behavior is undefined.
I believe this interface contract consistency between various types of collections leads to your misunderstanding: it would possible to implement a mutable list, but it is not required by the interfaces contract.
Imagine you want to implement a list that is mutable during enumeration. Would the enumerator hold a reference to (or a way to retrieve) the entry or the entry itself ? The entry itself would make it unmutable, a reference would be invalid when inserting elements in a linked list for example.
The simple for loop proposed by #Igor seems to be the best way to go if you want to use the standard Collections library. Otherwise, you may need to reimplement it yourself.

Use something like this:
List<T> GetModified<T>(List<T> list, Func<T, bool> condition, Func<T> replacement)
{
return list.Select(m => if (condition(m))
{ return m; }
else
{ return replacement(); }).ToList();
}
Usage:
originalList = GetModified(originalList, i => i.IsAwesome(), null);
But this can also get you into trouble with cross-thread operations. Try to use immutable instances where possible, especially with IEnumerable.
If you really really want to modify the instance of list:
//if you ever want to also remove items, this is magic (why I iterate backwards)
for (int i = list.Count - 1; i >= 0; i--)
{
if (condition)
{
list[i] = whateverYouWant;
}
}

Help me understand the code snippet in c#

I am reading this blog: Pipes and filters pattern
I am confused by this code snippet:
public class Pipeline<T>
{
private readonly List<IOperation<T>> operations = new List<IOperation<T>>();
public Pipeline<T> Register(IOperation<T> operation)
{
operations.Add(operation);
return this;
}
public void Execute()
{
IEnumerable<T> current = new List<T>();
foreach (IOperation<T> operation in operations)
{
current = operation.Execute(current);
}
IEnumerator<T> enumerator = current.GetEnumerator();
while (enumerator.MoveNext());
}
}
what is the purpose of this statement: while (enumerator.MoveNext());? seems this code is a noop.

First consider this:
IEnumerable<T> current = new List<T>();
foreach (IOperation<T> operation in operations)
{
current = operation.Execute(current);
}
This code appears to be creating nested enumerables, each of which takes elements from the previous, applies some operation to them, and passes the result to the next. But it only constructs the enumerables. Nothing actually happens yet. It's just ready to go, stored in the variable current. There are lots of ways to implement IOperation.Execute but it could be something like this.
IEnumerable<T> Execute(IEnumerable<T> ts)
{
foreach (T t in ts)
yield return this.operation(t); // Perform some operation on t.
}
Another option suggested in the article is a sort:
IEnumerable<T> Execute(IEnumerable<T> ts)
{
// Thank-you LINQ!
// This was 10 lines of non-LINQ code in the original article.
return ts.OrderBy(t => t.Foo);
}
Now look at this:
IEnumerator<T> enumerator = current.GetEnumerator();
while (enumerator.MoveNext());
This actually causes the chain of operations to be performed. When the elements are requested from the enumeration, it causes elements from the original enumerable to be passed through the chain of IOperations, each of which performs some operation on them. The end result is discarded so only the side-effect of the operation is interesting - such as writing to the console or logging to a file. This would have been a simpler way to write the last two lines:
foreach (T t in current) {}
Another thing to observe is that the initial list that starts the process is an empty list so for this to make sense some instances of T have to be created inside the first operation. In the article this is done by asking the user for input from the console.

In this case, the while (enumerator.MoveNext()); is simply evaluating all the items that are returned by the final IOperation<T>. It looks a little confusing, but the empty List<T> is only created in order to supply a value to the first IOperation<T>.
In many collections this would do exaclty nothing as you suggest, but given that we are talking about the pipes and filters pattern it is likely that the final value is some sort of iterator that will cause code to be executed. It could be something like this, for example (assuming that is an integer):
public class WriteToConsoleOperation : IOperation<int>
{
public IEnumerable<int> Execute(IEnumerable<int> ints)
{
foreach (var i in ints)
{
Console.WriteLine(i);
yield return i;
}
}
}
So calling MoveNext() for each item on the IEnumerator<int> returned by this iterator will return each of the values (which are ignored in the while loop) but also output each of the values to the console.
Does that make sense?

while (enumerator.MoveNext());
Inside the current block of code, there is no affect (it moves through all the items in the enumeration). The displayed code doesn't act on the current element in the enumeration. What might be happening is that the MoveNext() method is moving to the next element, and it is doing something to the objects in the collection (updating an internal value, pull the next from the database etc.). Since the type is List<T> this is probably not the case, but in other instances it could be.

Is yield useful outside of LINQ?

When ever I think I can use the yield keyword, I take a step back and look at how it will impact my project. I always end up returning a collection instead of yeilding because I feel the overhead of maintaining the state of the yeilding method doesn't buy me much. In almost all cases where I am returning a collection I feel that 90% of the time, the calling method will be iterating over all elements in the collection, or will be seeking a series of elements throughout the entire collection.
I do understand its usefulness in linq, but I feel that only the linq team is writing such complex queriable objects that yield is useful.
Has anyone written anything like or not like linq where yield was useful?

Note that with yield, you are iterating over the collection once, but when you build a list, you'll be iterating over it twice.
Take, for example, a filter iterator:
IEnumerator<T> Filter(this IEnumerator<T> coll, Func<T, bool> func)
{
foreach(T t in coll)
if (func(t)) yield return t;
}
Now, you can chain this:
MyColl.Filter(x=> x.id > 100).Filter(x => x.val < 200).Filter (etc)
You method would be creating (and tossing) three lists. My method iterates over it just once.
Also, when you return a collection, you are forcing a particular implementation on you users. An iterator is more generic.

I do understand its usefulness in linq, but I feel that only the linq team is writing such complex queriable objects that yield is useful.
Yield was useful as soon as it got implemented in .NET 2.0, which was long before anyone ever thought of LINQ.
Why would I write this function:
IList<string> LoadStuff() {
var ret = new List<string>();
foreach(var x in SomeExternalResource)
ret.Add(x);
return ret;
}
When I can use yield, and save the effort and complexity of creating a temporary list for no good reason:
IEnumerable<string> LoadStuff() {
foreach(var x in SomeExternalResource)
yield return x;
}
It can also have huge performance advantages. If your code only happens to use the first 5 elements of the collection, then using yield will often avoid the effort of loading anything past that point. If you build a collection then return it, you waste a ton of time and space loading things you'll never need.
I could go on and on....

I recently had to make a representation of mathematical expressions in the form of an Expression class. When evaluating the expression I have to traverse the tree structure with a post-order treewalk. To achieve this I implemented IEnumerable<T> like this:
public IEnumerator<Expression<T>> GetEnumerator()
{
if (IsLeaf)
{
yield return this;
}
else
{
foreach (Expression<T> expr in LeftExpression)
{
yield return expr;
}
foreach (Expression<T> expr in RightExpression)
{
yield return expr;
}
yield return this;
}
}
Then I can simply use a foreach to traverse the expression. You can also add a Property to change the traversal algorithm as needed.

At a previous company, I found myself writing loops like this:
for (DateTime date = schedule.StartDate; date <= schedule.EndDate;
date = date.AddDays(1))
With a very simple iterator block, I was able to change this to:
foreach (DateTime date in schedule.DateRange)
It made the code a lot easier to read, IMO.

yield was developed for C#2 (before Linq in C#3).
We used it heavily in a large enterprise C#2 web application when dealing with data access and heavily repeated calculations.
Collections are great any time you have a few elements that you're going to hit multiple times.
However in lots of data access scenarios you have large numbers of elements that you don't necessarily need to pass round in a great big collection.
This is essentially what the SqlDataReader does - it's a forward only custom enumerator.
What yield lets you do is quickly and with minimal code write your own custom enumerators.
Everything yield does could be done in C#1 - it just took reams of code to do it.
Linq really maximises the value of the yield behaviour, but it certainly isn't the only application.

Whenever your function returns IEnumerable you should use "yielding". Not in .Net > 3.0 only.
.Net 2.0 example:
public static class FuncUtils
{
public delegate T Func<T>();
public delegate T Func<A0, T>(A0 arg0);
public delegate T Func<A0, A1, T>(A0 arg0, A1 arg1);
...
public static IEnumerable<T> Filter<T>(IEnumerable<T> e, Func<T, bool> filterFunc)
{
foreach (T el in e)
if (filterFunc(el))
yield return el;
}
public static IEnumerable<R> Map<T, R>(IEnumerable<T> e, Func<T, R> mapFunc)
{
foreach (T el in e)
yield return mapFunc(el);
}
...

I'm not sure about C#'s implementation of yield(), but on dynamic languages, it's far more efficient than creating the whole collection. on many cases, it makes it easy to work with datasets much bigger than RAM.

I am a huge Yield fan in C#. This is especially true in large homegrown frameworks where often methods or properties return List that is a sub-set of another IEnumerable. The benefits that I see are:
the return value of a method that uses yield is immutable
you are only iterating over the list once
it a late or lazy execution variable, meaning the code to return the values are not executed until needed (though this can bite you if you dont know what your doing)
of the source list changes, you dont have to call to get another IEnumerable, you just iterate over IEnumeable again
many more
One other HUGE benefit of yield is when your method potentially will return millions of values. So many that there is the potential of running out of memory just building the List before the method can even return it. With yield, the method can just create and return millions of values, and as long the caller also doesnt store every value. So its good for large scale data processing / aggregating operations

Personnally, I haven't found I'm using yield in my normal day-to-day programming. However, I've recently started playing with the Robotics Studio samples and found that yield is used extensively there, so I also see it being used in conjunction with the CCR (Concurrency and Coordination Runtime) where you have async and concurrency issues.
Anyway, still trying to get my head around it as well.

Yield is useful because it saves you space. Most optimizations in programming makes a trade off between space (disk, memory, networking) and processing. Yield as a programming construct allows you to iterate over a collection many times in sequence without needing a separate copy of the collection for each iteration.
consider this example:
static IEnumerable<Person> GetAllPeople()
{
return new List<Person>()
{
new Person() { Name = "George", Surname = "Bush", City = "Washington" },
new Person() { Name = "Abraham", Surname = "Lincoln", City = "Washington" },
new Person() { Name = "Joe", Surname = "Average", City = "New York" }
};
}
static IEnumerable<Person> GetPeopleFrom(this IEnumerable<Person> people, string where)
{
foreach (var person in people)
{
if (person.City == where) yield return person;
}
yield break;
}
static IEnumerable<Person> GetPeopleWithInitial(this IEnumerable<Person> people, string initial)
{
foreach (var person in people)
{
if (person.Name.StartsWith(initial)) yield return person;
}
yield break;
}
static void Main(string[] args)
{
var people = GetAllPeople();
foreach (var p in people.GetPeopleFrom("Washington"))
{
// do something with washingtonites
}
foreach (var p in people.GetPeopleWithInitial("G"))
{
// do something with people with initial G
}
foreach (var p in people.GetPeopleWithInitial("P").GetPeopleFrom("New York"))
{
// etc
}
}
(Obviously you are not required to use yield with extension methods, it just creates a powerful paradigm to think about data.)
As you can see, if you have a lot of these "filter" methods (but it can be any kind of method that does some work on a list of people) you can chain many of them together without requiring extra storage space for each step. This is one way of raising the programming language (C#) up to express your solutions better.
The first side-effect of yield is that it delays execution of the filtering logic until you actually require it. If you therefore create a variable of type IEnumerable<> (with yields) but never iterate through it, you never execute the logic or consume the space which is a powerful and free optimization.
The other side-effect is that yield operates on the lowest common collection interface (IEnumerable<>) which enables the creation of library-like code with wide applicability.

Note that yield allows you to do things in a "lazy" way. By lazy, I mean that the evaluation of the next element in the IEnumberable is not done until the element is actually requested. This allows you the power to do a couple of different things. One is that you could yield an infinitely long list without the need to actually make infinite calculations. Second, you could return an enumeration of function applications. The functions would only be applied when you iterate through the list.

I've used yeild in non-linq code things like this (assuming functions do not live in same class):
public IEnumerable<string> GetData()
{
foreach(String name in _someInternalDataCollection)
{
yield return name;
}
}
...
public void DoSomething()
{
foreach(String value in GetData())
{
//... Do something with value that doesn't modify _someInternalDataCollection
}
}
You have to be careful not to inadvertently modify the collection that your GetData() function is iterating over though, or it will throw an exception.

Yield is very useful in general. It's in ruby among other languages that support functional style programming, so its like it's tied to linq. It's more the other way around, that linq is functional in style, so it uses yield.
I had a problem where my program was using a lot of cpu in some background tasks. What I really wanted was to still be able to write functions like normal, so that I could easily read them (i.e. the whole threading vs. event based argument). And still be able to break the functions up if they took too much cpu. Yield is perfect for this. I wrote a blog post about this and the source is available for all to grok :)

The System.Linq IEnumerable extensions are great, but sometime you want more. For example, consider the following extension:
public static class CollectionSampling
{
public static IEnumerable<T> Sample<T>(this IEnumerable<T> coll, int max)
{
var rand = new Random();
using (var enumerator = coll.GetEnumerator());
{
while (enumerator.MoveNext())
{
yield return enumerator.Current;
int currentSample = rand.Next(max);
for (int i = 1; i <= currentSample; i++)
enumerator.MoveNext();
}
}
}
}
Another interesting advantage of yielding is that the caller cannot cast the return value to the original collection type and modify your internal collection