Develop Reference

Get the first item of an IEnumerable and return the rest as IEnumerable, iterating through only once

I've got an enumerable that contains responses from a service call that come in gradually.
I can't do ToList on the enumerable as that would block until all responses are received instead of listing them as they come.
I also can't iterate twice as that would trigger another service call.
How to get the first element in the enumerable and return the continuation of the enumerable? I can't use an iterator method as I get a compilation error:
Iterators cannot have ref, in or out parameters.
I've tried this code:
public IEnumerable<object> GetFirstAndRemainder(IEnumerable<object> enumerable, out object first)
{
first = enumerable.Take(1).FirstOrDefault();
return enumerable.Skip(1); // Second interation - unexceptable
}
// This one has a compilation error: Iterators cannot have ref, in or out parameters
public IEnumerable<object> GetFirstAndRemainder2(IEnumerable<object> enumerable, out object first)
{
var enumerator = enumerable.GetEnumerator();
enumerator.MoveNext();
first = enumerator.Current;
while (enumerator.MoveNext())
{
yield return enumerator.Current;
}
}

Instead of using an out parameter, you can use ValueTuple<T1, T2> (as of C# 7.0, documented here) to return two elements: the first item of the IEnumerable<T>, and the remainder as another IEnumerable<T>.
using System.Linq;
class Program {
static void Main(string[] args) {
(int first, IEnumerable<int> remainder) = GetFirstAndRemainder(Enumerable.Range(1, 5));
// first = 1
// remainder yields (2, 3, 4, 5)
}
// Returns the first item and the remainders as an IEnumerable
static (T, IEnumerable<T>) GetFirstAndRemainder<T>(IEnumerable<T> sequence) {
var enumerator = sequence.GetEnumerator();
enumerator.MoveNext();
return (enumerator.Current, enumerator.AsEnumerable());
}
}
You also need to convert from an IEnumerator to an IEnumerable which I did with an extension method:
static class Extensions {
public static IEnumerable<T> AsEnumerable<T>(this IEnumerator<T> enumerator) {
while (enumerator.MoveNext()) {
yield return enumerator.Current;
}
}
}
Note that due to your requirements, iterating once over the remainder will exhaust it even though it has the type IEnumerable<T>.

There is also the possibility to do cheeky deconstruction like so:
var (x, xs) = new int?[] { 1, 2, 3 };
// x = 1, xs = 2, 3
All you need is to implement Deconstruct() on IEnumerable<T>. Based on the implementations from previous answers:
public static class Ext {
public static void Deconstruct<T>(this IEnumerable<T> source, out T? first, out IEnumerable<T> tail) {
using var e = source.GetEnumerator();
(first, tail) = e.MoveNext()
? (e.Current, e.AsEnumerable())
: (default, Enumerable.Empty<T>());
}
public static IEnumerable<T> AsEnumerable<T>(this IEnumerator<T> enumerator) {
while (enumerator.MoveNext()) yield return enumerator.Current;
}
}
If you feel particularly funny you can extend that approach to do things like this:
var (fst, snd, trd, rest) = new int?[] { 1, 2, 3, 4, 5 };
public static class Ext {
// ...
public static void Deconstruct<T>(this IEnumerable<T> source, out T? first, out T? second, out IEnumerable<T> tail) {
using var e = source.GetEnumerator();
if (e.MoveNext())
(first, (second, tail)) = (e.Current, e.AsEnumerable());
else
(first, second, tail) = (default, default, Enumerable.Empty<T>());
}
public static void Deconstruct<T>(this IEnumerable<T> source, out T? first, out T? second, out T? third, out IEnumerable<T> tail) {
using var e = source.GetEnumerator();
if (e.MoveNext())
(first, (second, third, tail)) = (e.Current, e.AsEnumerable());
else
(first, second, third, tail) = (default, default, default, Enumerable.Empty<T>());
}
}

As the answer of #corentin-pane has too many pending edits, I will share the extension methods I ended up implementing as a full new answer
public static class ExtraLinqExtensions
{
public static (T, IEnumerable<T>) GetFirstAndRemainder<T>(this IEnumerable<T> sequence)
{
using var enumerator = sequence.GetEnumerator();
return enumerator.MoveNext()
? (enumerator.Current, enumerator.AsEnumerable())
: (default, Enumerable.Empty<T>());
}
public static IEnumerable<T> AsEnumerable<T>(this IEnumerator<T> enumerator)
{
while (enumerator.MoveNext())
{
yield return enumerator.Current;
}
}
}
Where the most notable change is the check for whether the enumerator succeeded in doing a MoveNext().

Calling method with IEnumerable<T> sequence as argument, if that sequence is not empty

I have method Foo, which do some CPU intensive computations and returns IEnumerable<T> sequence. I need to check, if that sequence is empty. And if not, call method Bar with that sequence as argument.
I thought about three approaches...
Check, if sequence is empty with Any(). This is ok, if sequence is really empty, which will be case most of the times. But it will have horrible performance, if sequence will contains some elements and Foo will need them compute again...
Convert sequence to list, check if that list it empty... and pass it to Bar. This have also limitation. Bar will need only first x items, so Foo will be doing unnecessary work...
Check, if sequence is empty without actually reset the sequence. This sounds like win-win, but I can't find any easy build-in way, how to do it. So I create this obscure workaround and wondering, whether this is really a best approach.
Condition
var source = Foo();
if (!IsEmpty(ref source))
Bar(source);
with IsEmpty implemented as
bool IsEmpty<T>(ref IEnumerable<T> source)
{
var enumerator = source.GetEnumerator();
if (enumerator.MoveNext())
{
source = CreateIEnumerable(enumerator);
return false;
}
return true;
IEnumerable<T> CreateIEnumerable(IEnumerator<T> usedEnumerator)
{
yield return usedEnumerator.Current;
while (usedEnumerator.MoveNext())
{
yield return usedEnumerator.Current;
}
}
}
Also note, that calling Bar with empty sequence is not option...
EDIT:
After some consideration, best answer for my case is from Olivier Jacot-Descombes - avoid that scenario completely. Accepted solution answers this question - if it is really no other way.

I don't know whether your algorithm in Foo allows to determine if the enumeration will be empty without doing the calculations. But if this is the case, return null if the sequence would be empty:
public IEnumerable<T> Foo()
{
if (<check if sequence will be empty>) {
return null;
}
return GetSequence();
}
private IEnumerable<T> GetSequence()
{
...
yield return item;
...
}
Note that if a method uses yield return, it cannot use a simple return to return null. Therefore a second method is needed.
var sequence = Foo();
if (sequence != null) {
Bar(sequence);
}
After reading one of your comments
Foo need to initialize some resources, parse XML file and fill some HashSets, which will be used to filter (yield) returned data.
I suggest another approach. The time consuming part seems to be the initialization. To be able to separate it from the iteration, create a foo calculator class. Something like:
public class FooCalculator<T>
{
private bool _isInitialized;
private string _file;
public FooCalculator(string file)
{
_file = file;
}
private EnsureInitialized()
{
if (_isInitialized) return;
// Parse XML.
// Fill some HashSets.
_isInitialized = true;
}
public IEnumerable<T> Result
{
get {
EnsureInitialized();
...
yield return ...;
...
}
}
}
This ensures that the costly initialization stuff is executed only once. Now you can safely use Any().
Other optimizations are conceivable. The Result property could remember the position of the first returned element, so that if it is called again, it could skip to it immediately.

You would like to call some function Bar<T>(IEnumerable<T> source) if and only if the enumerable source contains at least one element, but you're running into two problems:
There is no method T Peek() in IEnumerable<T> so you would need to actually begin to evaluate the enumerable to see if it's nonempty, but...
You don't want to even partially double-evaluate the enumerable since setting up the enumerable might be expensive.
In that case your approach looks reasonable. You do, however, have some issues with your imlementation:
You need to dispose enumerator after using it.
As pointed out by Ivan Stoev in comments, if the Bar() method attempts to evaluate the IEnumerable<T> more than once (e.g. by calling Any() then foreach (...)) then the results will be undefined because usedEnumerator will have been exhausted by the first enumeration.
To resolve these issues, I'd suggest modifying your API a little and create an extension method IfNonEmpty<T>(this IEnumerable<T> source, Action<IEnumerable<T>> func) that calls a specified method only if the sequence is nonempty, as shown below:
public static partial class EnumerableExtensions
{
public static bool IfNonEmpty<T>(this IEnumerable<T> source, Action<IEnumerable<T>> func)
{
if (source == null|| func == null)
throw new ArgumentNullException();
using (var enumerator = source.GetEnumerator())
{
if (!enumerator.MoveNext())
return false;
func(new UsedEnumerator<T>(enumerator));
return true;
}
}
class UsedEnumerator<T> : IEnumerable<T>
{
IEnumerator<T> usedEnumerator;
public UsedEnumerator(IEnumerator<T> usedEnumerator)
{
if (usedEnumerator == null)
throw new ArgumentNullException();
this.usedEnumerator = usedEnumerator;
}
public IEnumerator<T> GetEnumerator()
{
var localEnumerator = System.Threading.Interlocked.Exchange(ref usedEnumerator, null);
if (localEnumerator == null)
// An attempt has been made to enumerate usedEnumerator more than once;
// throw an exception since this is not allowed.
throw new InvalidOperationException();
yield return localEnumerator.Current;
while (localEnumerator.MoveNext())
{
yield return localEnumerator.Current;
}
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
}
Demo fiddle with unit tests here.

If you can change Bar then how about change it to TryBar that returns false when IEnumerable<T> was empty?
bool TryBar(IEnumerable<Foo> source)
{
var count = 0;
foreach (var x in source)
{
count++;
}
return count > 0;
}
If that doesn't work for you could always create your own IEnumerable<T> wrapper that caches values after they have been iterated once.

One improvement for your IsEmpty would be to check if source is ICollection<T>, and if it is, check .Count (also, dispose the enumerator):
bool IsEmpty<T>(ref IEnumerable<T> source)
{
if (source is ICollection<T> collection)
{
return collection.Count == 0;
}
var enumerator = source.GetEnumerator();
if (enumerator.MoveNext())
{
source = CreateIEnumerable(enumerator);
return false;
}
enumerator.Dispose();
return true;
IEnumerable<T> CreateIEnumerable(IEnumerator<T> usedEnumerator)
{
yield return usedEnumerator.Current;
while (usedEnumerator.MoveNext())
{
yield return usedEnumerator.Current;
}
usedEnumerator.Dispose();
}
}
This will work for arrays and lists.
I would, however, rework IsEmpty to return:
IEnumerable<T> NotEmpty<T>(IEnumerable<T> source)
{
if (source is ICollection<T> collection)
{
if (collection.Count == 0)
{
return null;
}
return source;
}
var enumerator = source.GetEnumerator();
if (enumerator.MoveNext())
{
return CreateIEnumerable(enumerator);
}
enumerator.Dispose();
return null;
IEnumerable<T> CreateIEnumerable(IEnumerator<T> usedEnumerator)
{
yield return usedEnumerator.Current;
while (usedEnumerator.MoveNext())
{
yield return usedEnumerator.Current;
}
usedEnumerator.Dispose();
}
}
Now, you would check if it returned null.

The accepted answer is probably the best approach but, based on, and I quote:
Convert sequence to list, check if that list it empty... and pass it to Bar. This have also limitation. Bar will need only first x items, so Foo will be doing unnecessary work...
Another take would be creating an IEnumerable<T> that partially caches the underlying enumeration. Something along the following lines:
interface IDisposableEnumerable<T>
:IEnumerable<T>, IDisposable
{
}
static class PartiallyCachedEnumerable
{
public static IDisposableEnumerable<T> Create<T>(
IEnumerable<T> source,
int cachedCount)
{
if (source == null)
throw new NullReferenceException(
nameof(source));
if (cachedCount < 1)
throw new ArgumentOutOfRangeException(
nameof(cachedCount));
return new partiallyCachedEnumerable<T>(
source, cachedCount);
}
private class partiallyCachedEnumerable<T>
: IDisposableEnumerable<T>
{
private readonly IEnumerator<T> enumerator;
private bool disposed;
private readonly List<T> cache;
private readonly bool hasMoreItems;
public partiallyCachedEnumerable(
IEnumerable<T> source,
int cachedCount)
{
Debug.Assert(source != null);
Debug.Assert(cachedCount > 0);
enumerator = source.GetEnumerator();
cache = new List<T>(cachedCount);
var count = 0;
while (enumerator.MoveNext() &&
count < cachedCount)
{
cache.Add(enumerator.Current);
count += 1;
}
hasMoreItems = !(count < cachedCount);
}
public void Dispose()
{
if (disposed)
return;
enumerator.Dispose();
disposed = true;
}
public IEnumerator<T> GetEnumerator()
{
foreach (var t in cache)
yield return t;
if (disposed)
yield break;
while (enumerator.MoveNext())
{
yield return enumerator.Current;
cache.Add(enumerator.Current)
}
Dispose();
}
IEnumerator IEnumerable.GetEnumerator()
=> GetEnumerator();
}
}

Copy IEnumerable, replacing an item

I want to make a copy of an IEnumerable<T> in which a single item at a given index has been replaced by a given value.
I defined the following method which does what I want:
public static IEnumerable<T> ReplaceAt<T>(this IEnumerable<T> source, T item, int index)
{
foreach (T before in source.Take(index))
{
yield return before;
}
yield return item;
foreach (T after in source.Skip(index + 1))
{
yield return after;
}
}
However, while perhaps easy to understand, it seems 'inefficient' to create two iterators, one of which skips the items already taken by the first iterator.
Is there a better way of defining this?

How about:
public static IEnumerable<T> ReplaceAt<T>(this IEnumerable<T> source, T item, int index)
{
return source.Select((value, i) => index == i ? item : value);
}

Not sure about efficient, but have you tried this?
public static IEnumerable<T> ReplaceAt<T>(this IEnumerable<T> source, T item, int index)
{
return source.Select((x, i) => i == index ? item : x);
}

If you want to go crazy, you can unroll the foreach manually:
public static IEnumerable<T> ReplaceAt<T>(this IEnumerable<T> source, T item, int index)
{
int itemIndex = 0;
using(var iter = source.GetEnumerator())
{
while(iter.MoveNext())
{
yield return itemIndex++ == index ? item : iter.Current;
}
}
}

IEnumerable Method with AsParallel

I got the following extension method:
static class ExtensionMethods
{
public static IEnumerable<IEnumerable<T>> Subsequencise<T>(
this IEnumerable<T> input,
int subsequenceLength)
{
var enumerator = input.GetEnumerator();
SubsequenciseParameter parameter = new SubsequenciseParameter
{
Next = enumerator.MoveNext()
};
while (parameter.Next)
yield return getSubSequence(
enumerator,
subsequenceLength,
parameter);
}
private static IEnumerable<T> getSubSequence<T>(
IEnumerator<T> enumerator,
int subsequenceLength,
SubsequenciseParameter parameter)
{
do
{
lock (enumerator) // this lock makes it "work"
{ // removing this causes exceptions.
if (parameter.Next)
yield return enumerator.Current;
}
} while ((parameter.Next = enumerator.MoveNext())
&& --subsequenceLength > 0);
}
// Needed since you cant use out or ref in yield-return methods...
class SubsequenciseParameter
{
public bool Next { get; set; }
}
}
Its purpose is to split a sequence into subsequences of a given size.
Calling it like this:
foreach (var sub in "abcdefghijklmnopqrstuvwxyz"
.Subsequencise(3)
.**AsParallel**()
.Select(sub =>new String(sub.ToArray()))
{
Console.WriteLine(sub);
}
Console.ReadKey();
works, however there are some empty lines in-between since some of the threads are "too late" and enter the first yield return.
I tried putting more locks everywhere, however I cannot achieve to make this work correct in combination with as parallel.
It's obvious that this example doesn't justify the use of as parallel at all. It is just to demonstrate how the method could be called.

The problem is that using iterators is lazy evaluated, so you return a lazily evaluated iterator which gets used from multiple threads.
You can fix this by rewriting your method as follows:
public static IEnumerable<IEnumerable<T>> Subsequencise<T>(this IEnumerable<T> input, int subsequenceLength)
{
var syncObj = new object();
var enumerator = input.GetEnumerator();
if (!enumerator.MoveNext())
{
yield break;
}
List<T> currentList = new List<T> { enumerator.Current };
int length = 1;
while (enumerator.MoveNext())
{
if (length == subsequenceLength)
{
length = 0;
yield return currentList;
currentList = new List<T>();
}
currentList.Add(enumerator.Current);
++length;
}
yield return currentList;
}
This performs the same function, but doesn't use an iterator to implement the "nested" IEnumerable<T>, avoiding the problem. Note that this also avoids the locking as well as the custom SubsequenciseParameter type.

linq extension method to take elements from the end of the sequence

There is the enumerable extension method
Take<TSource>(
IEnumerable<TSource> source,
int count
)
which takes the first count elements from the start.
Is there a way to take the elements from the end?
or even better a way to take the elements from an offset to the end?
Thanks

finiteList.Reverse().Take(count).Reverse();
or
finiteList.Skip(finiteList.Count() - count)
There is some overhead in doing this so a custom method would be better.
Update: A custom method
public static class EnumerableExtensions
{
public static IEnumerable<T> TakeLast<T>(this IEnumerable<T> source, int count)
{
if (source == null) throw new ArgumentNullException("source");
if (count < 0) throw new ArgumentOutOfRangeException("count");
if (count == 0) yield break;
var queue = new Queue<T>(count);
foreach (var t in source)
{
if (queue.Count == count) queue.Dequeue();
queue.Enqueue(t);
}
foreach (var t in queue)
yield return t;
}
}
Update: Changed the code a littlebit with ideas from dtb´s answer :-)
Comment to Bear: Look at this example:
var lastFive = Enumerable.Range(1, 10).TakeLast(5);
var lastFive2 = Enumerable.Range(1, 10).TakeLast2(5); //Bear´s way
Queue<int> q = (Queue<int>)lastFive2;
q.Dequeue();
//Is lastFive2 still last five? no...
You could potentially change the values of lastFive2 and therefore that approach can be unsafe or at least it´s not the functional way.
To Bear:
What I meant about safe is this:
var lastFive2 = Enumerable.Range(1, 10).TakeLast2(5); //Bear´s way
//some = Some method which you don't control - it could be from another assembly which represents a crazy plugin etc.
some(lastFive2);
//Now what?
In these cases you would have to make a copy to be sure. But in most cases your way would be fine - and a little bit more efficient than this so +1 :)
An idea is to use a queue which only have internal Enqueue etc.

MoreLINQ provides a TakeLast extension method:
var last10 = finiteList.TakeLast(10);
To take the elements from an offset to the end, Enumerable.Skip should do the trick:
var allFromOffsetToEnd = finiteList.Skip(offset);

#lasseespeholt:
public static class EnumerableExtensions
{
public static ReadOnlyEnumerable<T> AsReadOnly<T>(
this IEnumerable<T> source)
{
return new ReadOnlyEnumerable<T>(source);
}
}
public sealed class ReadOnlyEnumerable<T> : IEnumerable<T>
{
private readonly IEnumerable<T> _source;
public ReadOnlyEnumerable(IEnumerable<T> source)
{
if (_source == null)
{
throw new ArgumentNullException("source");
}
_source = source;
}
public IEnumerator<T> GetEnumerator()
{
return _source.GetEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
{
return _source.GetEnumerator();
}
}
public static IEnumerable<T> TakeLast<T>(this IEnumerable<T> source, int count)
{
if (source == null) throw new ArgumentNullException("source");
if (count < 0) throw new ArgumentOutOfRangeException("count");
if (count == 0)
return Enumerable.Empty<T>();
var queue = new Queue<T>(count);
foreach (var t in source)
{
if (queue.Count == count) queue.Dequeue();
queue.Enqueue(t);
}
return queue.AsReadOnly();
}

A note on performance. Plenty of answers here operating on IEnumerable<> and that is probably what you need and should use.
But if the datasets are large and of type List<> or similar, you can prevent a lot of unnecessary iterating with something like:
// demo, no errorhandling
public static IEnumerable<T> TakeFrom<T>(this IList<T> list, int offset)
{
for (int i = offset; i < list.Count; i += 1)
{
yield return list[i];
}
}