Related
I have a very long sequence of data is the form of IEnumerable, and I would like to check it for a number of conditions. Each condition returns a value of true or false, and I want to know if all conditions are true. My problem is that I can not afford to materialize the IEnumerable by calling ToList, because it is simply too long (> 10,000,000,000 elements). Neither I can afford to enumerate the sequence multiple times, one for each condition, because each time I will get a different sequence. I am searching for an efficient way to perform this check, using the existing LINQ functionality if possible.
Clarification: I am asking for a general solution, not for a solution of the specific example problem that is presented bellow.
Here is a dummy version of my sequence:
static IEnumerable<int> GetLongSequence()
{
var random = new Random();
for (long i = 0; i < 10_000_000_000; i++) yield return random.Next(0, 100_000_000);
}
And here is an example of the conditions that the sequence must satisfy:
var source = GetLongSequence();
var result = source.Any(n => n % 28_413_803 == 0)
&& source.All(n => n < 99_999_999)
&& source.Average(n => n) > 50_000_001;
Unfortunately this approach invokes three times the GetLongSequence, so it doesn't satisfy the requirements of the problem.
I tried to write a Linqy extension method of the above, hoping that this could give me some ideas:
public static bool AllConditions<TSource>(this IEnumerable<TSource> source,
params Func<IEnumerable<TSource>, bool>[] conditions)
{
foreach (var condition in conditions)
{
if (!condition(source)) return false;
}
return true;
}
This is how I intend to use it:
var result = source.AllConditions
(
s => s.Any(n => n % 28_413_803 == 0),
s => s.All(n => n < 99_999_999),
s => s.Average(n => n) > 50_000_001,
// more conditions...
);
Unfortunately this offers no improvement. The GetLongSequence is again invoked three times.
After hitting my head against the wall for an hour, without making any progress, I figured out a possible solution. I could run each condition in a separate thread, and synchronize their access to a single shared enumerator of the sequence. So I ended up with this monstrosity:
public static bool AllConditions<TSource>(this IEnumerable<TSource> source,
params Func<IEnumerable<TSource>, bool>[] conditions)
{
var locker = new object();
var enumerator = source.GetEnumerator();
var barrier = new Barrier(conditions.Length);
long index = -1;
bool finished = false;
IEnumerable<TSource> OneByOne()
{
try
{
while (true)
{
TSource current;
lock (locker)
{
if (finished) break;
if (barrier.CurrentPhaseNumber > index)
{
index = barrier.CurrentPhaseNumber;
finished = !enumerator.MoveNext();
if (finished)
{
enumerator.Dispose(); break;
}
}
current = enumerator.Current;
}
yield return current;
barrier.SignalAndWait();
}
}
finally
{
barrier.RemoveParticipant();
}
}
var results = new ConcurrentQueue<bool>();
var threads = conditions.Select(condition => new Thread(() =>
{
var result = condition(OneByOne());
results.Enqueue(result);
})
{ IsBackground = true }).ToArray();
foreach (var thread in threads) thread.Start();
foreach (var thread in threads) thread.Join();
return results.All(r => r);
}
For the synchronization a used a Barrier. This solution actually works way better than I thought. It can process almost 1,000,000 elements per second in my machine. It is not fast enough though, since it needs almost 3 hours to process the full sequence of 10,000,000,000 elements. And I can't wait for the result for longer than 5 minutes. Any ideas about how I could run these conditions efficiently in a single thread?
If you need to ensure that the sequence is enumerated only once, conditions operating on the whole sequence are not useful.
One possibility that comes to my mind is to have an interface which is called for each element of the sequence and implement this interface in different ways for your specific conditions:
bool Example()
{
var source = GetLongSequence();
var conditions = new List<IEvaluate<int>>
{
new Any<int>(n => n % 28_413_803 == 0),
new All<int>(n => n < 99_999_999),
new Average(d => d > 50_000_001)
};
foreach (var item in source)
{
foreach (var condition in conditions)
{
condition.Evaluate(item);
}
}
return conditions.All(c => c.Result);
}
static IEnumerable<int> GetLongSequence()
{
var random = new Random();
for (long i = 0; i < 10_000_000_000; i++) yield return random.Next(0, 100_000_000);
}
interface IEvaluate<T>
{
void Evaluate(T item);
bool Result { get; }
}
class Any<T> : IEvaluate<T>
{
private bool _result;
private readonly Func<T, bool> _predicate;
public Any(Func<T, bool> predicate)
{
_predicate = predicate;
_result = false;
}
public void Evaluate(T item)
{
if (_predicate(item))
{
_result = true;
}
}
public bool Result => _result;
}
class All<T> : IEvaluate<T>
{
private bool _result;
private readonly Func<T, bool> _predicate;
public All(Func<T, bool> predicate)
{
_predicate = predicate;
_result = true;
}
public void Evaluate(T item)
{
if (!_predicate(item))
{
_result = false;
}
}
public bool Result => _result;
}
class Average : IEvaluate<int>
{
private long _sum;
private int _count;
Func<double, bool> _evaluate;
public Average(Func<double, bool> evaluate)
{
}
public void Evaluate(int item)
{
_sum += item;
_count++;
}
public bool Result => _evaluate((double)_sum / _count);
}
If all you want is check for these three conditions on a single thread in only one enumeration, I wouldn't use LINQ and manually aggregate the checks:
bool anyVerified = false;
bool allVerified = true;
double averageSoFar = 0;
foreach (int n in GetLongSequence()) {
anyVerified = anyVerified || n % 28_413_803 == 0;
allVerified = allVerified && n < 99_999_999;
averageSoFar += n / 10_000_000_000;
// Early out conditions here...
}
return anyVerified && allVerified && averageSoFar > 50_000_001;
This could be made more generic if you plan to do these checks often but it looks like it satisfies all your requirements.
Can I also suggest you another method based on the Enumerable.Aggregate LINQ extension method.
public static class Parsing {
public static bool ParseOnceAndCheck(this IEnumerable<int> collection, Func<int, bool> all, Func<int, bool> any, Func<double, bool> average) {
// Aggregate the two boolean results, the sum of all values and the count of values...
(bool allVerified, bool anyVerified, int sum, int count) = collection.Aggregate(
ValueTuple.Create(true, false, 0, 0),
(tuple, item) => ValueTuple.Create(tuple.Item1 && all(item), tuple.Item2 || any(item), tuple.Item3 + item, tuple.Item4 + 1)
);
// ... and summarizes the result
return allVerified && anyVerified && average(sum / count);
}
}
You could call this extension method in a very similar way than you would usual LINQ methods but there would be only one enumeration of your sequence:
IEnumerable<int> sequence = GetLongSequence();
bool result = sequence.ParseOnceAndCheck(
all: n => n < 99_999_999,
any: n => n % 28_413_803 == 0,
average: a => a > 50_000_001
);
I found a single-threaded solution that uses the Reactive Extensions library. On the one hand it's an excellent solution regarding features and ease of use, since all methods that are available in LINQ for IEnumerable are also available in RX for IObservable. On the other hand it is a bit disappointing regarding performance, as it is as slow as my wacky multi-threaded solution that is presented inside my question.
Update: I discarded the previous two implementations (one using the method Replay, the other using the method Publish) with a new one that uses the class Subject. This class is a low-level combination of an IObservable and IObserver. I am posting to it the items of the source IEnumerable, which are then propagated to all the IObservable<bool>'s provided by the caller. The performance is now decent, only 40% slower than Klaus Gütter's excellent solution. Also I can now break from the loop early if a condition (like All) can be determined to be false before the end of the enumeration.
public static bool AllConditions<TSource>(this IEnumerable<TSource> source,
params Func<IObservable<TSource>, IObservable<bool>>[] conditions)
{
var subject = new Subject<TSource>();
var result = true;
foreach (var condition in conditions)
{
condition(subject).SingleAsync().Subscribe(onNext: value =>
{
if (value) return;
result = false;
});
}
foreach (var item in source)
{
if (!result) break;
subject.OnNext(item);
}
return result;
}
Usage example:
var result = source.AllConditions
(
o => o.Any(n => n % 28_413_803 == 0),
o => o.All(n => n < 99_999_999),
o => o.Average(n => n).Select(v => v > 50_000_001)
);
Each condition should return an IObservable containing a single boolean value. This is not enforcible by the RX API, so I used the System.Reactive.Linq.SingleAsync method to enforce it at runtime (by throwing an exception if a result doesn't comply to this contract).
I am trying to port some code from Python to C#.
The goal is to compare adjacent elements and see if four consecutive values are bigger than one another.
Following this post (Compare two adjacent elements in same list) I was able to use pairwise on a deque to succesfully compare adjacent elements.
//In python
from more_itertools import pairwise
for x, y in pairwise(a_deque):
if (x < y):
i += 1
if (i == 4)
print("Hello world!")
The problem is that C# does not contain the more_itertools library so I am currently looking for a similar technique to achieve the same solution.
FYI. The deque implementation is from https://www.nuget.org/packages/Nito.Collections.Deque/
Any suggestions?
You can implement the python pairwise method yourself like this:
public static IEnumerable<(T, T)> Pairwise<T>(IEnumerable<T> collection)
{
using (var enumerator = collection.GetEnumerator())
{
enumerator.MoveNext();
var previous = enumerator.Current;
while (enumerator.MoveNext())
{
yield return (previous, enumerator.Current);
previous = enumerator.Current;
}
}
}
Then the algorithm in c# is structural very similar to the python version:
static void Main(string[] args)
{
var values = new[] { 5, 7, 8, 9, 10, 10, 8, 2, 1 };
var i = 0;
foreach (var (x, y) in Pairwise(values))
{
if (x < y)
{
i++;
if (i == 4)
Console.WriteLine("Hello world!");
}
}
Console.ReadLine();
}
There is a project called MoreLINQ with clever LINQ extensions. Most of the time though the code is really simple thanks to LINQ's simplicity. You can add it as a NuGet package or as individual source packages that add just the operators you want.
Pairwise.cs implements an operator that can apply a function to pairs of elements :
int[] numbers = { 123, 456, 789 };
var result = numbers.Pairwise((a, b) => a + b);
The source is really simple - retrieve an item and if we haven't reached the end, retrieve another one and apply the function :
public static IEnumerable<TResult> Pairwise<TSource, TResult>(this IEnumerable<TSource> source, Func<TSource, TSource, TResult> resultSelector)
{
if (source == null) throw new ArgumentNullException(nameof(source));
if (resultSelector == null) throw new ArgumentNullException(nameof(resultSelector));
return _();
IEnumerable<TResult> _()
{
using (var e = source.GetEnumerator())
{
if (!e.MoveNext())
yield break;
var previous = e.Current;
while (e.MoveNext())
{
yield return resultSelector(previous, e.Current);
previous = e.Current;
}
}
}
}
The only "trick" is the use of the local iterator function named ... _
You can thing of the Pairwise operator as an optimized Window operator for just 2 items.
There's another Window operator that can return sequences of N items.
This expression :
var x=Enumerable.Range(1,5).Window(3);
Produces the following arrays :
{1,2,3}
{2,3,4}
{3,4,5}
Just create the function:
static IEnumerable<(T, T)> PairWise<T>(IEnumerable<T> collection)
{
var queue = new Queue<T>();
foreach (T item in collection)
{
queue.Enqueue(item);
if (queue.Count == 2)
{
T x = queue.Dequeue();
T y = queue.Peek();
yield return (x, y);
}
}
}
and use it:
foreach ((int x, int y) in PairWise(new[] {1, 2, 3, 4, 5}))
{
Console.WriteLine($"{x} {y}");
}
You could try like this:
for (int i = 0; i < arr.Length - 1; ++i)
{
int a = arr[i];
int b = arr[i + 1];
Console.WriteLine($"{a} {b}");
}
I have the following peace of code, that factors an int to prime numbers:
private static IEnumerable<int> Factor(int input)
{
IList<int> result = new List<int>();
while (true)
{
var theSmallestDivisor = GetTheSmallestDivisor(input);
if (theSmallestDivisor == 0)
{
result.Add(input);
return result;
}
result.Add(theSmallestDivisor);
input = input/theSmallestDivisor;
}
}
I'm looking for hints on how to improve it, possibly using LINQ.
Here's an iterator version:
private static IEnumerable<int> Factor(int input)
{
while (true)
{
var theSmallestDivisor = GetTheSmallestDivisor(input);
if (theSmallestDivisor == 0)
{
yield return input;
yield break;
}
yield return theSmallestDivisor;
input = input / theSmallestDivisor;
}
}
LINQ would only make this code less readable in that particular case.
LINQ's operators are mostly designed to generating a new list from an existing list. e.g.
IEnumerable<B> LinqOperator(this IEnumerable<A> list, ...)
Not so much for generating a list from scratch as you a trying to do.
But, since you are returning IEnumerable, you may as well make it lazy:
private static IEnumerable<int> Factor(int input)
{
while (true)
{
var theSmallestDivisor = GetTheSmallestDivisor(input);
if (theSmallestDivisor == 0)
{
yield return input;
yield break;
}
yield return theSmallestDivisor;
input = input/theSmallestDivisor;
}
}
public IEnumerable<int> GetFactors(int input)
{
int first = Primes()
.TakeWhile(x => x <= Math.Sqrt(input))
.FirstOrDefault(x => input % x == 0);
return first == 0
? new[] { input }
: new[] { first }.Concat(GetFactors(input / first));
}
Let's say I have a sequence.
IEnumerable<int> sequence = GetSequenceFromExpensiveSource();
// sequence now contains: 0,1,2,3,...,999999,1000000
Getting the sequence is not cheap and is dynamically generated, and I want to iterate through it once only.
I want to get 0 - 999999 (i.e. everything but the last element)
I recognize that I could do something like:
sequence.Take(sequence.Count() - 1);
but that results in two enumerations over the big sequence.
Is there a LINQ construct that lets me do:
sequence.TakeAllButTheLastElement();
The Enumerable.SkipLast(IEnumerable<TSource>, Int32) method was added in .NET Standard 2.1. It does exactly what you want.
IEnumerable<int> sequence = GetSequenceFromExpensiveSource();
var allExceptLast = sequence.SkipLast(1);
From https://learn.microsoft.com/en-us/dotnet/api/system.linq.enumerable.skiplast
Returns a new enumerable collection that contains the elements from source with the last count elements of the source collection omitted.
I don't know a Linq solution - But you can easily code the algorithm by yourself using generators (yield return).
public static IEnumerable<T> TakeAllButLast<T>(this IEnumerable<T> source) {
var it = source.GetEnumerator();
bool hasRemainingItems = false;
bool isFirst = true;
T item = default(T);
do {
hasRemainingItems = it.MoveNext();
if (hasRemainingItems) {
if (!isFirst) yield return item;
item = it.Current;
isFirst = false;
}
} while (hasRemainingItems);
}
static void Main(string[] args) {
var Seq = Enumerable.Range(1, 10);
Console.WriteLine(string.Join(", ", Seq.Select(x => x.ToString()).ToArray()));
Console.WriteLine(string.Join(", ", Seq.TakeAllButLast().Select(x => x.ToString()).ToArray()));
}
Or as a generalized solution discarding the last n items (using a queue like suggested in the comments):
public static IEnumerable<T> SkipLastN<T>(this IEnumerable<T> source, int n) {
var it = source.GetEnumerator();
bool hasRemainingItems = false;
var cache = new Queue<T>(n + 1);
do {
if (hasRemainingItems = it.MoveNext()) {
cache.Enqueue(it.Current);
if (cache.Count > n)
yield return cache.Dequeue();
}
} while (hasRemainingItems);
}
static void Main(string[] args) {
var Seq = Enumerable.Range(1, 4);
Console.WriteLine(string.Join(", ", Seq.Select(x => x.ToString()).ToArray()));
Console.WriteLine(string.Join(", ", Seq.SkipLastN(3).Select(x => x.ToString()).ToArray()));
}
As an alternative to creating your own method and in a case the elements order is not important, the next will work:
var result = sequence.Reverse().Skip(1);
Because I'm not a fan of explicitly using an Enumerator, here's an alternative. Note that the wrapper methods are needed to let invalid arguments throw early, rather than deferring the checks until the sequence is actually enumerated.
public static IEnumerable<T> DropLast<T>(this IEnumerable<T> source)
{
if (source == null)
throw new ArgumentNullException("source");
return InternalDropLast(source);
}
private static IEnumerable<T> InternalDropLast<T>(IEnumerable<T> source)
{
T buffer = default(T);
bool buffered = false;
foreach (T x in source)
{
if (buffered)
yield return buffer;
buffer = x;
buffered = true;
}
}
As per Eric Lippert's suggestion, it easily generalizes to n items:
public static IEnumerable<T> DropLast<T>(this IEnumerable<T> source, int n)
{
if (source == null)
throw new ArgumentNullException("source");
if (n < 0)
throw new ArgumentOutOfRangeException("n",
"Argument n should be non-negative.");
return InternalDropLast(source, n);
}
private static IEnumerable<T> InternalDropLast<T>(IEnumerable<T> source, int n)
{
Queue<T> buffer = new Queue<T>(n + 1);
foreach (T x in source)
{
buffer.Enqueue(x);
if (buffer.Count == n + 1)
yield return buffer.Dequeue();
}
}
Where I now buffer before yielding instead of after yielding, so that the n == 0 case does not need special handling.
With C# 8.0 you can use Ranges and indices for that.
var allButLast = sequence[..^1];
By default C# 8.0 requires .NET Core 3.0 or .NET Standard 2.1 (or above). Check this thread to use with older implementations.
Nothing in the BCL (or MoreLinq I believe), but you could create your own extension method.
public static IEnumerable<T> TakeAllButLast<T>(this IEnumerable<T> source)
{
using (var enumerator = source.GetEnumerator())
bool first = true;
T prev;
while(enumerator.MoveNext())
{
if (!first)
yield return prev;
first = false;
prev = enumerator.Current;
}
}
}
It would be helpful if .NET Framework was shipped with extension method like this.
public static IEnumerable<T> SkipLast<T>(this IEnumerable<T> source, int count)
{
var enumerator = source.GetEnumerator();
var queue = new Queue<T>(count + 1);
while (true)
{
if (!enumerator.MoveNext())
break;
queue.Enqueue(enumerator.Current);
if (queue.Count > count)
yield return queue.Dequeue();
}
}
if you don't have time to roll out your own extension, here's a quicker way:
var next = sequence.First();
sequence.Skip(1)
.Select(s =>
{
var selected = next;
next = s;
return selected;
});
A slight expansion on Joren's elegant solution:
public static IEnumerable<T> Shrink<T>(this IEnumerable<T> source, int left, int right)
{
int i = 0;
var buffer = new Queue<T>(right + 1);
foreach (T x in source)
{
if (i >= left) // Read past left many elements at the start
{
buffer.Enqueue(x);
if (buffer.Count > right) // Build a buffer to drop right many elements at the end
yield return buffer.Dequeue();
}
else i++;
}
}
public static IEnumerable<T> WithoutLast<T>(this IEnumerable<T> source, int n = 1)
{
return source.Shrink(0, n);
}
public static IEnumerable<T> WithoutFirst<T>(this IEnumerable<T> source, int n = 1)
{
return source.Shrink(n, 0);
}
Where shrink implements a simple count forward to drop the first left many elements and the same discarded buffer to drop the last right many elements.
If you can get the Count or Length of an enumerable, which in most cases you can, then just Take(n - 1)
Example with arrays
int[] arr = new int[] { 1, 2, 3, 4, 5 };
int[] sub = arr.Take(arr.Length - 1).ToArray();
Example with IEnumerable<T>
IEnumerable<int> enu = Enumerable.Range(1, 100);
IEnumerable<int> sub = enu.Take(enu.Count() - 1);
A slight variation on the accepted answer, which (for my tastes) is a bit simpler:
public static IEnumerable<T> AllButLast<T>(this IEnumerable<T> enumerable, int n = 1)
{
// for efficiency, handle degenerate n == 0 case separately
if (n == 0)
{
foreach (var item in enumerable)
yield return item;
yield break;
}
var queue = new Queue<T>(n);
foreach (var item in enumerable)
{
if (queue.Count == n)
yield return queue.Dequeue();
queue.Enqueue(item);
}
}
Why not just .ToList<type>() on the sequence, then call count and take like you did originally..but since it's been pulled into a list, it shouldnt do an expensive enumeration twice. Right?
The solution that I use for this problem is slightly more elaborate.
My util static class contains an extension method MarkEnd which converts the T-items in EndMarkedItem<T>-items. Each element is marked with an extra int, which is either 0; or (in case one is particularly interested in the last 3 items) -3, -2, or -1 for the last 3 items.
This could be useful on its own, e.g. when you want to create a list in a simple foreach-loop with commas after each element except the last 2, with the second-to-last item followed by a conjunction word (such as “and” or “or”), and the last element followed by a point.
For generating the entire list without the last n items, the extension method ButLast simply iterates over the EndMarkedItem<T>s while EndMark == 0.
If you don’t specify tailLength, only the last item is marked (in MarkEnd()) or dropped (in ButLast()).
Like the other solutions, this works by buffering.
using System;
using System.Collections.Generic;
using System.Linq;
namespace Adhemar.Util.Linq {
public struct EndMarkedItem<T> {
public T Item { get; private set; }
public int EndMark { get; private set; }
public EndMarkedItem(T item, int endMark) : this() {
Item = item;
EndMark = endMark;
}
}
public static class TailEnumerables {
public static IEnumerable<T> ButLast<T>(this IEnumerable<T> ts) {
return ts.ButLast(1);
}
public static IEnumerable<T> ButLast<T>(this IEnumerable<T> ts, int tailLength) {
return ts.MarkEnd(tailLength).TakeWhile(te => te.EndMark == 0).Select(te => te.Item);
}
public static IEnumerable<EndMarkedItem<T>> MarkEnd<T>(this IEnumerable<T> ts) {
return ts.MarkEnd(1);
}
public static IEnumerable<EndMarkedItem<T>> MarkEnd<T>(this IEnumerable<T> ts, int tailLength) {
if (tailLength < 0) {
throw new ArgumentOutOfRangeException("tailLength");
}
else if (tailLength == 0) {
foreach (var t in ts) {
yield return new EndMarkedItem<T>(t, 0);
}
}
else {
var buffer = new T[tailLength];
var index = -buffer.Length;
foreach (var t in ts) {
if (index < 0) {
buffer[buffer.Length + index] = t;
index++;
}
else {
yield return new EndMarkedItem<T>(buffer[index], 0);
buffer[index] = t;
index++;
if (index == buffer.Length) {
index = 0;
}
}
}
if (index >= 0) {
for (var i = index; i < buffer.Length; i++) {
yield return new EndMarkedItem<T>(buffer[i], i - buffer.Length - index);
}
for (var j = 0; j < index; j++) {
yield return new EndMarkedItem<T>(buffer[j], j - index);
}
}
else {
for (var k = 0; k < buffer.Length + index; k++) {
yield return new EndMarkedItem<T>(buffer[k], k - buffer.Length - index);
}
}
}
}
}
}
public static IEnumerable<T> NoLast<T> (this IEnumerable<T> items) {
if (items != null) {
var e = items.GetEnumerator();
if (e.MoveNext ()) {
T head = e.Current;
while (e.MoveNext ()) {
yield return head; ;
head = e.Current;
}
}
}
}
I don't think it can get more succinct than this - also ensuring to Dispose the IEnumerator<T>:
public static IEnumerable<T> SkipLast<T>(this IEnumerable<T> source)
{
using (var it = source.GetEnumerator())
{
if (it.MoveNext())
{
var item = it.Current;
while (it.MoveNext())
{
yield return item;
item = it.Current;
}
}
}
}
Edit: technically identical to this answer.
This is a general and IMHO elegant solution that will handle all cases correctly:
using System;
using System.Collections.Generic;
using System.Linq;
public class Program
{
public static void Main()
{
IEnumerable<int> r = Enumerable.Range(1, 20);
foreach (int i in r.AllButLast(3))
Console.WriteLine(i);
Console.ReadKey();
}
}
public static class LinqExt
{
public static IEnumerable<T> AllButLast<T>(this IEnumerable<T> enumerable, int n = 1)
{
using (IEnumerator<T> enumerator = enumerable.GetEnumerator())
{
Queue<T> queue = new Queue<T>(n);
for (int i = 0; i < n && enumerator.MoveNext(); i++)
queue.Enqueue(enumerator.Current);
while (enumerator.MoveNext())
{
queue.Enqueue(enumerator.Current);
yield return queue.Dequeue();
}
}
}
}
You could write:
var list = xyz.Select(x=>x.Id).ToList();
list.RemoveAt(list.Count - 1);
My traditional IEnumerable approach:
/// <summary>
/// Skips first element of an IEnumerable
/// </summary>
/// <typeparam name="U">Enumerable type</typeparam>
/// <param name="models">The enumerable</param>
/// <returns>IEnumerable of type skipping first element</returns>
private IEnumerable<U> SkipFirstEnumerable<U>(IEnumerable<U> models)
{
using (var e = models.GetEnumerator())
{
if (!e.MoveNext()) return;
for (;e.MoveNext();) yield return e.Current;
yield return e.Current;
}
}
/// <summary>
/// Skips last element of an IEnumerable
/// </summary>
/// <typeparam name="U">Enumerable type</typeparam>
/// <param name="models">The enumerable</param>
/// <returns>IEnumerable of type skipping last element</returns>
private IEnumerable<U> SkipLastEnumerable<U>(IEnumerable<U> models)
{
using (var e = models.GetEnumerator())
{
if (!e.MoveNext()) return;
yield return e.Current;
for (;e.MoveNext();) yield return e.Current;
}
}
Could be:
var allBuLast = sequence.TakeWhile(e => e != sequence.Last());
I guess it should be like de "Where" but preserving the order(?).
If speed is a requirement, this old school way should be the fastest, even though the code doesn't look as smooth as linq could make it.
int[] newSequence = int[sequence.Length - 1];
for (int x = 0; x < sequence.Length - 1; x++)
{
newSequence[x] = sequence[x];
}
This requires that the sequence is an array since it has a fixed length and indexed items.
A simple way would be to just convert to a queue and dequeue until only the number of items you want to skip is left.
public static IEnumerable<T> SkipLast<T>(this IEnumerable<T> source, int n)
{
var queue = new Queue<T>(source);
while (queue.Count() > n)
{
yield return queue.Dequeue();
}
}
I would probably do something like this:
sequence.Where(x => x != sequence.LastOrDefault())
This is one iteration with a check that it isn't the last one for each time though.
If I have an IEnumerable like:
string[] items = new string[] { "a", "b", "c", "d" };
I would like to loop thru all the pairs of consecutive items (sliding window of size 2). Which would be
("a", "b"), ("b", "c"), ("c", "d")
My solution was is this
public static IEnumerable<Pair<T, T>> Pairs(IEnumerable<T> enumerable) {
IEnumerator<T> e = enumerable.GetEnumerator(); e.MoveNext();
T current = e.Current;
while ( e.MoveNext() ) {
T next = e.Current;
yield return new Pair<T, T>(current, next);
current = next;
}
}
// used like this :
foreach (Pair<String,String> pair in IterTools<String>.Pairs(items)) {
System.Out.PrintLine("{0}, {1}", pair.First, pair.Second)
}
When I wrote this code, I wondered if there are already functions in the .NET framework that do the same thing and do it not just for pairs but for any size tuples.
IMHO there should be a nice way to do this kind of sliding window operations.
I use C# 2.0 and I can imagine that with C# 3.0 (using LINQ) there are more and nicer ways to do this, but I'm primarily interested in C# 2.0 solutions. Though, I will also appreciate C# 3.0 solutions.
In .NET 4 this becomes even easier:-
var input = new[] { "a", "b", "c", "d", "e", "f" };
var result = input.Zip(input.Skip(1), (a, b) => Tuple.Create(a, b));
Rather than require a tuple (pair) type, why not just accept a selector:
public static IEnumerable<TResult> Pairwise<TSource, TResult>(this IEnumerable<TSource> source, Func<TSource, TSource, TResult> resultSelector)
{
TSource previous = default(TSource);
using (var it = source.GetEnumerator())
{
if (it.MoveNext())
previous = it.Current;
while (it.MoveNext())
yield return resultSelector(previous, previous = it.Current);
}
}
Which allows you to skip the intermediate object if you want:
string[] items = new string[] { "a", "b", "c", "d" };
var pairs = items.Pairwise((x, y) => string.Format("{0},{1}", x, y));
foreach(var pair in pairs)
Console.WriteLine(pair);
Or you can use an anonymous type:
var pairs = items.Pairwise((x, y) => new { First = x, Second = y });
Update: I just implemented this on a real project and used C# 7.0 ValueTuple instead:
public static IEnumerable<(T, T)> Pairwise<T>(this IEnumerable<T> source)
{
var previous = default(T);
using (var it = source.GetEnumerator())
{
if (it.MoveNext())
previous = it.Current;
while (it.MoveNext())
yield return (previous, previous = it.Current);
}
}
The easiest way is to use ReactiveExtensions
using System.Reactive;
using System.Reactive.Linq;
and make yourself an extension method to kit bash this together
public static IEnumerable<IList<T>> Buffer<T>(this IEnumerable<T> seq, int bufferSize, int stepSize)
{
return seq.ToObservable().Buffer(bufferSize, stepSize).ToEnumerable();
}
Just for convenience, here is a selector-less version of #dahlbyk's answer.
public static IEnumerable<Tuple<T, T>> Pairwise<T>(this IEnumerable<T> enumerable)
{
var previous = default(T);
using (var e = enumerable.GetEnumerator())
{
if (e.MoveNext())
previous = e.Current;
while (e.MoveNext())
yield return Tuple.Create(previous, previous = e.Current);
}
}
A little late to the party, but as an alternative to all these extension methods, one might use an actual "sliding" Collection to hold (and discard) the data.
Here is one I ended up making today:
public class SlidingWindowCollection<T> : ICollection<T>
{
private int _windowSize;
private Queue<T> _source;
public SlidingWindowCollection(int windowSize)
{
_windowSize = windowSize;
_source = new Queue<T>(windowSize);
}
public void Add(T item)
{
if (_source.Count == _windowSize)
{
_source.Dequeue();
}
_source.Enqueue(item);
}
public void Clear()
{
_source.Clear();
}
...and just keep forwarding all other ICollection<T> methods to _source.
}
Usage:
int pairSize = 2;
var slider = new SlidingWindowCollection<string>(pairSize);
foreach(var item in items)
{
slider.Add(item);
Console.WriteLine(string.Join(", ", slider));
}
Here is my solution using a Stack. It is short and concise.
string[] items = new string[] { "a", "b", "c", "d" };
Stack<string> stack = new Stack<string>(items.Reverse());
while(stack.Count > 1)
{
Console.WriteLine("{0},{1}", stack.Pop(), stack.Peek());
}
You can take the same concept and use a queue which avoids the need for reversing the items and is even simpler:
var queue = new Queue<string>(items);
while (queue.Count > 1)
{
Console.WriteLine("{0},{1}", queue.Dequeue(), queue.Peek());
}
A short word about performance:
I believe it's important to realize that unless you know that a task is causing a bottleneck in your real application, it's probably not worth working out what the truly fastest way of doing is. Instead, write the code which does the job for you. Also, use code you can remember, so it easily flows out of your hand the next time you need it.
Nevertheless, in case you care for some performance data for 10.000.000 random strings:
Run #1
InputZip 00:00:00.7355567
PairwiseExtension 00:00:00.5290042
Stack 00:00:00.6451204
Queue 00:00:00.3245580
ForLoop 00:00:00.7808004
TupleExtension 00:00:03.9661995
Run #2
InputZip 00:00:00.7386347
PairwiseExtension 00:00:00.5369850
Stack 00:00:00.6910079
Queue 00:00:00.3246276
ForLoop 00:00:00.8272945
TupleExtension 00:00:03.9415258
Tested using Jon Skeet's micro benchmarking tool.
If you want to take a look at the source for the test go here: gist here
Something like this:
public static IEnumerable<TResult> Pairwise<T, TResult>(this IEnumerable<T> enumerable, Func<T, T, TResult> selector)
{
var previous = enumerable.First();
foreach (var item in enumerable.Skip(1))
{
yield return selector(previous, item);
previous = item;
}
}
Expanding on the previous answer to avoid of O(n2) approach by explicitly using the passed iterator:
public static IEnumerable<IEnumerable<T>> Tuples<T>(this IEnumerable<T> input, int groupCount) {
if (null == input) throw new ArgumentException("input");
if (groupCount < 1) throw new ArgumentException("groupCount");
var e = input.GetEnumerator();
bool done = false;
while (!done) {
var l = new List<T>();
for (var n = 0; n < groupCount; ++n) {
if (!e.MoveNext()) {
if (n != 0) {
yield return l;
}
yield break;
}
l.Add(e.Current);
}
yield return l;
}
}
For C# 2, before extension methods, drop the "this" from the input parameter and call as a static method.
Forgive me if I'm overlooking something, but why not something simple, like a for loop?:
public static List <int []> ListOfPairs (int [] items)
{
List <int> output = new List <int>();
for (int i=0; i < items.Length-1; i++)
{
Int [] pair = new int [2];
pair [0]=items [i];
pair [1]=items [i+1];
output.Add (pair);
}
return output;
}
C# 3.0 solution (sorry:)
public static IEnumerable<IEnumerable<T>> Tuples<T>(this IEnumerable<T> sequence, int nTuple)
{
if(nTuple <= 0) throw new ArgumentOutOfRangeException("nTuple");
for(int i = 0; i <= sequence.Count() - nTuple; i++)
yield return sequence.Skip(i).Take(nTuple);
}
This isn't the most performant in the world, but it's sure pleasant to look at.
Really, the only thing making this a C# 3.0 solution is the .Skip.Take construct, so if you just change that to adding the elements in that range to a list instead, it should be golden for 2.0. That said, it's still not performant.
Alternate Pairs implementation, using last pair to store previous value:
static IEnumerable<Pair<T, T>> Pairs( IEnumerable<T> collection ) {
Pair<T, T> pair = null;
foreach( T item in collection ) {
if( pair == null )
pair = Pair.Create( default( T ), item );
else
yield return pair = Pair.Create( pair.Second, item );
}
}
Simple Window implementation (only safe for private use, if caller does not save returned arrays; see note):
static IEnumerable<T[]> Window( IEnumerable<T> collection, int windowSize ) {
if( windowSize < 1 )
yield break;
int index = 0;
T[] window = new T[windowSize];
foreach( var item in collection ) {
bool initializing = index < windowSize;
// Shift initialized window to accomodate new item.
if( !initializing )
Array.Copy( window, 1, window, 0, windowSize - 1 );
// Add current item to window.
int itemIndex = initializing ? index : windowSize - 1;
window[itemIndex] = item;
index++;
bool initialized = index >= windowSize;
if( initialized )
//NOTE: For public API, should return array copy to prevent
// modifcation by user, or use a different type for the window.
yield return window;
}
}
Example use:
for( int i = 0; i <= items.Length; ++i ) {
Console.WriteLine( "Window size {0}:", i );
foreach( string[] window in IterTools<string>.Window( items, i ) )
Console.WriteLine( string.Join( ", ", window ) );
Console.WriteLine( );
}
The F# Seq module defines the pairwise function over IEnumerable<T>, but this function is not in the .NET framework.
If it were already in the .NET framework, instead of returning pairs it would probably accept a selector function due to the lack of support for tuples in languages like C# and VB.
var pairs = ns.Pairwise( (a, b) => new { First = a, Second = b };
I don't think any of the answers here really improve on your simple iterator implementation, which seemed the most natural to me (and the poster dahlbyk by the looks of things!) too.
I created a slightly modified version of the late-2020-updated code in #dahlbyk's answer. It is better suited for projects with nullable reference types enabled (<Nullable>enable</Nullable>). I also added basic docs.
/// <summary>
/// Enumerates over tuples of pairs of the elements from the original sequence. I.e. { 1, 2, 3 } becomes { (1, 2), (2, 3) }. Note that { 1 } becomes { }.
/// </summary>
public static IEnumerable<(T, T)> Pairwise<T>(this IEnumerable<T> source)
{
using var it = source.GetEnumerator();
if (!it.MoveNext())
yield break;
var previous = it.Current;
while (it.MoveNext())
yield return (previous, previous = it.Current);
}
This will make a single pass through the IEnumerable.
items.Aggregate(new List<string[]>(), (list, next) => { if (list.Count > 0) list[^1][1] = next; list.Add(new[] { next, next }); return list; }, (list) => { list.RemoveAt(list.Count -1); return list; });
New C# language allows to do something like this:
var pairlist = new (string, string)[] { ("a", "b"), ("b", "c"), ("c", "d") };
foreach (var pair in pairlist)
{
//do something with pair.Item1 & pair.Item2