Related
This question already has answers here:
How to take all but the last element in a sequence using LINQ?
(22 answers)
Drop the last item with LINQ [duplicate]
(3 answers)
Closed 3 years ago.
We all know that Skip() can omit records that are not needed at the start of a collection.
But is there a way to Skip() records at the end of a collection?
How do you not take the last record in a collection?
Or do you have to do it via Take()
ie, the below code,
var collection = MyCollection
var listCount = collection.Count();
var takeList = collection.Take(listCount - 1);
Is this the only way exclude the last record in a collection?
With enumerator you can efficiently delay yielding by one enumeration.
public static IEnumerable<T> WithoutLast<T>(this IEnumerable<T> source)
{
using (IEnumerator<T> e = source.GetEnumerator())
{
if (e.MoveNext() == false) yield break;
var current = e.Current;
while (e.MoveNext())
{
yield return current;
current = e.Current;
}
}
}
Usage
var items = new int[] {};
items.WithoutLast(); // returns empty
var items = new int[] { 1 };
items.WithoutLast(); // returns empty
var items = new int[] { 1, 2 };
items.WithoutLast(); // returns { 1 }
var items = new int[] { 1, 2, 3 };
items.WithoutLast(); // returns { 1, 2 }
A slightly different version of Henrik Hansen's answer:
static public IEnumerable<TSource> SkipLast<TSource>(
this IEnumerable<TSource> source, int count)
{
if (count < 0) count = 0;
var queue = new Queue<TSource>(count + 1);
foreach (TSource item in source)
{
queue.Enqueue(item);
if (queue.Count > count) yield return queue.Dequeue();
}
}
What about this:
static public IEnumerable<T> SkipLast<T>(this IEnumerable<T> data, int count)
{
if (data == null || count < 0) yield break;
Queue<T> queue = new Queue<T>(data.Take(count));
foreach (T item in data.Skip(count))
{
queue.Enqueue(item);
yield return queue.Dequeue();
}
}
Update
With help from some reviews an optimized version building on the same idea could be:
static public IEnumerable<T> SkipLast<T>(this IEnumerable<T> data, int count)
{
if (data == null) throw new ArgumentNullException(nameof(data));
if (count <= 0) return data;
if (data is ICollection<T> collection)
return collection.Take(collection.Count - count);
IEnumerable<T> Skipper()
{
using (var enumer = data.GetEnumerator())
{
T[] queue = new T[count];
int index = 0;
while (index < count && enumer.MoveNext())
queue[index++] = enumer.Current;
index = -1;
while (enumer.MoveNext())
{
index = (index + 1) % count;
yield return queue[index];
queue[index] = enumer.Current;
}
}
}
return Skipper();
}
One way would be:
var result = l.Reverse().Skip(1);
And if needed another Reverse to get them back in the original order.
This question already has answers here:
Split List into Sublists with LINQ
(34 answers)
Closed 12 months ago.
I am attempting to split a list into a series of smaller lists.
My Problem: My function to split lists doesn't split them into lists of the correct size. It should split them into lists of size 30 but instead it splits them into lists of size 114?
How can I make my function split a list into X number of Lists of size 30 or less?
public static List<List<float[]>> splitList(List <float[]> locations, int nSize=30)
{
List<List<float[]>> list = new List<List<float[]>>();
for (int i=(int)(Math.Ceiling((decimal)(locations.Count/nSize))); i>=0; i--) {
List <float[]> subLocat = new List <float[]>(locations);
if (subLocat.Count >= ((i*nSize)+nSize))
subLocat.RemoveRange(i*nSize, nSize);
else subLocat.RemoveRange(i*nSize, subLocat.Count-(i*nSize));
Debug.Log ("Index: "+i.ToString()+", Size: "+subLocat.Count.ToString());
list.Add (subLocat);
}
return list;
}
If I use the function on a list of size 144 then the output is:
Index: 4, Size: 120
Index: 3, Size: 114
Index: 2, Size: 114
Index: 1, Size: 114
Index: 0, Size: 114
I would suggest to use this extension method to chunk the source list to the sub-lists by specified chunk size:
/// <summary>
/// Helper methods for the lists.
/// </summary>
public static class ListExtensions
{
public static List<List<T>> ChunkBy<T>(this List<T> source, int chunkSize)
{
return source
.Select((x, i) => new { Index = i, Value = x })
.GroupBy(x => x.Index / chunkSize)
.Select(x => x.Select(v => v.Value).ToList())
.ToList();
}
}
For example, if you chunk the list of 18 items by 5 items per chunk, it gives you the list of 4 sub-lists with the following items inside: 5-5-5-3.
NOTE: at the upcoming improvements to LINQ in .NET 6 chunking
will come out of the box like this:
const int PAGE_SIZE = 5;
IEnumerable<Movie[]> chunks = movies.Chunk(PAGE_SIZE);
public static List<List<float[]>> SplitList(List<float[]> locations, int nSize=30)
{
var list = new List<List<float[]>>();
for (int i = 0; i < locations.Count; i += nSize)
{
list.Add(locations.GetRange(i, Math.Min(nSize, locations.Count - i)));
}
return list;
}
Generic version:
public static IEnumerable<List<T>> SplitList<T>(List<T> locations, int nSize=30)
{
for (int i = 0; i < locations.Count; i += nSize)
{
yield return locations.GetRange(i, Math.Min(nSize, locations.Count - i));
}
}
how about:
while(locations.Any())
{
list.Add(locations.Take(nSize).ToList());
locations= locations.Skip(nSize).ToList();
}
Library MoreLinq have method called Batch
List<int> ids = new List<int>() { 1, 2, 3, 4, 5, 6, 7, 8, 9, 0 }; // 10 elements
int counter = 1;
foreach(var batch in ids.Batch(2))
{
foreach(var eachId in batch)
{
Console.WriteLine("Batch: {0}, Id: {1}", counter, eachId);
}
counter++;
}
Result is
Batch: 1, Id: 1
Batch: 1, Id: 2
Batch: 2, Id: 3
Batch: 2, Id: 4
Batch: 3, Id: 5
Batch: 3, Id: 6
Batch: 4, Id: 7
Batch: 4, Id: 8
Batch: 5, Id: 9
Batch: 5, Id: 0
ids are splitted into 5 chunks with 2 elements.
Update for .NET 6
var originalList = new List<int>{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}
// split into arrays of no more than three
IEnumerable<int[]> chunks = originalList.Chunk(3);
Prior to .NET 6
public static IEnumerable<IEnumerable<T>> SplitIntoSets<T>
(this IEnumerable<T> source, int itemsPerSet)
{
var sourceList = source as List<T> ?? source.ToList();
for (var index = 0; index < sourceList.Count; index += itemsPerSet)
{
yield return sourceList.Skip(index).Take(itemsPerSet);
}
}
Serj-Tm solution is fine, also this is the generic version as extension method for lists (put it into a static class):
public static List<List<T>> Split<T>(this List<T> items, int sliceSize = 30)
{
List<List<T>> list = new List<List<T>>();
for (int i = 0; i < items.Count; i += sliceSize)
list.Add(items.GetRange(i, Math.Min(sliceSize, items.Count - i)));
return list;
}
I find accepted answer (Serj-Tm) most robust, but I'd like to suggest a generic version.
public static List<List<T>> splitList<T>(List<T> locations, int nSize = 30)
{
var list = new List<List<T>>();
for (int i = 0; i < locations.Count; i += nSize)
{
list.Add(locations.GetRange(i, Math.Min(nSize, locations.Count - i)));
}
return list;
}
Addition after very useful comment of mhand at the end
Original answer
Although most solutions might work, I think they are not very efficiently. Suppose if you only want the first few items of the first few chunks. Then you wouldn't want to iterate over all (zillion) items in your sequence.
The following will at utmost enumerate twice: once for the Take and once for the Skip. It won't enumerate over any more elements than you will use:
public static IEnumerable<IEnumerable<TSource>> ChunkBy<TSource>
(this IEnumerable<TSource> source, int chunkSize)
{
while (source.Any()) // while there are elements left
{ // still something to chunk:
yield return source.Take(chunkSize); // return a chunk of chunkSize
source = source.Skip(chunkSize); // skip the returned chunk
}
}
How many times will this Enumerate the sequence?
Suppose you divide your source into chunks of chunkSize. You enumerate only the first N chunks. From every enumerated chunk you'll only enumerate the first M elements.
While(source.Any())
{
...
}
the Any will get the Enumerator, do 1 MoveNext() and returns the returned value after Disposing the Enumerator. This will be done N times
yield return source.Take(chunkSize);
According to the reference source this will do something like:
public static IEnumerable<TSource> Take<TSource>(this IEnumerable<TSource> source, int count)
{
return TakeIterator<TSource>(source, count);
}
static IEnumerable<TSource> TakeIterator<TSource>(IEnumerable<TSource> source, int count)
{
foreach (TSource element in source)
{
yield return element;
if (--count == 0) break;
}
}
This doesn't do a lot until you start enumerating over the fetched Chunk. If you fetch several Chunks, but decide not to enumerate over the first Chunk, the foreach is not executed, as your debugger will show you.
If you decide to take the first M elements of the first chunk then the yield return is executed exactly M times. This means:
get the enumerator
call MoveNext() and Current M times.
Dispose the enumerator
After the first chunk has been yield returned, we skip this first Chunk:
source = source.Skip(chunkSize);
Once again: we'll take a look at reference source to find the skipiterator
static IEnumerable<TSource> SkipIterator<TSource>(IEnumerable<TSource> source, int count)
{
using (IEnumerator<TSource> e = source.GetEnumerator())
{
while (count > 0 && e.MoveNext()) count--;
if (count <= 0)
{
while (e.MoveNext()) yield return e.Current;
}
}
}
As you see, the SkipIterator calls MoveNext() once for every element in the Chunk. It doesn't call Current.
So per Chunk we see that the following is done:
Any(): GetEnumerator; 1 MoveNext(); Dispose Enumerator;
Take():
nothing if the content of the chunk is not enumerated.
If the content is enumerated: GetEnumerator(), one MoveNext and one Current per enumerated item, Dispose enumerator;
Skip(): for every chunk that is enumerated (NOT the contents of the chunk):
GetEnumerator(), MoveNext() chunkSize times, no Current! Dispose enumerator
If you look at what happens with the enumerator, you'll see that there are a lot of calls to MoveNext(), and only calls to Current for the TSource items you actually decide to access.
If you take N Chunks of size chunkSize, then calls to MoveNext()
N times for Any()
not yet any time for Take, as long as you don't enumerate the Chunks
N times chunkSize for Skip()
If you decide to enumerate only the first M elements of every fetched chunk, then you need to call MoveNext M times per enumerated Chunk.
The total
MoveNext calls: N + N*M + N*chunkSize
Current calls: N*M; (only the items you really access)
So if you decide to enumerate all elements of all chunks:
MoveNext: numberOfChunks + all elements + all elements = about twice the sequence
Current: every item is accessed exactly once
Whether MoveNext is a lot of work or not, depends on the type of source sequence. For lists and arrays it is a simple index increment, with maybe an out of range check.
But if your IEnumerable is the result of a database query, make sure that the data is really materialized on your computer, otherwise the data will be fetched several times. DbContext and Dapper will properly transfer the data to local process before it can be accessed. If you enumerate the same sequence several times it is not fetched several times. Dapper returns an object that is a List, DbContext remembers that the data is already fetched.
It depends on your Repository whether it is wise to call AsEnumerable() or ToLists() before you start to divide the items in Chunks
While plenty of the answers above do the job, they all fail horribly on a never ending sequence (or a really long sequence). The following is a completely on-line implementation which guarantees best time and memory complexity possible. We only iterate the source enumerable exactly once and use yield return for lazy evaluation. The consumer could throw away the list on each iteration making the memory footprint equal to that of the list w/ batchSize number of elements.
public static IEnumerable<List<T>> BatchBy<T>(this IEnumerable<T> enumerable, int batchSize)
{
using (var enumerator = enumerable.GetEnumerator())
{
List<T> list = null;
while (enumerator.MoveNext())
{
if (list == null)
{
list = new List<T> {enumerator.Current};
}
else if (list.Count < batchSize)
{
list.Add(enumerator.Current);
}
else
{
yield return list;
list = new List<T> {enumerator.Current};
}
}
if (list?.Count > 0)
{
yield return list;
}
}
}
EDIT: Just now realizing the OP asks about breaking a List<T> into smaller List<T>, so my comments regarding infinite enumerables aren't applicable to the OP, but may help others who end up here. These comments were in response to other posted solutions that do use IEnumerable<T> as an input to their function, yet enumerate the source enumerable multiple times.
I have a generic method that would take any types include float, and it's been unit-tested, hope it helps:
/// <summary>
/// Breaks the list into groups with each group containing no more than the specified group size
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="values">The values.</param>
/// <param name="groupSize">Size of the group.</param>
/// <returns></returns>
public static List<List<T>> SplitList<T>(IEnumerable<T> values, int groupSize, int? maxCount = null)
{
List<List<T>> result = new List<List<T>>();
// Quick and special scenario
if (values.Count() <= groupSize)
{
result.Add(values.ToList());
}
else
{
List<T> valueList = values.ToList();
int startIndex = 0;
int count = valueList.Count;
int elementCount = 0;
while (startIndex < count && (!maxCount.HasValue || (maxCount.HasValue && startIndex < maxCount)))
{
elementCount = (startIndex + groupSize > count) ? count - startIndex : groupSize;
result.Add(valueList.GetRange(startIndex, elementCount));
startIndex += elementCount;
}
}
return result;
}
As of .NET 6.0, you can use the LINQ extension Chunk<T>() to split enumerations into chunks. Docs
var chars = new List<char>() { 'h', 'e', 'l', 'l', 'o', 'w','o','r' ,'l','d' };
foreach (var batch in chars.Chunk(2))
{
foreach (var ch in batch)
{
// iterates 2 letters at a time
}
}
public static IEnumerable<IEnumerable<T>> Batch<T>(this IEnumerable<T> items, int maxItems)
{
return items.Select((item, index) => new { item, index })
.GroupBy(x => x.index / maxItems)
.Select(g => g.Select(x => x.item));
}
How about this one? The idea was to use only one loop. And, who knows, maybe you're using only IList implementations thorough your code and you don't want to cast to List.
private IEnumerable<IList<T>> SplitList<T>(IList<T> list, int totalChunks)
{
IList<T> auxList = new List<T>();
int totalItems = list.Count();
if (totalChunks <= 0)
{
yield return auxList;
}
else
{
for (int i = 0; i < totalItems; i++)
{
auxList.Add(list[i]);
if ((i + 1) % totalChunks == 0)
{
yield return auxList;
auxList = new List<T>();
}
else if (i == totalItems - 1)
{
yield return auxList;
}
}
}
}
In .NET 6 you can just use source.Chunk(chunkSize)
A more generic version based on the accepted answer by Serj-Tm.
public static IEnumerable<IEnumerable<T>> Split<T>(IEnumerable<T> source, int size = 30)
{
var count = source.Count();
for (int i = 0; i < count; i += size)
{
yield return source
.Skip(Math.Min(size, count - i))
.Take(size);
}
}
One more
public static IList<IList<T>> SplitList<T>(this IList<T> list, int chunkSize)
{
var chunks = new List<IList<T>>();
List<T> chunk = null;
for (var i = 0; i < list.Count; i++)
{
if (i % chunkSize == 0)
{
chunk = new List<T>(chunkSize);
chunks.Add(chunk);
}
chunk.Add(list[i]);
}
return chunks;
}
public static List<List<T>> ChunkBy<T>(this List<T> source, int chunkSize)
{
var result = new List<List<T>>();
for (int i = 0; i < source.Count; i += chunkSize)
{
var rows = new List<T>();
for (int j = i; j < i + chunkSize; j++)
{
if (j >= source.Count) break;
rows.Add(source[j]);
}
result.Add(rows);
}
return result;
}
I had encountered this same need, and I used a combination of Linq's Skip() and Take() methods. I multiply the number I take by the number of iterations this far, and that gives me the number of items to skip, then I take the next group.
var categories = Properties.Settings.Default.MovementStatsCategories;
var items = summariesWithinYear
.Select(s => s.sku).Distinct().ToList();
//need to run by chunks of 10,000
var count = items.Count;
var counter = 0;
var numToTake = 10000;
while (count > 0)
{
var itemsChunk = items.Skip(numToTake * counter).Take(numToTake).ToList();
counter += 1;
MovementHistoryUtilities.RecordMovementHistoryStatsBulk(itemsChunk, categories, nLogger);
count -= numToTake;
}
Based on Dimitry Pavlov answere I would remove .ToList(). And also avoid the anonymous class.
Instead I like to use a struct which does not require a heap memory allocation. (A ValueTuple would also do job.)
public static IEnumerable<IEnumerable<TSource>> ChunkBy<TSource>(this IEnumerable<TSource> source, int chunkSize)
{
if (source is null)
{
throw new ArgumentNullException(nameof(source));
}
if (chunkSize <= 0)
{
throw new ArgumentOutOfRangeException(nameof(chunkSize), chunkSize, "The argument must be greater than zero.");
}
return source
.Select((x, i) => new ChunkedValue<TSource>(x, i / chunkSize))
.GroupBy(cv => cv.ChunkIndex)
.Select(g => g.Select(cv => cv.Value));
}
[StructLayout(LayoutKind.Auto)]
[DebuggerDisplay("{" + nameof(ChunkedValue<T>.ChunkIndex) + "}: {" + nameof(ChunkedValue<T>.Value) + "}")]
private struct ChunkedValue<T>
{
public ChunkedValue(T value, int chunkIndex)
{
this.ChunkIndex = chunkIndex;
this.Value = value;
}
public int ChunkIndex { get; }
public T Value { get; }
}
This can be used like the following which only iterates over the collection once and
also does not allocate any significant memory.
int chunkSize = 30;
foreach (var chunk in collection.ChunkBy(chunkSize))
{
foreach (var item in chunk)
{
// your code for item here.
}
}
If a concrete list is actually needed then I would do it like this:
int chunkSize = 30;
var chunkList = new List<List<T>>();
foreach (var chunk in collection.ChunkBy(chunkSize))
{
// create a list with the correct capacity to be able to contain one chunk
// to avoid the resizing (additional memory allocation and memory copy) within the List<T>.
var list = new List<T>(chunkSize);
list.AddRange(chunk);
chunkList.Add(list);
}
List<int> orginalList =new List<int>(){1,2,3,4,5,6,7,8,9,10,12};
Dictionary<int,List<int>> dic = new Dictionary <int,List<int>> ();
int batchcount = orginalList.Count/2; //To List into two 2 parts if you
want three give three
List<int> lst = new List<int>();
for (int i=0;i<orginalList.Count; i++)
{
lst.Add(orginalList[i]);
if (i % batchCount == 0 && i!=0)
{
Dic.Add(threadId, lst);
lst = new List<int>();**strong text**
threadId++;
}
}
if(lst.Count>0)
Dic.Add(threadId, lst); //in case if any dayleft
foreach(int BatchId in Dic.Keys)
{
Console.Writeline("BatchId:"+BatchId);
Console.Writeline('Batch Count:"+Dic[BatchId].Count);
}
in case you wanna split it with condition instead of fixed number :
///<summary>
/// splits a list based on a condition (similar to the split function for strings)
///</summary>
public static IEnumerable<List<T>> Split<T>(this IEnumerable<T> src, Func<T, bool> pred)
{
var list = new List<T>();
foreach(T item in src)
{
if(pred(item))
{
if(list != null && list.Count > 0)
yield return list;
list = new List<T>();
}
else
{
list.Add(item);
}
}
}
You can simply try the following code with only using LINQ :
public static IList<IList<T>> Split<T>(IList<T> source)
{
return source
.Select((x, i) => new { Index = i, Value = x })
.GroupBy(x => x.Index / 3)
.Select(x => x.Select(v => v.Value).ToList())
.ToList();
}
I need an elegant method that takes an enumerable and gets the enumerable of enumerables each of the same number of elements in it but the last one:
public static IEnumerable<IEnumerable<TValue>> Chunk<TValue>(this IEnumerable<TValue> values, Int32 chunkSize)
{
// TODO: code that chunks
}
This is what I have tried:
public static IEnumerable<IEnumerable<TValue>> Chunk<TValue>(this IEnumerable<TValue> values, Int32 chunkSize)
{
var count = values.Count();
var numberOfFullChunks = count / chunkSize;
var lastChunkSize = count % chunkSize;
for (var chunkIndex = 0; chunkSize < numberOfFullChunks; chunkSize++)
{
yield return values.Skip(chunkSize * chunkIndex).Take(chunkSize);
}
if (lastChunkSize > 0)
{
yield return values.Skip(chunkSize * count).Take(lastChunkSize);
}
}
UPDATE
Just discovered there was a similar topic about splitting a list Split List into Sublists with LINQ
If memory consumption isn't a concern, then like this?
static class Ex
{
public static IEnumerable<IEnumerable<TValue>> Chunk<TValue>(
this IEnumerable<TValue> values,
int chunkSize)
{
return values
.Select((v, i) => new {v, groupIndex = i / chunkSize})
.GroupBy(x => x.groupIndex)
.Select(g => g.Select(x => x.v));
}
}
Otherwise you could get creative with the yield keyword, like so:
static class Ex
{
public static IEnumerable<IEnumerable<TValue>> Chunk<TValue>(
this IEnumerable<TValue> values,
int chunkSize)
{
using(var enumerator = values.GetEnumerator())
{
while(enumerator.MoveNext())
{
yield return GetChunk(enumerator, chunkSize).ToList();
}
}
}
private static IEnumerable<T> GetChunk<T>(
IEnumerator<T> enumerator,
int chunkSize)
{
do
{
yield return enumerator.Current;
} while(--chunkSize > 0 && enumerator.MoveNext());
}
}
>= .Net 6
Built-in Enumerable.Chunk method:
// Giving an enumerable
var e = Enumerable.Range(1, 999);
// Here it is. Enjoy :)
var chunks = e.Chunk(29);
// Sample, iterating over chunks
foreach(var chunk in chunks) // for each chunk
{
foreach(var item in chunk) // for each item in a chunk
{
Console.WriteLine(item);
}
}
< .Net 6 ?
Copy and paste MS Chunk source code into your project. Just few lines of code.
Here is a extension method using Take and Skip:
public static IList<IList<T>> Chunk<T>(this IList<T> source, int chunksize)
{
while (source.Any())
{
yield return source.Take(chunksize);
source = source.Skip(chunksize);
}
}
(updated to use IList rather than IEnumerable)
If you don't have .net 6, you might opt to patch the Chunk method from it into your project. The only adaptations you'll likely need to make are in relation to the exception helpers the .net source uses, as your own project probably won't have ThrowHelper in.
Their code:
ThrowHelper.ThrowArgumentNullException(ExceptionArgument.source);
would probably be more like:
throw new ArgumentNullException(nameof(source));
The following code block has had these adjustments applied; you can make a new file called Chunk.cs and drop the following code into it:
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
using System.Collections.Generic;
namespace System.Linq
{
public static partial class Enumerable
{
/// <summary>
/// Split the elements of a sequence into chunks of size at most <paramref name="size"/>.
/// </summary>
/// <remarks>
/// Every chunk except the last will be of size <paramref name="size"/>.
/// The last chunk will contain the remaining elements and may be of a smaller size.
/// </remarks>
/// <param name="source">
/// An <see cref="IEnumerable{T}"/> whose elements to chunk.
/// </param>
/// <param name="size">
/// Maximum size of each chunk.
/// </param>
/// <typeparam name="TSource">
/// The type of the elements of source.
/// </typeparam>
/// <returns>
/// An <see cref="IEnumerable{T}"/> that contains the elements the input sequence split into chunks of size <paramref name="size"/>.
/// </returns>
/// <exception cref="ArgumentNullException">
/// <paramref name="source"/> is null.
/// </exception>
/// <exception cref="ArgumentOutOfRangeException">
/// <paramref name="size"/> is below 1.
/// </exception>
public static IEnumerable<TSource[]> Chunk<TSource>(this IEnumerable<TSource> source, int size)
{
if (source == null)
{
throw new ArgumentNullException(nameof(source));
}
if (size < 1)
{
throw new ArgumentOutOfRangeException(nameof(size));
}
return ChunkIterator(source, size);
}
private static IEnumerable<TSource[]> ChunkIterator<TSource>(IEnumerable<TSource> source, int size)
{
using IEnumerator<TSource> e = source.GetEnumerator();
while (e.MoveNext())
{
TSource[] chunk = new TSource[size];
chunk[0] = e.Current;
int i = 1;
for (; i < chunk.Length && e.MoveNext(); i++)
{
chunk[i] = e.Current;
}
if (i == chunk.Length)
{
yield return chunk;
}
else
{
Array.Resize(ref chunk, i);
yield return chunk;
yield break;
}
}
}
}
}
You should verify that incorporating their MIT licensed code into your project does not unduly impact your own license intentions
Only had some quick testing but this seems to work:
public static IEnumerable<IEnumerable<TValue>> Chunk<TValue>(this IEnumerable<TValue> values, Int32 chunkSize)
{
var valuesList = values.ToList();
var count = valuesList.Count();
for (var i = 0; i < (count / chunkSize) + (count % chunkSize == 0 ? 0 : 1); i++)
{
yield return valuesList.Skip(i * chunkSize).Take(chunkSize);
}
}
As other answers have already pointed out, from .NET 6 upwards, there is an Enumerable.Chunk extension method.
Unfortunately (in my opinion), this method returns an IEnumerable<T[]>, which undermines the memory-conservation benefits of processing an IEnumerable<T> one element at a time:
public IEnumerable<HugeObject> CreateHugeObjects(int count) {
for (var i = 0; i < count; ++i) {
yield return new HugeObject(i);
}
}
public static int AggregateSomehow(IEnumerable<HugeObject> chunk) {
return 0;
}
public void Consume() {
var source = CreateHugeObjects(1000);
var chunks = source.Chunk(100);
var result = chunks.Select(AggregateSomehow);
}
In this example, the underlying type of chunk in AggregateSomehow will be HugeObject[100], meaning that 100 instances of HugeObject have to be loaded into memory simultaneously to perform the method call.
Before the availability of Enumerable.Chunk, I used to write my own extension named Partition like so:
public static IEnumerable<IEnumerable<T>> Partition<T>(this IEnumerable<T> source, int size) {
if (source == null) {
throw new ArgumentNullException(nameof(source));
}
if (size < 1) {
throw new ArgumentOutOfRangeException(nameof(size));
}
using var e = source.GetEnumerator();
while (e.MoveNext()) {
yield return new Partitioner<T>(e, size);
}
}
private class Partitioner<T> : IEnumerable<T>
{
private class PartitionerEnumerator : IEnumerator<T>
{
private readonly IEnumerator<T> m_Source;
private readonly int m_Size;
private int m_Index = -1;
private bool m_Disposed = false;
public PartitionerEnumerator(IEnumerator<T> source, int size) {
m_Source = source;
m_Size = size;
}
public T Current => m_Source.Current;
object IEnumerator.Current => Current;
public void Dispose() {
if (!m_Disposed) {
m_Disposed = true;
while (++m_Index < m_Size && m_Source.MoveNext()) { }
}
}
public bool MoveNext() {
if (m_Index == -1) {
++m_Index;
return true;
} else {
return ++m_Index < m_Size && m_Source.MoveNext();
}
}
public void Reset() => throw new NotImplementedException();
}
private readonly PartitionerEnumerator m_Enumerator;
public Partitioner(IEnumerator<T> source, int size) {
m_Enumerator = new PartitionerEnumerator(source, size);
}
public IEnumerator<T> GetEnumerator() => m_Enumerator;
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
This approach takes into account three considerations:
The original source is only enumerated once (which is often missed by Skip/Take implementations)
Within normal nested/chained LINQ expressions, only one element at a time has to be in memory (which is ignored by the current implementation)
When any partition is processed only partly and then disposed prematurely, the PartitionerEnumerator.Dispose method makes sure that the underlying enumerator still gets forwarded the rest of the count (which is often missed by nested-loop approaches:)
public static IEnumerable<IEnumerable<T>> PartitionWrong<T>(this IEnumerable<T> source, int size) {
if (source == null) {
throw new ArgumentNullException(nameof(source));
}
if (size < 1) {
throw new ArgumentOutOfRangeException(nameof(size));
}
static IEnumerable<T> EnumeratePartition(IEnumerator<T> e, int size) {
var i = 0;
do {
yield return e.Current;
} while (++i < size && e.MoveNext())
}
using (var e = source.GetEnumerator()) {
while (e.MoveNext()) {
yield return EnumeratePartition(e, size);
}
}
}
This approach will work if all sub-sequences are fully enumerated, e. g. by calling Count or Sum on them, but it fails for partial enumeration like calling First on them:
var source = new int[] { 1, 2, 3, 4, 5, 6, 7, 8, 9 };
source.PartitionWrong(3).Select(c => c.Count()); // 3, 3, 3
source.PartitionWrong(3).Select(c => c.Sum()); // 6, 15, 24
source.PartitionWrong(3).Select(c => c.First()); // 1, 2, 3, 4, 5, 6, 7, 8, 9 but should be 1, 4, 7
My implementation will work for all of the above, but still has several shortcomings, which weren't relevant for my applications, but the first two are probably the reason why the .NET team chose "the easy way out" and use an array that gets filled immediately:
If you save all Partition objects at first, then were to process them round-robin, one element at a time, the order of the original IEnumerable is not preserved in the partitions, i. e. the first partition is not guaranteed to contain the first three elements. As a side effect, if the number of elements does not divide evenly into the partition size, it is "random" which partition is shorter than size. It is not even necessarily guaranteed that only one partition is shorter.
Using this in a parallelized context suffers from the same issues as (1), but TBH I never even looked into the thread-safetiness of my code.
The benefits of pre-mature enumeration abortion (like calling Any or All on the sub-sequences) will not prevent the rest of the currently enumerated Partion's elements to be created (although this is obviously true for Chunk as well, where all elements got created upon entering the chunk)
So in a nutshell - if you are not planning on using parallelization or aren't dependant on ordered processing, and run into a memory problem when using .NET 6's Chunk, my old code might be helpful to you.
Given a collection, is there a way to get the last N elements of that collection? If there isn't a method in the framework, what would be the best way to write an extension method to do this?
collection.Skip(Math.Max(0, collection.Count() - N));
This approach preserves item order without a dependency on any sorting, and has broad compatibility across several LINQ providers.
It is important to take care not to call Skip with a negative number. Some providers, such as the Entity Framework, will produce an ArgumentException when presented with a negative argument. The call to Math.Max avoids this neatly.
The class below has all of the essentials for extension methods, which are: a static class, a static method, and use of the this keyword.
public static class MiscExtensions
{
// Ex: collection.TakeLast(5);
public static IEnumerable<T> TakeLast<T>(this IEnumerable<T> source, int N)
{
return source.Skip(Math.Max(0, source.Count() - N));
}
}
A brief note on performance:
Because the call to Count() can cause enumeration of certain data structures, this approach has the risk of causing two passes over the data. This isn't really a problem with most enumerables; in fact, optimizations exist already for Lists, Arrays, and even EF queries to evaluate the Count() operation in O(1) time.
If, however, you must use a forward-only enumerable and would like to avoid making two passes, consider a one-pass algorithm like Lasse V. Karlsen or Mark Byers describe. Both of these approaches use a temporary buffer to hold items while enumerating, which are yielded once the end of the collection is found.
coll.Reverse().Take(N).Reverse().ToList();
public static IEnumerable<T> TakeLast<T>(this IEnumerable<T> coll, int N)
{
return coll.Reverse().Take(N).Reverse();
}
UPDATE: To address clintp's problem: a) Using the TakeLast() method I defined above solves the problem, but if you really want the do it without the extra method, then you just have to recognize that while Enumerable.Reverse() can be used as an extension method, you aren't required to use it that way:
List<string> mystring = new List<string>() { "one", "two", "three" };
mystring = Enumerable.Reverse(mystring).Take(2).Reverse().ToList();
.NET Core 2.0+ provides the LINQ method TakeLast():
https://learn.microsoft.com/en-us/dotnet/api/system.linq.enumerable.takelast
example:
Enumerable
.Range(1, 10)
.TakeLast(3) // <--- takes last 3 items
.ToList()
.ForEach(i => System.Console.WriteLine(i))
// outputs:
// 8
// 9
// 10
Note: I missed your question title which said Using Linq, so my answer does not in fact use Linq.
If you want to avoid caching a non-lazy copy of the entire collection, you could write a simple method that does it using a linked list.
The following method will add each value it finds in the original collection into a linked list, and trim the linked list down to the number of items required. Since it keeps the linked list trimmed to this number of items the entire time through iterating through the collection, it will only keep a copy of at most N items from the original collection.
It does not require you to know the number of items in the original collection, nor iterate over it more than once.
Usage:
IEnumerable<int> sequence = Enumerable.Range(1, 10000);
IEnumerable<int> last10 = sequence.TakeLast(10);
...
Extension method:
public static class Extensions
{
public static IEnumerable<T> TakeLast<T>(this IEnumerable<T> collection,
int n)
{
if (collection == null)
throw new ArgumentNullException(nameof(collection));
if (n < 0)
throw new ArgumentOutOfRangeException(nameof(n), $"{nameof(n)} must be 0 or greater");
LinkedList<T> temp = new LinkedList<T>();
foreach (var value in collection)
{
temp.AddLast(value);
if (temp.Count > n)
temp.RemoveFirst();
}
return temp;
}
}
Here's a method that works on any enumerable but uses only O(N) temporary storage:
public static class TakeLastExtension
{
public static IEnumerable<T> TakeLast<T>(this IEnumerable<T> source, int takeCount)
{
if (source == null) { throw new ArgumentNullException("source"); }
if (takeCount < 0) { throw new ArgumentOutOfRangeException("takeCount", "must not be negative"); }
if (takeCount == 0) { yield break; }
T[] result = new T[takeCount];
int i = 0;
int sourceCount = 0;
foreach (T element in source)
{
result[i] = element;
i = (i + 1) % takeCount;
sourceCount++;
}
if (sourceCount < takeCount)
{
takeCount = sourceCount;
i = 0;
}
for (int j = 0; j < takeCount; ++j)
{
yield return result[(i + j) % takeCount];
}
}
}
Usage:
List<int> l = new List<int> {4, 6, 3, 6, 2, 5, 7};
List<int> lastElements = l.TakeLast(3).ToList();
It works by using a ring buffer of size N to store the elements as it sees them, overwriting old elements with new ones. When the end of the enumerable is reached the ring buffer contains the last N elements.
I am surprised that no one has mentioned it, but SkipWhile does have a method that uses the element's index.
public static IEnumerable<T> TakeLastN<T>(this IEnumerable<T> source, int n)
{
if (source == null)
throw new ArgumentNullException("Source cannot be null");
int goldenIndex = source.Count() - n;
return source.SkipWhile((val, index) => index < goldenIndex);
}
//Or if you like them one-liners (in the spirit of the current accepted answer);
//However, this is most likely impractical due to the repeated calculations
collection.SkipWhile((val, index) => index < collection.Count() - N)
The only perceivable benefit that this solution presents over others is that you can have the option to add in a predicate to make a more powerful and efficient LINQ query, instead of having two separate operations that traverse the IEnumerable twice.
public static IEnumerable<T> FilterLastN<T>(this IEnumerable<T> source, int n, Predicate<T> pred)
{
int goldenIndex = source.Count() - n;
return source.SkipWhile((val, index) => index < goldenIndex && pred(val));
}
Use EnumerableEx.TakeLast in RX's System.Interactive assembly. It's an O(N) implementation like #Mark's, but it uses a queue rather than a ring-buffer construct (and dequeues items when it reaches buffer capacity).
(NB: This is the IEnumerable version - not the IObservable version, though the implementation of the two is pretty much identical)
If you are dealing with a collection with a key (e.g. entries from a database) a quick (i.e. faster than the selected answer) solution would be
collection.OrderByDescending(c => c.Key).Take(3).OrderBy(c => c.Key);
If you don't mind dipping into Rx as part of the monad, you can use TakeLast:
IEnumerable<int> source = Enumerable.Range(1, 10000);
IEnumerable<int> lastThree = source.AsObservable().TakeLast(3).AsEnumerable();
I tried to combine efficiency and simplicity and end up with this :
public static IEnumerable<T> TakeLast<T>(this IEnumerable<T> source, int count)
{
if (source == null) { throw new ArgumentNullException("source"); }
Queue<T> lastElements = new Queue<T>();
foreach (T element in source)
{
lastElements.Enqueue(element);
if (lastElements.Count > count)
{
lastElements.Dequeue();
}
}
return lastElements;
}
About
performance : In C#, Queue<T> is implemented using a circular buffer so there is no object instantiation done each loop (only when the queue is growing up). I did not set queue capacity (using dedicated constructor) because someone might call this extension with count = int.MaxValue . For extra performance you might check if source implement IList<T> and if yes, directly extract the last values using array indexes.
If using a third-party library is an option, MoreLinq defines TakeLast() which does exactly this.
It is a little inefficient to take the last N of a collection using LINQ as all the above solutions require iterating across the collection. TakeLast(int n) in System.Interactive also has this problem.
If you have a list a more efficient thing to do is slice it using the following method
/// Select from start to end exclusive of end using the same semantics
/// as python slice.
/// <param name="list"> the list to slice</param>
/// <param name="start">The starting index</param>
/// <param name="end">The ending index. The result does not include this index</param>
public static List<T> Slice<T>
(this IReadOnlyList<T> list, int start, int? end = null)
{
if (end == null)
{
end = list.Count();
}
if (start < 0)
{
start = list.Count + start;
}
if (start >= 0 && end.Value > 0 && end.Value > start)
{
return list.GetRange(start, end.Value - start);
}
if (end < 0)
{
return list.GetRange(start, (list.Count() + end.Value) - start);
}
if (end == start)
{
return new List<T>();
}
throw new IndexOutOfRangeException(
"count = " + list.Count() +
" start = " + start +
" end = " + end);
}
with
public static List<T> GetRange<T>( this IReadOnlyList<T> list, int index, int count )
{
List<T> r = new List<T>(count);
for ( int i = 0; i < count; i++ )
{
int j=i + index;
if ( j >= list.Count )
{
break;
}
r.Add(list[j]);
}
return r;
}
and some test cases
[Fact]
public void GetRange()
{
IReadOnlyList<int> l = new List<int>() { 0, 10, 20, 30, 40, 50, 60 };
l
.GetRange(2, 3)
.ShouldAllBeEquivalentTo(new[] { 20, 30, 40 });
l
.GetRange(5, 10)
.ShouldAllBeEquivalentTo(new[] { 50, 60 });
}
[Fact]
void SliceMethodShouldWork()
{
var list = new List<int>() { 1, 3, 5, 7, 9, 11 };
list.Slice(1, 4).ShouldBeEquivalentTo(new[] { 3, 5, 7 });
list.Slice(1, -2).ShouldBeEquivalentTo(new[] { 3, 5, 7 });
list.Slice(1, null).ShouldBeEquivalentTo(new[] { 3, 5, 7, 9, 11 });
list.Slice(-2)
.Should()
.BeEquivalentTo(new[] {9, 11});
list.Slice(-2,-1 )
.Should()
.BeEquivalentTo(new[] {9});
}
I know it's to late to answer this question. But if you are working with collection of type IList<> and you don't care about an order of the returned collection, then this method is working faster. I've used Mark Byers answer and made a little changes. So now method TakeLast is:
public static IEnumerable<T> TakeLast<T>(IList<T> source, int takeCount)
{
if (source == null) { throw new ArgumentNullException("source"); }
if (takeCount < 0) { throw new ArgumentOutOfRangeException("takeCount", "must not be negative"); }
if (takeCount == 0) { yield break; }
if (source.Count > takeCount)
{
for (int z = source.Count - 1; takeCount > 0; z--)
{
takeCount--;
yield return source[z];
}
}
else
{
for(int i = 0; i < source.Count; i++)
{
yield return source[i];
}
}
}
For test I have used Mark Byers method and kbrimington's andswer. This is test:
IList<int> test = new List<int>();
for(int i = 0; i<1000000; i++)
{
test.Add(i);
}
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
IList<int> result = TakeLast(test, 10).ToList();
stopwatch.Stop();
Stopwatch stopwatch1 = new Stopwatch();
stopwatch1.Start();
IList<int> result1 = TakeLast2(test, 10).ToList();
stopwatch1.Stop();
Stopwatch stopwatch2 = new Stopwatch();
stopwatch2.Start();
IList<int> result2 = test.Skip(Math.Max(0, test.Count - 10)).Take(10).ToList();
stopwatch2.Stop();
And here are results for taking 10 elements:
and for taking 1000001 elements results are:
Here's my solution:
public static class EnumerationExtensions
{
public static IEnumerable<T> TakeLast<T>(this IEnumerable<T> input, int count)
{
if (count <= 0)
yield break;
var inputList = input as IList<T>;
if (inputList != null)
{
int last = inputList.Count;
int first = last - count;
if (first < 0)
first = 0;
for (int i = first; i < last; i++)
yield return inputList[i];
}
else
{
// Use a ring buffer. We have to enumerate the input, and we don't know in advance how many elements it will contain.
T[] buffer = new T[count];
int index = 0;
count = 0;
foreach (T item in input)
{
buffer[index] = item;
index = (index + 1) % buffer.Length;
count++;
}
// The index variable now points at the next buffer entry that would be filled. If the buffer isn't completely
// full, then there are 'count' elements preceding index. If the buffer *is* full, then index is pointing at
// the oldest entry, which is the first one to return.
//
// If the buffer isn't full, which means that the enumeration has fewer than 'count' elements, we'll fix up
// 'index' to point at the first entry to return. That's easy to do; if the buffer isn't full, then the oldest
// entry is the first one. :-)
//
// We'll also set 'count' to the number of elements to be returned. It only needs adjustment if we've wrapped
// past the end of the buffer and have enumerated more than the original count value.
if (count < buffer.Length)
index = 0;
else
count = buffer.Length;
// Return the values in the correct order.
while (count > 0)
{
yield return buffer[index];
index = (index + 1) % buffer.Length;
count--;
}
}
}
public static IEnumerable<T> SkipLast<T>(this IEnumerable<T> input, int count)
{
if (count <= 0)
return input;
else
return input.SkipLastIter(count);
}
private static IEnumerable<T> SkipLastIter<T>(this IEnumerable<T> input, int count)
{
var inputList = input as IList<T>;
if (inputList != null)
{
int first = 0;
int last = inputList.Count - count;
if (last < 0)
last = 0;
for (int i = first; i < last; i++)
yield return inputList[i];
}
else
{
// Aim to leave 'count' items in the queue. If the input has fewer than 'count'
// items, then the queue won't ever fill and we return nothing.
Queue<T> elements = new Queue<T>();
foreach (T item in input)
{
elements.Enqueue(item);
if (elements.Count > count)
yield return elements.Dequeue();
}
}
}
}
The code is a bit chunky, but as a drop-in reusable component, it should perform as well as it can in most scenarios, and it'll keep the code that's using it nice and concise. :-)
My TakeLast for non-IList`1 is based on the same ring buffer algorithm as that in the answers by #Mark Byers and #MackieChan further up. It's interesting how similar they are -- I wrote mine completely independently. Guess there's really just one way to do a ring buffer properly. :-)
Looking at #kbrimington's answer, an additional check could be added to this for IQuerable<T> to fall back to the approach that works well with Entity Framework -- assuming that what I have at this point does not.
Honestly I'm not super proud of the answer, but for small collections you could use the following:
var lastN = collection.Reverse().Take(n).Reverse();
A bit hacky but it does the job ;)
My solution is based on ranges, introduced in C# version 8.
public static IEnumerable<T> TakeLast<T>(this IEnumerable<T> source, int N)
{
return source.ToArray()[(source.Count()-N)..];
}
After running a benchmark with most rated solutions (and my humbly proposed solution):
public static class TakeLastExtension
{
public static IEnumerable<T> TakeLastMarkByers<T>(this IEnumerable<T> source, int takeCount)
{
if (source == null) { throw new ArgumentNullException("source"); }
if (takeCount < 0) { throw new ArgumentOutOfRangeException("takeCount", "must not be negative"); }
if (takeCount == 0) { yield break; }
T[] result = new T[takeCount];
int i = 0;
int sourceCount = 0;
foreach (T element in source)
{
result[i] = element;
i = (i + 1) % takeCount;
sourceCount++;
}
if (sourceCount < takeCount)
{
takeCount = sourceCount;
i = 0;
}
for (int j = 0; j < takeCount; ++j)
{
yield return result[(i + j) % takeCount];
}
}
public static IEnumerable<T> TakeLastKbrimington<T>(this IEnumerable<T> source, int N)
{
return source.Skip(Math.Max(0, source.Count() - N));
}
public static IEnumerable<T> TakeLastJamesCurran<T>(this IEnumerable<T> source, int N)
{
return source.Reverse().Take(N).Reverse();
}
public static IEnumerable<T> TakeLastAlex<T>(this IEnumerable<T> source, int N)
{
return source.ToArray()[(source.Count()-N)..];
}
}
Test
[MemoryDiagnoser]
public class TakeLastBenchmark
{
[Params(10000)]
public int N;
private readonly List<string> l = new();
[GlobalSetup]
public void Setup()
{
for (var i = 0; i < this.N; i++)
{
this.l.Add($"i");
}
}
[Benchmark]
public void Benchmark1_MarkByers()
{
var lastElements = l.TakeLastMarkByers(3).ToList();
}
[Benchmark]
public void Benchmark2_Kbrimington()
{
var lastElements = l.TakeLastKbrimington(3).ToList();
}
[Benchmark]
public void Benchmark3_JamesCurran()
{
var lastElements = l.TakeLastJamesCurran(3).ToList();
}
[Benchmark]
public void Benchmark4_Alex()
{
var lastElements = l.TakeLastAlex(3).ToList();
}
}
Program.cs:
var summary = BenchmarkRunner.Run(typeof(TakeLastBenchmark).Assembly);
Command dotnet run --project .\TestsConsole2.csproj -c Release --logBuildOutput
The results were following:
// * Summary *
BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19044.1889/21H2/November2021Update)
AMD Ryzen 5 5600X, 1 CPU, 12 logical and 6 physical cores
.NET SDK=6.0.401
[Host] : .NET 6.0.9 (6.0.922.41905), X64 RyuJIT AVX2
DefaultJob : .NET 6.0.9 (6.0.922.41905), X64 RyuJIT AVX2
Method
N
Mean
Error
StdDev
Gen0
Gen1
Allocated
Benchmark1_MarkByers
10000
89,390.53 ns
1,735.464 ns
1,704.457 ns
-
-
248 B
Benchmark2_Kbrimington
10000
46.15 ns
0.410 ns
0.363 ns
0.0076
-
128 B
Benchmark3_JamesCurran
10000
2,703.15 ns
46.298 ns
67.862 ns
4.7836
0.0038
80264 B
Benchmark4_Alex
10000
2,513.48 ns
48.661 ns
45.517 ns
4.7607
-
80152 B
Turns out the solution proposed by #Kbrimington to be the most efficient in terms of memory alloc as well as raw performance.
Below the real example how to take last 3 elements from a collection (array):
// split address by spaces into array
string[] adrParts = adr.Split(new string[] { " " },StringSplitOptions.RemoveEmptyEntries);
// take only 3 last items in array
adrParts = adrParts.SkipWhile((value, index) => { return adrParts.Length - index > 3; }).ToArray();
Using This Method To Get All Range Without Error
public List<T> GetTsRate( List<T> AllT,int Index,int Count)
{
List<T> Ts = null;
try
{
Ts = AllT.ToList().GetRange(Index, Count);
}
catch (Exception ex)
{
Ts = AllT.Skip(Index).ToList();
}
return Ts ;
}
Little different implementation with usage of circular buffer. The benchmarks show that the method is circa two times faster than ones using Queue (implementation of TakeLast in System.Linq), however not without a cost - it needs a buffer which grows along with the requested number of elements, even if you have a small collection you can get huge memory allocation.
public IEnumerable<T> TakeLast<T>(IEnumerable<T> source, int count)
{
int i = 0;
if (count < 1)
yield break;
if (source is IList<T> listSource)
{
if (listSource.Count < 1)
yield break;
for (i = listSource.Count < count ? 0 : listSource.Count - count; i < listSource.Count; i++)
yield return listSource[i];
}
else
{
bool move = true;
bool filled = false;
T[] result = new T[count];
using (var enumerator = source.GetEnumerator())
while (move)
{
for (i = 0; (move = enumerator.MoveNext()) && i < count; i++)
result[i] = enumerator.Current;
filled |= move;
}
if (filled)
for (int j = i; j < count; j++)
yield return result[j];
for (int j = 0; j < i; j++)
yield return result[j];
}
}
//detailed code for the problem
//suppose we have a enumerable collection 'collection'
var lastIndexOfCollection=collection.Count-1 ;
var nthIndexFromLast= lastIndexOfCollection- N;
var desiredCollection=collection.GetRange(nthIndexFromLast, N);
---------------------------------------------------------------------
// use this one liner
var desiredCollection=collection.GetRange((collection.Count-(1+N)), N);
Let's say I have a sequence.
IEnumerable<int> sequence = GetSequenceFromExpensiveSource();
// sequence now contains: 0,1,2,3,...,999999,1000000
Getting the sequence is not cheap and is dynamically generated, and I want to iterate through it once only.
I want to get 0 - 999999 (i.e. everything but the last element)
I recognize that I could do something like:
sequence.Take(sequence.Count() - 1);
but that results in two enumerations over the big sequence.
Is there a LINQ construct that lets me do:
sequence.TakeAllButTheLastElement();
The Enumerable.SkipLast(IEnumerable<TSource>, Int32) method was added in .NET Standard 2.1. It does exactly what you want.
IEnumerable<int> sequence = GetSequenceFromExpensiveSource();
var allExceptLast = sequence.SkipLast(1);
From https://learn.microsoft.com/en-us/dotnet/api/system.linq.enumerable.skiplast
Returns a new enumerable collection that contains the elements from source with the last count elements of the source collection omitted.
I don't know a Linq solution - But you can easily code the algorithm by yourself using generators (yield return).
public static IEnumerable<T> TakeAllButLast<T>(this IEnumerable<T> source) {
var it = source.GetEnumerator();
bool hasRemainingItems = false;
bool isFirst = true;
T item = default(T);
do {
hasRemainingItems = it.MoveNext();
if (hasRemainingItems) {
if (!isFirst) yield return item;
item = it.Current;
isFirst = false;
}
} while (hasRemainingItems);
}
static void Main(string[] args) {
var Seq = Enumerable.Range(1, 10);
Console.WriteLine(string.Join(", ", Seq.Select(x => x.ToString()).ToArray()));
Console.WriteLine(string.Join(", ", Seq.TakeAllButLast().Select(x => x.ToString()).ToArray()));
}
Or as a generalized solution discarding the last n items (using a queue like suggested in the comments):
public static IEnumerable<T> SkipLastN<T>(this IEnumerable<T> source, int n) {
var it = source.GetEnumerator();
bool hasRemainingItems = false;
var cache = new Queue<T>(n + 1);
do {
if (hasRemainingItems = it.MoveNext()) {
cache.Enqueue(it.Current);
if (cache.Count > n)
yield return cache.Dequeue();
}
} while (hasRemainingItems);
}
static void Main(string[] args) {
var Seq = Enumerable.Range(1, 4);
Console.WriteLine(string.Join(", ", Seq.Select(x => x.ToString()).ToArray()));
Console.WriteLine(string.Join(", ", Seq.SkipLastN(3).Select(x => x.ToString()).ToArray()));
}
As an alternative to creating your own method and in a case the elements order is not important, the next will work:
var result = sequence.Reverse().Skip(1);
Because I'm not a fan of explicitly using an Enumerator, here's an alternative. Note that the wrapper methods are needed to let invalid arguments throw early, rather than deferring the checks until the sequence is actually enumerated.
public static IEnumerable<T> DropLast<T>(this IEnumerable<T> source)
{
if (source == null)
throw new ArgumentNullException("source");
return InternalDropLast(source);
}
private static IEnumerable<T> InternalDropLast<T>(IEnumerable<T> source)
{
T buffer = default(T);
bool buffered = false;
foreach (T x in source)
{
if (buffered)
yield return buffer;
buffer = x;
buffered = true;
}
}
As per Eric Lippert's suggestion, it easily generalizes to n items:
public static IEnumerable<T> DropLast<T>(this IEnumerable<T> source, int n)
{
if (source == null)
throw new ArgumentNullException("source");
if (n < 0)
throw new ArgumentOutOfRangeException("n",
"Argument n should be non-negative.");
return InternalDropLast(source, n);
}
private static IEnumerable<T> InternalDropLast<T>(IEnumerable<T> source, int n)
{
Queue<T> buffer = new Queue<T>(n + 1);
foreach (T x in source)
{
buffer.Enqueue(x);
if (buffer.Count == n + 1)
yield return buffer.Dequeue();
}
}
Where I now buffer before yielding instead of after yielding, so that the n == 0 case does not need special handling.
With C# 8.0 you can use Ranges and indices for that.
var allButLast = sequence[..^1];
By default C# 8.0 requires .NET Core 3.0 or .NET Standard 2.1 (or above). Check this thread to use with older implementations.
Nothing in the BCL (or MoreLinq I believe), but you could create your own extension method.
public static IEnumerable<T> TakeAllButLast<T>(this IEnumerable<T> source)
{
using (var enumerator = source.GetEnumerator())
bool first = true;
T prev;
while(enumerator.MoveNext())
{
if (!first)
yield return prev;
first = false;
prev = enumerator.Current;
}
}
}
It would be helpful if .NET Framework was shipped with extension method like this.
public static IEnumerable<T> SkipLast<T>(this IEnumerable<T> source, int count)
{
var enumerator = source.GetEnumerator();
var queue = new Queue<T>(count + 1);
while (true)
{
if (!enumerator.MoveNext())
break;
queue.Enqueue(enumerator.Current);
if (queue.Count > count)
yield return queue.Dequeue();
}
}
if you don't have time to roll out your own extension, here's a quicker way:
var next = sequence.First();
sequence.Skip(1)
.Select(s =>
{
var selected = next;
next = s;
return selected;
});
A slight expansion on Joren's elegant solution:
public static IEnumerable<T> Shrink<T>(this IEnumerable<T> source, int left, int right)
{
int i = 0;
var buffer = new Queue<T>(right + 1);
foreach (T x in source)
{
if (i >= left) // Read past left many elements at the start
{
buffer.Enqueue(x);
if (buffer.Count > right) // Build a buffer to drop right many elements at the end
yield return buffer.Dequeue();
}
else i++;
}
}
public static IEnumerable<T> WithoutLast<T>(this IEnumerable<T> source, int n = 1)
{
return source.Shrink(0, n);
}
public static IEnumerable<T> WithoutFirst<T>(this IEnumerable<T> source, int n = 1)
{
return source.Shrink(n, 0);
}
Where shrink implements a simple count forward to drop the first left many elements and the same discarded buffer to drop the last right many elements.
If you can get the Count or Length of an enumerable, which in most cases you can, then just Take(n - 1)
Example with arrays
int[] arr = new int[] { 1, 2, 3, 4, 5 };
int[] sub = arr.Take(arr.Length - 1).ToArray();
Example with IEnumerable<T>
IEnumerable<int> enu = Enumerable.Range(1, 100);
IEnumerable<int> sub = enu.Take(enu.Count() - 1);
A slight variation on the accepted answer, which (for my tastes) is a bit simpler:
public static IEnumerable<T> AllButLast<T>(this IEnumerable<T> enumerable, int n = 1)
{
// for efficiency, handle degenerate n == 0 case separately
if (n == 0)
{
foreach (var item in enumerable)
yield return item;
yield break;
}
var queue = new Queue<T>(n);
foreach (var item in enumerable)
{
if (queue.Count == n)
yield return queue.Dequeue();
queue.Enqueue(item);
}
}
Why not just .ToList<type>() on the sequence, then call count and take like you did originally..but since it's been pulled into a list, it shouldnt do an expensive enumeration twice. Right?
The solution that I use for this problem is slightly more elaborate.
My util static class contains an extension method MarkEnd which converts the T-items in EndMarkedItem<T>-items. Each element is marked with an extra int, which is either 0; or (in case one is particularly interested in the last 3 items) -3, -2, or -1 for the last 3 items.
This could be useful on its own, e.g. when you want to create a list in a simple foreach-loop with commas after each element except the last 2, with the second-to-last item followed by a conjunction word (such as “and” or “or”), and the last element followed by a point.
For generating the entire list without the last n items, the extension method ButLast simply iterates over the EndMarkedItem<T>s while EndMark == 0.
If you don’t specify tailLength, only the last item is marked (in MarkEnd()) or dropped (in ButLast()).
Like the other solutions, this works by buffering.
using System;
using System.Collections.Generic;
using System.Linq;
namespace Adhemar.Util.Linq {
public struct EndMarkedItem<T> {
public T Item { get; private set; }
public int EndMark { get; private set; }
public EndMarkedItem(T item, int endMark) : this() {
Item = item;
EndMark = endMark;
}
}
public static class TailEnumerables {
public static IEnumerable<T> ButLast<T>(this IEnumerable<T> ts) {
return ts.ButLast(1);
}
public static IEnumerable<T> ButLast<T>(this IEnumerable<T> ts, int tailLength) {
return ts.MarkEnd(tailLength).TakeWhile(te => te.EndMark == 0).Select(te => te.Item);
}
public static IEnumerable<EndMarkedItem<T>> MarkEnd<T>(this IEnumerable<T> ts) {
return ts.MarkEnd(1);
}
public static IEnumerable<EndMarkedItem<T>> MarkEnd<T>(this IEnumerable<T> ts, int tailLength) {
if (tailLength < 0) {
throw new ArgumentOutOfRangeException("tailLength");
}
else if (tailLength == 0) {
foreach (var t in ts) {
yield return new EndMarkedItem<T>(t, 0);
}
}
else {
var buffer = new T[tailLength];
var index = -buffer.Length;
foreach (var t in ts) {
if (index < 0) {
buffer[buffer.Length + index] = t;
index++;
}
else {
yield return new EndMarkedItem<T>(buffer[index], 0);
buffer[index] = t;
index++;
if (index == buffer.Length) {
index = 0;
}
}
}
if (index >= 0) {
for (var i = index; i < buffer.Length; i++) {
yield return new EndMarkedItem<T>(buffer[i], i - buffer.Length - index);
}
for (var j = 0; j < index; j++) {
yield return new EndMarkedItem<T>(buffer[j], j - index);
}
}
else {
for (var k = 0; k < buffer.Length + index; k++) {
yield return new EndMarkedItem<T>(buffer[k], k - buffer.Length - index);
}
}
}
}
}
}
public static IEnumerable<T> NoLast<T> (this IEnumerable<T> items) {
if (items != null) {
var e = items.GetEnumerator();
if (e.MoveNext ()) {
T head = e.Current;
while (e.MoveNext ()) {
yield return head; ;
head = e.Current;
}
}
}
}
I don't think it can get more succinct than this - also ensuring to Dispose the IEnumerator<T>:
public static IEnumerable<T> SkipLast<T>(this IEnumerable<T> source)
{
using (var it = source.GetEnumerator())
{
if (it.MoveNext())
{
var item = it.Current;
while (it.MoveNext())
{
yield return item;
item = it.Current;
}
}
}
}
Edit: technically identical to this answer.
This is a general and IMHO elegant solution that will handle all cases correctly:
using System;
using System.Collections.Generic;
using System.Linq;
public class Program
{
public static void Main()
{
IEnumerable<int> r = Enumerable.Range(1, 20);
foreach (int i in r.AllButLast(3))
Console.WriteLine(i);
Console.ReadKey();
}
}
public static class LinqExt
{
public static IEnumerable<T> AllButLast<T>(this IEnumerable<T> enumerable, int n = 1)
{
using (IEnumerator<T> enumerator = enumerable.GetEnumerator())
{
Queue<T> queue = new Queue<T>(n);
for (int i = 0; i < n && enumerator.MoveNext(); i++)
queue.Enqueue(enumerator.Current);
while (enumerator.MoveNext())
{
queue.Enqueue(enumerator.Current);
yield return queue.Dequeue();
}
}
}
}
You could write:
var list = xyz.Select(x=>x.Id).ToList();
list.RemoveAt(list.Count - 1);
My traditional IEnumerable approach:
/// <summary>
/// Skips first element of an IEnumerable
/// </summary>
/// <typeparam name="U">Enumerable type</typeparam>
/// <param name="models">The enumerable</param>
/// <returns>IEnumerable of type skipping first element</returns>
private IEnumerable<U> SkipFirstEnumerable<U>(IEnumerable<U> models)
{
using (var e = models.GetEnumerator())
{
if (!e.MoveNext()) return;
for (;e.MoveNext();) yield return e.Current;
yield return e.Current;
}
}
/// <summary>
/// Skips last element of an IEnumerable
/// </summary>
/// <typeparam name="U">Enumerable type</typeparam>
/// <param name="models">The enumerable</param>
/// <returns>IEnumerable of type skipping last element</returns>
private IEnumerable<U> SkipLastEnumerable<U>(IEnumerable<U> models)
{
using (var e = models.GetEnumerator())
{
if (!e.MoveNext()) return;
yield return e.Current;
for (;e.MoveNext();) yield return e.Current;
}
}
Could be:
var allBuLast = sequence.TakeWhile(e => e != sequence.Last());
I guess it should be like de "Where" but preserving the order(?).
If speed is a requirement, this old school way should be the fastest, even though the code doesn't look as smooth as linq could make it.
int[] newSequence = int[sequence.Length - 1];
for (int x = 0; x < sequence.Length - 1; x++)
{
newSequence[x] = sequence[x];
}
This requires that the sequence is an array since it has a fixed length and indexed items.
A simple way would be to just convert to a queue and dequeue until only the number of items you want to skip is left.
public static IEnumerable<T> SkipLast<T>(this IEnumerable<T> source, int n)
{
var queue = new Queue<T>(source);
while (queue.Count() > n)
{
yield return queue.Dequeue();
}
}
I would probably do something like this:
sequence.Where(x => x != sequence.LastOrDefault())
This is one iteration with a check that it isn't the last one for each time though.