Develop Reference

How to concatenate immutable collections efficiently?

I'm using System.Collections.Immutable and I want to find a way to concatenate several immutable collections without copying all the items (better than O(n)). All I need from resulting collection is IReadOnlyCollection<T> implementation.
My first idea was to use immutable double-linked list, but it seems that only prototypes of it exists over the Internet, and there's no reliable implementation. See, for example: Efficient implementation of immutable (double) LinkedList
Another idea is to create an immutable list of immutable lists and implement IReadOnlyCollection<T> on top of it. But again, it's a self-made solution for quite a popular problem, and I'm afraid I'm overlooking something.

IEnumerable's Concat() will return an enumerable implementation that simply enumerates the passed in enumerables without making a copy of them.
Be aware that a similar method IEnumerable Append allows adding single element to an enumerable.
Here is a passing test that verifies the original enumerable isn't ran when Concat and Append is called, execution is delayed until the concat or append result is enumerated (I wasn't sure given the wording of Append()'s documentation):
[Fact]
public void Test()
{
var selectClauseExecutionCount = 0;
var original = Enumerable.Range(1, 100);
var enumerated = original.Select(a =>
{
selectClauseExecutionCount++; ;
return a;
});
var concated = enumerated.Concat(new[] { 1, 2, 3 });
Assert.Equal(0, selectClauseExecutionCount);
var appended = concated.Append(5);
Assert.Equal(0, selectClauseExecutionCount);
Assert.Equal(5, appended.Last());
Assert.Equal(100, selectClauseExecutionCount);
}

If a double-linked immutable list or a list of lists is suitable for you then I'm guessing that you're just looking for a good way to merge and iterate any number of immutable lists as one, without creating unnecessary new copies of their elements.
From the docs you can see that IReadOnlyCollection<T> derives directly from IEnumerable<T> so if you can relax the constraint and have the resulting collection as IEnumerable<T> then your problem can be solved with LINQ and the ref keyword (as the parent interface is essentially readonly as well).
public IEnumerable<T> Concat<T>(params IReadOnlyCollection<T>[] things)
{
return things.SelectMany(x => x.Select(y => SelectByReference(ref y)));
}
private static ref T SelectByReference<T>(ref T t)
{
return ref t;
}
private void Example()
{
var c1 = new ReadOnlyCollection<string>(new[] { "1", "2" });
var c2 = new ReadOnlyCollection<string>(new[] { "3", "4" });
var c3 = new ReadOnlyCollection<string>(new[] { "5", "6" });
var resulting = Concat(c1, c2, c3);
foreach (var item in resulting)
{
// read the item etc without any copies being created
}
}

What is the best way to trim a list?

I have a List of strings. Its being generated elsewhere but i will generate it below to help describe this simplified example
var list = new List<string>();
list.Add("Joe");
list.Add("");
list.Add("Bill");
list.Add("Bill");
list.Add("");
list.Add("Scott");
list.Add("Joe");
list.Add("");
list.Add("");
list = TrimList(list);
I would like a function that "trims" this list and by trim I want to remove all items at the end of the array that are blank strings (the final two in this case).
NOTE: I still want to keep the blank one that is the second item in the array (or any other one that is just not at the end) so I can't do a .Where(r=> String.isNullOrEmpty(r))

I would just write it without any LINQ, to be honest- after all, you're modifying a collection rather than just querying it:
void TrimList(List<string> list)
{
int lastNonEmpty = list.FindLastIndex(x => !string.IsNullOrEmpty(x));
int firstToRemove = lastNonEmpty + 1;
list.RemoveRange(firstToRemove, list.Count - firstToRemove);
}
If you actually want to create a new list, then the LINQ-based solutions are okay... although potentially somewhat inefficient (as Reverse has to buffer everything).

Take advantage of Reverse and SkipWhile.
list = list.Reverse().SkipWhile(s => String.IsNullOrEmpty(s)).Reverse().ToList();

List<T> (not the interface) has a FindLastIndex method. Therefore you can wrap that in a method:
static IList<string> TrimList(List<string> input) {
return input.Take(input.FindLastIndex(x => !string.IsNullOrEmpty(x)) + 1)
.ToList();
}
This produces a copy, whereas Jon's modifies the list.

The only solution I can think of is to code a loop that starts at the end of the list and searches for an element that is not an empty string. Don't know of any library functions that would help. Once you know the last good element, you know which ones to remove.
Be careful not to modify the collection while you are iterating over it. Tends to break the iterator.

I always like to come up with the most generic solution possible. Why restrict yourself with lists and strings? Let's make an algorithm for generic enumerable!
public static class EnumerableExtensions
{
public static IEnumerable<T> TrimEnd<T>(this IEnumerable<T> enumerable, Predicate<T> predicate)
{
if (predicate == null)
{
throw new ArgumentNullException("predicate");
}
var accumulator = new LinkedList<T>();
foreach (var item in enumerable)
{
if (predicate(item))
{
accumulator.AddLast(item);
}
else
{
foreach (var accumulated in accumulator)
{
yield return accumulated;
}
accumulator.Clear();
yield return item;
}
}
}
}
Use it like this:
var list = new[]
{
"Joe",
"",
"Bill",
"Bill",
"",
"Scott",
"Joe",
"",
""
};
foreach (var item in list.TrimEnd(string.IsNullOrEmpty))
{
Console.WriteLine(item);
}

Simple IEnumerator use (with example)

I am having trouble remembering how (but not why) to use IEnumerators in C#. I am used to Java with its wonderful documentation that explains everything to beginners quite nicely. So please, bear with me.
I have tried learning from other answers on these boards to no avail. Rather than ask a generic question that has already been asked before, I have a specific example that would clarify things for me.
Suppose I have a method that needs to be passed an IEnumerable<String> object. All the method needs to do is concatenate the letters roxxors to the end of every String in the iterator. It then will return this new iterator (of course the original IEnumerable object is left as it was).
How would I go about this? The answer here should help many with basic questions about these objects in addition to me, of course.

Here is the documentation on IEnumerator. They are used to get the values of lists, where the length is not necessarily known ahead of time (even though it could be). The word comes from enumerate, which means "to count off or name one by one".
IEnumerator and IEnumerator<T> is provided by all IEnumerable and IEnumerable<T> interfaces (the latter providing both) in .NET via GetEnumerator(). This is important because the foreach statement is designed to work directly with enumerators through those interface methods.
So for example:
IEnumerator enumerator = enumerable.GetEnumerator();
while (enumerator.MoveNext())
{
object item = enumerator.Current;
// Perform logic on the item
}
Becomes:
foreach(object item in enumerable)
{
// Perform logic on the item
}
As to your specific scenario, almost all collections in .NET implement IEnumerable. Because of that, you can do the following:
public IEnumerator Enumerate(IEnumerable enumerable)
{
// List implements IEnumerable, but could be any collection.
List<string> list = new List<string>();
foreach(string value in enumerable)
{
list.Add(value + "roxxors");
}
return list.GetEnumerator();
}

public IEnumerable<string> Appender(IEnumerable<string> strings)
{
List<string> myList = new List<string>();
foreach(string str in strings)
{
myList.Add(str + "roxxors");
}
return myList;
}
or
public IEnumerable<string> Appender(IEnumerable<string> strings)
{
foreach(string str in strings)
{
yield return str + "roxxors";
}
}
using the yield construct, or simply
var newCollection = strings.Select(str => str + "roxxors"); //(*)
or
var newCollection = from str in strings select str + "roxxors"; //(**)
where the two latter use LINQ and (**) is just syntactic sugar for (*).

If i understand you correctly then in c# the yield return compiler magic is all you need i think.
e.g.
IEnumerable<string> myMethod(IEnumerable<string> sequence)
{
foreach(string item in sequence)
{
yield return item + "roxxors";
}
}

I'd do something like:
private IEnumerable<string> DoWork(IEnumerable<string> data)
{
List<string> newData = new List<string>();
foreach(string item in data)
{
newData.Add(item + "roxxors");
}
return newData;
}
Simple stuff :)

Also you can use LINQ's Select Method:
var source = new[] { "Line 1", "Line 2" };
var result = source.Select(s => s + " roxxors");
Read more here Enumerable.Select Method

Collection was modified; enumeration operation may not execute [duplicate]

This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
Collection was modified; enumeration operation may not execute - why?
First off, I believe this is NOT a duplicate question, as the context here deals with Extension methods.
I am getting this exception when moving items from one list to another via an Extension method, and I'm a bit confused because the list I'm removing items from is not the list I'm iterating over.
For example, the extension method at a minimum would look like this:
public static void MoveItemsTo(this IList source, IList target, IList items)
{
foreach (var item in items) {
target.Add(item);
source.Remove(item);
}
}
This is method is then called like this:
myCollectionOne.MoveItemsTo(myCollectionTwo, itemsToMove);
I only receive the error if I attempt to remove the item from the source list... This is a bit confusing as the method is iterating over a different list. Something must be going on behind the scenes that I am unaware of...

Instead of using a foreach loop, use a regular for loop. You're actually removing an item from the items list, since that's what you're iterating through.
for (int i = 0; i < items.Count; i++)
{
target.Add(item);
source.Remove(source[i]);
}
On a side note, you could probably just clear the entire source list in one go after the for loop, if that is indeed what you're trying to accomplish.

This will return a new list that is the result of the item move from source to destination, without altering the source lists. This avoids the refence problem in the question by treating the sources lists as immutable lists.
void Main()
{
var a = new List<string>() { "a","b","c" };
var b = new List<string>() { "d" };
var c = a.MoveItemsTo(i=>i=="b",b);
c.Dump(); // { "d", "b" }
}
public static class Extensions
{
public static IEnumerable<T> MoveItemsTo<T>(this IEnumerable<T> source, Func<T,bool> Predicate,IEnumerable<T> DestSource)
{
var newList = new List<T>(DestSource);
newList.AddRange(source.Where(Predicate).ToList());
return newList;
}
}

Best way to remove items from a collection

What is the best way to approach removing items from a collection in C#, once the item is known, but not it's index. This is one way to do it, but it seems inelegant at best.
//Remove the existing role assignment for the user.
int cnt = 0;
int assToDelete = 0;
foreach (SPRoleAssignment spAssignment in workspace.RoleAssignments)
{
if (spAssignment.Member.Name == shortName)
{
assToDelete = cnt;
}
cnt++;
}
workspace.RoleAssignments.Remove(assToDelete);
What I would really like to do is find the item to remove by property (in this case, name) without looping through the entire collection and using 2 additional variables.

If RoleAssignments is a List<T> you can use the following code.
workSpace.RoleAssignments.RemoveAll(x =>x.Member.Name == shortName);

If you want to access members of the collection by one of their properties, you might consider using a Dictionary<T> or KeyedCollection<T> instead. This way you don't have to search for the item you're looking for.
Otherwise, you could at least do this:
foreach (SPRoleAssignment spAssignment in workspace.RoleAssignments)
{
if (spAssignment.Member.Name == shortName)
{
workspace.RoleAssignments.Remove(spAssignment);
break;
}
}

#smaclell asked why reverse iteration was more efficient in in a comment to #sambo99.
Sometimes it's more efficient. Consider you have a list of people, and you want to remove or filter all customers with a credit rating < 1000;
We have the following data
"Bob" 999
"Mary" 999
"Ted" 1000
If we were to iterate forward, we'd soon get into trouble
for( int idx = 0; idx < list.Count ; idx++ )
{
if( list[idx].Rating < 1000 )
{
list.RemoveAt(idx); // whoops!
}
}
At idx = 0 we remove Bob, which then shifts all remaining elements left. The next time through the loop idx = 1, but
list[1] is now Ted instead of Mary. We end up skipping Mary by mistake. We could use a while loop, and we could introduce more variables.
Or, we just reverse iterate:
for (int idx = list.Count-1; idx >= 0; idx--)
{
if (list[idx].Rating < 1000)
{
list.RemoveAt(idx);
}
}
All the indexes to the left of the removed item stay the same, so you don't skip any items.
The same principle applies if you're given a list of indexes to remove from an array. In order to keep things straight you need to sort the list and then remove the items from highest index to lowest.
Now you can just use Linq and declare what you're doing in a straightforward manner.
list.RemoveAll(o => o.Rating < 1000);
For this case of removing a single item, it's no more efficient iterating forwards or backwards. You could also use Linq for this.
int removeIndex = list.FindIndex(o => o.Name == "Ted");
if( removeIndex != -1 )
{
list.RemoveAt(removeIndex);
}

If it's an ICollection then you won't have a RemoveAll method. Here's an extension method that will do it:
public static void RemoveAll<T>(this ICollection<T> source,
Func<T, bool> predicate)
{
if (source == null)
throw new ArgumentNullException("source", "source is null.");
if (predicate == null)
throw new ArgumentNullException("predicate", "predicate is null.");
source.Where(predicate).ToList().ForEach(e => source.Remove(e));
}
Based on:
http://phejndorf.wordpress.com/2011/03/09/a-removeall-extension-for-the-collection-class/

For a simple List structure the most efficient way seems to be using the Predicate RemoveAll implementation.
Eg.
workSpace.RoleAssignments.RemoveAll(x =>x.Member.Name == shortName);
The reasons are:
The Predicate/Linq RemoveAll method is implemented in List and has access to the internal array storing the actual data. It will shift the data and resize the internal array.
The RemoveAt method implementation is quite slow, and will copy the entire underlying array of data into a new array. This means reverse iteration is useless for List
If you are stuck implementing this in a the pre c# 3.0 era. You have 2 options.
The easily maintainable option. Copy all the matching items into a new list and and swap the underlying list.
Eg.
List<int> list2 = new List<int>() ;
foreach (int i in GetList())
{
if (!(i % 2 == 0))
{
list2.Add(i);
}
}
list2 = list2;
Or
The tricky slightly faster option, which involves shifting all the data in the list down when it does not match and then resizing the array.
If you are removing stuff really frequently from a list, perhaps another structure like a HashTable (.net 1.1) or a Dictionary (.net 2.0) or a HashSet (.net 3.5) are better suited for this purpose.

What type is the collection? If it's List, you can use the helpful "RemoveAll":
int cnt = workspace.RoleAssignments
.RemoveAll(spa => spa.Member.Name == shortName)
(This works in .NET 2.0. Of course, if you don't have the newer compiler, you'll have to use "delegate (SPRoleAssignment spa) { return spa.Member.Name == shortName; }" instead of the nice lambda syntax.)
Another approach if it's not a List, but still an ICollection:
var toRemove = workspace.RoleAssignments
.FirstOrDefault(spa => spa.Member.Name == shortName)
if (toRemove != null) workspace.RoleAssignments.Remove(toRemove);
This requires the Enumerable extension methods. (You can copy the Mono ones in, if you are stuck on .NET 2.0). If it's some custom collection that cannot take an item, but MUST take an index, some of the other Enumerable methods, such as Select, pass in the integer index for you.

This is my generic solution
public static IEnumerable<T> Remove<T>(this IEnumerable<T> items, Func<T, bool> match)
{
var list = items.ToList();
for (int idx = 0; idx < list.Count(); idx++)
{
if (match(list[idx]))
{
list.RemoveAt(idx);
idx--; // the list is 1 item shorter
}
}
return list.AsEnumerable();
}
It would look much simpler if extension methods support passing by reference !
usage:
var result = string[]{"mike", "john", "ali"}
result = result.Remove(x => x.Username == "mike").ToArray();
Assert.IsTrue(result.Length == 2);
EDIT: ensured that the list looping remains valid even when deleting items by decrementing the index (idx).

Here is a pretty good way to do it
http://support.microsoft.com/kb/555972
System.Collections.ArrayList arr = new System.Collections.ArrayList();
arr.Add("1");
arr.Add("2");
arr.Add("3");
/*This throws an exception
foreach (string s in arr)
{
arr.Remove(s);
}
*/
//where as this works correctly
Console.WriteLine(arr.Count);
foreach (string s in new System.Collections.ArrayList(arr))
{
arr.Remove(s);
}
Console.WriteLine(arr.Count);
Console.ReadKey();

There is another approach you can take depending on how you're using your collection. If you're downloading the assignments one time (e.g., when the app runs), you could translate the collection on the fly into a hashtable where:
shortname => SPRoleAssignment
If you do this, then when you want to remove an item by short name, all you need to do is remove the item from the hashtable by key.
Unfortunately, if you're loading these SPRoleAssignments a lot, that obviously isn't going to be any more cost efficient in terms of time. The suggestions other people made about using Linq would be good if you're using a new version of the .NET Framework, but otherwise, you'll have to stick to the method you're using.

Similar to Dictionary Collection point of view, I have done this.
Dictionary<string, bool> sourceDict = new Dictionary<string, bool>();
sourceDict.Add("Sai", true);
sourceDict.Add("Sri", false);
sourceDict.Add("SaiSri", true);
sourceDict.Add("SaiSriMahi", true);
var itemsToDelete = sourceDict.Where(DictItem => DictItem.Value == false);
foreach (var item in itemsToDelete)
{
sourceDict.Remove(item.Key);
}
Note:
Above code will fail in .Net Client Profile (3.5 and 4.5) also some viewers mentioned it is
Failing for them in .Net4.0 as well not sure which settings are causing the problem.
So replace with below code (.ToList()) for Where statement, to avoid that error. “Collection was modified; enumeration operation may not execute.”
var itemsToDelete = sourceDict.Where(DictItem => DictItem.Value == false).ToList();
Per MSDN From .Net4.5 onwards Client Profile are discontinued. http://msdn.microsoft.com/en-us/library/cc656912(v=vs.110).aspx

Save your items first, than delete them.
var itemsToDelete = Items.Where(x => !!!your condition!!!).ToArray();
for (int i = 0; i < itemsToDelete.Length; ++i)
Items.Remove(itemsToDelete[i]);
You need to override GetHashCode() in your Item class.

The best way to do it is by using linq.
Example class:
public class Product
{
public string Name { get; set; }
public string Price { get; set; }
}
Linq query:
var subCollection = collection1.RemoveAll(w => collection2.Any(q => q.Name == w.Name));
This query will remove all elements from collection1 if Name match any element Name from collection2
Remember to use: using System.Linq;

To do this while looping through the collection and not to get the modifying a collection exception, this is the approach I've taken in the past (note the .ToList() at the end of the original collection, this creates another collection in memory, then you can modify the existing collection)
foreach (SPRoleAssignment spAssignment in workspace.RoleAssignments.ToList())
{
if (spAssignment.Member.Name == shortName)
{
workspace.RoleAssignments.Remove(spAssignment);
}
}

If you have got a List<T>, then List<T>.RemoveAll is your best bet. There can't be anything more efficient. Internally it does the array moving in one shot, not to mention it is O(N).
If all you got is an IList<T> or an ICollection<T> you got roughly these three options:
public static void RemoveAll<T>(this IList<T> ilist, Predicate<T> predicate) // O(N^2)
{
for (var index = ilist.Count - 1; index >= 0; index--)
{
var item = ilist[index];
if (predicate(item))
{
ilist.RemoveAt(index);
}
}
}
or
public static void RemoveAll<T>(this ICollection<T> icollection, Predicate<T> predicate) // O(N)
{
var nonMatchingItems = new List<T>();
// Move all the items that do not match to another collection.
foreach (var item in icollection)
{
if (!predicate(item))
{
nonMatchingItems.Add(item);
}
}
// Clear the collection and then copy back the non-matched items.
icollection.Clear();
foreach (var item in nonMatchingItems)
{
icollection.Add(item);
}
}
or
public static void RemoveAll<T>(this ICollection<T> icollection, Func<T, bool> predicate) // O(N^2)
{
foreach (var item in icollection.Where(predicate).ToList())
{
icollection.Remove(item);
}
}
Go for either 1 or 2.
1 is lighter on memory and faster if you have less deletes to perform (i.e. predicate is false most of the times).
2 is faster if you have more deletes to perform.
3 is the cleanest code but performs poorly IMO. Again all that depends on input data.
For some benchmarking details see https://github.com/dotnet/BenchmarkDotNet/issues/1505

A lot of good responses here; I especially like the lambda expressions...very clean. I was remiss, however, in not specifying the type of Collection. This is a SPRoleAssignmentCollection (from MOSS) that only has Remove(int) and Remove(SPPrincipal), not the handy RemoveAll(). So, I have settled on this, unless there is a better suggestion.
foreach (SPRoleAssignment spAssignment in workspace.RoleAssignments)
{
if (spAssignment.Member.Name != shortName) continue;
workspace.RoleAssignments.Remove((SPPrincipal)spAssignment.Member);
break;
}