Develop Reference

Check if an object equals another object through a transitive relationship

I have a class ValuePair with two properties defined in it:
public class ValuePair: IEquatable<ValuePair>
{
public string value1;
public string value2;
public ValuePair(string v1, string v1)
{
this.value1 = v1;
this.value2 = v2;
}
...
}
I have some test data in a List as defined below:
List<ValuePair> pairs = new ValuePair<ValuePair>();
pairs.Add(new ValuePair("A","B"));
pairs.Add(new ValuePair("A","C"));
pairs.Add(new ValuePair("B","C"));
pairs.Add(new ValuePair("C","D"));
My goal is to keep pairs[0] and pairs[1] because the pairs "A,B" and "A,C" are unique, but to remove pair[2] because the relationship "B,C" has already been captured in the first two relationships. pairs[3] should remain since the "C,D" relationship is unique.
I have a feeling the solution to this problem will be recursive, which is something that I have very little experience with. I started going down a path of adding a method to the class ValuePair that looks something like this:
public string EqualToEither(ValuePair v)
{
if (v.value1 == this.value1 || v.value1 == this.value2)
return v.value1;
else if (v.value2 == this.value1 || v.value2 == this.value2)
return v.value2;
else
return string.Empty;
}
I've started to try to use the above method inside of a function like this, but I am getting hung up on what to do next:
for (int i = 0; i < pairs.Count; i++)
{
for (int j = pairs.Count - 1; j >= 0; j--)
{
if (pairs[j].EqualToEither(pairs[i]) != string.Empty)
{
pairs[j].EqualToEither(pairs[i]);
}
else
{
continue;
}
}
}
I feel like I am close but still unable to get it. Can anyone please offer some guidance? If I'm approaching this the completely wrong way please let me know, thank you!

I had to solve a similar problem recently, here is how I solved it:
Transitivity is best represented, in my opinion, by grouping interrelated elements together.
For each pair you have to validate if it already belongs to a group (both values are already in the group) or if it extends the relation of a group (only one of the values belong to the group).
In the case it does not belong in any group, it becomes a new group.
In the case both values belong in different groups then you have to merge them.
As mentioned, this is closely related to a spanning tree.
One solution could be to use HashSets to represent the transitivity of your relations (I did not use HashSets in my case, there are many possible solutions).
Each HashSet would represent a group of interrelated elements.
Example implementation using HashSets:
List<ValuePair> pairs = new List<ValuePair>();
pairs.Add(new ValuePair("A", "B"));
pairs.Add(new ValuePair("A", "C"));
pairs.Add(new ValuePair("B", "C"));
pairs.Add(new ValuePair("C", "D"));
List<ValuePair> uniquePairs = new List<ValuePair>();
// this list is not really needed if all you care about
// is getting the resulting groups
List<HashSet<string>> sets = new List<HashSet<string>>();
foreach (ValuePair pair in pairs)
{
int value1Set = -1;
int value2Set = -1;
for (int i = 0; i < sets.Count; i++)
{
HashSet<string> set = sets[i];
if (set.Contains(pair.value1))
value1Set = i;
if (set.Contains(pair.value2))
value2Set = i;
}
if (value1Set == -1 && value2Set == -1)
{
// we have a new set
sets.Add(new HashSet<string> {pair.value1, pair.value2});
}
else if (value1Set == -1)
{
sets[value2Set].Add(pair.value1);
}
else if (value2Set == -1)
{
sets[value1Set].Add(pair.value2);
}
else
{
if (value1Set == value2Set)
{
// duplicate entry, skip the add
continue;
}
// merge the sets at value1Set and value2Set
foreach (string value in sets[value2Set])
{
sets[value1Set].Add(value);
}
sets.RemoveAt(value2Set);
}
uniquePairs.Add(pair);
}

OutOfMemoryException when updating a large list?

I have a large list and I would like to overwrite one value if required. To do this, I create two subsets of the list which seems to give me an OutOfMemoryException. Here is my code snippet:
if (ownRG != "")
{
List<string> maclist = ownRG.Split(',').ToList();
List<IVFile> temp = powlist.Where(a => maclist.Contains(a.Machine)).ToList();
powlist = powlist.Where(a => !maclist.Contains(a.Machine)).ToList(); // OOME Here
temp.ForEach(a => { a.ReportingGroup = ownRG; });
powlist.AddRange(temp);
}
Essentially I'm splitting the list into the part that needs updating and the part that doesn't, then I perform the update and put the list back together. This works fine for smaller lists, but breaks with an OutOfMemoryException on the third row within the if for a large list. Can I make this more efficient?
NOTE
powlist is the large list (>1m) items. maclist only has between 1 and 10 but even with 1 item this breaks.

Solving your issue
Here is how to rearrange your code using the enumerator code from my answer:
if (!string.IsNullOrEmpty(ownRG))
{
var maclist = new CommaSeparatedStringEnumerable(str);
var temp = powlist.Where(a => maclist.Contains(a.Machine));
foreach (var p in temp)
{
p.ReportingGroup = ownRG;
}
}
You should not use ToList in your code.
You don't need to remove thee contents of temp from powlist (you are re-adding them anyway)
Streaming over a large comma-separated string
You can iterate over the list manually instead of doing what you do now, by looking for , characters and remembering the position of the last found one and the one before. This will definitely make your app work because then it won't need to store the entire set in the memory at once.
Code example:
var str = "aaa,bbb,ccc";
var previousComma = -1;
var currentComma = 0;
for (; (currentComma = str.IndexOf(',', previousComma + 1)) != -1; previousComma = currentComma)
{
var currentItem = str.Substring(previousComma + 1, currentComma - previousComma - 1);
Console.WriteLine(currentItem);
}
var lastItem = str.Substring(previousComma + 1);
Console.WriteLine(lastItem);
Custom iterator
If you want to do it 'properly' in a fancy way, you can even write a custom enumerator:
public class CommaSeparatedStringEnumerator : IEnumerator<string>
{
int previousComma = -1;
int currentComma = -1;
string bigString = null;
bool atEnd = false;
public CommaSeparatedStringEnumerator(string s)
{
if (s == null)
throw new ArgumentNullException("s");
bigString = s;
this.Reset();
}
public string Current { get; private set; }
public void Dispose() { /* No need to do anything here */ }
object IEnumerator.Current { get { return this.Current; } }
public bool MoveNext()
{
if (atEnd)
return false;
atEnd = (currentComma = bigString.IndexOf(',', previousComma + 1)) == -1;
if (!atEnd)
Current = bigString.Substring(previousComma + 1, currentComma - previousComma - 1);
else
Current = bigString.Substring(previousComma + 1);
previousComma = currentComma;
return true;
}
public void Reset()
{
previousComma = -1;
currentComma = -1;
atEnd = false;
this.Current = null;
}
}
public class CommaSeparatedStringEnumerable : IEnumerable<string>
{
string bigString = null;
public CommaSeparatedStringEnumerable(string s)
{
if (s == null)
throw new ArgumentNullException("s");
bigString = s;
}
public IEnumerator<string> GetEnumerator()
{
return new CommaSeparatedStringEnumerator(bigString);
}
IEnumerator IEnumerable.GetEnumerator()
{
return this.GetEnumerator();
}
}
Then you can iterate over it like this:
var str = "aaa,bbb,ccc";
var enumerable = new CommaSeparatedStringEnumerable(str);
foreach (var item in enumerable)
{
Console.WriteLine(item);
}
Other thoughts
Can I make this more efficient?
Yes, you can. I suggest to either work with a more efficient data format (you can take a look around databases or XML, JSON, etc. depending on your needs). If you really want to work with comma-separated items, see my code examples above.

There's no need to create a bunch of sub-lists from powlist and reconstruct it. Simply loop over the powlist and update the ReportingGroup property accordingly.
var maclist = new HashSet<string>( ownRG.Split(',') );
foreach( var item in powlist) {
if( maclist.Contains( item.Machine ) ){
item.ReportingGroup = ownRG;
}
}
Since this changes powlist in place, you won't allocate any extra memory and shouldn't run into an OutOfMemoryException.

In a loop find the next ',' char. Take the substring between the ',' and the previous ',' position. At the end of the loop save a reference to the previous ',' position (which is initially set to 0). So you parse the items one-by-one rather than all at once.

You can try looping the items of your lists, but this will increase processing time.
foreach(var item in powlist)
{
//do your opeartions
}

How to select 2 items at a time?

I was writing a PascalCaseParser using Regex.Split and I came to the desire to select two items from an collection at a time.
This example code demonstrates.
void Main()
{
string pascalCasedString = "JustLikeYouAndMe";
var words = WordsFromPascalCasedString(pascalCasedString);
words.Dump();
}
IEnumerable<string> WordsFromPascalCasedString(string pascalCasedString)
{
var rx = new Regex("([A-Z])");
return rx.Split(pascalCasedString)
.Where(c => !string.IsNullOrEmpty(c))
// how to select 2 elements at a time?
;
}
The result of above code is:
IEnumerable<String> (10 items)
J
ust
L
ike
Y
ou
A
nd
M
e
Every two elements of the collection make one result that I want the function WordsFromPascalCasedString to yield.
My questions is: How would you, in general, deal with a requirement to return two items at a time. I'm curious if there are any interesting non-brute-force approaches.

Personally, I'd go with Simon Belanger's answer in this particular case. But in general, to select consecutive pairs, from an IEnumerable, you'd use this:
IEnumerable<Tuple<string, string>> WordsFromPascalCasedString(string pascalCasedString)
{
var rx = new Regex("([A-Z])");
var array = rx.Split(pascalCasedString)
.Where(c => !string.IsNullOrEmpty(c))
.ToArray();
var items = Enumerable.Range(0, array.Length / 2)
.Select(i => Tuple.Create(array[i * 2], array[i * 2 + 1]);
}
Or this, which takes more effort, but it's reusable and more efficient:
IEnumerable<Tuple<T, T>> Pairs<T>(IEnumerable<T> input)
{
var array = new T[2];
int i = 0;
foreach(var x in input)
{
array[i] = x;
i = (i + 1) % 2;
if (i == 0)
{
yield return Tuple.Create(array[0], array[1]);
}
}
}
IEnumerable<Tuple<string, string>> WordsFromPascalCasedString(string pascalCasedString)
{
var rx = new Regex("([A-Z])");
var output = rx.Split(pascalCasedString)
.Where(c => !string.IsNullOrEmpty(c));
var items = Pairs(output);
}
It can easily be extended to groups of n:
IEnumerable<IEnumerable<T>> Batches<T>(IEnumerable<T> input, int n)
{
var array = new T[n];
int i = 0;
foreach(var x in input)
{
array[i] = x;
i = (i + 1) % n;
if (i == 0)
{
yield return array.ToArray();
}
}
if (i != 0)
{
yield return array.Take(i);
}
}
A similar method exists in MoreLINQ.

The regex should be ([A-Z][a-z]*). Adjust the last portion if you want to include numbers too. Use + instead of * if you want at least one lowercase element after the uppercase delimiter.
Edit As for the actual question, you will need to materialize and iterate in a for loop for better performance (passing the list once). In your specific problem, you can just use Regex.Matches
var result = Regex.Matches("([A-Z][a-z]*)([A-Z][a-z]*)?", "AbCdEfGhIj")
.OfType<Match>()
.Where(m => m.Success)
.Select(m => Tuple.Create(m.Groups[1].Value, m.Groups[2].Value));

This is just to share, I'm throwing the solution I came up with after getting inspired by the other answers. It is not better than the others...
void Main()
{
string pascalCasedString = "JustLikeYouAndMe";
var words = WordsFromPascalCasedString(pascalCasedString);
words.Dump();
}
IEnumerable<string> WordsFromPascalCasedString(string pascalCasedString)
{
var rx = new Regex("([A-Z])");
return rx.Split(pascalCasedString)
.Where(c => !string.IsNullOrEmpty(c))
.InPieces(2)
.Select(c => c.ElementAt(0) + c.ElementAt(1));
}
static class Ext
{
public static IEnumerable<IEnumerable<T>> InPieces<T>(this IEnumerable<T> seq, int len)
{
if(!seq.Any())
yield break;
yield return seq.Take(len);
foreach (var element in InPieces(seq.Skip(len), len))
yield return element;
}
}

Easiest is to write function that simply returns pairs.
Something like:
IEnumerable<Tuple<T,T>> Pairs<T>(IEnumerable<T> items)
{
T first = default(T);
bool hasFirst = false;
foreach(T item in items)
{
if (hasFirst)
yield return Tuple.Create(first, item);
else
first = item;
hasFirst = !hasFirst;
}
}
Aggregate is likely only one one-line approach. This is purely entertainment code due to amount of garbage created on a way, but there is no mutable objects used.
IEnumerable<Tuple<T,T>> Pairs<T>(IEnumerable<T> collection)
{
return collection
.Aggregate(
Tuple.Create(false, default(T), Enumerable.Empty<Tuple<T,T>>()),
(accumulate, item)=> !accumulate.Item1 ?
Tuple.Create(true, item, accumulate.Item3) :
Tuple.Create(false, default(T),
accumulate.Item3.Concat(
Enumerable.Repeat(Tuple.Create(accumulate.Item2, item), 1))),
accumulate => accumulate.Item3);
}
Zip of odd and even elements (index %2 ==/!= 0) is 2 line approach. Note that iterates source collection twice.
IEnumerable<Tuple<T,T>> Pairs<T>(IEnumerable<T> collection)
{
return collection
.Where((item, index)=>index %2 == 0)
.Zip(collection.Where((item, index)=>index %2 != 0),
(first,second)=> Tuple.Create(first,second));
}

Remove duplicates from a List<T> in C#

Anyone have a quick method for de-duplicating a generic List in C#?

If you're using .Net 3+, you can use Linq.
List<T> withDupes = LoadSomeData();
List<T> noDupes = withDupes.Distinct().ToList();

Perhaps you should consider using a HashSet.
From the MSDN link:
using System;
using System.Collections.Generic;
class Program
{
static void Main()
{
HashSet<int> evenNumbers = new HashSet<int>();
HashSet<int> oddNumbers = new HashSet<int>();
for (int i = 0; i < 5; i++)
{
// Populate numbers with just even numbers.
evenNumbers.Add(i * 2);
// Populate oddNumbers with just odd numbers.
oddNumbers.Add((i * 2) + 1);
}
Console.Write("evenNumbers contains {0} elements: ", evenNumbers.Count);
DisplaySet(evenNumbers);
Console.Write("oddNumbers contains {0} elements: ", oddNumbers.Count);
DisplaySet(oddNumbers);
// Create a new HashSet populated with even numbers.
HashSet<int> numbers = new HashSet<int>(evenNumbers);
Console.WriteLine("numbers UnionWith oddNumbers...");
numbers.UnionWith(oddNumbers);
Console.Write("numbers contains {0} elements: ", numbers.Count);
DisplaySet(numbers);
}
private static void DisplaySet(HashSet<int> set)
{
Console.Write("{");
foreach (int i in set)
{
Console.Write(" {0}", i);
}
Console.WriteLine(" }");
}
}
/* This example produces output similar to the following:
* evenNumbers contains 5 elements: { 0 2 4 6 8 }
* oddNumbers contains 5 elements: { 1 3 5 7 9 }
* numbers UnionWith oddNumbers...
* numbers contains 10 elements: { 0 2 4 6 8 1 3 5 7 9 }
*/

How about:
var noDupes = list.Distinct().ToList();
In .net 3.5?

Simply initialize a HashSet with a List of the same type:
var noDupes = new HashSet<T>(withDupes);
Or, if you want a List returned:
var noDupsList = new HashSet<T>(withDupes).ToList();

Sort it, then check two and two next to each others, as the duplicates will clump together.
Something like this:
list.Sort();
Int32 index = list.Count - 1;
while (index > 0)
{
if (list[index] == list[index - 1])
{
if (index < list.Count - 1)
(list[index], list[list.Count - 1]) = (list[list.Count - 1], list[index]);
list.RemoveAt(list.Count - 1);
index--;
}
else
index--;
}
Notes:
Comparison is done from back to front, to avoid having to resort list after each removal
This example now uses C# Value Tuples to do the swapping, substitute with appropriate code if you can't use that
The end-result is no longer sorted

I like to use this command:
List<Store> myStoreList = Service.GetStoreListbyProvince(provinceId)
.GroupBy(s => s.City)
.Select(grp => grp.FirstOrDefault())
.OrderBy(s => s.City)
.ToList();
I have these fields in my list: Id, StoreName, City, PostalCode
I wanted to show list of cities in a dropdown which has duplicate values.
solution: Group by city then pick the first one for the list.

It worked for me. simply use
List<Type> liIDs = liIDs.Distinct().ToList<Type>();
Replace "Type" with your desired type e.g. int.

As kronoz said in .Net 3.5 you can use Distinct().
In .Net 2 you could mimic it:
public IEnumerable<T> DedupCollection<T> (IEnumerable<T> input)
{
var passedValues = new HashSet<T>();
// Relatively simple dupe check alg used as example
foreach(T item in input)
if(passedValues.Add(item)) // True if item is new
yield return item;
}
This could be used to dedupe any collection and will return the values in the original order.
It's normally much quicker to filter a collection (as both Distinct() and this sample does) than it would be to remove items from it.

An extension method might be a decent way to go... something like this:
public static List<T> Deduplicate<T>(this List<T> listToDeduplicate)
{
return listToDeduplicate.Distinct().ToList();
}
And then call like this, for example:
List<int> myFilteredList = unfilteredList.Deduplicate();

In Java (I assume C# is more or less identical):
list = new ArrayList<T>(new HashSet<T>(list))
If you really wanted to mutate the original list:
List<T> noDupes = new ArrayList<T>(new HashSet<T>(list));
list.clear();
list.addAll(noDupes);
To preserve order, simply replace HashSet with LinkedHashSet.

This takes distinct (the elements without duplicating elements) and convert it into a list again:
List<type> myNoneDuplicateValue = listValueWithDuplicate.Distinct().ToList();

Use Linq's Union method.
Note: This solution requires no knowledge of Linq, aside from that it exists.
Code
Begin by adding the following to the top of your class file:
using System.Linq;
Now, you can use the following to remove duplicates from an object called, obj1:
obj1 = obj1.Union(obj1).ToList();
Note: Rename obj1 to the name of your object.
How it works
The Union command lists one of each entry of two source objects. Since obj1 is both source objects, this reduces obj1 to one of each entry.
The ToList() returns a new List. This is necessary, because Linq commands like Union returns the result as an IEnumerable result instead of modifying the original List or returning a new List.

As a helper method (without Linq):
public static List<T> Distinct<T>(this List<T> list)
{
return (new HashSet<T>(list)).ToList();
}

Here's an extension method for removing adjacent duplicates in-situ. Call Sort() first and pass in the same IComparer. This should be more efficient than Lasse V. Karlsen's version which calls RemoveAt repeatedly (resulting in multiple block memory moves).
public static void RemoveAdjacentDuplicates<T>(this List<T> List, IComparer<T> Comparer)
{
int NumUnique = 0;
for (int i = 0; i < List.Count; i++)
if ((i == 0) || (Comparer.Compare(List[NumUnique - 1], List[i]) != 0))
List[NumUnique++] = List[i];
List.RemoveRange(NumUnique, List.Count - NumUnique);
}

Installing the MoreLINQ package via Nuget, you can easily distinct object list by a property
IEnumerable<Catalogue> distinctCatalogues = catalogues.DistinctBy(c => c.CatalogueCode);

If you have tow classes Product and Customer and we want to remove duplicate items from their list
public class Product
{
public int Id { get; set; }
public string ProductName { get; set; }
}
public class Customer
{
public int Id { get; set; }
public string CustomerName { get; set; }
}
You must define a generic class in the form below
public class ItemEqualityComparer<T> : IEqualityComparer<T> where T : class
{
private readonly PropertyInfo _propertyInfo;
public ItemEqualityComparer(string keyItem)
{
_propertyInfo = typeof(T).GetProperty(keyItem, BindingFlags.GetProperty | BindingFlags.Instance | BindingFlags.Public);
}
public bool Equals(T x, T y)
{
var xValue = _propertyInfo?.GetValue(x, null);
var yValue = _propertyInfo?.GetValue(y, null);
return xValue != null && yValue != null && xValue.Equals(yValue);
}
public int GetHashCode(T obj)
{
var propertyValue = _propertyInfo.GetValue(obj, null);
return propertyValue == null ? 0 : propertyValue.GetHashCode();
}
}
then, You can remove duplicate items in your list.
var products = new List<Product>
{
new Product{ProductName = "product 1" ,Id = 1,},
new Product{ProductName = "product 2" ,Id = 2,},
new Product{ProductName = "product 2" ,Id = 4,},
new Product{ProductName = "product 2" ,Id = 4,},
};
var productList = products.Distinct(new ItemEqualityComparer<Product>(nameof(Product.Id))).ToList();
var customers = new List<Customer>
{
new Customer{CustomerName = "Customer 1" ,Id = 5,},
new Customer{CustomerName = "Customer 2" ,Id = 5,},
new Customer{CustomerName = "Customer 2" ,Id = 5,},
new Customer{CustomerName = "Customer 2" ,Id = 5,},
};
var customerList = customers.Distinct(new ItemEqualityComparer<Customer>(nameof(Customer.Id))).ToList();
this code remove duplicate items by Id if you want remove duplicate items by other property, you can change nameof(YourClass.DuplicateProperty) same nameof(Customer.CustomerName) then remove duplicate items by CustomerName Property.

If you don't care about the order you can just shove the items into a HashSet, if you do want to maintain the order you can do something like this:
var unique = new List<T>();
var hs = new HashSet<T>();
foreach (T t in list)
if (hs.Add(t))
unique.Add(t);
Or the Linq way:
var hs = new HashSet<T>();
list.All( x => hs.Add(x) );
Edit: The HashSet method is O(N) time and O(N) space while sorting and then making unique (as suggested by #lassevk and others) is O(N*lgN) time and O(1) space so it's not so clear to me (as it was at first glance) that the sorting way is inferior

Might be easier to simply make sure that duplicates are not added to the list.
if(items.IndexOf(new_item) < 0)
items.add(new_item)

You can use Union
obj2 = obj1.Union(obj1).ToList();

Another way in .Net 2.0
static void Main(string[] args)
{
List<string> alpha = new List<string>();
for(char a = 'a'; a <= 'd'; a++)
{
alpha.Add(a.ToString());
alpha.Add(a.ToString());
}
Console.WriteLine("Data :");
alpha.ForEach(delegate(string t) { Console.WriteLine(t); });
alpha.ForEach(delegate (string v)
{
if (alpha.FindAll(delegate(string t) { return t == v; }).Count > 1)
alpha.Remove(v);
});
Console.WriteLine("Unique Result :");
alpha.ForEach(delegate(string t) { Console.WriteLine(t);});
Console.ReadKey();
}

There are many ways to solve - the duplicates issue in the List, below is one of them:
List<Container> containerList = LoadContainer();//Assume it has duplicates
List<Container> filteredList = new List<Container>();
foreach (var container in containerList)
{
Container duplicateContainer = containerList.Find(delegate(Container checkContainer)
{ return (checkContainer.UniqueId == container.UniqueId); });
//Assume 'UniqueId' is the property of the Container class on which u r making a search
if(!containerList.Contains(duplicateContainer) //Add object when not found in the new class object
{
filteredList.Add(container);
}
}
Cheers
Ravi Ganesan

Here's a simple solution that doesn't require any hard-to-read LINQ or any prior sorting of the list.
private static void CheckForDuplicateItems(List<string> items)
{
if (items == null ||
items.Count == 0)
return;
for (int outerIndex = 0; outerIndex < items.Count; outerIndex++)
{
for (int innerIndex = 0; innerIndex < items.Count; innerIndex++)
{
if (innerIndex == outerIndex) continue;
if (items[outerIndex].Equals(items[innerIndex]))
{
// Duplicate Found
}
}
}
}

David J.'s answer is a good method, no need for extra objects, sorting, etc. It can be improved on however:
for (int innerIndex = items.Count - 1; innerIndex > outerIndex ; innerIndex--)
So the outer loop goes top bottom for the entire list, but the inner loop goes bottom "until the outer loop position is reached".
The outer loop makes sure the entire list is processed, the inner loop finds the actual duplicates, those can only happen in the part that the outer loop hasn't processed yet.
Or if you don't want to do bottom up for the inner loop you could have the inner loop start at outerIndex + 1.

A simple intuitive implementation:
public static List<PointF> RemoveDuplicates(List<PointF> listPoints)
{
List<PointF> result = new List<PointF>();
for (int i = 0; i < listPoints.Count; i++)
{
if (!result.Contains(listPoints[i]))
result.Add(listPoints[i]);
}
return result;
}

All answers copy lists, or create a new list, or use slow functions, or are just painfully slow.
To my understanding, this is the fastest and cheapest method I know (also, backed by a very experienced programmer specialized on real-time physics optimization).
// Duplicates will be noticed after a sort O(nLogn)
list.Sort();
// Store the current and last items. Current item declaration is not really needed, and probably optimized by the compiler, but in case it's not...
int lastItem = -1;
int currItem = -1;
int size = list.Count;
// Store the index pointing to the last item we want to keep in the list
int last = size - 1;
// Travel the items from last to first O(n)
for (int i = last; i >= 0; --i)
{
currItem = list[i];
// If this item was the same as the previous one, we don't want it
if (currItem == lastItem)
{
// Overwrite last in current place. It is a swap but we don't need the last
list[i] = list[last];
// Reduce the last index, we don't want that one anymore
last--;
}
// A new item, we store it and continue
else
lastItem = currItem;
}
// We now have an unsorted list with the duplicates at the end.
// Remove the last items just once
list.RemoveRange(last + 1, size - last - 1);
// Sort again O(n logn)
list.Sort();
Final cost is:
nlogn + n + nlogn = n + 2nlogn = O(nlogn) which is pretty nice.
Note about RemoveRange:
Since we cannot set the count of the list and avoid using the Remove funcions, I don't know exactly the speed of this operation but I guess it is the fastest way.

Using HashSet this can be done easily.
List<int> listWithDuplicates = new List<int> { 1, 2, 1, 2, 3, 4, 5 };
HashSet<int> hashWithoutDuplicates = new HashSet<int> ( listWithDuplicates );
List<int> listWithoutDuplicates = hashWithoutDuplicates.ToList();

Using HashSet:
list = new HashSet<T>(list).ToList();

public static void RemoveDuplicates<T>(IList<T> list )
{
if (list == null)
{
return;
}
int i = 1;
while(i<list.Count)
{
int j = 0;
bool remove = false;
while (j < i && !remove)
{
if (list[i].Equals(list[j]))
{
remove = true;
}
j++;
}
if (remove)
{
list.RemoveAt(i);
}
else
{
i++;
}
}
}

If you need to compare complex objects, you will need to pass a Comparer object inside the Distinct() method.
private void GetDistinctItemList(List<MyListItem> _listWithDuplicates)
{
//It might be a good idea to create MyListItemComparer
//elsewhere and cache it for performance.
List<MyListItem> _listWithoutDuplicates = _listWithDuplicates.Distinct(new MyListItemComparer()).ToList();
//Choose the line below instead, if you have a situation where there is a chance to change the list while Distinct() is running.
//ToArray() is used to solve "Collection was modified; enumeration operation may not execute" error.
//List<MyListItem> _listWithoutDuplicates = _listWithDuplicates.ToArray().Distinct(new MyListItemComparer()).ToList();
return _listWithoutDuplicates;
}
Assuming you have 2 other classes like:
public class MyListItemComparer : IEqualityComparer<MyListItem>
{
public bool Equals(MyListItem x, MyListItem y)
{
return x != null
&& y != null
&& x.A == y.A
&& x.B.Equals(y.B);
&& x.C.ToString().Equals(y.C.ToString());
}
public int GetHashCode(MyListItem codeh)
{
return codeh.GetHashCode();
}
}
And:
public class MyListItem
{
public int A { get; }
public string B { get; }
public MyEnum C { get; }
public MyListItem(int a, string b, MyEnum c)
{
A = a;
B = b;
C = c;
}
}

I think the simplest way is:
Create a new list and add unique item.
Example:
class MyList{
int id;
string date;
string email;
}
List<MyList> ml = new Mylist();
ml.Add(new MyList(){
id = 1;
date = "2020/09/06";
email = "zarezadeh#gmailcom"
});
ml.Add(new MyList(){
id = 2;
date = "2020/09/01";
email = "zarezadeh#gmailcom"
});
List<MyList> New_ml = new Mylist();
foreach (var item in ml)
{
if (New_ml.Where(w => w.email == item.email).SingleOrDefault() == null)
{
New_ml.Add(new MyList()
{
id = item.id,
date = item.date,
email = item.email
});
}
}

Fastest way to find common items across multiple lists in C#

Given the following:
List<List<Option>> optionLists;
what would be a quick way to determine the subset of Option objects that appear in all N lists? Equality is determined through some string property such as option1.Value == option2.Value.
So we should end up with List<Option> where each item appears only once.

Ok, this will find the list of Option objects that have a Value appearing in every list.
var x = from list in optionLists
from option in list
where optionLists.All(l => l.Any(o => o.Value == option.Value))
orderby option.Value
select option;
It doesn't do a "distinct" select so it'll return multiple Option objects, some of them with the same Value.

Building on Matt's answer, since we are only interested in options that all lists have in common, we can simply check for any options in the first list that the others share:
var sharedOptions =
from option in optionLists.First( ).Distinct( )
where optionLists.Skip( 1 ).All( l => l.Contains( option ) )
select option;
If an option list cannot contain duplicate entires, the Distinct call is unnecessary. If the lists vary greatly in size, it would be better to iterate over the options in the shortest list, rather than whatever list happens to be First. Sorted or hashed collections could be used to improve the lookup time of the Contains call, though it should not make much difference for a moderate number of items.

Here's a much more efficent implementation:
static SortedDictionary<T,bool>.KeyCollection FindCommon<T> (List<List<T>> items)
{
SortedDictionary<T, bool>
current_common = new SortedDictionary<T, bool> (),
common = new SortedDictionary<T, bool> ();
foreach (List<T> list in items)
{
if (current_common.Count == 0)
{
foreach (T item in list)
{
common [item] = true;
}
}
else
{
foreach (T item in list)
{
if (current_common.ContainsKey(item))
common[item] = true;
else
common[item] = false;
}
}
if (common.Count == 0)
{
current_common.Clear ();
break;
}
SortedDictionary<T, bool>
swap = current_common;
current_common = common;
common = swap;
common.Clear ();
}
return current_common.Keys;
}
It works by creating a set of all items common to all lists processed so far and comparing each list with this set, creating a temporary set of the items common to the current list and the list of common items so far. Effectively an O(n.m) where n is the number of lists and m the number of items in the lists.
An example of using it:
static void Main (string [] args)
{
Random
random = new Random();
List<List<int>>
items = new List<List<int>>();
for (int i = 0 ; i < 10 ; ++i)
{
List<int>
list = new List<int> ();
items.Add (list);
for (int j = 0 ; j < 100 ; ++j)
{
list.Add (random.Next (70));
}
}
SortedDictionary<int, bool>.KeyCollection
common = FindCommon (items);
foreach (List<int> list in items)
{
list.Sort ();
}
for (int i = 0 ; i < 100 ; ++i)
{
for (int j = 0 ; j < 10 ; ++j)
{
System.Diagnostics.Trace.Write (String.Format ("{0,-4:D} ", items [j] [i]));
}
System.Diagnostics.Trace.WriteLine ("");
}
foreach (int item in common)
{
System.Diagnostics.Trace.WriteLine (String.Format ("{0,-4:D} ", item));
}
}

Fastest to write :)
var subset = optionLists.Aggregate((x, y) => x.Intersect(y))

what about using a hashSet? that way you can do what you want in O(n) where n is the number of items in all the lists combined, and I think that's the fastest way to do it.
you just have to iterate over every list and insert the values you find into the hashset
When you insert a key that already exists you will receive false as the return value of the .add method, otherwise true is returned

Sort, then do something akin to a merge-sort.
Basically you would do this:
Retrieve the first item from each list
Compare the items, if equal, output
If any of the items are before the others, sort-wise, retrieve a new item from the corresponding list to replace it, otherwise, retrieve new items to replace them all, from all the list
As long as you still got items, go back to 2.

I don't have the performance stats, but if you don't want to roll your own method, various collections libraries have a 'Set' or 'Set(T)' object that offer the usual set procedures. (listed in the order I would use them).
IESI Collections (literally just Set classes)
PowerCollections (not updated in a while)
C5 (never personally used)

You can do this by counting occurrences of all items in all lists - those items whose occurrence count is equal to the number of lists, are common to all lists:
static List<T> FindCommon<T>(IEnumerable<List<T>> lists)
{
Dictionary<T, int> map = new Dictionary<T, int>();
int listCount = 0; // number of lists
foreach (IEnumerable<T> list in lists)
{
listCount++;
foreach (T item in list)
{
// Item encountered, increment count
int currCount;
if (!map.TryGetValue(item, out currCount))
currCount = 0;
currCount++;
map[item] = currCount;
}
}
List<T> result= new List<T>();
foreach (KeyValuePair<T,int> kvp in map)
{
// Items whose occurrence count is equal to the number of lists are common to all the lists
if (kvp.Value == listCount)
result.Add(kvp.Key);
}
return result;
}

/// <summary>
/// The method FindCommonItems, returns a list of all the COMMON ITEMS in the lists contained in the listOfLists.
/// The method expects lists containing NO DUPLICATE ITEMS.
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="allSets"></param>
/// <returns></returns>
public static List<T> FindCommonItems<T>(IEnumerable<List<T>> allSets)
{
Dictionary<T, int> map = new Dictionary<T, int>();
int listCount = 0; // Number of lists.
foreach (IEnumerable<T> currentSet in allSets)
{
int itemsCount = currentSet.ToList().Count;
HashSet<T> uniqueItems = new HashSet<T>();
bool duplicateItemEncountered = false;
listCount++;
foreach (T item in currentSet)
{
if (!uniqueItems.Add(item))
{
duplicateItemEncountered = true;
}
if (map.ContainsKey(item))
{
map[item]++;
}
else
{
map.Add(item, 1);
}
}
if (duplicateItemEncountered)
{
uniqueItems.Clear();
List<T> duplicateItems = new List<T>();
StringBuilder currentSetItems = new StringBuilder();
List<T> currentSetAsList = new List<T>(currentSet);
for (int i = 0; i < itemsCount; i++)
{
T currentItem = currentSetAsList[i];
if (!uniqueItems.Add(currentItem))
{
duplicateItems.Add(currentItem);
}
currentSetItems.Append(currentItem);
if (i < itemsCount - 1)
{
currentSetItems.Append(", ");
}
}
StringBuilder duplicateItemsNamesEnumeration = new StringBuilder();
int j = 0;
foreach (T item in duplicateItems)
{
duplicateItemsNamesEnumeration.Append(item.ToString());
if (j < uniqueItems.Count - 1)
{
duplicateItemsNamesEnumeration.Append(", ");
}
}
throw new Exception("The list " + currentSetItems.ToString() + " contains the following duplicate items: " + duplicateItemsNamesEnumeration.ToString());
}
}
List<T> result= new List<T>();
foreach (KeyValuePair<T, int> itemAndItsCount in map)
{
if (itemAndItsCount.Value == listCount) // Items whose occurrence count is equal to the number of lists are common to all the lists.
{
result.Add(itemAndItsCount.Key);
}
}
return result;
}

#Skizz The method is not correct. It returns also items that are not common to all the lists in items.
Here is the corrected method:
/// <summary>.
/// The method FindAllCommonItemsInAllTheLists, returns a HashSet that contains all the common items in the lists contained in the listOfLists,
/// regardless of the order of the items in the various lists.
/// </summary>
/// <typeparam name="T"></typeparam>
/// <param name="listOfLists"></param>
/// <returns></returns>
public static HashSet<T> FindAllCommonItemsInAllTheLists<T>(List<List<T>> listOfLists)
{
if (listOfLists == null || listOfLists.Count == 0)
{
return null;
}
HashSet<T> currentCommon = new HashSet<T>();
HashSet<T> common = new HashSet<T>();
foreach (List<T> currentList in listOfLists)
{
if (currentCommon.Count == 0)
{
foreach (T item in currentList)
{
common.Add(item);
}
}
else
{
foreach (T item in currentList)
{
if (currentCommon.Contains(item))
{
common.Add(item);
}
}
}
if (common.Count == 0)
{
currentCommon.Clear();
break;
}
currentCommon.Clear(); // Empty currentCommon for a new iteration.
foreach (T item in common) /* Copy all the items contained in common to currentCommon.
* currentCommon = common;
* does not work because thus currentCommon and common would point at the same object and
* the next statement:
* common.Clear();
* will also clear currentCommon.
*/
{
if (!currentCommon.Contains(item))
{
currentCommon.Add(item);
}
}
common.Clear();
}
return currentCommon;
}

After searching the 'net and not really coming up with something I liked (or that worked), I slept on it and came up with this. My SearchResult is similar to your Option. It has an EmployeeId in it and that's the thing I need to be common across lists. I return all records that have an EmployeeId in every list. It's not fancy, but it's simple and easy to understand, just what I like. For small lists (my case) it should perform just fine—and anyone can understand it!
private List<SearchResult> GetFinalSearchResults(IEnumerable<IEnumerable<SearchResult>> lists)
{
Dictionary<int, SearchResult> oldList = new Dictionary<int, SearchResult>();
Dictionary<int, SearchResult> newList = new Dictionary<int, SearchResult>();
oldList = lists.First().ToDictionary(x => x.EmployeeId, x => x);
foreach (List<SearchResult> list in lists.Skip(1))
{
foreach (SearchResult emp in list)
{
if (oldList.Keys.Contains(emp.EmployeeId))
{
newList.Add(emp.EmployeeId, emp);
}
}
oldList = new Dictionary<int, SearchResult>(newList);
newList.Clear();
}
return oldList.Values.ToList();
}