Develop Reference

Custom IEqualityComparer to get distinct objects from two lists faster than using .Where() + .Any()

so I've got two lists of objects, objects have multiple fields, but I'd like to distinct them basing on only two of them.
To give you the picture, object KeyAndValue consists of fields Key and Tag, so:
list1 = { obj1(key=1,tag=A), obj2(key=2,tag=A) }
list2 = { obj3(key=1,tag=A), obj4(key=2,tag=B) }
I'm currently using:
list1.Where(x => !list2.Any(y => y.key == x.key)).ToList();
The correct result is: obj1, obj2 and obj4 as obj3 has the same key and tag as obj1
What I'm trying to accomplish is to speed this process up as with a lot of objects it takes much too long. I found that custom IEqualityComparer could help here, so I've written my own basing on MS Specification
It looks like this:
class KeyComparer : IEqualityComparer<KeyAndValue>
{
public bool Equals(KeyAndValue x, KeyAndValue y)
{
if (Object.ReferenceEquals(x, y))
return true;
if (Object.ReferenceEquals(x, null) || Object.ReferenceEquals(y, null))
return false;
return x.key == y.key && x.tag == y.tag;
}
public int GetHashCode(KeyAndValue keyAndValue)
{
if (Object.ReferenceEquals(keyAndValue, null))
return 0;
int hashKeyAndValueKey = keyAndValue.key == null ? 0 : keyAndValue.key.GetHashCode();
int hashKeyAndValueTag = keyAndValue.tag == null ? 0 : keyAndValue.tag.GetHashCode();
return hashKeyAndValueKey ^ hashKeyAndValueTag;
}
}
And I use it like this:
list1.Except(list2, new KeyComparer()).ToList();
Unfortunately it does only remove duplicates from list2. It seems that it does not even touch list1 and I do not know if it's the fault of my custom comparer, the way I use it or maybe I should use another method. I've been looking through other questions but could not find a working answer (or at least one that I'd actually know how to implement properly).

I think you don't need Except. You want to have distinct values of both?
var distinctValues = list1.Union(list2).Distinct();
You need to either implement GetHashCode/Equals in KeyAndValue or use a comparer to compare the objects by key and value.
(Old stuff bellow)
Not sure whether I understand the question correctly. Could it be that you didn't recognize that Except create a new IEnumerable and ToList a new List?
try:
var list1AdditionalItems = list1.Except(list2, new KeyComparer()).ToList();
and probably also:
var list2AdditionalItems = list2.Except(list1, new KeyComparer()).ToList();
Another observation. in this code:
list1.Where(x => !list2.Any(y => y.key == x.key)).ToList();
You only check the key. If you want this behaviour, you should write the comparer accordingly:
class KeyComparer : IEqualityComparer<KeyAndValue>
{
public bool Equals(KeyAndValue x, KeyAndValue y)
{
if (ReferenceEquals(x, y))
return true;
if (ReferenceEquals(x, null) || ReferenceEquals(y, null))
return false;
return x.key == y.key;
}
public int GetHashCode(KeyAndValue keyAndValue)
{
if (ReferenceEquals(keyAndValue, null))
return 0;
return keyAndValue.key == null ? 0 : keyAndValue.key.GetHashCode();
}
}
Last but not least: When you need performance, consider using a dictionary instead.

As Stefan Steinegger (whom I thank grately for the effort and time spent) mentioned in the first comment that neither of my methods return obj4 I found an issue and decided to implement totally different approach. Now my KeyAndValue class has also an int Hash field, when a constructor is called Hash field is filled with key.GetHashCode() ^ tag.GetHashCode(). It simplified the comparison as now I'm first combining list1 with list2 and then sending it through: CombinedList.GroupBy(x => x.Hash).Select(y => y.First()).ToList(); Results seem to be correct :)

How can I remove a list from another list using a field value of the element's type if i can't modify the class directly?

Suppose I have a List<Product> named big of some object Product which has 2 fields, ID and Name. The list is fully populated, meaning that each element has both fields not null, although they can be.
I have another List<Product>, smaller than the first, named small, but this time the field Name is null for some elements, while ID is always present.
I want to remove small from big where ID is the same.
Example:
List<Product> big = { {1,A},{2,B},{3,C},{4,D} };
List<Product> small = { {1,null},{3,C} };
List<Product> result = { {2,B},{4,D}};
I can't modify the Product object, i.e. I can't implement IEquatable<Product>, and such interface isn't implemented, meaning that big.Except(small) or big.RemoveAll(small) or big.Contains(anElementOfSmall) won't work (for the sake of the question, I've tried them).
I want to avoid the double loop with iterator removal or similar, I'm searching for built-in functions with specific predicates.

Implement an IEqualityComparer for your Product that compares two Product based on ID. Then just use Except and pass the comparer:
var result = big.Except(small, new ProductComparer()).ToList();

Using a simple predicate you can easily achieve it:
big.Where(p => !small.Any(o => o.id == p.id)).ToList();
which translates in: select from big where the element (p) doesn't suffice the case where an element o in small has the same ID.

You need to tell Except how to compare instances of Product.
public class ProductEqualityComparer : IEqualityComparer<Product>
{
public bool Equals(Product x, Product y)
{
//they're both the same instance or they're both null
if(ReferanceEquals(x, y))
return true;
//only one of them is null
if(x == null || y == null)
return false;
return x.Id == y.Id;
}
public int GetHashCode(Product prod)
{
return prod == null? 0 : prod.Id.GetHashCode();
}
}
big.Except(small, new ProductEqualityComparer())

LINQ grouping multiple columns with nested collection

I am grouping a collection, by multiple columns. It is easy to group them if they are on this same level of nesting:
var groupedAirProducts = airproductsPerClient.GroupBy(ac => new
{
ac.AirProduct.PassengerNameRecord,
ac.AirProduct.Flights.First().MarketingAirlineCode,
ac.AirProduct.Flights.First().FlightNo,
ac.AirProduct.Flights.First().DepartureDate
})
The problem is that I don't want only to group by a first flight, but I want to include all the flights. What I would like to do is something like:
var groupedAirProducts= airproductsPerClient.GroupBy(ac =>
{
ac.AirProduct.PassengerNameRecord,
foreach(var flight in ac.AirProduct.Flights){
flight.MarketingAirlineCode,
flight.FlightNo,
flight.DepartureDate
}
})
Please note that the code above is just an illustration of an idea. It is not a working/proper code. The only way I know how to do it is pretty ugly and complex. I was wondering if there is a nice way to do it using LINQ, or a simple way to tackle this problem.
Edit:
To explain more of what is expected. As a result of grouping I want to have collections of airProducts that share same PassengerNameRecord, and that flights that belong to a given air product share same MarketingAirlineCode, FlightNo, DepartureDate. I know how to implement it by flattering a collection of PassengerNameRecord, so the Flights are included in it, grouping it, so I have a groups of air products that share grouped properties. And then rebuilding the flattered structure. I was hoping that there is a way to either add groupBy properties by iterating thru some collection, or there is a way to merge grouped collection - if there is a way to have a collection grouped by PassengerNameRecord, and collection grouped by Flight properties and merging them together, unfortunately I doubt that there is an easy way to do such merge.

You can implement a custom IEqualityComparer<AirClient> for GroupBy(or other LINQ methods).
You can implement it in the following way:
public class AirClientComparer : IEqualityComparer<AirClient>
{
public bool Equals(AirClient lhs, AirClient rhs)
{
if (lhs == null || rhs == null) return false;
if(object.ReferenceEquals(lhs, rhs)) return true;
if(lhs.PassengerNameRecord != rhs.PassengerNameRecord) return false;
if(object.ReferenceEquals(lhs.AirProduct, rhs.AirProduct)) return true;
if(lhs.AirProduct == null || rhs.AirProduct == null) return false;
if(object.ReferenceEquals(lhs.AirProduct.Flights , rhs.AirProduct.Flights )) return true;
if(lhs.AirProduct.Flights.Count != rhs.AirProduct.Flights.Count) return false;
if(lhs.AirProduct.Flights.Count == 0 && rhs.AirProduct.Flights.Count == 0) return true;
return lhs.AirProduct.Flights.All(f =>
rhs.AirProduct.Flights.Any(f2 =>
f.MarketingAirlineCode == f2.MarketingAirlineCode
&& f.FlightNo == f2.FlightNo
&& f.DepartureDate == f2.DepartureDate));
}
public int GetHashCode(AirClient obj)
{
if(obj.AirProduct == null) return 0;
int hash = obj.AirProduct.PassengerNameRecord == null
? 17 : 17 * obj.AirProduct.PassengerNameRecord.GetHashCode();
unchecked
{
foreach(var flight in obj.AirProduct.Flights)
{
hash = (hash * 31) + flight.MarketingAirlineCode == null ? 0 : flight.MarketingAirlineCode.GetHashCode();
hash = (hash * 31) + flight.FlightNo == null ? 0 : flight.FlightNo.GetHashCode();
hash = (hash * 31) + flight.DepartureDate.GetHashCode();
}
}
return hash;
}
}
Now you can use it for example in GroupBy:
var groupedAirProducts = airproductsPerClient.GroupBy(ac => new AirClientComparer());

In such cases, you need to start with the inner table.
The inner table also needs a reference to the outer table, such like you have a
ICollection<Flight> Flights in AirProduct
you also need a Airproduct Member inside of Flight.
Then group your flights like this (pseudo code)
AirProduct.Flights.groupby( flight => new{ flight.MarketingAirlineCode,
flight.FlightNo,
flight.DepartureDate,
flight.product.PassengerNameRecord
});

Poorly performing query needs rewriting

I have a piece of code that's performing badly, and need to rewite it to introduce a proper where clause before starting the .ToList however, that's where I'm getting stuck.
Currently the code looks lke this (roughly, I've taken some of the search criteria out to make it easier to display)
var Widgets = from b in _caspEntities.Widgets.Include("WidgetRegionLogs")
.Include("WidgetStatusLogs").Include("WidgetVoltageTests")
select b;
IEnumerable<Widget> results = Widgets.ToList();
if (comboBoxRegion.SelectedValue.ToString() != "0")
{
results = from b in results
where b.CurrentRegionLog != null && b.CurrentRegionLog.RegionId == int.Parse(comboBoxRegion.SelectedValue.ToString())
select b;
}
if (comboBoxStatus.SelectedValue != null)
{
results = from b in results
where b.CurrentStatusLog != null && b.CurrentStatusLog.StatusId == comboBoxStatus.SelectedValue.ToString()
select b;
}
if (txtCode.Text.Trim().Length > 0)
{
results = from b in results
where b.CodeNumber == txtCode.Text.Trim()
select b;
}
dataGridViewWidget.DataSource = results.ToList();
I can write the SQL easily enough, essentially the model is simple, I have a Widget it has a RegionLog and a StatusLog, both of which store a history. The current region and status are retrieved from this by grouping by WidgetID and selecting the most recent Date Updated (and then going off to Region and Status tables to get the actual value).
So, I need to translate this into LINQ, but to be honest I don't have a clue but am ken and willing to learn. In my head, I think I need to add some better where clauses, and then do the Widget.toList after I have applied the where clauses. I'm struggling with the CurrentRegionLog and CurrentStatusLog concepts as they are not populated until I run the IEnumerable.
If anyone can give some pointers, I'd be grateful,
Thanks
Edit - Added
public BatteryRegionLog CurrentRegionLog
{
get { return _currentRegionLog; }
}
private BatteryRegionLog _currentRegionLog
{
get
{
if (this.BatteryRegionLogs.Count > 0)
{
BatteryRegionLog log = this.BatteryRegionLogs.OrderByDescending(item => item.LastModifiedDate).First();
return log;
}
else
{
return null;
}
}
}

You can compose the query like this:
if (comboBoxRegion.SelectedValue.ToString() != "0")
{
var id = int.Parse(comboBoxRegion.SelectedValue.ToString()
Widgets = from b in Widgets
let currentRegionLog =
b.BatteryRegionLogs
.OrderByDescending(item => item.LastModifiedDate)
.FirstOrDefault()
where currentRegionLog.RegionId == id)
select b;
}
... // Same for the other criteria.
dataGridViewWidget.DataSource = Widgets.ToList();
The whole query is not executed before you do ToList(). As everything is translated to SQL you don't need the null check b.CurrentRegionLog != null. SQL will evaluate b.CurrentRegionLog.RegionId == id just fine when there is no CurrentRegionLog.
Edit
Since CurrentRegionLog is a calculated property of your Widget class it cannot be translated into SQL. I made an effort to incorporate the code of calculated property into the query in a way that only the basic navigation property is used, so EF can translate it to SQL again.

try remove this line:
IEnumerable<Widget> results = Widgets.ToList();
and just use the Widgets variable you get in at the top
The .ToList() goes to the database and materialiaze all the data into entities.
if you don't call the .ToList() the query is still "open" for a where clause

Get one unique List<T> from three non-unique List<T>'s

I have three lists of User:
ICollection<User> listOne = _somewhere.GetUsers(1);
ICollection<User> listTwo = _somewhere.GetUsers(2);
ICollection<User> listThree = _somewhere.GetUsers(3);
The "unique" identifier to compare on is a string field called "Email".
How do i get a unique list from the three (e.g no dupes).
I've got a unique list from two lists before using Except, but not sure how to do it with three? Do i have to use Except on the first two, then do it again on the result of the first two and the third?
Also, i should mention that the list of User's comes from external Web API calls, and there is no guarantee that each list has a unique list of email addresses.
So it's like i need two steps:
In each list, get rid of dupes
Combine the three unique lists to get one unique list.

You can just union the lists and do a dedupe (using Distinct()) once on the combined list.
var uniqueList = listOne.Union(listTwo)
.Union(listThree)
.Distinct(new EmailComparer())
.ToList();
For the comparer could be as simple as this:
class EmailComparer : IEqualityComparer<User>
{
public bool Equals(User x, User y)
{
return x.Email == y.Email;
}
public int GetHashCode(User obj)
{
return obj.Email.GetHashCode();
}
}
Edit:
As pointed out in comments Distinct() is not needed if we apply the custom email comparer to Union():
var emailComparer = new EmailComparer();
var uniqueList = listOne.Union(listTwo, emailComparer)
.Union(listThree, emailComparer)
.ToList();

If it does not matter which user you pick from the list of users with the same e-mail, you can do this:
var res = listOne.Concat(listTwo).Concat(listThree)
.GroupBy(u => u.Email)
.Select(g => g.First());
Again, this assumes that when e-mail addresses are the same, it does not matter which user you'd pick.

First define how we want to define uniqueness:
private class EmailEqComparer : IEqualityComparer<User>
{
public bool Equals(User x, User y)
{
//don't bother shortcutting on reference equality, since if they come from
//separate web-calls it's unlikely to happen, though it could
//with some optimisations on the web-client code.
if(x == null)
return y == null;
if(y == null)
return false;
return x.Email == y.Email;
}
public int GetHashCode(User obj)
{
return obj == null ? 0 : obj.Email.GetHashCode();
}
}
Now call Distinct on the items of each, and put the results into a list:
var distinctUnion = listOne.Concat(listTwo).Concat(listThree).Distinct(new EmailEqComparer());