Develop Reference

Comparing two lists of nested lists and returning the added/changed/removed items

I've looked at many similar questions on stackoverflow, but I haven't seen an exact match for my problem.
I need to compare two "lists of nested lists" and capture the differences. One is an "old" list and the other is a "new" list. When comparing the nested lists, they can be considered equal if all of the NESTED list items (the MyObject.Ids) are present in both lists in order (you can assume that the nested MyObject.Ids lists are already sorted and that there are no duplicates). The MyObject.Id and MyObject.Name properties are not considering in the equality comparison, but they are still important metadata for MyObject's which should not get lost.
I am not looking for a boolean indicator of equality. Instead I need to create three new lists which capture the differences between the old and new lists (e.g. a list of items which were Added, a list of items which were Removed, and a list of items which were present in both lists).
Below is an example of some code which does exactly what I want! What I would like to know is how to make this shorter/better/simpler (cutting out one of the for loops would be a good start). To make things trickier, please assume that you cannot make any changes to the MyObject class or use any custom Equals/IEqualityComparer etc implementations.
public class MyObject
{
public Guid Id { get; set; }
public string Name { get; set; }
public List<Guid> Ids { get; set; }
}
...
// Get the list of existing objects (assume this returns some populated list)
List<MyObject> existingObjects = GetExistingObjects();
// Create a list of updated objects
List<MyObject> updatedObjects = new List<MyObject>()
{
new MyObject()
{
Ids = new List<Guid>() { new Guid("48af3cb9-945a-4ab9-91e4-7ee5765e5304"), new Guid("54b5128a-cf53-436c-9d88-2ef7abd15140") }
},
new MyObject()
{
Ids = new List<Guid>() { new Guid("0485382f-8f92-4a71-9eba-09831392ceb9"), new Guid("3d8b98df-caee-41ce-b802-2f0c5f9742de") }
}
};
// Do the comparison and capture the differences
List<MyObject> addedObjects = new List<MyObject>();
List<MyObject> removedObjects = new List<MyObject>();
List<MyObject> sameObjects = new List<MyObject>();
foreach (MyObject obj in updatedObjects)
{
if (existingObjects.Any(list => list.Ids.SequenceEqual(obj.Ids)))
{
sameObjects.Add(obj);
continue;
}
addedObjects.Add(obj);
}
foreach (MyObject obj in existingObjects)
{
if (!updatedObjects.Any(list => list.Ids.SequenceEqual(obj.Ids)))
{
removedObjects.Add(obj);
}
}

Here is a little shorter (due to elimination of the second loop) and little better (due to elimination of the ineffective search contained in the second loop). Still O(N^2) time complexity due to ineffective search contained in the loop though.
var addedObjects = new List<MyObject>();
var removedObjects = new List<MyObject>(existingObjects);
var sameObjects = new List<MyObject>();
foreach (var newObject in updatedObjects)
{
int index = removedObjects.FindIndex(oldObject => oldObject.Ids.SequenceEqual(newObject.Ids));
if (index < 0)
addedObjects.Add(newObject);
else
{
removedObjects.RemoveAt(index);
sameObjects.Add(newObject);
}
}
Update: A shorter, but IMO definitely not better (in fact worse performance wise) version
var addedObjects = updatedObjects.Where(newObject => !existingObjects.Any(oldObject => oldObject.Ids.SequenceEqual(newObject.Ids))).ToList();
var removedObjects = existingObjects.Where(oldObject => !updatedObjects.Any(newObject => newObject.Ids.SequenceEqual(oldObject.Ids))).ToList();
var sameObjects = updatedObjects.Where(newObject => !addedObjects.Any(addedObject => addedObject.Ids.SequenceEqual(newObject.Ids))).ToList();
If MyObject does not define custom equality comparison, i.e. uses default reference equality, the last line could be replaced with shorter and better performing
var sameObjects = updatedObjects.Except(addedObjects);

You can use Intersect and Except function in Linq
With Intersect you will get existing object,
and with Except you will get new objects.
Example of Except from MSDN:
double[] numbers1 = { 2.0, 2.1, 2.2, 2.3, 2.4, 2.5 };
double[] numbers2 = { 2.2 };
IEnumerable<double> onlyInFirstSet = numbers1.Except(numbers2);
foreach (double number in onlyInFirstSet)
Console.WriteLine(number);

C# failing to set property inside IEnumerable

Seen a weird bit of behaviour in some C# code that I'm at a loss to explain. Could be I'm missing an important bit of understanding, so hoping someone out there can switch on the light for me.
Got a block of code that looks like this:
IEnumberable<myObject> objects = GetObjectsFromApiCall();
for (int i = 0; i < objects.Count(); i++)
{
if (String.IsNullOrEmpty(objects.ElementAt(i).SubObject.Title))
{
SubObject sub = GetSubObjectFromDatabase((long)objects.ElementAt(i).SubObject.Id);
if (sub != null)
{
objects.ElementAt(i).SubObject.Title = sub.Title;
}
}
}
When you step through it, everything about this code seems to work properly. The "objects" collection is populated as expected. "sub" is fetched as collected and has a full set of expected properties, including a populated Title property. No errors are thrown during execution.
But ... the SubObject.Title property (which just has standard get; set; code) that exists in each Object stubbornly remains empty.
I'm at a loss. Anyone explain what's going on?
EDIT: For those who suggested I shouldn't use a for loop and ElementAt, I started with a foreach loop but thought it might be the source of the problem because it was fetching a new SubObject each time round. Fixed now, thanks to your help, and the ForEach restored.
Cheers,
Matt

I would fix it this way:
var objects = GetObjectsFromApiCall().ToList();
Then you could keep the loop as is (it works), or optimize it a bit using foreach and some Linq as suggested by other answers, but it does not really matter: the problem was that you attempted to change an element on an IEnumerator<> as explained in this question pointed by #Ahmet Kakıcı.

Try this
List<myObject> objects = GetObjectsFromApiCall().ToList();
foreach(var obj in objects.Where(o => string.IsNullOrEmpty(objects.SubObject.Title)).ToList())
{
var subObject = GetSubObjectFromDatabase(obj.SubObject.Id);
if(subObject == null) continue;
obj.SubObject.Title = subObject.Title;
}

First of all, you should not use ElementAt() for this kind of code, use
foreach (var o in objects)
{
if (string.IsNullOrEmpty(o.SubObject.Title))
{
o.SubObject.Title = ...;
}
}
Also you should note that if your method returns a dynamic IEnumerable then every time you call objects.Something() the API is called again and a fresh copy is retrieved. If this is the case, you should copy the enumerable into a list using .ToList() method.
There is also a way of not putting a copy in the list - by creating a dynamic enumerator like this:
objects = objects.Select(o =>
{
if (string.IsNullOrEmpty(o.SubObject.Title))
{
o.SubObject.Title = ...;
}
return o;
});
As for the value not being set correctly (if previous things did not help) - try adding a throw new Exception(value) in the setter for Title property - see if that is being called with the correct value.

I guest the function GetObjectsFromApiCall looks like following:
public IEnumberable<myObject> GetObjectsFromApiCall(){
for(var i = 0; i < 10; i++)
{
yield return new myObject();
}
}
If I'm right, every time you call objects.ElementAt(i) function to get the object, you will get a new object by "yield return new myObject()".

But how do you check if the Title property is changed? Do you call GetObjectsFromApiCall() again? Or do you foreach through the same objects instance again?
An IEnumerable instance may create and yield new objects each time it is "enumerated". So here's a simple example for illustration. For the example, define:
class SomeObject
{
public string Title { get; set; }
}
Then we will consider two types of "source", first an array, and then an iterator block defined like this:
static IEnumerable<SomeObject> GetSomeSequence()
{
yield return new SomeObject { Title = "Alpha", };
yield return new SomeObject { Title = "Beta", };
yield return new SomeObject { Title = "Gamma", };
}
Then test it this way:
static void Main()
{
IEnumerable<SomeObject> thingsToModify;
// set source to an array
thingsToModify = new[] { new SomeObject { Title = "Alpha", }, new SomeObject { Title = "Beta", }, new SomeObject { Title = "Gamma", }, };
foreach (var t in thingsToModify)
Console.WriteLine(t.Title);
foreach (var t in thingsToModify)
t.Title = "Changed!";
foreach (var t in thingsToModify)
Console.WriteLine(t.Title); // OK, modified
// set source to something which yields new object each time a new GetEnumerator() call is made
thingsToModify = GetSomeSequence();
foreach (var t in thingsToModify)
Console.WriteLine(t.Title);
foreach (var t in thingsToModify)
t.Title = "Changed!"; // no-one keeps these modified objects
foreach (var t in thingsToModify)
Console.WriteLine(t.Title); // new objects, titles not modified
}
Conclusion: It's perfectly possible to modify the state of a mutable object which belongs to the source we're iterating over. But some types of IEnumerable sources yield new copies of the data each time they are called, and then it's useless to make modifications to the copy.

Adding a generic object to a Dictionary<string, T> in C# - Null Reference Exception

I am getting a Null Reference Exception, but can't figure out why (see code).
I have the following code as part of a deserializer I'm making. The following code adds the default key/values to a set of dictionaries when the deserialised instance of my class is missing some keys. The reason for this is so that when a customer deploys a new version of my plugin, their old settings will still deserilize properly, even if they don't have all the keys.
Here's my code:
private Dictionary<string, T> AddMissingSettings<T>(Dictionary<string, T> settingsObject, bool forceDefaults, string[] settingsList)
{
if (forceDefaults || settingsObject == null)
{
settingsObject = new Dictionary<string, T>();
}
foreach (string t in settingsList)
{
if (!settingsObject.ContainsKey(t))
{
settingsObject.Add(t, default(T));// Null Reference Exception Here
}
}
return settingsObject;
}
Here is an example of me envoking the method:
Dictionary<string, DateTime> Times = getlistoftimesettingsfromdata();
//Note: Not the real way Times gets set, but whatever
Times = AddMissingSettings<DateTime>(Times, forceDefaults, new string[] { "LastEmailUpdate", "IdealRegularReportTime", "LastRegularReport" });
//Also, pretty sure the "Times = " here is redundant, but let it pass
EDIT
VPNd into work to post the actual code I'm using. I can confirm that it compiles and runs.
private void LoadDefaults(bool forceDefaults)
{
if (RegularReportDays == null || forceDefaults) { RegularReportDays = new List<DayOfWeek>(); }
if (RegularReportSubscribers == null || forceDefaults) { RegularReportSubscribers = new List<string>(); }
Times = AddMissingSettings<DateTime>(Times, forceDefaults, DateTime.MinValue, new string[] { "LastEmailUpdate", "IdealRegularReportTime", "LastRegularReport" });
Toggles = AddMissingSettings<bool>(Toggles, forceDefaults, false, new string[] { "AutomaticUpdates", "AutomaticUpdatesAlerts", "SendRegularReports" });
}
private Dictionary<string, T> AddMissingSettings<T>(Dictionary<string, T> settingsObject, bool forceDefaults, T defaultValue, string[] settingsList)
{
if (forceDefaults || settingsObject == null)
{
settingsObject = new Dictionary<string, T>();
}
foreach (string t in settingsList)
{
if (!settingsObject.ContainsKey(t))
{
settingsObject.Add(t, defaultValue);
}
}
return settingsObject;
}

Okay, so I've come back fresh and bright on Monday and I see what the issue is. The problem was happening - as others have sniffed out - further up the stack. The key to this solution is in a place I didn't mention because I didn't think it was relevant (sorry).
I was trying to run the following stack
private Dictionary<string, T> AddMissingSettings<T>(Dictionary<string, T> settingsObject, bool forceDefaults, string[] settingsList)
private void LoadDefaults(bool forceDefaults)
public BookEasyImportServiceSettings(SerializationInfo info, StreamingContext cxt)
You might notice that I'm trying to alter the members of an object, whilst it is being deserialized. It seems you can't do that. So I added a public LoadMissingDefaults method to my interface and I call it when the BinaryFormatter.Deserialize(Stream) has finished.
Now, I really don't like awarding answers to myself (although equally, I don't like solving my problem and not sharing my solution - so I do it sometimes). If anyone can submit a good answer as to why this is the case and/or a better way of doing it, I'll switch the awarded answer to them.

You cannot use null as a key in Dictionary. From your code settingsList is string[] and string can be null. I would suggest you to check null before you add it to settingObject.
foreach (string t in settingsList)
{
if (t != null && !settingsObject.ContainsKey(t))
{
settingsObject.Add(t, default(T));// Null Reference Exception Here
}
}

Sort List without creating new variable

I'm attempting to use Enumerable.OrderBy to sort a List because ultimately I want to be able to sort by more than a single field. At the moment it only appears to work if I create a new variable var to hold the results view which means (I think) the types need to be re-cast.
Is there a method to sort a List by more than 1 field whilst retaining the original List variable and types? I.e. I'd rather end up with variable _orderedbins of type List<orderedbins>
Below is what I currently have but everything from var test = ... onwards seems a bit wrong.
public class orderedBins
{
public string Bin { get; set; }
public int Order { get; set; }
}
List<orderedbins> _orderedbins = new List<orderedbins>();
foreach (string item in splitbins)
{
string[] spbinSetting = item.Split(',');
bool bchecked = bool.Parse(spbinSetting[1]);
int border = int.Parse(spbinSetting[2]);
if (bchecked == true)
{
_orderedbins.Add(new orderedbins { bin = spbinSetting[0], Order = border });
}
}
var test =_orderedbins.OrderBy(x => x.Order);
foreach (var item in test)
{
string f = item.Bin;
int g = item.Order;
}

You know, you can perform multiple sub-sorts for an order by...
lst.OrderBy(x => x.Prop1).ThenBy(x => x.Prop2).ThenByDescending(x => x.Prop3)...
Just add a .ToList(); and introduce it with a variable, to have the result in a list variable.
EDIT:
Great suggestion by Willem, for more readability:
from x in lst
order by x.Prop1, x.Prop2, x.Prop3
select x

You can create a new sorted list without creating a new variable using
list = list.OrderBy(item => item.Field1).ThenBy(item => item.Field1).ToList();
It will still create an entirely new list though (it's not actually much of a problem to add a new variable; those are cheap. Creating a new list, doing this, is fine as long as the list isn't very large.
If you need to sort the list in place then you'll want to use a custom comparer with the List's sort method:
public class MyComparer : IComparer<MyClass>
{
public int Compare(MyClass x, MyClass y)
{
if(x.Field1 != y.Field1)
return x.Field1.CompareTo(y.Field1)
else
return x.Field2.CompareTo(y.Field2);
}
}
List<MyClass> list = new List<MyClass>();
//Populate list
list.Sort(new MyComparer());

As others suggested, using Linq's OrderBy(...).ToList() would be a cleaner way, but this will give you a new instance of the list.
To retain the original instance, consider to use List<T>.Sort():
_orderedbins.Sort(new Comparison<orderedBins>((obj1, obj2) =>
{
int result = obj1.Order.CompareTo(obj2.Order);
return result != 0 ? result : obj1.Bin.CompareTo(obj2.Bin);
}));

This will do the trick:
_orderedbins = _orderedbins.OrderBy(x => x.Order).ToList();
...but there's no real issue creating a new variable/reference.

I think this will do it (it's already a list of orderbins so no casting is required):
_orderbins = _orderbins.OrderBy(x => x.Order).ToList();

Is there a more efficient way of creating a list based on an existing list and a lookup list?

I have the following method that takes an extremely long time to run and would love some help to make it run faster and or be more efficient.
The main responsibility of the method is to take a list of data points created from a CSV file, map the Name property of the file datapoints to the to the HistorianTagname property in a list of tagnames by the DataLoggerTagname property and create a resulting list from the mapping. If the mapping does not exist, the file datapoint is ignored.
I know it that was long-winded, but I hope it makes sense. It may be easier just to look at the method:
private IEnumerable<DataPoint> GetHistorianDatapoints(IEnumerable<DataPoint> fileDatapoints, IEnumerable<Tagname> historianTagnames)
{
/**
** REFACTOR THIS
**/
foreach (var fileDatapoint in fileDatapoints)
{
var historianTagname = historianTagnames.FirstOrDefault(x => x.DataLoggerTagname.Equals(fileDatapoint.Name, StringComparison.OrdinalIgnoreCase));
if (historianTagname != null)
{
var historianDatapoint = new DataPoint();
historianDatapoint.Name = historianTagname.HistorianTagname;
historianDatapoint.Date = fileDatapoint.Date;
historianDatapoint.Value = fileDatapoint.Value;
yield return historianDatapoint;
}
}
}
Notes:
I have complete control of classes and methods of mapping, so if I am doing something fundamentally wrong. I would love to know!
Thanks!

I would start by fixing up:
var historianTagname = historianTagnames.FirstOrDefault(x => x.DataLoggerTagname.Equals(fileDatapoint.Name, StringComparison.OrdinalIgnoreCase))
That's a pretty expensive operation to run every iteration through this loop.

Below is my proposition:
private IEnumerable<DataPoint> GetHistorianDatapoints(IEnumerable<DataPoint> fileDatapoints, IEnumerable<Tagname> historianTagnames)
{
var tagNameDictionary = historianTagnames.ToDictionary(t => t.DataLoggerTagname, StringComparer.OrdinalIgnoreCase);
foreach (var fileDatapoint in fileDatapoints)
{
if (tagNameDictionary.ContainsKey(fileDatapoint.Name))
{
var historianTagname = tagNameDictionary[fileDatapoint.Name];
var historianDatapoint = new DataPoint();
historianDatapoint.Name = historianTagname.HistorianTagname;
historianDatapoint.Date = fileDatapoint.Date;
historianDatapoint.Value = fileDatapoint.Value;
yield return historianDatapoint;
}
}
}

Like #Sheldon Warkentin said FirstOrDefault is probably bottle neck of your function, i s better to create historianTagnames a Dictionary where Name is key, then in your function you can get value by key.
Something like bellow:
// this is passed to method
IDictionary<string, Tagname> historianTagnames;
// .. method body
var historianTagname = historianTagnames[fileDatapoint.Name];
ofcourse you need to add proper if's.

As others have said, a Dictionary<string, Tagname> might perform better.
var historianDict = new Dictionary<string, Tagname>();
foreach (var tagName in historianTagnames) {
historianDict[tagName.DataLoggerTagname.ToLowerInvariant()] = tagName;
}
foreach (var fileDatapoint in fileDatapoints) {
if (historianDict.ContainsKey(fileDatapoint.Name.ToLowerInvariant()) {
// ...
}
}