Develop Reference

Collections manipulation, need help optimizing this code from a report generator

I'm creating a report generating tool that use custom data type of different sources from our system. The user can create a report schema and depending on what asked, the data get associated based different index keys, time, time ranges, etc. The project is NOT doing queries in a relational database, it's pure C# code in collections from RAM.
I'm having a huge performance issue and I'm looking at my code since a few days and struggle with trying to optimize it.
I stripped down the code to the minimum for a short example of what the profiler point as the problematic algorithm, but the real version is a bit more complex with more conditions and working with dates.
In short, this function return a subset of "values" satisfying the conditions depending on the keys of the values that were selected from the "index rows".
private List<LoadedDataSource> GetAssociatedValues(IReadOnlyCollection<List<LoadedDataSource>> indexRows, List<LoadedDataSource> values)
{
var checkContainers = ((ValueColumn.LinkKeys & ReportLinkKeys.ContainerId) > 0 &&
values.Any(t => t.ContainerId.HasValue));
var checkEnterpriseId = ((ValueColumn.LinkKeys & ReportLinkKeys.EnterpriseId) > 0 &&
values.Any(t => t.EnterpriseId.HasValue));
var ret = new List<LoadedDataSource>();
foreach (var value in values)
{
var valid = true;
foreach (var index in indexRows)
{
// ContainerId
var indexConservedSource = index.AsEnumerable();
if (checkContainers && index.CheckContainer && value.ContainerId.HasValue)
{
indexConservedSource = indexConservedSource.Where(t => t.ContainerId.HasValue && t.ContainerId.Value == value.ContainerId.Value);
if (!indexConservedSource.Any())
{
valid = false;
break;
}
}
//EnterpriseId
if (checkEnterpriseId && index.CheckEnterpriseId && value.EnterpriseId.HasValue)
{
indexConservedSource = indexConservedSource.Where(t => t.EnterpriseId.HasValue && t.EnterpriseId.Value == value.EnterpriseId.Value);
if (!indexConservedSource.Any())
{
valid = false;
break;
}
}
}
if (valid)
ret.Add(value);
}
return ret;
}
This works for small samples, but as soon as I have thousands of values, and 2-3 index rows with a few dozens values too, it can take hours to generate.
As you can see, I try to break as soon as a index condition fail and pass to the next value.
I could probably do everything in a single "values.Where(####).ToList()", but that condition get complex fast.
I tried generating a IQueryable around indexConservedSource but it was even worse. I tried using a Parallel.ForEach with a ConcurrentBag for "ret", and it was also slower.
What else can be done?

What you are doing, in principle, is calculating intersection of two sequences. You use two nested loops and that is slow as the time is O(m*n). You have two other options:
sort both sequences and merge them
convert one sequence into hash table and test the second against it
The second approach seems better for this scenario. Just convert those index lists into HashSet and test values against it. I added some code for inspiration:
private List<LoadedDataSource> GetAssociatedValues(IReadOnlyCollection<List<LoadedDataSource>> indexRows, List<LoadedDataSource> values)
{
var ret = values;
if ((ValueColumn.LinkKeys & ReportLinkKeys.ContainerId) > 0 &&
ret.Any(t => t.ContainerId.HasValue))
{
var indexes = indexRows
.Where(i => i.CheckContainer)
.Select(i => new HashSet<int>(i
.Where(h => h.ContainerId.HasValue)
.Select(h => h.ContainerId.Value)))
.ToList();
ret = ret.Where(v => v.ContainerId == null
|| indexes.All(i => i.Contains(v.ContainerId)))
.ToList();
}
if ((ValueColumn.LinkKeys & ReportLinkKeys.EnterpriseId) > 0 &&
ret.Any(t => t.EnterpriseId.HasValue))
{
var indexes = indexRows
.Where(i => i.CheckEnterpriseId)
.Select(i => new HashSet<int>(i
.Where(h => h.EnterpriseId.HasValue)
.Select(h => h.EnterpriseId.Value)))
.ToList();
ret = ret.Where(v => v.EnterpriseId == null
|| indexes.All(i => i.Contains(v.EnterpriseId)))
.ToList();
}
return ret;
}

Date falls between two dates and matches one date from table in Linq

I need to check if an application date falls between a range of policy applicationBeginDate and policy applicationEndDate I have a table that stores these dates and the Effective date that needs to match as well. My method needs to return a boolean value of true or false and I am lost.
The error I am receiving is that > operator cannot be applied to operands of type DateTime and Iqueryable.
var applicationBeginDates = _app2Lead.DataAccess.App2BonusPromos.FindAll(x => x.ApplicationBeginDate);
var effectiveDates = _app2Lead.DataAccess.App2BonusPromos.FindAll(x => x.EffectiveDate);
var applicationEndDate = _app2Lead.DataAccess.App2BonusPromos.FindAll(x => x.ApplicationEndDate);
if (policyApplicationDate > applicationBeginDates && policyApplicationDate < applicationEndDate && policyEffectiveDate == effectiveDates)
{
return true;
}
return false;

Thanks to the comment by #stephen.vakil and the suggestion from #JonSkeet i was able to use this to get what i needed. Thank you for your help!.
var duh = _app2Lead.DataAccess.App2BonusPromos.FindByExp(
x =>
x.ApplicationBeginDate < policyApplicationDate && x.ApplicationEndDate > policyApplicationDate &&
x.EffectiveDate == policyEffectiveDate).Any();
if (duh)
{
return true;
}
return false;

Iterate through dbset to get next n rows

I'm trying to find out what's a good way to continually iterate through a dbset over various function calls, looping once at the end.
Essentially I've got a bunch of Ads in the database, and I want to getNext(count) on the dbset.Ads
Here's an example
ID Text Other Columns...
1 Ad1 ...
2 Ad2 ...
3 Ad3 ...
4 Ad4 ...
5 Ad5 ...
6 Ad6 ...
Let's say that in my View, I determine I need 2 ads to display for User 1. I want to return Ads 1-2. User 2 then requests 3 ads. I want it to return 3-5. User 3 needs 2 ads, and the function should return ads 6 and 1, looping back to the beginning.
Here's my code that I've been working with (it's in class AdsManager):
Ad NextAd = db.Ads.First();
public IEnumerable<Ad> getAds(count)
{
var output = new List<Ad>();
IEnumerable<Ad> Ads = db.Ads.OrderBy(x=>x.Id).SkipWhile(x=>x.Id != NextAd.Id);
output.AddRange(Ads);
//If we're at the end, handle that case
if(output.Count != count)
{
NextAd = db.Ads.First();
output.AddRange(getAds(count - output.Count));
}
NextAd = output[count-1];
return output;
}
The problem is that the function call IEnumerable<Ad> Ads = db.Ads.OrderBy(x=>x.Id).SkipWhile(x=>x.Id != NextAd.Id); throws an error on AddRange(Ads):
LINQ to Entities does not recognize the method 'System.Linq.IQueryable'1[Domain.Entities.Ad] SkipWhile[Ad](System.Linq.IQueryable'1[Domain.Entities.Ad], System.Linq.Expressions.Expression'1[System.Func`2[Domain.Entities.Ad,System.Boolean]])' method, and this method cannot be translated into a store expression.
I originally had loaded the entire dbset into a Queue, and did enqueue/dequeue, but that would not updat when a change was made to the database. I got the idea for this algorithm based on Get the next and previous sql row by Id and Name, EF?
What call should I be making to the database to get what I want?
UPDATE: Here's the working Code:
public IEnumerable<Ad> getAds(int count)
{
List<Ad> output = new List<Ad>();
output.AddRange(db.Ads.OrderBy(x => x.Id).Where(x => x.Id >= NextAd.Id).Take(count + 1));
if(output.Count != count+1)
{
NextAd = db.Ads.First();
output.AddRange(db.Ads.OrderBy(x => x.Id).Where(x => x.Id >= NextAd.Id).Take(count - output.Count+1));
}
NextAd = output[count];
output.RemoveAt(count);
return output;
}

SkipWhile is not supported in the EF; it can't be translated into SQL code. EF basically works on sets and not sequences (I hope that sentence makes sense).
A workaround is to simply use Where, e.g.:
IEnumerable<Ad> Ads = db.Ads.OrderBy(x=>x.Id).Where(x=>x.Id >= NextAd.Id);

Maybe you can simplify this into something like this:
Ad NextAd = db.Ads.First();
public IQueryable<Ad> getAds(count)
{
var firstTake = db.Ads
.OrderBy(x => x.Id)
.Where(x => x.Id >= NextAd.Id);
var secondTake = db.Ads
.OrderBy(x => x.Id)
.Take(count - result.Count());
var result = firstTake.Concat(secondTake);
NextAd = result.LastOrDefault()
return result;
}
Sorry, haven't tested this, but it should work.

How do i get this conditional include projection to instantiate the "premis" entity-object property of the ref_calendar_premis entity-object?

I'm working on my first EntityFramework-solution, and I'm stumped.
When I'm iterating through the resulting "calendar"-entity objects, the property calendar.ref_calendar_premisis.premisis is un-instantiated (null). I understand why, I deactivated lazy-loading because i wanted to control the conditional "includes" in a syntax-tree projection like so:
private List<CalendarBlockDTO> FillUserCalendar(DateTime start, DateTime end, int uid, bool everything)
{
var result = new List<CalendarBlockDTO>();
using (var db = new ViggoEntities())
{
//We need to disable lazyloading to use the "expression tree" syntax
db.Configuration.LazyLoadingEnabled = false;
//flip our "everything" so we can compare it to our "special" column
int specialScope = Convert.ToInt32(!everything);
//Build a query "Projection" with "expression tree" syntax
var query = from c in db.calendars
select new
{
calendarEntry = c,
createdByUser = c.MadeByUser,
premisesBookings = c.ref_calendar_premises.Where
(
rcp => rcp.deleted == 0 &&
(
//started before the start-parameter AND ended after start-parameter
(rcp.timestart < start && rcp.timeend > start) ||
//OR startet before the end-parameter AND ended after the end-parameter
(rcp.timestart < end && rcp.timeend > end) ||
//OR startet before the start-parameter AND ended after the end-paremeter
(rcp.timestart < start && rcp.timeend > end) ||
//OR startet after the start-parameter AND ended before the end-parameter
(rcp.timestart > start && rcp.timeend < end)
)
),
attendingGroups = c.ref_groups_calendar.Where
(
rug => rug.deleted == 0
),
groups = c.ref_groups_calendar.Select( rgc => rgc.usergroup ),
////Assignments not implemented yet
////assignments = c.
schedules = c.ref_calendar_schedule.Where
(
sch => sch.deleted == 0
)
};
var calEntries =
query.ToArray().Select(c => c.calendarEntry).
Where(
//If only special requested, show only special
c => c.special >= specialScope &&
//If not "MadeInInfo", show for creator as well
(c.madeininfo==0 && c.made_by == uid) ||
//Else, show to involved users
(c.ref_groups_calendar.Any(rgc => rgc.usergroup.ref_users_groups.Any(rug => rug.userid == uid)))
);
foreach (var calendar in calEntries)
{
//I WANT THIS TO NOT THROW AN EXCEPTION, PREMIS SHOULD NOT BE NULL
if (calendar.name == "Dinner with Allan" && calendar.ref_calendar_premises.Any(rcp => rcp.premis == null))
throw new Exception("Premis not instantiated!");
result.AddRange(CalendarToCalendarBlockDTOs(calendar));
}
}
return result;
}
I tried adding something like:
...
room = c.ref_calendar_premises.Select(r => r.premis),
...
... But to no avail. A room has been booked for the "Dinner with Allan" event in our test data, but i cant seem to get it to load the premis-entyties.
I have no previous experience with EntityFramework, LINQ to SQL or any other ORM's, so I might be missing something plainly obvious.
Any suggestions?

I twisted the arm of our DB-responsible, and turns out premisisid is null for all rows because of a conversion error (i was looking at testdata documentation, not the actual data).
So thanks for the input and your time, I'll just show myself out 0_0

Why does my Linq Where clause produce more results instead of less?

I just had the weirdest debug experience in a very long time. It's a bit embarassing to admit, but it lead me to be believe that my Linq query produces MORE results when adding an additional Where clause.
I know it's not possible, so I've refactored my offending function plus the unit test belonging to it into this:
[Test]
public void LoadUserBySearchString()
{
//Setup
var AllUsers = new List<User>
{
new User
{
FirstName = "Luke",
LastName = "Skywalker",
Email = "luke#jedinet.org"
},
new User
{
FirstName = "Leia",
LastName = "Skywalker",
Email = "faeryprincess#winxmail.com"
}
};
//Execution
List<User> SearchResults = LoadUserBySearchString("princess", AllUsers.AsQueryable());
List<User> SearchResults2 = LoadUserBySearchString("princess Skywalker", AllUsers.AsQueryable());
//Assertion
Assert.AreEqual(1, SearchResults.Count); //test passed!
Assert.AreEqual(1, SearchResults2.Count); //test failed! got 2 instead of 1 User???
}
//search CustID, fname, lname, email for substring(s)
public List<User> LoadUserBySearchString(string SearchString, IQueryable<User> AllUsers)
{
IQueryable<User> Result = AllUsers;
//split into substrings and apply each substring as additional search criterium
foreach (string SubString in Regex.Split(SearchString, " "))
{
int SubStringAsInteger = -1;
if (SubString.IsInteger())
{
SubStringAsInteger = Convert.ToInt32(SubString);
}
if (SubString != null && SubString.Length > 0)
{
Result = Result.Where(c => (c.FirstName.Contains(SubString)
|| c.LastName.Contains(SubString)
|| c.Email.Contains(SubString)
|| (c.ID == SubStringAsInteger)
));
}
}
return Result.ToList();
}
I have debugged the LoadUserBySearchString function and asserted that the second call to the function actually produces a linq query with two where clauses instead of one. So it seems that the additional where clause is increasing the amount of results.
What's even more weird, the LoadUserBySearchString function works great when I test it by hand (with real users from the database). It only shows this weird behavior when running the unit test.
I guess I just need some sleep (or even an extended vacation). If anyone could please help me shed some light on this, I could go stop questioning my sanity and go back to work.
Thanks,
Adrian
Edit (to clarify on several responses I go so far): I know it looks like it is the or clause, but unfortuantely it is not that simple. LoadUserBySearchString splits the search string into several strings and attaches a Where clause for each of them. "Skywalker" matches both luke and Leia, but "princess" only matches Leia.
This is the Linq query for the search string "princess":
+ Result {System.Collections.Generic.List`1[TestProject.Models.User].Where(c => (((c.FirstName.Contains(value(TestProject.Controllers.SearchController+<>c__DisplayClass1).SubString) || c.LastName.Contains(value(TestProject.Controllers.SearchController+<>c__DisplayClass1).SubString)) || c.Email.Contains(value(TestProject.Controllers.SearchController+<>c__DisplayClass1).SubString)) || (c.ID = value(TestProject.Controllers.SearchController+<>c__DisplayClass3).SubStringAsInteger)))} System.Linq.IQueryable<TestProject.Models.User> {System.Linq.EnumerableQuery<TestProject.Models.User>}
And this is the Linq clause for the search string "princess Skywalker"
+ Result {System.Collections.Generic.List`1[TestProject.Models.User].Where(c => (((c.FirstName.Contains(value(TestProject.Controllers.SearchController+<>c__DisplayClass1).SubString) || c.LastName.Contains(value(TestProject.Controllers.SearchController+<>c__DisplayClass1).SubString)) || c.Email.Contains(value(TestProject.Controllers.SearchController+<>c__DisplayClass1).SubString)) || (c.ID = value(TestProject.Controllers.SearchController+<>c__DisplayClass3).SubStringAsInteger))).Where(c => (((c.FirstName.Contains(value(TestProject.Controllers.SearchController+<>c__DisplayClass1).SubString) || c.LastName.Contains(value(TestProject.Controllers.SearchController+<>c__DisplayClass1).SubString)) || c.Email.Contains(value(TestProject.Controllers.SearchController+<>c__DisplayClass1).SubString)) || (c.ID = value(TestProject.Controllers.SearchController+<>c__DisplayClass3).SubStringAsInteger)))} System.Linq.IQueryable<TestProject.Models.User> {System.Linq.EnumerableQuery<TestProject.Models.User>}
Same as above, just with one additional where clause.

This is a nice little gotcha.
What is happening is that, because of anonymous methods, and deferred execution, you're actually not filtering on "princess". Instead, you're building a filter that will filter on the contents of the subString variable.
But, you then change this variable, and build another filter, which again uses the same variable.
Basically, this is what you will execute, in short form:
Where(...contains(SubString)).Where(...contains(SubString))
so, you're actually only filtering on the last word, which exists in both, simply because by the time these filters are actually applied, there is only one SubString value left, the last one.
If you change the code so that you capture the SubString variables inside the scope of the loop, it'll work:
if (SubString != null && SubString.Length > 0)
{
String captured = SubString;
Int32 capturedId = SubStringAsInteger;
Result = Result.Where(c => (c.FirstName.Contains(captured)
|| c.LastName.Contains(captured)
|| c.Email.Contains(captured)
|| (c.ID == capturedId)
));
}

Your algorithm amounts to "select records which match any of the words in the search string".
This is because of deferred execution. The query is not actually performed until you call the .ToList(). If you move the .ToList() inside the loop, you'll get the behaviour you want.