C# on .NET 3.5 Framework
So, I'm trying to make an efficient query to find all emails to purge from my system. What should be purged are emails that have been attempted (Datetime column LastAttempted is not null) and the # of days to retain (int column RetentionDays) has elapsed. This works, but I know it's pulling everything back and filtering in memory.
This is what I have right now
var emails = dbContext.Emails
.Where(x => x.LastAttempted.HasValue == true)
.ToList()
.Where(x => ((DateTime)x.LastAttempted).AddDays(x.RetentionDays) <= DateTime.Now);
How can I update this so it only pull the records I'm caring about from SQL Server?
Use EntityFunctions.AddDays to add days to a DateTime in an EF query.
var emails = dbContext.Emails
.Where(x => x.LastAttempted.HasValue &&
EntityFunctions.AddDays(x.LastAttempted, x.RetentionDays)
<= DateTime.Now);
This part of query:
dbContext.Emails
.Where(x => x.LastAttempted.HasValue == true)
.ToList();
will be executed in the sql server and an in memory collection will be returned, when ToList is called.
Then the last part of your query:
.Where(x =>
((DateTime)x.LastAttempted).AddDays(x.RetentionDays) <= DateTime.Now);
will be executed in the items of the above mentioned in memory collection.
So the problem relies on the last part of your query. You could avoid this using the method AddDays of EntityFunctions and avoid making the second filtering in the in memory collection. This can be done like below:
dbContext.Emails
.Where(x => x.LastAttempted.HasValue == true &&
EntityFunctions.AddDays(x.LastAttempted, x.RetentionDays)
<= DateTime.Now);
Move ToList() at the end of the query
var now = DateTime.Now;
var emails = dbContext.Emails
.Where(x => x.LastAttempted.HasValue &&
EntityFunctions.AddDays((DateTime)x.LastAttempted, x.RetentionDays) <= now).
ToList();
You can always query the database directly, like this:
var items = dbContext.Database.SqlQuery<Email>("select * from Emails where LastAttempted Is Not Null And LastAttempted + RetentionDaya >= GetDate()", new SqlParameter[0]);
NOTE: I made some guesses with respect to your entity, table, and column names.
Related
I am attempting to get a list of Order Ids from a EF linq query. The sql query is returning back quickly but I think the EF framework is trying to create the full entity. I only want the ID of the order. It seems that it creates the whole entity and then it parses it out to only the id. Which seems to be a complete waste of resources.
Orders are a complex object that includes lots of child entitys. I dont need anything but the Ids of the orders in the list. Orders are organized into OrderCollection which is a many to many relationship.
The basic query in English is get the order ids in the specified order collection and have a cart date newer then the date specified and only send the specified page (skip and take).
example:
_repo.Orders.Where(o => o.OrderCollection.Any(r => r.Id == RoutingRuleId)).ToList()
.Where(o => o.OrderDate >= StartDateTime)
.OrderBy(x => x.OrderDate )
.Skip(RecordsToSkipCount)
.Take(BatchSize).Select(x => new { x.Id }).ToArray();
The sql runs in just 102ms for this in debug mode. But afterwards I see the memory go to up 4GB before failing. The batchsize is only 100. Its like it grabbing everything.
I tried moving the select around but that failed also or gave syntax errors or poor performance in running the sql (SQL taking 16 seconds).
Example
_repo.Orders.Select(x => new { x.Id, x.OrderCollection, x.OrderDate})
.Where(o => o.OrderCollection.Any(r => r.Id == RoutingRuleId)).ToList()
.Where(o => o.OrderDate >= StartDateTime)
.OrderBy(x => x.OrderDate )
.Skip(RecordsToSkipCount)
.Take(BatchSize).Select(x => new { x.Id }).ToArray();
The database has millions of records.
What you have is roughly;
List<Order> list = _repo.Orders
.Where(o => o.OrderCollection.Any(r => r.Id == RoutingRuleId))
.ToList();
list.Where(o => o.OrderDate >= StartDateTime)
.OrderBy(x => x.OrderDate )
.Skip(RecordsToSkipCount)
.Take(BatchSize)
.Select(x => new { x.Id })
.ToArray();
That first .ToList is forcing EF Core to load every order with a matching routing rule into memory. The rest of the expression is then using IEnumerable extension methods to process those results.
I think you want to rearrange that to;
IQueryable<Order> query = _repo.Orders
.Where(o => o.OrderCollection.Any(r => r.Id == RoutingRuleId)
&& o.OrderDate >= StartDateTime)
.OrderBy(x => x.OrderDate )
.Skip(RecordsToSkipCount)
.Take(BatchSize)
.Select(x => new { x.Id });
query.ToArray();
Creating an IQueryable doesn't trigger EF Core to execute any SQL. An IQueryable is just a description of the query you would like to run. Then it's the .ToArray method that will finally cause EF Core to compile and execute an sql statement.
I am currently building a custom MVC widget for a Sitefinity (v9.2) project.
As part of this widget there is a query to the database to retrieve a collection of a dynamic module type (articles). I am trying to get article types that contain all the labels in my Labels.PublicLabels guid list.
My current query is:
var collection = dynamicModuleManager.GetDataItems(articleType)
.Where(a => a.Status == Telerik.Sitefinity.GenericContent.Model.ContentLifecycleStatus.Live
&& a.Visible == true
&& Labels.PublicLabels.All(l => a.GetValue<IList<Guid>>("Public").Contains(l)));
At runtime I get an exception mentioning that 'server side not implemented'.
Could this be a limitation of OpenAccess?
I have tried a wide range of LINQ to SQL query combinations which have been successful, I am struggling to understand the problem here.
Any ideas would be greatly appreciated!
UPDATE:
I have tried a few variations on the same query such as:
var collection = dynamicModuleManager.GetDataItems(articleType)
.Where(a => a.Status == Telerik.Sitefinity.GenericContent.Model.ContentLifecycleStatus.Live
&& a.Visible == true
&& Labels.PublicLabels.Any(l => a.GetValue<IList<Guid>>("Public").Contains(l)));
No result, I still recieve the following exception message:
Execution of 'System.Linq.Enumerable:Any(IEnumerable1,Func2)' on the database server side currently not implemented.
Any further advice would be greatly appreciated, for now I will assume this is to do with OpenAccess limitations around LINQ to SQL.
Managed to solve this issue. Definitely down to Open Access LINQ to SQL limitations. What I did was add an additional where clause after filtering the collection down as much as I needed to. Here is the complete query:
var collection = dynamicModuleManager.GetDataItems(articleType).Where(a => a.Status == ContentLifecycleStatus.Live && a.Visible == true)
.OrderByDescending(a => a.PublicationDate)
.Distinct()
.Where(a => new HashSet<Guid>(a.GetValue<IList<Guid>>("Public")).IsSupersetOf(Labels.PublicLabels));
I will be reporting this to Sitefinity and if they manage to put a proper fix in place I will report back.
Thank you everyone for your support!
I had the same problem before, this way worked for me
var listIds = Labels.PublicLabels;
var collection = dynamicModuleManager.GetDataItems(articleType)
.Where(a => a.Status == ContentLifecycleStatus.Live && a.Visible == true)
.Where(a => a.GetValue<TrackedList<Guid>>("Public").Where(al => listIds.Contains(al)).Any());
I didn't compile that, let me know if it will not work for you
I have a query that looks like this
var query = db.Customer
.Include(c => c.Address)
.Where(c => c.Address.Id > 10)
.ToList();
when i do this instead
var query = db.Customer
.Where(c => c.Address.Id > 10)
.ToList();
db.Address
.Where(a => a.Id > 10)
.Load();
I get the same result as far as I see.
My question is: is there any difference between what these two queries return and is one preferred over the other?
var query = db.Customer
.Include(c => c.Address)
.Where(c => c.Address.Id > 10)
.ToList();
On above query where it brings all the related data using single database trip.
var query = db.Customer
.Where(c => c.Address.Id > 10)
.ToList();
db.Address
.Where(a => a.Id > 10)
.Load();
Here it uses 2 database trips to bring the data.
Load :
There are several scenarios where you may want to load entities from
the database into the context without immediately doing anything with
those entities. A good example of this is loading entities for data
binding as described in Local Data. One common way to do this is to
write a LINQ query and then call ToList on it, only to immediately
discard the created list. The Load extension method works just like
ToList except that it avoids the creation of the list altogether.
Note : We cannot say which one is better.Most of the time we use eager loading method (Include).It is nice and simple.But sometimes it is slow.So you need to decide which one to use according to your data size and etc.
In RavenDB, I would like to query the database using a date field. How can I write a LINQ query to query the RavenDB that involves date comparison..
Session.Query<Movie>()
.Where(x => x.Status == "New" && x.ReleaseDate > DateTime.Parse("04/03/2012 00:00:00"))
.Dump();
is returning zero records inm LINQ pad. Something is not right the way I wrote the query.. thanks for any help.
Instead of DateTime.Parse() try instantiating a new DateTime.
Session.Query<Movie>()
.Where(x => x.Status == "New" && x.ReleaseDate > new DateTime(2012, 4, 3))
.Dump();
I need to order the articles stored in a database by descending publication date and then take the first 20 records after the article with Id == 100.
This is what I would like to do with Linq:
IQueryable<Article> articles =
db.Articles
.OrderByDescending(a => a.PublicationDate)
.SkipWhile(a => a.Id != 100)
.Take(20);
However, this generates a NotSupportedException because SkipWhile is not supported in Linq to Sql (see here).
A possible solution is to execute the query and then apply SkipWhile using Linq to Object:
IEnumerable<ArticleDescriptor> articles =
db.Articles
.OrderByDescending(a => a.PublicationDate)
.ToList()
.SkipWhile(a => a.Article.Id != 100)
.Take(20);
But this means I need to load the whole ordered list into memory first and then take 20 articles after the one with Id == 100.
Is there a way to avoid this huge memory consumption?
More in general, what is the best way to achieve this in SQL?
If, as I'm guessing from the column name, PublicationDate doesn't change, you can do this in two separate queries:
Establish the PublicationDate of the Article with Id == 100
Retrieve the 20 articles from that date onwards
Something like:
var thresholdDate = db.Articles.Single(a => a.Id == 100).PublicationDate;
var articles =
db.Articles
.Where(a => a.PublicationDate <= thresholdDate)
.OrderByDescending(a => a.PublicationDate)
.Take(20);
It might even be that LINQ to SQL can translate this:
var articles =
db.Articles
.Where(a => a.PublicationDate
<= db.Articles.Single(aa => aa.Id == 100).PublicationDate)
.OrderByDescending(a => a.PublicationDate)
.Take(20);
but that may be too complex for it. Try it and see.
You can try like this
var articles =
db.Articles
.Where(a => a.PublicationDate < db.Articles
.Where(aa => aa.Id==100)
.Select(aa => aa.PublicationDate)
.SingleOrDefault())
.OrderByDescending(a => a.PublicationDate)
.Take(20);
Isnt the solution to just add a where statement?
IQueryable<Article> articles = db.Articles.Where(a => a.id != 100).OrderByDescending(a => a.PublicationDate).Take(20);