I want to use Take() in projection but I dont want to produce N+1 query, in addition without Take() in projection I'm facing with performance issue.
I used Take() with EF6 but I faced with N+1 issue on EF Core.
example projection:
source.Select(post => new PostProject
{
PostDisableCoins = post.PostDisableCoins
.OrderBy(x=>x.CoinAmount)
.Take(3)
.ToList(),
WarStartTime = post.WarStartTime,
WarEndTime = post.WarEndTime,
WarWinner = post.WarWinner,
WarDeclarer = post.WarDeclarer
});
I want to have Take(3) whithout N+1, any suggestion?!?
This is EF Core 2.1 implementation defect. Following is the workaround, but use it only if you really have performance issues, because it requires breaking the navigation property join abstraction and using manual join, which I always say should not be used with EF (Core). Also might not work if used to project more than one collection, or as part of a more complex query.
It requires replacing the usage of the collection navigation property post.PostDisableCoins with SelectMany using lateral join and hiding the OrderBy / Take operators (update with proper types and PK/FK names):
var postDisableCoinsQuery = source.SelectMany(p =>
db.Set<PostDisableCoin>()
.Where(c => c.PostId == p.Id)
.OrderByDescending(c => c.CoinAmount)
.Take(3)
);
Then do GroupJoin to it:
var query =
from p in source
join c in postDisableCoinsQuery on p.Id equals c.PostId into postDisableCoins
select new PostProject
{
PostDisableCoins = postDisableCoins.ToList(),
WarStartTime = p.WarStartTime,
WarEndTime = p.WarEndTime,
WarWinner = p.WarWinner,
WarDeclarer = post.WarDeclarer
};
When executed, the above will produce the desired result with single SQL query.
Please note the doc regarding EF core 2.1 new features:
We have improved our query translation to avoid executing "N + 1" SQL
queries in many common scenarios in which the usage of a navigation
property in the projection leads to joining data from the root query
with data from a correlated subquery. The optimization requires
buffering the results from the subquery, and we require that you
modify the query to opt-in the new behavior.
for example:
var query = context.Customers.Select(
c => c.Orders.Where(o => o.Amount > 100).Select(o => o.Amount).ToList());
Notice where the .ToList() is included.
You need to modify your projection query accordingly, in order to enable the optimization feature.
In your case it might be:
source.Select(post => new PostProject
{
PostDisableCoins = post.PostDisableCoins
.Select(x => x.OrderBy(x=>x.CoinAmount))
.Select(x => x)
.Take(3)
.ToList(),
WarStartTime = post.WarStartTime,
WarEndTime = post.WarEndTime,
WarWinner = post.WarWinner,
WarDeclarer = post.WarDeclarer
});
Related
Why does one method work but not the other, when it seems they are both doing the same thing, i.e. constructing an entity. My question then, is there a way to construct the entity in a L2E query instead of having to use just Linq or indeed both?
This works fine...
var queryToList = (from ac in ctx.AuthorisationChecks
where wedNumbers.Contains(ac.WedNo)
orderby ac.WedNo, ac.ExpAuthDate, ac.ActAuthDate
select new AuthorisationCheck
{
Blah = ac.Blah
}).ToList();
model.AuthorisationChecks = queryToList.Select(x => new AuthorisationCheck
{
Blah = x.Blah
}).ToList();
However, if i change...
var queryToList
to
model.AuthorisationChecks queryToList // Of type List<AuthorisationCheck>
i get the error in the Title...
The entity or complex type 'Model.AuthorisationCheck' cannot be constructed in a LINQ to Entities query.
EDIT:
In the model it is simply, nothing fancy here.
public List<AuthorisationCheck> AuthorisationChecks { get; set; }
EDIT2:
Tidied this up a little to be (which works fine)...
model.AuthorisationChecks = (from ac in ctx.AuthorisationChecks
where wedNumbers.Contains(ac.WedNo)
orderby ac.WedNo, ac.ExpAuthDate, ac.ActAuthDate
select ac).ToList()
.Select(x => new AuthorisationCheck
{
Blah = x.Blah
}).ToList();
EDIT2: My Solution
I wasn't happy with the anonymous type method and so went ahead and created a simple model containing only the properties I required to be used in the viewmodel.
Changed the type of model.AuthorisationChecks
from
List<AuthorisationCheck> // List of Entities
to
List<AuthorisationCheckModel> // List of models
which allows the following code to work, and without profiling it seems a lot quicker than using an anonymous type (and of course I don't cast to a list twice!).
model.AuthorisationChecks = (from ac in ctx.AuthorisationChecks
where wedNumbers.Contains(ac.WedNo)
orderby ac.WedNo, ac.ExpAuthDate, ac.ActAuthDate
select new AuthorisationCheckModel
{
Blah = x.Blah
}).ToList();
P.S. I was once warned by a coworker (who used to work for Microsoft) that it isn't a good idea to directly use Entities in this manner, maybe this was one of the reasons he was thinking of, I've also noticed some odd behavior using Entities directly in other cases (mainly corruptions).
Note: I generally use lambda expression instead of Fluent API, but the root issue should be the same.
I have historically noticed LINQ is unable to use C# classes for Select statements if the original datasource (i.e. ctx for you) is accessed by translating your query into SQL.
In other words, there are issues when getting something from the database and casting it to a custom class in the same chain.
LINQ is smart enough that it actually doesn't immediately execute your chained calls. It simply internally builds a query, and when you actually access your results (i.e. retrieve the value from memory), it executes the query.
I assume this is also the reason why you are faced with this error, because LINQ translates everything (including the Select) to SQL, and fails because there's no SQL-way to express it. In short, LINQ can't do a built query half-SQL, half-code. It's either all in SQL, or all in code.
You can usually confirm that this is the case by first making a List<> of the database table, then run the exact same query on it.
var myTable = db.AuthorizationCheck.ToList();
var myResult = myTable. //query here
Note: That is not a solution!
Taking the entire table in memory is an overly intensive way to work around this. It just proves the point that the problem isn't encountered if the datasource is in memory, but the error does occur if it's in a database.
There are ways I've fixed this, although I've never found a uniform way to approach this problem (generally depends on the opinion of my code reviewer, whether he likes the fix or not)
Using anonymous types, you can select what you want, and later cast it to the correct class. You can use the exact same fields, making a later cast easier to understand.
//Simpler query for clarity's sake
var myAnonymousResult = ctx.AuthorizationChecks
.Where(x => x.IsActive)
.Select(x => new { Name = x.Name, IsActive = x.IsActive })
.ToList();
var myCastResult = myAnonymousResult.Select(x => new Check() { Name = x.Name, IsActive = x.IsActive }).ToList();
If you use lambda expressions instead of the fluent API, you can call .ToList() after applying the filters but before calling the .Select() method. This ensures the current query will be executed, retrieved from the database, and put into an actual List<> in memory. At that point, you can call the .Select() statement without running into the same problem.
//Simpler query for clarity's sake
var myCastResult = ctx.AuthorizationChecks
.Where(x => x.IsActive)
.ToList()
.Select(x => new Check() { Name = x.Name, IsActive = x.IsActive });
Unfortunately though, my experience with this problem is anecdotal. I've never been able to officially confirm my suspicions as to the root cause of this issue; but the workarounds I mentioned should work as I've applied them numerous times in the past.
If anyone has an explanation of the root cause, I'd be very interested in hearing it!
If this query works fine:
var queryToList = (from ac in ctx.AuthorisationChecks
where wedNumbers.Contains(ac.WedNo)
orderby ac.WedNo, ac.ExpAuthDate, ac.ActAuthDate
select new AuthorisationCheck
{
Blah = ac.Blah
}).ToList();
then this should also work:
model.AuthorisationChecks = (from ac in ctx.AuthorisationChecks
where wedNumbers.Contains(ac.WedNo)
orderby ac.WedNo, ac.ExpAuthDate, ac.ActAuthDate
select new AuthorisationCheck
{
Blah = ac.Blah
}).ToList();
and in your first case you don't need to project again, you can directly assign it to model propeerty:
model.AuthorisationChecks = queryToList;
UPDATE:
As it is Linq To Entities,you have to do something like this using anonymous type:
var queryToList = (from ac in ctx.AuthorisationChecks
where wedNumbers.Contains(ac.WedNo)
orderby ac.WedNo, ac.ExpAuthDate, ac.ActAuthDate
select new
{
Blah = ac.Blah
}).ToList();
and then:
model.AuthorisationChecks = queryToList.Select(x => new AuthorisationCheck
{
Blah = x.Blah
}).ToList();
I have a database that contains 3 tables:
Phones
PhoneListings
PhoneConditions
PhoneListings has a FK from the Phones table(PhoneID), and a FK from the Phone Conditions table(conditionID)
I am working on a function that adds a Phone Listing to the user's cart, and returns all of the necessary information for the user. The phone make and model are contained in the PHONES table, and the details about the Condition are contained in the PhoneConditions table.
Currently I am using 3 queries to obtain all the neccesary information. Is there a way to combine all of this into one query?
public ActionResult phoneAdd(int listingID, int qty)
{
ShoppingBasket myBasket = new ShoppingBasket();
string BasketID = myBasket.GetBasketID(this.HttpContext);
var PhoneListingQuery = (from x in myDB.phoneListings
where x.phonelistingID == listingID
select x).Single();
var PhoneCondition = myDB.phoneConditions
.Where(x => x.conditionID == PhoneListingQuery.phonelistingID).Single();
var PhoneDataQuery = (from ph in myDB.Phones
where ph.PhoneID == PhoneListingQuery.phonePageID
select ph).SingleOrDefault();
}
You could project the result into an anonymous class, or a Tuple, or even a custom shaped entity in a single line, however the overall database performance might not be any better:
var phoneObjects = myDB.phoneListings
.Where(pl => pl.phonelistingID == listingID)
.Select(pl => new
{
PhoneListingQuery = pl,
PhoneCondition = myDB.phoneConditions
.Single(pc => pc.conditionID == pl.phonelistingID),
PhoneDataQuery = myDB.Phones
.SingleOrDefault(ph => ph.PhoneID == pl.phonePageID)
})
.Single();
// Access phoneObjects.PhoneListingQuery / PhoneCondition / PhoneDataQuery as needed
There are also slightly more compact overloads of the LINQ Single and SingleOrDefault extensions which take a predicate as a parameter, which will help reduce the code slightly.
Edit
As an alternative to multiple retrievals from the ORM DbContext, or doing explicit manual Joins, if you set up navigation relationships between entities in your model via the navigable join keys (usually the Foreign Keys in the underlying tables), you can specify the depth of fetch with an eager load, using Include:
var phoneListingWithAssociations = myDB.phoneListings
.Include(pl => pl.PhoneConditions)
.Include(pl => pl.Phones)
.Single(pl => pl.phonelistingID == listingID);
Which will return the entity graph in phoneListingWithAssociations
(Assuming foreign keys PhoneListing.phonePageID => Phones.phoneId and
PhoneCondition.conditionID => PhoneListing.phonelistingID)
You should be able to pull it all in one query with join, I think.
But as pointed out you might not achieve alot of speed from this, as you are just picking the first match and then moving on, not really doing any inner comparisons.
If you know there exist atleast one data point in each table then you might aswell pull all at the same time. if not then waiting with the "sub queries" is nice as done by StuartLC.
var Phone = (from a in myDB.phoneListings
join b in myDB.phoneConditions on a.phonelistingID equals b.conditionID
join c in ph in myDB.Phones on a.phonePageID equals c.PhoneID
where
a.phonelistingID == listingID
select new {
Listing = a,
Condition = b,
Data = c
}).FirstOrDefault();
FirstOrDefault because single throws error if there exists more than one element.
I'm executing following linq query in ASP.NET MVC project:
Version baseVersion = new Version("1.0.0.0");
var v = (from a in db.table where new Version(a.version) > baseVersion select new { a.id, a.version, a.name}).ToList();
While executing this I'm getting this error:
Only parameterless constructors and initializers are supported in LINQ to Entities
When I removed version check and used this query:
var v = (from a in db.table select new { a.id, a.version, a.name}).ToList();
it executed successfully. How can I make the linq query with version condition to work?
If you want to use the Version class to do the comparison, then I see no other option than importing the whole table into memory, and do the comparison there. You can use IQueryable<T>.AsEnumerable() for to specify you want to do anything after in memory.
Version baseVersion = new Version("1.0.0.0");
var v = db.Table
.Select(x => new { x.id, x.version, x.name })
.AsEnumerable()
.Where(x => new Version(x.version) > baseVersion)
.ToList();
Note: I've moved the .Select(...) to immediately after the db.Table to only select the columns you need, which minimizes the memory footprint.
The alternative is that you recreate the Version comparison in SQL. But then you are reinventing the wheel.
I have a sql statement like this:
DECLARE #destinations table(destinationId int)
INSERT INTO #destinations
VALUES (414),(416)
SELECT *
FROM GroupOrder grp (NOLOCK)
JOIN DestinationGroupItem destItem (NOLOCK)
ON destItem.GroupOrderId = grp.GroupOrderId
JOIN #destinations dests
ON destItem.DestinationId = dests.destinationId
WHERE OrderId = 5662
I am using entity framework and I am having a hard time getting this query into Linq. (The only reason I wrote the query above was to help me conceptualize what I was looking for.)
I have an IQueryable of GroupOrder entities and a List of integers that are my destinations.
After looking at this I realize that I can probably just do two joins (like my SQL query) and get to what I want.
But it seems a bit odd to do that because a GroupOrder object already has a list of DestinationGroupItem objects on it.
I am a bit confused how to use the Navigation property on the GroupOrder when I have an IQueryable listing of GroupOrders.
Also, if possible, I would like to do this in one trip to the database. (I think I could do a few foreach loops to get this done, but it would not be as efficient as a single IQueryable run to the database.)
NOTE: I prefer fluent linq syntax over the query linq syntax. But beggars can't be choosers so I will take whatever I can get.
If you already have the DestinationGroupItem as a Navigation-property, then you already have your SQL-JOIN equivalent - example. Load the related entities with Include. Use List's Contains extension method to see if the desired DestinationId(s) is(are) hit:
var destinations = new List<int> { 414, 416 };
var query = from order in GroupOrder.Include(o => o.DestinationGroupItem) // this is the join via the navigation property
where order.OrderId == 5662 && destinations.Contain(order.DestinationGroupItem.DestinationId)
select order;
// OR
var query = dataContext.GroupOrder
.Include(o => o.DestinationGroupItem)
.Where(order => order.OrderId == 5662 && destinations.Contain(order.DestinationGroupItem.DestinationId));
I have an Entity Framework entity Provider, with a list of rating votes for that provider. My current queries look something like this:
int previousVote = provider.ProviderRankings.FirstOrDefault(r => r.UserId == CurrUserId);
double averageVote = provider.ProviderRankings.Average(r => r.Rating);
int totalVotes = provider.ProviderRankings.Count();
This seems functionally correct. However, I believe this will result in three additional trips to the database. Is there anyway to have these requests combined into a single query such that only one SQL query will be sent, and all results can be returned with only one additional trip to the server?
You could combine the two aggregates fairly easily using a Group By:
Multiple SQL aggregate functions in a single Linq-to-Entities query
I am pretty sure the FirstOrDefault will work if you choose a suitably vague key for the grouping (for example key = 0) i.e:
from t in ProviderRankings
group t by key = 0
into g
select new {
previousVote = g.FirstOrDefault(r => r.UserId == CurrUserId),
totalVotes = g.Count(),
averageVote = g.Average(x => x.Rating)
}