Develop Reference

With Entity Framework how to create nested objects without one massive query resultset or hundreds of small queries?

I'm using EF to populate objects which I then interact with in my business layer code. The objects have several levels but let's first simplify it to a typical master-detail example with Order and OrderLine.
Let's say I need to retrieve 50 orders each of which has about 100 order lines, and I need all that data. What's the most efficient way of doing this in EF?
If I do it like this:
var orders = context.Orders.Where(o => o.Status == "Whatever")
.Include(order => order.OrderLines)
.ToList();
Then it'll run a single query that pulls back 50x100 rows, then it cleverly assembles that into 50 Order objects, each with their own OrderLine objects.
If instead I don't do the Include() and I later iterate over each Order and its OrderLines, then it'll issue 50 separate queries to get the OrderLine data, i.e. one for each Order.
So far the .Include() seems great: I pull back a bit more data than needed from SQL but that's much better than issuing 50 extra queries. But is there a way I can choose to issue 2 queries, one to get Order and one to get OrderLine, and have EF connect the objects automatically?
A more realistic scenario where I want this is if the data is more complex. Let's say I want objects like this (where Product is the thing being bought in the OrderLine and ProductPart is a number of bits used to make the Product):
- Order
- OrderLine (avg 100 per Order)
- Product (1 per OrderLine)
- ProductPart (avg 20 per Product)
Now if I do a .Include() with ProductPart, it'll make the query results much bigger. Or if I don't Include() then it'll issue separate queries for each Product.
Is there a third way where I can get all the Order, OrderLine and Product data in one query and all the ProductPart data in another query, and magically have EF connect the objects up for me?
UPDATE:
I just read about AsSplitQuery() which seems to be what I'm looking for but is only available in EF Core 5 which isn't stable till Nov 2020 (?). I'm looking for a way to achieve this in EF6.
A bit more research and I found https://entityframework.net/improve-ef-include-performance which suggests two approaches when you have multiple object types coming off the parent object:
Execute multiple EF commands to pull back the same list of parent objects, but each time Include()-ing different child object types. EF will hook up the related objects it's already pulled from the db, apparently.
Use the EF+ library which seems it can do something like AsSplitQuery()
I'm not sure if this works with my case where there's more levels of nesting rather than just different types of objects off the parent. My guess is yes, if I'm clever about it. Any tips?

Something like this might destructure your object results a bit, but should perform two separate queries.
var ordersQuery = context.Orders.Where(o => o.Status == "Whatever");
var orderLineGroups = ordersQuery
.SelectMany(o => o.OrderLines)
.ToLookup(l => l.OrderID); // <- Not sure what your FK name is
var orders = ordersQuery.Select(o => new {
Order = o,
OrderLines = orderLineGroups[o.Id]
}).ToList();

Not sure that it will work in EF6 but you can try rely on tracking and relationship fix up. Next one works for me in the latest EF Core:
public class Make
{
public int MakeId { get; set; }
public string Name { get; set; }
public ICollection<Model> Models { get; set; }
public Make()
{
Models = new HashSet<Model>();
}
}
public class Model
{
public int ModelId { get; set; }
public string Name { get; set; }
public Make Make { get; set; }
public int MakeId { get; set; }
}
ctx.Makes.Where(m => m.Name=="3").SelectMany(m=> m.Models).ToList();
ctx.Makes.Where(m => m.Name=="3").ToList(); // makes have models filled in

If you are issuing tracking queries, then EF will fixup all the references while the results are being returned. You can also use the Load method to load the results & fixup references without constructing a list.
For example;
var orders = context.Orders
.Where(o => o.Status == "Whatever")
.ToList();
context.OrderLines
.Where(l => l.Order.Status == "Whatever")
.Include(l => l.Product) // maybe this is a reasonable tradeoff?
.Load();
context.ProductPart
.Where(p => p.Product.OrderLines.Any(l => l.Order.Status == "Whatever"))
.Load();

Handle Linq queries outputs

I've been struggling for the last 3 days on that topic.
I'm sure i'm doing something wrong but there, i need help.
During the load of a form, i'm doing a Linq query (on a global dataset) to populate fields on that form. As i want to be able to change the views of the form, i want queries that will make the data available in a specific format (to avoid having to query every now on then (the dataset is 20,000 lines)).
so i came up with that first queries :
var results =
from row in Globals.ds.Tables["Song"].AsEnumerable()
group row by (row.Field<int>("year"), row.Field<int>("rating")) into grp
orderby grp.Key
select new
{
year = grp.Key.Item1,
conte = grp.ToList().Count,
rating = grp.Key.Item2,
duree = grp.Sum(r => r.Field<int>("duree"))
};
It works and i'm pasting the result in the following screenshot (conte is the count)
Result of the query
1 have 2 issues :
1/ I really dont know how to handle that result : i would like to filter for a specific year and list all the subsequent ratings (i have from 1 to 6 per year). I tried the .ToList() but it only helped to get the count. The CopyToDataTable is not available for the query.
2/ i have buttons in the form that will need to access to that query, yet the var result is only available in the load and i can't manage to declare it at the class level.
Thanks for the help :)

So:
Your first point have been answered by #jdweng
It is possible to use LinQ also for collections (ex. List), not only Db queries.
The reason is that the result of the query is an anonymous type, and it can't be declared outside local scope. You must create a new class with the same structure.
public class MyResultClass
{
public int year;
public int conte;
public int rating;
public int duree;
}
Define your field:
List<MyResultClass> data;
And then use both:
var result =
from row in Globals.ds.Tables["Song"].AsEnumerable()
group row by (row.Field<int>("year"), row.Field<int>("rating")) into grp
orderby grp.Key
select new MyResultClass
{
year = grp.Key.Item1,
conte = grp.ToList().Count,
rating = grp.Key.Item2,
duree = grp.Sum(r => r.Field<int>("duree"))
};
data = result.ToList();
I hope I was helpful.

LINQ: Is there a way to combine these queries into one?

I have a database that contains 3 tables:
Phones
PhoneListings
PhoneConditions
PhoneListings has a FK from the Phones table(PhoneID), and a FK from the Phone Conditions table(conditionID)
I am working on a function that adds a Phone Listing to the user's cart, and returns all of the necessary information for the user. The phone make and model are contained in the PHONES table, and the details about the Condition are contained in the PhoneConditions table.
Currently I am using 3 queries to obtain all the neccesary information. Is there a way to combine all of this into one query?
public ActionResult phoneAdd(int listingID, int qty)
{
ShoppingBasket myBasket = new ShoppingBasket();
string BasketID = myBasket.GetBasketID(this.HttpContext);
var PhoneListingQuery = (from x in myDB.phoneListings
where x.phonelistingID == listingID
select x).Single();
var PhoneCondition = myDB.phoneConditions
.Where(x => x.conditionID == PhoneListingQuery.phonelistingID).Single();
var PhoneDataQuery = (from ph in myDB.Phones
where ph.PhoneID == PhoneListingQuery.phonePageID
select ph).SingleOrDefault();
}

You could project the result into an anonymous class, or a Tuple, or even a custom shaped entity in a single line, however the overall database performance might not be any better:
var phoneObjects = myDB.phoneListings
.Where(pl => pl.phonelistingID == listingID)
.Select(pl => new
{
PhoneListingQuery = pl,
PhoneCondition = myDB.phoneConditions
.Single(pc => pc.conditionID == pl.phonelistingID),
PhoneDataQuery = myDB.Phones
.SingleOrDefault(ph => ph.PhoneID == pl.phonePageID)
})
.Single();
// Access phoneObjects.PhoneListingQuery / PhoneCondition / PhoneDataQuery as needed
There are also slightly more compact overloads of the LINQ Single and SingleOrDefault extensions which take a predicate as a parameter, which will help reduce the code slightly.
Edit
As an alternative to multiple retrievals from the ORM DbContext, or doing explicit manual Joins, if you set up navigation relationships between entities in your model via the navigable join keys (usually the Foreign Keys in the underlying tables), you can specify the depth of fetch with an eager load, using Include:
var phoneListingWithAssociations = myDB.phoneListings
.Include(pl => pl.PhoneConditions)
.Include(pl => pl.Phones)
.Single(pl => pl.phonelistingID == listingID);
Which will return the entity graph in phoneListingWithAssociations
(Assuming foreign keys PhoneListing.phonePageID => Phones.phoneId and
PhoneCondition.conditionID => PhoneListing.phonelistingID)

You should be able to pull it all in one query with join, I think.
But as pointed out you might not achieve alot of speed from this, as you are just picking the first match and then moving on, not really doing any inner comparisons.
If you know there exist atleast one data point in each table then you might aswell pull all at the same time. if not then waiting with the "sub queries" is nice as done by StuartLC.
var Phone = (from a in myDB.phoneListings
join b in myDB.phoneConditions on a.phonelistingID equals b.conditionID
join c in ph in myDB.Phones on a.phonePageID equals c.PhoneID
where
a.phonelistingID == listingID
select new {
Listing = a,
Condition = b,
Data = c
}).FirstOrDefault();
FirstOrDefault because single throws error if there exists more than one element.

Dynamic where clause using Linq to SQL in a join query in a MVC application

I am looking for a way to query for products in a catalog using filters on properties which have been assigned to the product based on the category to which the product belongs. So I have the following entities involved:
Products
-Id
-CategoryId
Categories
[Id, Name, UrlName]
Properties
[Id, CategoryId, Name, UrlName]
PropertyValues
[Id, PropertyId, Text, UrlText]
ProductPropertyValues
[ProductId, PropertyValueId]
When I add a product to the catalog, multiple ProductPropertyValues will be added based on the category and I would like to be able to filter all products from a category by selecting values for one or more properties. The business logic and SQL indexes and constraints make sure that all UrlNames and texts are unique for values properties and categories.
The solution will be a MVC3 EF code first based application and the routing is setup as followed:
/products/{categoryUrlName}/{*filters}
The filter routing part has a variable length so multiple filters can be applied. Each filter contains the UrlName of the property and the UrlText of the value separated by an underscore.
An url could look like this /products/websites/framework_mvc3/language_csharp
I will gather all filters, which I will hold in a list, by reading the URL. Now it is time to actually get the products based on multiple properties and I have been trying to find the right strategy.
Maybe there is another way to implement the filters. All larger web shops use category depending filters and I am still looking for the best way to implement the persistence part for this type of functionality. The suggested solutions result in an "or" resultset if multiple filters are selected. I can imagine that adding a text property to the product table in which all property values are stores as a joined string can work as well. I have no idea what this would cost performance wise. At leased there will be no complex join and the properties and their values will be received as text anyway.
Maybe the filtering mechanism can be done client side ass well.

The tricky part about this is sending the whole list into the database as a filter. Your approach of building up more and more where clauses can work:
productsInCategory = ProductRepository
.Where(p => p.Category.Name == category);
foreach (PropertyFilter pf in filterList)
{
PropertyFilter localVariableCopy = pf;
productsInCategory = from product in productsInCategory
where product.ProductProperties
.Any(pp => pp.PropertyValueId == localVariableCopy.ValueId)
select product;
}
Another way to go is to send the whole list in using the List.Contains method
List<int> valueIds = filterList.Select(pf => pf.ValueId).ToList();
productsInCategory = ProductRepository
.Where(p => p.Category.Name == category)
.Where(p => p.ProductProperties
.Any(pp => valueIds.Contains(pp.PropertyValueId)
);

IEnumerable<int> filters = filterList.Select(pf => pf.ValueId);
var products = from pp in ProductPropertyRepository
where filters.Contains(pp.PropertyValueId)
&& pp.Product.Category.Name == category
select pp.Product;
Bear in mind that as Contains is used, the filters will be passed in as sproc parameters, this means that you have to be careful not to exceed the sproc parameter limit.

I came up with a solution that even I can understand... by using the 'Contains' method you can chain as many WHERE's as you like. If the WHERE is an empty string, it's ignored (or evaluated as a select all). Here is my example of joining 2 tables in LINQ, applying multiple where clauses and populating a model class to be returned to the view.
public ActionResult Index()
{
string AssetGroupCode = "";
string StatusCode = "";
string SearchString = "";
var mdl = from a in _db.Assets
join t in _db.Tags on a.ASSETID equals t.ASSETID
where a.ASSETGROUPCODE.Contains(AssetGroupCode)
&& a.STATUSCODE.Contains(StatusCode)
&& (
a.PO.Contains(SearchString)
|| a.MODEL.Contains(SearchString)
|| a.USERNAME.Contains(SearchString)
|| a.LOCATION.Contains(SearchString)
|| t.TAGNUMBER.Contains(SearchString)
|| t.SERIALNUMBER.Contains(SearchString)
)
select new AssetListView
{
AssetId = a.ASSETID,
TagId = t.TAGID,
PO = a.PO,
Model = a.MODEL,
UserName = a.USERNAME,
Location = a.LOCATION,
Tag = t.TAGNUMBER,
SerialNum = t.SERIALNUMBER
};
return View(mdl);
}

I know this an old answer but if someone see's this I've built this project:
https://github.com/PoweredSoft/DynamicLinq
Which should be downloadable on nuget as well:
https://www.nuget.org/packages/PoweredSoft.DynamicLinq
You could use this to loop through your filter coming from query string and do
something in the lines of
query = query.Query(q =>
{
q.Compare("AuthorId", ConditionOperators.Equal, 1);
q.And(sq =>
{
sq.Compare("Content", ConditionOperators.Equal, "World");
sq.Or("Title", ConditionOperators.Contains, 3);
});
});

How can I query this hierarchical data using LINQ?

I have 3 kinds of objects: Agency, BusinessUnit and Client (each with their own respective table)
In terms of hierarchy, Agencies own BusinessUnits, and BusinessUnits own Clients.
I have 3 C# POCO Objects to represent them (I usually select new {} into them, rather than use the LINQ generated classes):
public class Agency
{
public IEnumerable<BusinessUnit> BusinessUnits { get; set; }
}
public class BusinessUnit
{
public IEnumerable<Client> Clients { get; set; }
}
public class Client
{
public int NumberOfAccounts { get; set; }
public Decimal AmountOfPlacement { get; set; }
public Decimal AvgBalance { get; set; }
public Double NeuPlacementScore { get; set; }
}
You can see that Agencies contain a list of BusinessUnits, and BusinessUnits contain a list of Clients.
I also have a mapping table called BAC_Map in the database which says which owns which, and it looks something like this:
How can I construct a query, so I can query for and return a list of Agencies? Meaning that, I want each Agency to have its list of BusinessUnit objects set, and I want the list of BusinessObjects to have its list of Clients set.
I can do basic LINQ queries, but this is a little over my head concerning the Map table and the multiple? queries.
How could I construct a method like GetAllAgencies() which would query, for not only all agencies, but populate its BusinessUnits that Agency owns, and the Clients those BusinessUnits own?
Edit: Any tips or info is appreciated. Do I need to do joins? Does this need to be multiple queries to return an Agency list, with its submembers populated?

If you drop all four tables (Agency, BusinessUnit, Client, Map) on the linq to sql designer, and draw relationships from Map to the other three, there will be some useful properties on Map.
//construct a query to fetch the row/column shaped results.
var query =
from m in db.map
//where m.... ?
let a = m.Agency
let b = m.BusinessUnit
let c = m.Client
// where something about a or b or c ?
select new {
AgencyID = a.AgencyID,
AgencyName = a.Name,
BusinessUnitID = b.BusinessUnitID,
ClientID = c.ClientID,
NumberOfAccounts = c.NumberOfAccounts,
Score = c.Score
};
//hit the database
var rawRecords = query.ToList();
//shape the results further into a hierarchy.
List<Agency> results = rawRecords
.GroupBy(x => x.AgencyID)
.Select(g => new Agency()
{
Name = g.First().AgencyName,
BusinessUnits = g
.GroupBy(y => y.BusinessUnitID)
.Select(g2 => new BusinessUnit()
{
Clients = g2
.Select(z => new Client()
{
NumberOfAccounts = z.NumberOfAccounts,
Score = z.Score
})
})
})
.ToList();
If approriate filters are supplied (see the commented out where clauses), then only the needed portions of the tables will be pulled into memory. This is standard SQL joining at work here.

I created your tables in a SQL Server database, and tried to recreate your scenario in LinqPad. I ended up with the following LINQ statements, which basically result in the same structure of your POCO classes:
var map = from bac in BAC_Maps
join a in Agencies on bac.Agency_ID equals a.Agency_ID
join b in BusinessUnits on bac.Business_Unit_ID equals b.Business_Unit_ID
join c in Clients on bac.Client_ID equals c.Client_ID
select new
{
AgencyID = a.Agency_ID,
BusinessUnitID = b.Business_Unit_ID,
Client = c
};
var results = from m in map.ToList()
group m by m.AgencyID into g
select new
{
BusinessUnits = from m2 in g
group m2 by m2.BusinessUnitID into g2
select new
{
Clients = from m3 in g2
select m3.Client
}
};
results.Dump();
Note that I called map.ToList() in the second query. This actually resulted in a single, efficient query. My initial attempt did not include .ToList(), and resulted in nine separate queries to produce the same results. The query generated by the .ToList() version is as follows:
SELECT [t1].[Agency_ID] AS [AgencyID], [t2].[Business_Unit_ID] AS [BusinessUnitID], [t3].[Client_ID], [t3].[NumberOfAccounts], [t3].[AmountOfPlacement], [t3].[AvgBalance], [t3].[NeuPlacementScore]
FROM [BAC_Map] AS [t0]
INNER JOIN [Agencies] AS [t1] ON [t0].[Agency_ID] = [t1].[Agency_ID]
INNER JOIN [BusinessUnits] AS [t2] ON [t0].[Business_Unit_ID] = [t2].[Business_Unit_ID]
INNER JOIN [Clients] AS [t3] ON [t0].[Client_ID] = [t3].[Client_ID]
Here is a screenshot of the results:
alt text http://img411.imageshack.us/img411/5003/agencybusinessunitclien.png

If you are doing this with direct LINQ to SQL, there is no way to do this without some kind of recursion, whether you do it yourself or you hide it behind an extension method. Recursive SQL is very bad (many round trips, many single queries).
There are two options here. One is to pull the entire table(s) with the hierarchy into memory and use LINQ to Objects on it. Leave the "details" tables in SQL. If you have less than several thousand entities, this is probably the most efficient way to go. You can keep a single copy of the table(s) in cache and refresh them when necessary. When you need to fetch more detailed data from the DB for a single record, you can reattach that entity from your cached hierarchy to a new DataContext and fetch it.
The other option is to use a more complex relationship model in your database. Storing parent only by nature demands recursion, but you can use the adjacency list model to construct a single query which can span many levels of inheritance. This will mean your LINQ to SQL queries become less intuitive (querying against Entity.Right and Entity.Left isn't quite as pretty as Parent or Children...) but you can do in one query what might take hundreds or thousands in the literal recursive approach.