I have this query running ok in a stored procedure but now I want to do what sp does from c# with EF and linq, any idea?
I'm using .Net 6 in MVC EF project.
I have my db context working and entities for Productos and AlmacenesStock created
The query:
SELECT s.ProductoId, p.Descripcion, SUM(s.Cantidad) AS Cantidad
FROM Productos p INNER JOIN AlmacenesStock s
ON p.Id = s.ProductoId
GROUP BY s.ProductoId, p.Descripcion
Thanks!
Assume that in your DbContext you have these DbSet:
public DbSet<Producto> Productos { get; set; }
public DbSet<AlmacenesStock> ProductoAlmacenesStocks { get; set; }
With LINQ query syntax/expression which has some similarities with SQL query.
var result = (from a in _context.Productos
join b in _context.AlmacenesStocks on a.Id equals b.ProductoId
group new { a, b } by new { b.ProductoId, a.Descripcion } into g
select new
{
ProductoId = g.Keys.ProductoId,
Descripcion = g.Keys.Descripcion,
Cantidad = g.Sum(x => x.b.Cantidad)
}
)
.ToList();
The above result will return the value with the List of anonymous type. If you have your concrete class to store the value, modify the select part as:
select new YourEntity
{
ProductoId = g.Keys.ProductoId,
Descripcion = g.Keys.Descripcion,
Cantidad = g.Sum(x => x.b.Cantidad)
}
Recommended reading: Query expression basics
Make sure you set up the relationship of Products and AlmacenesStock
var products = _context.Products.Include(product => product.AlmacenesStock)
.GroupBy(product => new { product.ProductId, product.Description }
.Select(product => new { product.Key.ProductId, product.Key.Description, x.Sum(almnacenesStock => almnacenesStock.Cantidad) });
I'm using Dapper to map my database content to EmployeeModel objects.
The mapping of properties is working, but the grouping at the end is still giving me trouble:
EmployeeModel takes List<PhoneModel> and List<EmployeeModel> as properties.
Items are grouped according to EmployeeID, however multiple email and phone results are returned as I haven't been able to find the syntax for doing so.
I've tried looping through the EmployeeIDs in the employeeList after it's been grouped by ID, before it's been grouped by ID, and while it's being grouped by ID.
var sql = #"
SELECT
e.id,
e.FirstName, e.LastName, e.Nickname,
em.id as ID, em.Address, em.Type,
jt.id as ID, jt.Name,
e.id as ID, p.Number, p.Type,
d.id as ID, d.Name,
es.id as ID, es.Name
FROM
dbo.Employees e
LEFT JOIN dbo.Emails em ON em.EmployeeID = e.id
LEFT JOIN dbo.JobTitles jt ON e.JobTitleID = jt.id
LEFT JOIN Phones p ON p.EmployeeID = e.id
LEFT JOIN dbo.Departments d ON e.DepartmentID = d.id
LEFT JOIN dbo.EmployeeStatus es ON e.StatusID = es.id
";
IEnumerable<EmailModel> emailsGrouped = new List<EmailModel>();
var employees = await connection
.QueryAsync<
EmployeeModel,EmailModel,TitleModel,
PhoneModel,DepartmentModel,StatusModel,
EmployeeModel>
(
sql,
( e, em, t, p, d, s ) =>
{
e.EmailList.Add(em);
e.JobTitle = t;
e.PhoneList.Add(p);
e.Department = d;
e.Status = s;
return e;
},
splitOn: "ID, ID, ID, ID, ID"
);
foreach (EmployeeModel emod in employees)
{
emod.EmailList.GroupBy(em => em.ID);
}
var result = employees
.GroupBy(e => e.ID)
.Select(g =>
{
var groupedEmployee = g.First();
groupedEmployee.EmailList = g.Select(e => e.EmailList.Single()).ToList();
groupedEmployee.PhoneList = g.Select(e => e.PhoneList.Single()).ToList();
return groupedEmployee;
});
return result.ToList();
Here is my Email definition, as requested. It's inside my EmployeeClass, so I've posted the whole thing.
public class EmployeeModel
{
public int ID { get; set; }
public string LastName { get; set; }
public string FirstName { get; set; }
public string Nickname { get; set; }
public DepartmentModel Department { get; set; }
public TitleModel JobTitle { get; set; }
public DateTime HireDate { get; set; }
public StatusModel Status { get; set; }
public List<EmailModel> EmailList { get; set; } = new List<EmailModel>();
public List<PhoneModel> PhoneList { get; set; } = new List<PhoneModel>();
public List<RestrictionModel> RestrictionsList { get; set; } = new List<RestrictionModel>();
public List<CitationModel> CitationsList { get; set; } = new List<CitationModel>();
public List<CertificationModel> CertificationList { get; set; } = new List<CertificationModel>();
public string ListView
{
get
{
return $"{LastName}, {FirstName}";
}
}
public string ToEmailString()
{
IEnumerable<string> employeeEmailStrings = EmailList.Select(emmod => emmod.ToString());
string employeeEmailString = string.Join($"{Environment.NewLine}", employeeEmailStrings);
return $"{FirstName}, {LastName}: {Environment.NewLine} -{JobTitle.Name}- {Environment.NewLine}";
}
//IEnumerable<string> phoneStrings = PhoneList.Select(plistmod => plistmod.ToString());
//string phoneString = string.Join($"{Environment.NewLine}", phoneStrings);
public string ToCertificationString()
{
IEnumerable<string> certificationStrings = CertificationList.Select(clistmod => clistmod.ToString());
string certificationString = string.Join($"{Environment.NewLine}", certificationStrings);
return certificationString;
}
public class EmailModel
{
public int ID { get; set; }
public string Address { get; set; }
public string Type { get; set; }
public override string ToString()
{
return $"{Address} ({Type})";
}
}
public class PhoneModel
{
public int ID { get; set; }
public string Number { get; set; }
public string Type { get; set; }
public override string ToString()
{
return $"{Number} ({Type})";
}
}
}
}
What I'm trying now is to loop through the Emails in the EmployeeModel to create a new list of emails, and then set that new list as the EmployeeModel.List<EmailModel>.
So it looks like you're actually trying to load an object-graph (containing nodes of distinct types) from a database using SQL - and you're trying to do that using a single query.
That won't work. (Naïve, single-query) SQL is not suitable for querying object-graphs. This is why ORMs exist. However with some RDBMS-specific SQL extensions (e.g. T-SQL, PL/SQL, etc) to execute a query batch you can return an object-graph from a database.
The good news is that Dapper supports this scenario with QueryMultiple - however as far as I know it won't map collection properties, so you need to do that manually (so read on!)
(I note that Entity Framework, specifically, will generate single-SELECT queries that return redundant data in columns that represent lower-multiplicity data - this has its trade-offs but generally speaking separate queries can work faster overall with the right tweaks (such as using a table-valued variable to hold KEY values instead of re-evaluating the same WHERE criteria for every query in the batch - as always, check your indexes, STATISTICS objects, and execution plans!).
When querying for an object-graph, you'll write a SELECT query batch where each query returns all objects of the same type that has a JOIN with any other entities with a 1:1 or 1:0..1 multiplicity (if it isn't more efficient to load them in a separate query in the same batch).
In your case, I see you have:
[Employees]---(1:m)---[Phones]
[Employees]---(1:m)---[Emails]
[JobTitles]---(1:m)---[Employees]
[Departments]---(1:m)---[Employees]
[EmployeeStatus]---(1:m)---[Employees] // is this an enum table? if so, you can probably ditch it
So try this:
For the sake of simplicity, JobTitles, Departments, and EmployeeStatus can be done in a single query.
I assume the foreign-key columns are NOT NULL so an INNER JOIN should be used instead of LEFT OUTER JOIN.
const String EMPLOYEES_PHONES_EMAILS_SQL = #"
-- Query 1: Employees, Departments, EmployeeStatuses
SELECT
e.id,
e.FirstName,
e.LastName,
e.Nickname,
t.Name AS JobTitleName, -- This is to disambiguate column names. Never rely on column ordinals!
d.Name AS DepartmentName,
s.Name AS StatusName
FROM
dbo.Employees AS e
INNER JOIN dbo.JobTitles AS t ON e.JobTitleID = t.id
INNER JOIN dbo.Departments AS d ON e.DepartmentId = d.id
INNER JOIN dbo.EmployeeStatus AS s ON e.StatusID = s.id;
-- Query 2: Phones
SELECT
p.EmployeeId,
p.Number,
p.Type
FROM
dbo.Phones AS p;
-- Query 3: Emails
SELECT
m.id,
m.EmployeeId,
m.Address,
m.Type
FROM
dbo.Emails AS m;
";
using( SqlMapper.GridReader rdr = connection.QueryMultiple( EMPLOYEES_PHONES_EMAILS_SQL ) )
{
List<EmployeeModel> employees = ( await rdr.ReadAsync<EmployeeModel>() ).ToList();
var phonesByEmployeeId = ( await rdr.ReadAsync<PhoneModel> () ).GroupBy( p => p.EmployeeId ).Dictionary( grp => grp.Key grp => grp.ToList() );
var emailsByEmployeeId = ( await rdr.ReadAsync<EmailModel> () ).GroupBy( m => m.EmployeeId ).Dictionary( grp => grp.Key, grp => grp.ToList() );
foreach( EmployeeModel emp in employees )
{
if( phonesByEmployeeId.TryGetValue( emp.EmployeeId, out var phones ) )
{
emp.Phones.AddRange( phones );
}
if( emailsByEmployeeId.TryGetValue( emp.EmployeeId, out var emails ) )
{
emp.Emails.AddRange( emails );
}
}
}
I'll admit that I'm not intimately familiar with Dapper - and there is a problem with the code above: it doesn't instruct Dapper how to read the included Department, JobTitleModel, and EmployeeStatus data in the first query. I assume there's some overload of ReadAsync to specify other included data.
If you find yourself doing this kind of logic repetitively you can define your own extension-methods to handle the worst parts (such as GroupBy().ToDictionary(), and populating a collection property from a dictionary of loaded entities).
If you had a filter criteria, then you'd need to either store the resultant EmployeeId key values in a TVV, or repeat the criteria on Employees as the right-hand-side of an INNER JOIN in the queries for Phones and Emails.
For example, if you wanted to add an ability to find all Employees (and their phone-numbers and e-mail addresses) by name, you'd do this:
const String EMPLOYEES_PHONES_EMAILS_SQL = #"
-- Query 0: Get EmployeeIds:
DECLARE #empIds TABLE ( EmpId int NOT NULL PRIMARY KEY );
INSERT INTO #empIds ( EmpId )
SELECT
EmployeeId
FROM
dbo.Employees
WHERE
FirstName LIKE #likeFirst
OR
LastName LIKE #likeLast;
-- Query 1: Employees, Departments, EmployeeStatuses
SELECT
e.id,
e.FirstName,
e.LastName,
e.Nickname,
t.Name AS JobTitleName, -- This is to disambiguate column names. Never rely on column ordinals!
d.Name AS DepartmentName,
s.Name AS StatusName
FROM
dbo.Employees AS e
INNER JOIN dbo.JobTitles AS t ON e.JobTitleID = t.id
INNER JOIN dbo.Departments AS d ON e.DepartmentId = d.id
INNER JOIN dbo.EmployeeStatus AS s ON e.StatusID = s.id
INNER JOIN #empIds AS i ON i.EmpId = e.EmployeeId;
-- Query 2: Phones
SELECT
p.EmployeeId,
p.Number,
p.Type
FROM
dbo.Phones AS p
INNER JOIN #empIds AS i ON i.EmpId = p.EmployeeId;
-- Query 3: Emails
SELECT
m.id,
m.EmployeeId,
m.Address,
m.Type
FROM
dbo.Emails AS m
INNER JOIN #empIds AS i ON i.EmpId = m.EmployeeId;
";
using( SqlMapper.GridReader rdr = connection.QueryMultiple( EMPLOYEES_PHONES_EMAILS_SQL, new { likeFirst = "%john%", likeLast = "%smith%" } ) )
{
// same as before
}
I need to query two database tables for a search term and return the results. The following was working in EF Core 2:
var SearchTerm = "hello";
IQueryable<TableA> q;
q = (from a in context.TableA
join b in context.TableB on a equals b.A into leftjoin
from c in leftjoin.DefaultIfEmpty()
where c.Column1.Contains(SearchTerm)
|| a.Column1.Contains(SearchTerm)
|| a.Column2.Contains(SearchTerm)
select a);
return q.Include(a => a.TableD)
.GroupBy(a => a.Id)
.Select(group => group.First())
.ToList();
The idea of the above is to take a SearchTerm and query two columns from TableA, join to TableB and also query a column in this one then select distinct values from TableA.
In .NET 3 the above throws an error saying it can't be translated to SQL. I tried to rewrite this, the best I can do is the below:
var SearchTerm = "hello";
var q = (from a in context.TableA
join b in context.TableB on a equals b.A into leftjoin
from c in leftjoin.DefaultIfEmpty()
where c.Column1.Contains(SearchTerm)
|| a.Column1.Contains(SearchTerm)
|| a.Column2.Contains(SearchTerm)
select a.Id).Distinct().ToList();
return context.TableA
.Where(a => q.Contains(a.Id))
.Include(c => c.TableD)
.ToList();
Which works ok but involves two database queries, since I already have the list of TableA from the first query it would be great to be able to just use this without having to extract the Ids and performing the second query. Also making sure the database continues to handle the distinct part rather than C# would be preferable too.
The definitions of A and B are:
public class TableA
{
public int Id { get; set; }
public string Column1 { get; set; }
public string Column2 { get; set; }
public int TableDId { get; set; }
public TableD TableD { get; set; }
}
public class TableB
{
public int Id { get; set; }
public string Column1 { get; set; }
public int TableAId { get; set; }
public TableA TableA { get; set; }
}
If I understood you correctly you have one-to-many relation between TableA and TableB, so it should be possible to add collection navigation property to TableA like in this tutorial for example:
public class TableA
{
public int Id { get; set; }
...
public ICollection<TableB> TableBs { get; set; }
}
So you can try to do something like this:
context.TableA
.Where(ta => ta.TableBs.Any(tb => tb.Column1.Contains(SearchTerm))
|| ta.Column1.Contains(SearchTerm)
|| ta.Column2.Contains(SearchTerm))
.Include(c => c.TableD)
.ToList();
Another option is to try subquery:
var q = (from a in context.TableA
join b in context.TableB on a.Id equals b.TableAId into leftjoin
from c in leftjoin.DefaultIfEmpty()
where c.Column1.Contains(SearchTerm)
|| a.Column1.Contains(SearchTerm)
|| a.Column2.Contains(SearchTerm)
select a.Id); // remove Distinct and ToList
return context.TableA
.Where(a => q.Contains(a.Id))
.Include(c => c.TableD)
.ToList();
I have a nested data model and I'd like to get aggregated data from it grouped by a top level property.
My models for example:
public class Scan {
public long Id {get; set;}
public int ProjectId { get; set; }
public int ScanUserId { get; set; }
public ICollection<ScanItem>? Items { get; set; }
}
public class ScanItem
{
public long Id { get; set; }
public long InventoryScanId { get; set; }
public double Qty { get; set; }
}
I'd like to get all Scans grouped by Scan.ScanUserId and then then get the sum of ScanItems.Qty for example per user. My query looks ike this and EF gives the following error:
Processing of the LINQ expression 'AsQueryable((Unhandled
parameter: x).Items)' by 'NavigationExpandingExpressionVisitor' failed
from scan in Scans
.Include(x=>x.ScanUser)
.Include(x=>x.Items)
group scan by new { scan.ScanUser.Name, scan.ScanUser.Id } into g
select new
{
UserId = g.Key.Id,
Name = g.Key.Name,
LastSyncTime = g.Max(x => x.ScanDate),
ScanItems = g.Sum(x=>x.Items.Sum(i=>i.Qty))
}
How can I run aggregate functions on the properties of the nested table without evaluating it on the client side?
EF Core still can't translate nested aggregates on GroupBy result (grouping).
You have to pre calculate the nested aggregates in advance by utilizing the element selector of GroupBy (or element in query syntax group element by key):
from scan in Scans
group new { scan.ScanDate, Qty = scan.Items.Sum(i => i.Qty) } // <--
by new { scan.ScanUser.Name, scan.ScanUser.Id } into g
select new
{
UserId = g.Key.Id,
Name = g.Key.Name,
LastSyncTime = g.Max(x => x.ScanDate),
ScanItems = g.Sum(x => x.Qty) // <--
}
**Update: ** for SqlServer the above LINQ query translates to SQL query like this:
SELECT [s1].[Id] AS [UserId], [s1].[Name], MAX([s0].[ScanDate]) AS [LastSyncTime], SUM((
SELECT SUM([s].[Qty])
FROM [ScanItem] AS [s]
WHERE [s0].[Id] = [s].[InventoryScanId])) AS [ScanItems]
FROM [Scan] AS [s0]
INNER JOIN [ScanUser] AS [s1] ON [s0].[ScanUserId] = [s1].[Id]
GROUP BY [s1].[Name], [s1].[Id]
which as mentioned in the comment generates SQL execution exception "Cannot perform an aggregate function on an expression containing an aggregate or a subquery.".
So you really need another approach - use left join to flatten the result set before grouping, and then perform the grouping / aggregates on that set:
from scan in Scans
from item in scan.Items.DefaultIfEmpty() // <-- left outer join
group new { scan, item } by new { scan.ScanUser.Name, scan.ScanUser.Id } into g
select new
{
UserId = g.Key.Id,
Name = g.Key.Name,
LastSyncTime = g.Max(x => x.scan.ScanDate),
ScanItems = g.Sum(x => (double?)x.item.Qty) ?? 0
};
which now translates to hopefully valid SqlServer SQL query:
SELECT [s1].[Id] AS [UserId], [s1].[Name], MAX([s].[ScanDate]) AS [LastSyncTime], COALESCE(SUM([s0].[Qty]), 0.0E0) AS [ScanItems]
FROM [Scan] AS [s]
LEFT JOIN [ScanItem] AS [s0] ON [s].[Id] = [s0].[InventoryScanId]
INNER JOIN [ScanUser] AS [s1] ON [s].[ScanUserId] = [s1].[Id]
GROUP BY [s1].[Name], [s1].[Id]
I want to use Database.SqlQuery in Entity framework to run a custom JOIN operation. I don't want to use LINQ to do the JOIN because it is doing a horrible job of generating performant SQL on the backend and I just want to control what it does.
So my question is -- How can I get a set of objects (it's a JOIN from table A to table B, and I want both an object of type A and an object of type B) back out from an INNER JOIN operation on Database.SqlQuery?
As far as I know the SqlQuery method uses property names to map columns to properties.
So you can just declare the class with properties of query and then split it to pair A and B.
Example:
public class AB
{
public int Id { get; set; }
public string Name { get; set; }
public string Title { get; set; }
}
var abs = dbContext.Database.SqlQuery<AB>(#"SELECT A.Id, A.Name, B.Title
FROM A JOIN B ON A.Id = B.Id");
var a_and_bs = from ab in abs
select new
{
A = new A { ab.Id, ab.Name },
B = new B { ab.Title }
};