Optimize LINQ instead of creating new collections/loops - c#

I have two tables:
Invoices (InvoiceID, InvoiceNumber)
Invoices_Products (InvoiceID, ProductID, IsFinalized)
I show a list of all invoices, and there are buttons to filter by "finalized" or "not finalized" invoices. A finalized invoice is one where every product on it is IsFinalized==true.
At the moment I have the following code which is performing quite slowly:
IEnumerable<Invoice> invoices = db.Invoices;
if (isFinalized) // filter by finalized invoices
{
List<Invoice> unfinalizedInvoices = new List<Invoice>();
foreach (var invoice in invoices)
{
int invoicesProductsCountTotal = db.Invoices_Products.Where(l => l.InvoiceID == invoice.InvoiceID).Count();
int invoicesProductsCountFinalized = db.Invoices_Products.Where(l => l.InvoiceID == invoice.InvoiceID && l.IsFinalized == true).Count();
if (invoicesProductsCountTotal != invoicesProductsCountFinalized)
{
unfinalizedInvoices.Add(invoice);
}
}
invoices = invoices.Except(unfinalizedInvoices);
}
else
{
List<Invoice> finalizedInvoices = new List<Invoice>();
foreach (var invoice in invoices)
{
int invoicesProductsCountTotal = db.Invoices_Products.Where(l => l.InvoiceID == invoice.InvoiceID).Count();
int invoicesProductsCountFinalized = db.Invoices_Products.Where(l => l.InvoiceID == invoice.InvoiceID && l.IsFinalized == true).Count();
if (invoicesProductsCountTotal == invoicesProductsCountFinalized && invoicesProductsCountFinalized > 0)
{
finalizedInvoices.Add(invoice);
}
}
invoices = invoices.Except(finalizedInvoices);
}
I realize this isn't optimal but I like spreading out my LINQ so that I can read and understand it.My question: Is there any way I could make this query faster using .All or .Any or something, or do I need to rethink my database design (possibly adding an extra column to the Invoices table)
edit: Third table is Products (ProductID, ProductNumber) but you knew that


At the moment you're loading all your invoices and then loading the products for each invoice. This is bound to be slow (and it will become a lot slower when you start adding a lot of invoice).
You should create a many-to-many relationship in EntityFramework. (see example)
Your classes would look like this:
class Invoice
{
List<Product> Products {get; set;}
}
class Product
{
bool IsFinalized {get; set;}
}
Now you can use LINQ to make sure that only SQL statement is executed which fetches only the data you want:
var invoices = db.Invoices.Where(i => i.Products.All(p => p.IsFinalized == finalized));


Iterating over each Invoice and then make additional requests to the database will be very slow. Let your query get all the informations at once and iterate through the results instead.
var result = from invoice in db.Invoices
join invoicedProduct in db.Invoices_Products
on invoice.InvoiceId equals invoicedProduct.InvoiceId
select new
{
InvoiceId = invoice.InvoiceId,
ProductId = invoicedProduct.ProductId,
IsFinalized = invoicedProuct.IsFinalized
};
var grpResult = from record in result
group record by record.ProductId into productGrp
select productGrp;
foreach( var grp in grpResult )
{
Console.WriteLine( "ProductId: " + grp.Key.ToString( ) );
Console.WriteLine( "TotalCount: " + grp.Count( ).ToString( ) );
Console.WriteLine( "Finalized: " + grp.Where( item => item.IsFinalized ).Count( ).ToString( ) );
}


if (isFinalized)
{
invoices = invoices.Where(l => l.Invoices_Products.All(m => m.IsFinalized == true));
}
else
{
List<Invoice> finalizedInvoices = invoices.Where(l => l.Invoices_Products.All(m => m.IsFinalized == true)).ToList();
invoices = invoices.Except(finalizedInvoices);
}
^^ this seems to have improved performance dramatically. oh well, thanks for listening

Related

EF core query group by date and subquery

I have the following table:
CREATE TABLE "OrderStatusLogs" (
"Id" UNIQUEIDENTIFIER NOT NULL,
"OrderId" UNIQUEIDENTIFIER NOT NULL,
"Status" INT NOT NULL,
"StartDateTime" DATETIMEOFFSET NOT NULL,
"EndDateTime" DATETIMEOFFSET NULL DEFAULT NULL,
PRIMARY KEY ("Id"),
FOREIGN KEY INDEX "FK_OrderStatusLogs_Orders_OrderId" ("OrderId"),
CONSTRAINT "FK_OrderStatusLogs_Orders_OrderId" FOREIGN KEY ("OrderId") REFERENCES "Orders" ("Id") ON UPDATE NO_ACTION ON DELETE CASCADE
)
;
For the following entity:
[DebuggerDisplay(nameof(OrderStatusLog) + " {Status} {StartDateTime} - {EndDateTime}" )]
public class OrderStatusLog
{
public Guid Id { get; set; }
public Guid OrderId { get; set; }
public OrderStatus Status { get; set; }
public DateTimeOffset StartDateTime { get; set; }
public DateTimeOffset? EndDateTime { get; set; }
}
public enum OrderStatus
{
Unknown = 0,
Pending = 1,
Processing = 2,
Shipping = 3,
}
And i'm trying to generate a report which should show how many orders are set to a certain state for a given range.
For example, for the month oktober, we'd have the range 1 to 31 oktober.
The desired output would be something like this:
1/10/2021 Pending 21 orders
1/10/2021 Processing 23 orders
1/10/2021 Shipping 33 orders
1/10/2021 Unknown 0 orders
...
31/10/2021 Pending 1 orders
31/10/2021 Processing 3 orders
31/10/2021 Shipping 44 orders
31/10/2021 Unknown 5 orders
I'm having some difficulties writing a query in EF that would give me the right output. I can get things to work, but only client-side. I'm trying to make this work in the database instead.
So far i tried:
var logsByDayAndOrderId = orderStatusLogs.GroupBy(c => new { c.StartDateTime.Date, c.OrderId }, (key, values) => new
{
key.Date,
key.OrderId,
MaxStartDateTime = values.Max(x => x.StartDateTime)
});
var list = logsByDayAndOrderId.ToList();
var statusByDayAndOrderId = logsByDayAndOrderId.Select(c => new
{
c.Date,
c.OrderId,
orderStatusLogs.FirstOrDefault(x => x.StartDateTime == c.MaxStartDateTime && x.OrderId == c.OrderId).Status
});
//var statusByDayAndOrderId = logsByDayAndOrderId.Join(orderStatusLogs.def, inner => new { inner.OrderId, StartDateTime = inner.MaxStartDateTime }, outer => new { outer.OrderId, outer.StartDateTime }, (inner,outer) => new
//{
// inner.Date,
// inner.OrderId,
// outer.Status
//}); // TODO rem this query gives more results because of the join. we need an Outer join - but i could not get that to work. the version with select above works better, but then it does not use join so it may be slow(er).
var list1 = statusByDayAndOrderId.ToList();
var groupBy = statusByDayAndOrderId
.GroupBy(c => new { c.Date, c.Status })
.Select(c => new { c.Key.Date, c.Key.Status, Count = c.Count() });
var list2 = groupBy.ToList();
Another attempt:
var datesAndOrders = orderStatusLogs
.GroupBy(c => new { c.StartDateTime.Date, c.OrderId }, (key, values) => key);
var ordersByDateAndActiveStatusLog = orderStatusLogs
.Select(c => new
{
c.StartDateTime.Date,
c.OrderId,
ActiveStatusForDate = orderStatusLogs
.OrderByDescending(x => x.StartDateTime)
.FirstOrDefault(x => x.OrderId == c.OrderId && x.StartDateTime.Date == c.StartDateTime.Date)
.Status
});
var list = ordersByDateAndActiveStatusLog.ToList();
var orderCountByDateAndStatus = ordersByDateAndActiveStatusLog
.GroupBy(c => new { c.Date, c.ActiveStatusForDate }, (key, values) => new
{
key, count = values.Count()
});
var list1 = orderCountByDateAndStatus.ToList();
Both of these fail because of Cannot use an aggregate or a subquery in an expression used for the group by list of a GROUP BY clause..
This makes sense.
I'm hoping for someone that could help write a Linq query that generates the right data using ef core.
Notes:
I Solely use the fluent query syntax
I Have more places where i'd like to get data for each day so any other info or tips and tricks are welcome
I use net core 5 with ef core 5.0.11 with a MSSQL database

I would suggest to use EF Core extension linq2db.EntityFrameworkCore which has ability to work with local (in-memory) collections in database queries. Disclaimer: i'm one of the creators.
At first define function which generates days sequence:
public static IEnumerable<DateTime> GenerateDays(int year, int month)
{
var start = new DateTime(year, month, 1);
var endDate = start.AddMonths(1);
while (start < endDate)
{
yield return start;
start = start.AddDays(1);
}
}
Then we can use generated sequence in LINQ Query:
var days = GenerateDays(2021, 10).ToArray();
using var dc = ctx.CreateLinqToDbConnection();
var totalsQuery =
from d in days.AsQueryable(dc)
from l in orderStatusLogs.Where(l =>
(l.EndDateTime == null || l.EndDateTime >= d) && l.StartDateTime < d.AddDays(1))
.DefaultIfEmpty()
group l by new { Date = d, l.Status } into g
into g
select new
{
g.Key.Date,
g.Key.Status,
Count = g.Sum(x => x == null ? 0 : 1),
};
var result = totalsQuery.ToList();
The following SQL should be generated:
SELECT
[d].[item],
[e].[Status],
Sum(IIF([e].[OrderID] IS NULL, 0, 1))
FROM
(VALUES
('2021-05-01T00:00:00'), ('2021-05-02T00:00:00'),
('2021-05-03T00:00:00'), ('2021-05-04T00:00:00'),
('2021-05-05T00:00:00'), ('2021-05-06T00:00:00'),
('2021-05-07T00:00:00'), ('2021-05-08T00:00:00'),
('2021-05-09T00:00:00'), ('2021-05-10T00:00:00'),
('2021-05-11T00:00:00'), ('2021-05-12T00:00:00'),
('2021-05-13T00:00:00'), ('2021-05-14T00:00:00'),
('2021-05-15T00:00:00'), ('2021-05-16T00:00:00'),
('2021-05-17T00:00:00'), ('2021-05-18T00:00:00'),
('2021-05-19T00:00:00'), ('2021-05-20T00:00:00'),
('2021-05-21T00:00:00'), ('2021-05-22T00:00:00'),
('2021-05-23T00:00:00'), ('2021-05-24T00:00:00'),
('2021-05-25T00:00:00'), ('2021-05-26T00:00:00'),
('2021-05-27T00:00:00'), ('2021-05-28T00:00:00'),
('2021-05-29T00:00:00'), ('2021-05-30T00:00:00'),
('2021-05-31T00:00:00')
) [d]([item])
LEFT JOIN [OrderStatusLogs] [e] ON ([e].[EndDateTime] IS NULL OR [e].[EndDateTime] >= [d].[item]) AND [e].[StartDateTime] < DateAdd(day, 1, [d].[item])
GROUP BY
[d].[item],
[e].[Status]

How do I minimize C#, LINQ code in Deleting Multiple Records

Here is a method to delete zero Inventory records from Inventory Table. I would like to reduce code/no of times that LINQ executes on Database.
Inventory Table
public class Inventory
{
public int itemCode { get; set; }
public decimal price { get; set; }
public decimal availQty { get; set; } // Can have Negative values.
}
example data
itemCode price availQty
1 10 10
1 12 -10
2 10 10
From above records, i want to delete all records of itemCode == 1, as net availQty is 0.
Here is my method
private void RemoveZeroInvs()
{
// Remove individual zero Inventorys
var rinvs = from ri in _context.Inventorys
where ri.availQty == 0
select ri;
_context.Inventorys.RemoveRange(rinvs);
_context.SaveChanges();
// Remove if group is zero in availQty, as it allows Negative Qty.
var result = from d in _context.Inventorys
group d by new
{
d.itemCode
}
into g
select new
{
g.Key.itemCode,
availQty = g.Sum(y => y.availQty)
};
var zrs = from r in result
where r.availQty == 0
select r;
foreach (var zr in zrs) // Here, zrs length may be more than 500
{
var ri = _context.Inventorys.Where(w => w.itemCode == zr.itemCode);
_context.Inventorys.RemoveRange(ri);
_context.SaveChanges();
}
}
I use Asp.Net Core 2.2. Is there any such possibility?
Also I get following error at line _context.Inventorys.RemoveRange(ri); in the loop.
A command is already in progress: SELECT t."itemCode", t."availQty"
FROM (
SELECT d."itemCode", SUM(d."availQty") AS "availQty"
FROM "Inventorys" AS d
GROUP BY d."itemCode"
) AS t

var todelete = _context.Inventorys
.GroupBy(i => i.itemCode)
.Where(g => g.Sum(i => i.availQty) == 0)
.SelectMany(g => g);
Here is a shorter versions of your code, in terms of DB excecution, one would have to compare the raw queries. But it may be lighter that your code…

SQL Distinct records on multiple conditional query

I am trying to get Distinct results using EntityFramework and MS SQL query and filtering records on two conditions so I have used Distinct()
here is my code:
public List<ProductViewModel> GetPagedFilterProducts(int page, int type_id1, int type_id2)
{
int recordsPerPage = 20;
var skipRecords = page * recordsPerPage;
var results = _products.GetAll().Where(p => p.type1 == type_id1 && p.type2 == type_id2).Select(p => new ProductViewModel
{
productId = p.product_id,
productTitle = p.product_title,
}).OrderByDescending(p => p.productTitle).Skip(skipRecords).Take(recordsPerPage).ToList();
return results.Distinct().ToList();
}
I am using checkboxes to filter the records, so type_id can be more than two conditions (I mean there could be type_id3). How should I query so user can select more than two type conditions and it still gives me Distinct results.
How can I query on each condition separately and combine them together and get the Distinct results?
Please forgive me if I didn't explain my question properly.

public List<ProductViewModel> GetPagedFilterProducts(int page, int type_id1, int type_id2)
{
int recordsPerPage = 20;
var skipRecords = page * recordsPerPage;
var results = _products.GetAll().Where(p => p.type == type_id1 || p.type == type_id2).Select(p => new ProductViewModel
{
productId = p.product_id,
productTitle = p.product_title,
}).OrderByDescending(p => p.productTitle)
.Skip(skipRecords)
.Take(recordsPerPage).ToList();
return results.Distinct().ToList();
}
The basic idea behind the query,Value of checkbox for selecting items will compare in a table with a single column .

Count and Groupby using linq from sql query

Im trying to create a table that counts all orders and groups them in a table from sql to linq to use in a bar graph with google charts.
Table`
Orders Status
8 Created
3 Delayed
4 Enroute
sql
SELECT Count (OrderID) as 'Orders', order_status FROM [ORDER]
where order_status ='Created'OR order_status= 'Delayed' OR order_status='Enroute'
group by order_status
controller
public ActionResult GetChart()
{
var Orders = db.Order.Select(a => new { a.OrderID, a.order_status })
.GroupBy(a => a.order_status);
return Json(Orders, JsonRequestBehavior.AllowGet);
}
this is not displaying the correct results as the linq seems to be wrong.
can someone please point me in the right direction? I am relatively new to this.
thanks in advance.

This should work:-
var result = db.Order.Where(x => x.order_status == "Created"
|| x.order_status == "Delayed"
|| x.order_status == "Enroute")
.GroupBy(x => x.order_status)
.Select(x => new
{
order_status = x.Key,
Orders = x.Count()
});
Or if you prefer query syntax then:-
var result = from o in db.Order
where o.order_status == "Created" || o.order_status == "Delayed"
|| o.order_status == "Enroute"
group o by o.order_status
select new
{
orderStatus = x.Key,
Counts = x.Count()
};

I think you want to group by Status and count total number of orders in each group (I build a simple console program to demonstrate). I suppose the data is:
Orders Status
8 Created
3 Delayed
4 Enroute
2 Created
1 Delayed
Order.cs
public class Order
{
public Order(int orderId, string status)
{
OrderId = orderId;
Status = status;
}
public int OrderId { get; set; }
public string Status { get; set; }
}
Program.cs
class Program
{
static void Main(string[] args)
{
// Data
var orders = new List<Order>
{
new Order(8, "Created"),
new Order(3, "Delayed"),
new Order(4, "Enroute"),
new Order(2, "Created"),
new Order(1, "Delayed"),
};
// Query
var query = orders
.GroupBy(x => x.Status)
.Select(x => new {Status = x.Key, Total = x.Count()});
// Display
foreach (var item in query)
{
Console.WriteLine(item.Status + ": " + item.Total);
}
Console.ReadLine();
}
}
The one you need to focus in is query. After using GroupBy, you will have a list of groups. For each group, the Key is the criteria to group (here is the Status). Then, we call Count() to get the total number of element in that group.
So, from the program above, the output should be:
Created: 2
Delayed: 2
Enroute: 1

How to loop through multiple level master detail data through linq?

I have three levels of master detail relation, One purchase can contain multiple challan, one challan can contain multiple items. each item has some quantity. I need to calculate total item quantity for every purchase. I've done that by the following, but it takes a lot of time just for a handful of data. I'm worried what will happen when the amount of data becomes large. Is there a way to do this in a single query using join or anything else? Thanks in advance.
var allData = (from p in _context.Prq_Purchase.AsEnumerable()
//where p.RecordStatus == "NCF"
from s in _context.Sys_Supplier
where s.SupplierID == p.SupplierID
from sa in _context.Sys_SupplierAddress
where sa.SupplierAddressID == p.SupplierAddressID
orderby p.PurchaseID descending
select new PurchaseReceive
{
PurchaseID = (p.PurchaseID).ToString(),
PurchaseNo= p.PurchaseNo,
SupplierID = (p.SupplierID).ToString(),
SupplierName = s.SupplierName,
Address = sa.Address,
SupplierAddressID = (p.SupplierAddressID).ToString(),
PurchaseCategory = p.PurchaseCategory,
PurchaseType = p.PurchaseType,
PurchaseYear = p.PurchaseYear,
PurchaseDate = (p.PurchaseDate).ToString("dd'/'MM'/'yyyy"),
RecordStatus= DalCommon.ReturnRecordStatus(p.RecordStatus)
}).ToList();
foreach(var Purchase in allData)
{
decimal TotalQty = 0;
var ChallanList= (from c in _context.Prq_PurchaseChallan.AsEnumerable()
where (c.PurchaseID).ToString()==Purchase.PurchaseID
select c).ToList();
foreach(var Challan in ChallanList)
{
var ItemList = (from i in _context.Prq_PurchaseChallanItem.AsEnumerable()
where i.ChallanID == Challan.ChallanID
select i).ToList();
foreach(var Item in ItemList )
{
TotalQty = TotalQty + Item.ReceiveQty;
}
}
Purchase.TotalItem = TotalQty;
}

Categories

Resources