LINQ with joins and group generates enormous query and is slow - c#

My Entity structure is as follows:
public class DisbursementItem
{
public int DisbursementNumber;
public int IDDisbursementItem;
public int IDReceiptItem;
public decimal Amount;
public decimal MeasureUnit;
public decimal PricePerMU;
public decimal PriceTotal;
public Disbursement Disbursement_IDDisbursement;
public int IDDisbursementNumber;
}
public class Disbursement
{
public int DisbursementNumber;
DateTime date;
DisbursementType DType;
string Note;
string Subscriber;
Subscriber SubscriberModel;
string ItemType;
int ProcessNumber;
}
public class Subscriber
{
public string Name
public string Address;
public string City;
}
public class DisbursementDescription
{
public int IDDisbursementItem;
public string Description;
}
public class Receipt
{
public int IDReceiptItem;
public int ItemNumber;
}
public class StorageCard
{
public int ItemNumber;
public string StorageCardGroup;
public string StorageCardName;
}
And my EF6 LINQ query is:
DateTime from;
DateTime to;
var result = context.DisbursementItem
.Where(x => x.Disbursement_IDDisbursement.Date <= to && x.Disbursement_IDDisbursement.Date >= from)
.Join(context.DisbursementDescription, di => di.IDDisbursementItem, dd => dd.IDDisbursementItem, (di, dd) => new {di = di, desc = dd.Description})
.Join(context.Receipt, x => x.di.IDReceiptItem, r => r.IDReceiptItem, (x, r) => new { di = x.di, desc = x.desc, r = r })
.Join(context.StorageCard, x => x.r.ItemNumber, sc => sc.ItemNumber, (x, sc) => new { di = x.di, desc = x.desc, r = x.r, sc = sc})
.GroupBy(g => new {g.di.DisbursementNumber, g.sc.ItemNumber, g.di.MeasureUnit})
.Select(x => new
{
Date = x.FirstOrDefault().di.Disbursement_IDDisbursement.Date,
DisbursementNumber = x.Key.DisbursementNumber,
DType = x.FirstOrDefault().di.Disbursement_IDDisbursement.DType,
Note = x.FirstOrDefault().di.Disbursement_IDDisbursement.Note,
Subscriber = x.FirstOrDefault().di.Disbursement_IDDisbursement.Subscriber,
SubscriberName = x.FirstOrDefault().di.Disbursement_IDDisbursement.SubscriberModel.Name,
SubscriberAddress = x.FirstOrDefault().di.Disbursement_IDDisbursement.SubscriberModel.Address,
SubscriberCity = x.FirstOrDefault().di.Disbursement_IDDisbursement.SubscriberModel.City,
ItemNumber = x.FirstOrDefault().sc.ItemNumber,
StorageCardGroup = x.FirstOrDefault().sc.StorageCardGroup,
StorageCardName = x.FirstOrDefault().sc.StorageCardName,
Amount = x.Sum(y => y.di.Amount),
PricePerMU = x.FirstOrDefault().di.PricePerMU,
PriceTotal = x.Sum(y => y.di.PriceTotal),
MeasureUnit = x.Key.MeasureUnit
Desc = x.FirstOrDefault().desc,
})
SELECT
di.Date,
di.DisbursementNumber,
d.DType,
d.Note,
d.Subscriber,
subs.Name,
subs.Address,
subs.City,
sc.ItemNumber,
sc.StorageCardGroup,
sc.StorageCardName,
Sum(di.Amount) as Amount,
di.PricePerMU,
Sum(di.PriceTotal) as PriceTotal,
di.MeasureUnit,
dd.Description
FROM
DisbursementItem as di
INNER JOIN Disbursement as d
ON di.IDDisbursementNumber = d.DisbursementNumber
INNER JOIN Receipt as r
ON di.IDReceiptItem = r.IDReceiptItem
INNER JOIN StorageCard as sc
ON r.ItemNumber = sc.ItemNumber
INNER JOIN DisbursementDescription dd
ON di.IDDisbuzrsementItem = dd.IDDisbursementItem
WHERE
di.Date <= ... and di.Date >= ...
GROUP BY
di.DisbursementNumber, sc.ItemNumber, di.MeasureUnit
That is the query in SQL that I want to achieve in EF
This query can take over a minute for a few hundred rows. How can I optimize it? I suspect the multiple joins is a problem and maybe also the Sum of some fields.
Also the database schema cannot be modified.
The query it generate is enormous. It's like a SELECT in SELECT in SELECT for like 40 times.

Easiest way is to add all fields which are needed for result to grouping key. Rewritten query to Query syntax for readability and maintainability:
DateTime from;
DateTime to;
var query =
from di in context.DisbursementItem
where di.Disbursement_IDDisbursement.Date <= to && di.Disbursement_IDDisbursement.Date >= from
join dd in context.DisbursementDescription on di.IDDisbursementItem equals dd.IDDisbursementItem
join r in context.Receipt on di.IDReceiptItem equals r.IDReceiptItem
join sc in context.StorageCard on r.ItemNumber equals sc.ItemNumber
group di by new
{
di.DisbursementNumber,
sc.ItemNumber,
di.MeasureUnit,
di.Disbursement_IDDisbursement.Date,
di.Disbursement_IDDisbursement.DType,
di.Disbursement_IDDisbursement.Note,
Subscriber = di.Disbursement_IDDisbursement.Subscriber,
SubscriberName = di.Disbursement_IDDisbursement.SubscriberModel.Name,
SubscriberAddress = di.Disbursement_IDDisbursement.SubscriberModel.Address,
SubscriberCity = di.Disbursement_IDDisbursement.SubscriberModel.City,
sc.ItemNumber,
sc.StorageCardGroup,
sc.StorageCardName,
di.PricePerMU,
Desc = dd.Description
} into g
select new
{
g.Key.Date,
g.Key.DisbursementNumber,
g.Key.DType,
g.Key.Note,
g.Key.Subscriber,
g.Key.SubscriberName,
g.Key.SubscriberAddress,
g.Key.SubscriberCity,
g.Key.ItemNumber,
g.Key.StorageCardGroup,
g.Key.StorageCardName,
g.Key.PricePerMU,
g.Key.MeasureUnit,
g.Key.Desc,
Amount = g.Sum(x => x.Amount),
PriceTotal = g.Sum(x => x.PriceTotal)
}

you could try some kind of multithreading
the query could be splitted in parts and each part assigned to a task. in here you should find something useful (parallel section):
https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks?view=net-6.0

Related

Linq with Multiple Joins

I'm attempting to translate the following SQL statement to Linq and am having trouble with the multiple joins- I seem to be missing something.
SELECT DISTINCT
Test1 = Table1.Column1,
Test2 = 1,
Test3 = Table1.Column2,
Test4 = Table1.Column5,
Test5 = Table1.Column6
FROM Table1
LEFT JOIN Table2 ON Table1.Column1 = Table2.Column1
INNER JOIN Table3 ON Table1.Column3 = Table3.Column3
WHERE Table3.Column4 IN (1,2,6)
Here's the Linq so far:
var TestQuery = Table1_Collection.Select(x => new
{
Test1 = Table1.Column1,
Test2 = 1,
Test3 = Table1.Column2,
Test4 = Table1.Column5,
Test5 = Table1.Column6
})
[joins go here]
.Where("where stuff goes here");
Any ideas? I'm not so much seeking assistance with the .Where as I am the joins. I'm not sure about the formatting with the method syntax.
Here you go:
var results = Table3_Collection
.Where(i => column4s.Contains(i.Column4))
.Join(Table1_Collection, i => i.Column3, i => i.Column3, (i, j) => j)
.Join(Table2_Collection, i => i.Column1, i => i.Column1, (i, j) => i)
.Distinct(comparer);
In your original SQL query you weren't using selecting any columns from Table2, so you could omit that join. I included it above, but please feel free to remove it.
Also, your C# example didn't have Distinct, but I included it for you as it was in your original SQL query, and is most likely your intent. And, please, don't forget to implement your own IEqualityComparer. Here is an example of one:
class Table1Comparer : IEqualityComparer<Table1>
{
public bool Equals(Table1 x, Table1 y)
{
return x.Column1 == y.Column1
&& x.Column2 == y.Column2
&& x.Column3 == y.Column3
&& x.Column4 == y.Column4
&& x.Column5 == y.Column5
&& x.Column6 == y.Column6;
}
public int GetHashCode(Table1 obj)
{
return obj.GetHashCode();
}
}
try this example, I hope you help
class Table1
{
public int Id1 { get; set; }
public string Column1 { get; set; }
}
class Table2
{
public int Id2 { get; set; }
public string Column2 { get; set; }
}
class Table3
{
public int Id3 { get; set; }
public string Column3 { get; set; }
}
static void Main(string[] args)
{
var table1 = new List<Table1>();
var table2 = new List<Table2>();
var table3 = new List<Table3>();
for (int i = 0; i < 10; i++)
{
table1.Add(new Table1 { Id1 = i, Column1 = "column1_table1_" + i });
table2.Add(new Table2 { Id2 = i, Column2 = "column2_table2_" + i });
table3.Add(new Table3 { Id3 = i, Column3 = "column3_table3_" + i });
}
var table1JoinTable2 = table1.Join(table2, t1 => t1.Id1, t2 => t2.Id2, (t1, t2) => new { Id = t1.Id1, Column1 = t1.Column1, Column2 = t2.Column2 } );
var table1JoinTable2JoinTable3 = table1JoinTable2.Join(table3, t12 => t12.Id, t3 => t3.Id3, (t12, t3) => new { Id = t12.Id, Column1 = t12.Column1, Column2 = t12.Column2, Column3 = t3.Column3 });
var result1 = table1JoinTable2JoinTable3.Single(t123 => t123.Id == 1);
Console.WriteLine("Id={0} C1={1} C2={2} C3={3}", result1.Id, result1.Column1, result1.Column2, result1.Column3);
// prints "Id=1 C1=column_table1_1 C2=column_table2_1 C3=column_table3_1"
}
Here is an example of your original SQL statement in LINQ query syntax.
List<int> vals = new List<int> {1,2,6};
var qry = from rec1 in Table1
join rec2 in Table2 on rec1.Column1 equals rec2.Column2 into ljT2
from rec2 in ljT2.DefaultIfEmpty() //Handle left join
join rec3 in Table3 on rec1.Column1 equals rec3.Column3
where vals.Contains(rec3.Column4)
select new {
Test1 = rec1.Column1,
Test2 = 1,
Test3 = rec2 == null?null:rec2.Column2, //Must allow for rec2 to be null
Test4 = rec3.Column5,
Test5 = rec3.Column6
}
qry = qry.Distinct();

Nopcommerce MessageTemplate, can't add method as token

I'm making a MessageTemplate in nopCommerce and I want to add a token that implements a method from another class.
In the class "ProductService" I have a method called "DailyBestsellersReport" which I want to add as a token for my MessageTemplate in "MessageTokenProvider". However when I add "ProductService" as a reference in the MessageTokenProvider it tells me the method "DailyBestsellerReport" isn't valid in the current context which have me thinking there's a syntax error somewhere.
This is the method I want to add as token:
ProductService class:
public IList<BestsellersReportLine> DailyBestSellersReport(
int recordsToReturn = 5, int orderBy = 1, int groupBy = 1)
{
var yesterDay = DateTime.Now.Subtract(new TimeSpan(1, 0, 0, 0));
var earliest = new DateTime(yesterDay.Year, yesterDay.Month, yesterDay.Day, 0, 0, 0);
var latest = earliest.Add(new TimeSpan(1, 0, 0, 0, -1));
var currentDay = DateTime.Now;
var dayBefore = DateTime.Now.AddDays(-1);
var query1 = from opv in _opvRepository.Table
where earliest <= currentDay && latest >= dayBefore
join o in _orderRepository.Table on opv.OrderId equals o.Id
join pv in _productVariantRepository.Table on opv.ProductVariantId equals pv.Id
join p in _productRepository.Table on pv.ProductId equals p.Id
select opv;
var query2 = groupBy == 1 ?
//group by product variants
from opv in query1
group opv by opv.ProductVariantId into g
select new
{
EntityId = g.Key,
TotalAmount = g.Sum(x => x.PriceExclTax),
TotalQuantity = g.Sum(x => x.Quantity),
}
:
//group by products
from opv in query1
group opv by opv.ProductVariant.ProductId into g
select new
{
EntityId = g.Key,
TotalAmount = g.Sum(x => x.PriceExclTax),
TotalQuantity = g.Sum(x => x.Quantity),
}
;
switch (orderBy)
{
case 1:
{
query2 = query2.OrderByDescending(x => x.TotalQuantity);
}
break;
case 2:
{
query2 = query2.OrderByDescending(x => x.TotalAmount);
}
break;
default:
throw new ArgumentException("Wrong orderBy parameter", "orderBy");
}
if (recordsToReturn != 0 && recordsToReturn != int.MaxValue)
query2 = query2.Take(recordsToReturn);
var result = query2.ToList().Select(x =>
{
var reportLine = new BestsellersReportLine()
{
EntityId = x.EntityId,
TotalAmount = x.TotalAmount,
TotalQuantity = x.TotalQuantity
};
return reportLine;
}).ToList();
return result;
}
I want to add "DailyBestsellersReport" here:
MessageTokenProvider class
public void AddReportTokens(IList<Token> tokens, BestsellersReportLine DailyBestSellersReport, ProductService productService, int languageId)
{
tokens.Add(new Token("BestsellersReportLine.EntityId", DailyBestSellersReport.EntityId.ToString()));
tokens.Add(new Token("BestsellersReportLine.TotaAmount", DailyBestSellersReport.TotalAmount.ToString()));
tokens.Add(new Token("BestsellersReportLine.TotalQuantity", DailyBestSellersReport.TotalQuantity.ToString()));
tokens.Add(new Token("ProductService.DailyBestSellersReport", productService.DailyBestSellersReport.ToString)());
}
When I add:
tokens.Add(new Token("ProductService.DailyBestSellersReport", productService.DailyBestSellersReport.ToString)());
It tell me:
Error 10 'Nop.Services.Catalog.ProductService.DailyBestSellersReport(int, int, int)' is a 'method', which is not valid in the given context
I've also added tokens from "BestsellersReportLine" class which works fine, however these are properties and not methods like:
public partial class BestsellersReportLine
{
public int EntityId { get; set; }
public decimal TotalAmount { get; set; }
public int TotalQuantity { get; set; }
}
Any thoughts?
Thank you
The error itself tells the solution how can you call a method without passing any arguments to it when it accepts parameters.
You will need to write method which returns string depending on result in of productService.DailyBestSellersReport
public string ReturnTable()
{
var report = productService.DailyBestSellersReport(param1,param2,param2)
StringBuilder sb = new StringBuilder();
foreach (var r in report)
{
// I believe you are trying to build HTML table so you can append any string here to sb
}
return sb.ToString();
}
Then use
tokens.Add(new Token("ProductService.DailyBestSellersReport",ReturnTable());
sb.AppendLine("<table border=\"0\" style=\"width:100%;\">");//sb is stringbuilder's object
sb.AppendLine(string.Format("<tr style=\"background-color:{0};text-align:center;font-size:12px; \">", _templatesSettings.Color1));
sb.AppendLine(string.Format("<th>Sr. No.</th>"));
sb.AppendLine(string.Format("<th>Item1</th>"));
sb.AppendLine(string.Format("<th>Item2</th>"));
sb.AppendLine("</tr>");
// Header is closed
// Next is data which will be created using foreach on data from `IList<BestsellersReportLine>`
foreach (var item in result)
{
sb.AppendLine(string.Format("<tr style=\"background-color: {0};text-align: center;\">", _templatesSettings.Color2));
// you can place all your data in below tds, you can create any number of tds.
sb.AppendLine(string.Format("<td style=\"padding: 0.6em 0.4em;text-align: right;\">{0}</td>", item.Prop1));
sb.AppendLine("<td style=\"padding: 0.6em 0.4em;text-align: left;\">" + item.Prop2);
sb.AppendLine("</td>");
sb.AppendLine("</tr>");
}
sb.AppendLine("</table>"); // don't forget to close table
return sb.ToString();

Group and Map into Object with an inner aggregate Object

I have many Object1A, say IEnumerable<Object1A>.
public class Object1A {
public string text;
public datetime date;
public decimal percent;
public Object3 obj;
}
Many of these objects have the same text, date, and percent, but have a different obj. I want to transform the list such that the output will be a IEnumerable<Object1B> where
public class Object1B{
public string text;
public datetime date;
public decimal percent;
public IEnumerable<Object3> objs;
}
My current apporach is a bit clunky, and listed below
IEnumerable<Object1A> a = GetSomeConstruct();
var lookup = a.ToLookup( t => t.text);
var b = new List<Object1b>();
foreach(var group in lookup){
var itemA = group.first();
var itemB = new Object1b(){
text = itemA.text,
date = itemA.date,
percent = itemA.percent
};
itemB.objs = pair.Select(t => t.obj);
b.Add(itemB);
}
Can this approach be refined? It doesn't seem to run to slow, but it seems like it could be better. I'm looking for a more terse approach if possible.
edit: yeah, this was a dumb question, cudos to the downvote....
simple answer
var b_objects = a_objects.GroupBy(t => new {t.Text})
.Select( t => new Object1B
{ Text = t.Key.Text,
Percent = t.First().Percent,
Date = t.First().Date,
Objs = t.Select( o => o.Obj).ToList()
});
Guess you want something like this?
var b = from a in GetSomeConstruct()
group a.obj by new { a.text, a.date, a.percent } into grp
select new Object1B
{
text = grp.Key.text,
date = grp.Key.date,
percent = grp.Key.percent,
objs = grp
};
You can use anonymous types with join and group by. Their GetHashCode and Equals overloads operate on each member.

Linq Join tables, Group by date, Sum of values?

I have two tables (one-to-many). MeterReadings(0..1) and MeterReadingDetails(*)
I want to join these tables and group by date. Date field is in MeterReadings and Others are in MeterReadingDetails.
I used this code:
Linq
public static IEnumerable<MeterReadingsForChart> GetCustomerTotal(int CustomerId, int MeterTypeId, DateTime StartDate, DateTime EndDate, MeterReadingsTimeIntervals DateRangeType)
{
var customerReadings = from m in entity.MeterReadings
join n in entity.MeterReadingDetails on m.sno equals n.ReadingId
where m.Meters.CustomerId == CustomerId && m.ReadDate >= StartDate && m.ReadDate <= EndDate && m.Meters.TypeId == MeterTypeId
group n by new { date = new DateTime(m.ReadDate.Value.Year, m.ReadDate.Value.Month, 1) } into g
select new MeterReadingsForChart
{
ReadDate = g.Key.date,
Value = g.Sum(x => x.Value),
Name = g.FirstOrDefault().MeterReadingTypes.TypeName
};
return customerReadings;
}
MeterReadinsForChart.cs
public class MeterReadingsForChart
{
public DateTime ReadDate { get; set; }
public string Name { get; set; }
public double Value { get; set; }
}
But I got this error:
Only parameterless constructors and initializers are supported in LINQ to Entities
How can I join, group, and sum?
Try the following:
var customerReadings = (from m in entity.MeterReadings
join n in entity.MeterReadingDetails on m.sno equals n.ReadingId
where m.Meters.CustomerId == CustomerId && m.ReadDate >= StartDate && m.ReadDate <= EndDate && m.Meters.TypeId == MeterTypeId
group n by new { Year = m.ReadDate.Value.Year, Month = m.ReadDate.Value.Month} into g
select new
{
Key = g.Key,
Value = g.Sum(x => x.Value),
Name = g.FirstOrDefault().MeterReadingTypes.TypeName
}).AsEnumerable()
.Select(anon => new MeterReadingsForChart
{
ReadDate = new DateTime(anon.Key.Year, anon.Key.Month, 1),
Value = anon.Value,
Name = anon.Name
});
Unf. its ugly, but entity framework won't let you create a DateTime (being a struct it has no parameterless constructors). So in this case we want most of the result from the db and then as this streams we construct the date in memory.

linq-to-sql group by with count and custom object model

I'm looking to fill an object model with the count of a linq-to-sql query that groups by its key.
The object model looks somewhat like this:
public class MyCountModel()
{
int CountSomeByte1 { get; set; }
int CountSomeByte2 { get; set; }
int CountSomeByte3 { get; set; }
int CountSomeByte4 { get; set; }
int CountSomeByte5 { get; set; }
int CountSomeByte6 { get; set; }
}
This is what I have for the query:
var TheQuery = from x in MyDC.TheTable
where ListOfRecordIDs.Contains(x.RecordID) && x.SomeByte < 7
group x by x.SomeByte into TheCount
select new MyCountModel()
{
CountSomeByte1 = TheCount.Where(TheCount => TheCount.Key == 1)
.Select(TheCount).Count(),
CountSomeByte2 = TheCount.Where(TheCount => TheCount.Key == 2)
.Select(TheCount).Count(),
.....
CountSomeByte6 = TheCount.Where(TheCount => TheCount.Key == 6)
.Select(TheCount).Count(),
}.Single();
ListOfRecordIDs is list of longs that's passed in as a parameter. All the CountSomeByteN are underlined red. How do you do a count of grouped elements with the group's key mapped to an object model?
Thanks for your suggestions.
The select is taking each element of your group and projecting them to identical newly created MyCountModels, and you're only using one of them. Here's how I'd do it:
var dict = MyDC.TheTable
.Where(x => ListOfRecordIDs.Contains(x.RecordID) && x.SomeByte < 7)
.GroupBy(x => x.SomeByte)
.ToDictionary(grp => grp.Key, grp => grp.Count());
var result = new MyCountModel()
{
CountSomeByte1 = dict[1];
CountSomeByte2 = dict[2];
CountSomeByte3 = dict[3];
CountSomeByte4 = dict[4];
CountSomeByte5 = dict[5];
CountSomeByte6 = dict[6];
}
EDIT: Here's one way to do it in one statement. It uses an extension method called Into, which basically works as x.Into(f) == f(x). In this context, it can be viewed as like a Select that works on the whole enumerable rather than on its members. I find it handy for eliminating temporary variables in this sort of situation, and if I were to write this in one statement, it's probably how I'd do it:
public static U Into<T, U>(this T self, Func<T, U> func)
{
return func(self);
}
var result = MyDC.TheTable
.Where(x => ListOfRecordIDs.Contains(x.RecordID) && x.SomeByte < 7)
.GroupBy(x => x.SomeByte)
.ToDictionary(grp => grp.Key, grp => grp.Count())
.Into(dict => new MyCountModel()
{
CountSomeByte1 = dict[1];
CountSomeByte2 = dict[2];
CountSomeByte3 = dict[3];
CountSomeByte4 = dict[4];
CountSomeByte5 = dict[5];
CountSomeByte6 = dict[6];
});
Your range variable is not correct in the subqueries:
CountSomeByte6 = TheCount.Where(TheCount => TheCount.Key == 6)
.Select(TheCount).Count(),
In method notation you don't need the extra select:
CountSomeByte6 = TheCount.Where(theCount => theCount.Key == 6).Count(),
If you want to use it anyway:
CountSomeByte6 = TheCount.Where(theCount => theCount.Key == 6).Select(theCount => theCount).Count(),

Categories

Resources