I'm sending an IQueryable to a method that parses data for jQuery DataTables. It builds the filter from FORM data and creates a LINQ statement with filter, sorting, paging, etc that gets sent back to the page as JSON. I'd like to extend the parser and add Totals to the result set. When I add a GroupBy statement, the query is not evaluated on the server, but locally instead. It will only execute on the server if the original IQueryable has an anonymous projection...
This is a .Net Core 2.1 website. I know in the past GroupBy could not be executed locally but it does at 2.1. I've tried projection with a class and to an anonymous type, and it only works properly with an anonymous type. I really need to be able to do this with a class.
The following iQueryable gets sent to the parser:
var query = Context.InvoiceHeaders
.AsNoTracking()
.Where(x=>x.Slsno.Equals("13"))
.Select(x => new InvoiceHeaderSummary()
{
SalesNumber = x.Slsno,
OrderNumber = x.Ordnum,
ItemAmount = x.Itmamt,
SpecialChargeAmount = x.Sc1amt,
TaxAmount = x.Taxamt,
InvoiceTotal = x.Invamt
})
var parser = new Parser<InvoiceHeaderSummary>(Request.Form, query);
I'm trying to extend the parser by adding a List of totals to the output. But since I'm sending in an IQueryable with a class projection (InvoiceHeaderSummary), it doesn't execute on the server. I get the warning about it being evaluated locally:
var totalList = query
.GroupBy(i => 1)
.Select(g => new
{
TotalInvoice = g.Sum(i => i.InvoiceTotal)
})
.ToList();
I tried creating the entire LINQ method inline and it DOES run properly (notice I'm using an anonymous projection prior to the GroupBy rather than InvoiceHeaderSummary class):
var query = Context.InvoiceHeaders
.AsNoTracking()
.Where(x=>x.Slsno.Equals("13"))
.Select(x => new
{
SalesNumber = x.Slsno,
OrderNumber = x.Ordnum,
ItemAmount = x.Itmamt,
SpecialChargeAmount = x.Sc1amt,
TaxAmount = x.Taxamt,
InvoiceTotal = x.Invamt
})
.GroupBy(i => 1)
.Select(g => new
{
TotalInvoice = g.Sum(i => i.InvoiceTotal)
})
.ToList();
Is there a way to properly write this so it runs optimally??
Apparently one of the EF Core 2.x query translation defects / bugs.
The only workaround I've found is to use the GroupBy overload with element selector and select the data you want to aggregate into anonymous type:
var totalList = query
.GroupBy(i => 1, i => new { i.InvoiceTotal }) // <--
.Select(g => new
{
TotalInvoice = g.Sum(i => i.InvoiceTotal)
})
.ToList();
Related
I have a problem trying to reuse some subqueries. I have the following situation:
var rooms = dbContext.Rooms.Select(r => new
{
RoomId = r.Id,
Zones = r.Zones.Select(zr => zr.Zone),
Name = r.Name,
Levels = r.Levels.Select(lr => lr.Level),
IdealSetpoint = (double?)r.Group.Setpoints.First(sp => sp.ClimaticZoneId == dbContext.ClimaticZonesLogs.OrderByDescending(cz => cz.Timestamp).First().ClimaticZoneId).Setpoint??int.MinValue,
Devices = r.Devices.Select(rd => rd.Device)
}).ToList();
var tagsTypes = rooms.Select(r => r.Devices.Select(d => GetSetpointTagTypeId(d.DeviceTypeId))).ToList().SelectMany(x => x).Distinct().ToList();
predicate = predicate.And(pv => tagsTypes.Contains(pv.TagSettings.TagTypeId) &&
pv.ClimaticZoneId == dbContext.ClimaticZonesLogs.OrderByDescending(cz => cz.Timestamp).First().ClimaticZoneId);
var setpoints = valuesSubquery.Include(t=>t.TagSettings).Where(predicate).ToList();
This works fine, and generates the exact queries as wanted. The problem is that I want to have this subquery dbContext.ClimaticZonesLogs.OrderByDescending(cz => cz.Timestamp).First().ClimaticZoneId to be taken from a method and not repeat it every time I need it.
I've tested it with the database, where I have values in the corresponding tables, and I've tested the query with the database without any data in the corresponding tables. It works fine with no problems or exceptions.
But when I try to extract the repeating subquery in a separate method and execute it against empty database tables (no data) the .First() statement throws error. Here is the code:
protected long GetClimaticZoneId()
{
return dbContext.ClimaticZonesLogs.OrderByDescending(cz => cz.Timestamp).First().ClimaticZoneId;
}
and the query generation:
var rooms = dbContext.Rooms.Select(r => new
{
RoomId = r.Id,
Zones = r.Zones.Select(zr => zr.Zone),
Name = r.Name,
Levels = r.Levels.Select(lr => lr.Level),
IdealSetpoint = (double?)r.Group.Setpoints.First(sp => sp.ClimaticZoneId == GetClimaticZoneId()).Setpoint??int.MinValue,
Devices = r.Devices.Select(rd => rd.Device)
}).ToList();
var tagsTypes = rooms.Select(r => r.Devices.Select(d => GetSetpointTagTypeId(d.DeviceTypeId))).ToList().SelectMany(x => x).Distinct().ToList();
predicate = predicate.And(pv => tagsTypes.Contains(pv.TagSettings.TagTypeId) &&
pv.ClimaticZoneId == GetClimaticZoneId());
var setpoints = valuesSubquery.Include(t=>t.TagSettings).Where(predicate).ToList();
After execution I get InvalidOperationException "Sequence do not contain any elements" exception in the GetClimaticZoneId method:
I'm sure that I'm not doing something right.
Please help!
Regards,
Julian
As #Gert Arnold suggested, I used the GetClimaticZoneId() method to make a separate call to the database, get the Id and use it in the other queries. I gust modified the query to not generate exception when there is no data in the corresponding table:
protected long GetClimaticZoneId()
{
return dbContext.ClimaticZonesLogs.OrderByDescending(cz => cz.Timestamp).FirstOrDefault()?.ClimaticZoneId??0;
}
I am trying to cut this linq down
var sys = db.tlkpSystems
.Where(a => db.tlkpSettings.Where(e => e.Hidden < 3)
.Select(o => o.System)
.ToList().Contains(a.System)) //cannot get this part in?
.OrderBy(a => a.SystemName).ToList();
foreach (var item in sys)
model.Add(new SettingSystem {
System = item.System,
SystemName = item.SystemName
});
I have tried the following:
List<SettingSystem> model = new List<SettingSystem>();
model = db.tlkpSettings.Where(e => e.Hidden < 3)
.OrderBy(e => e.Setting)
.Select(e => new SettingSystem
{
System = e.System,
SystemName = e.Setting
}).ToList();
How can I call the .Contains(a.System) part in my query?
Thanks
Some general rules when working with LINQ to Entities:
Avoid using ToList inside the query. It prevents EF to build a correct SQL query.
Don't use Contains when working with entities (tables). Use Any or joins.
Here is your query (in case System is not an entity navigation property):
var sys = db.tlkpSystems
.Where(a => db.tlkpSettings.Any(e => e.Hidden < 3 && e.System == a.System))
.OrderBy(a => a.SystemName).ToList();
As an addendum, there is also AsEnumerable for when you must pull a query into memory (such as calling methods within another clause). This is generally better than ToList or ToArray since it'll enumerate the query, rather than enumerating, putting together a List/Array, and then enumerating that collection.
im having problems using String format on my LINQ but only on my Controller, using it on my view work well, so how can i change my LINQ to not give me an error in the Controller.
This is my LINQ
foreach (var item in db.Pos.Where(r => r.Fecha.Day.ToString() == "2").Select(g => new { Pdv = g.Pdv, Total = g.Total })
.GroupBy(l => l.Pdv).Select(z => new { Pdv = z.Key, Total = String.Format("{0:$#,##0.00;($#,##0.00);Zero}",Decimal.Round(z.Sum(l => l.Total), 0) }))
{
listadepuntos.Add(item.ToString());
}
var grupoPdv = new SelectList(listadepuntos.ToList());
ViewBag.GroupS = grupoPdv;
what i want is that the data of ViewBag.GroupS gets ',' each 3 digits , like for hundreds, thousands and millions right now i get the date plain without format.
what can i do?
You can call AsEnumerable extension method after your group by to execute your Select method using Linq to Objects instead Linq to Entities:
.GroupBy(l => l.Pdv)
.AsEnumerable()// Add this
.Select(z => new { Pdv = z.Key, Total = String.Format("{0:$#,##0.00;($#,##0.00);Zero}",Decimal.Round(z.Sum(l => l.Total), 0) })
The issue is because your Linq provider doesn't know how to convert your method calls to a proper expression tree, which later need to be translated to a SQL statement. There is a few string methods that are currently supported (you will find them in this link), String.Format is not one of them. In case of Decimal.Round which is either supported, you could use System.Math.Round instead.
I have a method that returns data from an EF model.
I'm getting the above message, but I can't wotk our how to circumvent the problem.
public static IEnumerable<FundedCount> GetFundedCount()
{
var today = DateTime.Now;
var daysInMonth = DateTime.DaysInMonth(today.Year, today.Month);
var day1 = DateTime.Now.AddDays(-1);
var day31 = DateTime.Now.AddDays(-31);
using (var uow = new UnitOfWork(ConnectionString.PaydayLenders))
{
var r = new Repository<MatchHistory>(uow.Context);
return r.Find()
.Where(x =>
x.AppliedOn >= day1 && x.AppliedOn <= day31 &&
x.ResultTypeId == (int)MatchResultType.Accepted)
.GroupBy(x => new { x.BuyerId, x.AppliedOn })
.Select(x => new FundedCount(
x.Key.BuyerId,
x.Count() / 30 * daysInMonth))
.ToList();
}
}
FundedCount is not an EF enity, MatchHistory is, so can't understand why it is complaining.
All advice appreciated.
The reason it is complaining is because it doesn't know how to translate your Select() into a SQL expression. If you need to do a data transformation to a POCO that is not an entity, you should first get the relevant data from EF and then transform it to the POCO.
In your case it should be as simple as calling ToList() earlier:
return r.Find()
.Where(x => x.AppliedOn >= day1 && x.AppliedOn <= day31 &&
x.ResultTypeId == (int)MatchResultType.Accepted)
.GroupBy(x => new { x.BuyerId, x.AppliedOn })
.ToList() // this causes the query to execute
.Select(x => new FundedCount(x.Key.BuyerId, x.Count() / 30 * daysInMonth));
Be careful with this, though, and make sure that you're limiting the size of the data set returned by ToList() as much as possible so that you're not trying to load an entire table into memory.
Message is clear : linq to entities doesn't support objects without a parameterless ctor.
So
Solution1
enumerate before (or use an intermediate anonymous type and enumerate on that one)
.ToList()
.Select(x => new FundedCount(
x.Key.BuyerId,
x.Count() / 30 * daysInMonth))
.ToList();
Solution2
add a parameterless ctor to your FundedCount class (if it's possible)
public FundedCount() {}
and use
.Select(x => new FundedCount{
<Property1> = x.Key.BuyerId,
<Property2> = x.Count() / 30 * daysInMonth
})
.ToList();
It's complaining because it can't convert references to FundedCount to SQL statements.
All LINQ providers convert LINQ statements and expressions to operations that their target can understand. LINQ to SQL and LINQ to EF will convert LINQ to SQL, PLINQ will convert it to Tasks and parallel operations, LINQ to Sharepoint will convert it to CAML etc.
What happens if they can't do the conversion, depends on the provider. Some providers will return intermediate results and convert the rest of the query to a LINQ to Objects query. Others will simply fail with an error message.
Failing with a message is actually a better option when talking to a database. Otherwise the server would have to return all columns to the client when only 1 or 2 would be actually necessary.
In your case you should modify your select to return an anonymous type with the data you want, call ToList() and THEN create the FundedCount objects, eg:
.Select( x=> new {Id=x.Key.BuyerId,Count=x.Count()/30 * daysInMonth)
.ToList()
.Select(y => new FundedCount(y.Id,y.Count))
.ToList();
The first ToList() will force the generation of the SQL statement and execute the query that will return only the data you need. The rest of the query is actually Linq to Objects and will get the data and create the final objects
I had the same exception in GroupBy. I found that the exception "Only parameterless constructors and initializers are supported in LINQ to Entities" is not 100% accurate description.
I had a GroupBy() in my "Linq to EntityFramework query" which used a struct as a Key in GroupBy. That did not work. When I changed that struct to normal class everything worked fine.
Code sample
var affectedRegistrationsGrouped = await db.Registrations
.Include(r => r.Person)
.Where(r =>
//whatever
)
.GroupBy(r => new GroupByKey
{
EventId = r.EventId,
SportId = r.SportId.Value
})
.ToListAsync();
...
...
// this does not work
private struct GroupByKey() {...}
// this works fine
private class GroupByKey() {...}
I have a query over a single entity (with some navigation properties) and need to project them into a model for consumption by API clients. It looks basically like this:
repository.CreateQuery<Reviews>()
.Where(/* criteria */)
.Select(m => new
{
ID = m.ID,
Reviewers = m.IsAnonymous
? m.Reviewers.Take(1).Select(r => new { Name = "Anonymous" })
: m.Reviewers.Select(r => new { Name = r.Name })
})
LINQ to Entities fails to execute this at runtime. In the Visual Studio debugger, the exception message is "Specified method is not supported". In LinqPad, the inner exception is "The nested query is not supported. Operation1='Case' Operations2='Collect'".
Any ideas how to work around this? I'd rather not force the query to execute to get the objects in memory because the point of this conditional query is to solve a performance issue, so as far as I can tell I really need to solve this within the scope of the L2E query.
Update: the projected types aren't anonymous in my real application. I just used anonymous ones here because it was convenient to contrive the example. In similar fashion, the real queries involve a number of additional properties which mean a great deal more data coming from the mapped tables.
You can use union to do what you want:
var query1 = repository.CreateQuery<Reviews>()
.Where(/* criteria */);
var queryAnonimous = query1.Where(m=>m.IsAnonymous)
.Select(m => new
{
ID = m.ID,
Reviewers = m.Reviewers.Take(1).Select(r => new { Name = "Anonymous" })
})
var queryNotAnonymous = query1.Where(m=>!m.IsAnonymous)
.Select(m => new
{
ID = m.ID,
Reviewers = m.Reviewers.Select(r => new { Name = r.Name })
})
var unionQuery = queryAnonimous.union(queryNotAnonymous);