Develop Reference

Using ternary conditional operator or expression in Entity Framework

I have an entity framework query that I inherited which includes several Sums, cutdown example:-
from c in db.Clients
where {where clauses}
select new
{
ClientID = c.ClientID,
ClientName = c.ClientName,
CurrentBalance = c.ClientTransactions.Select(ct => ct.Amount)
.DefaultIfEmpty(0m).Sum(),
}).ToList();
As the number of clients and the number of transactions has grown, this query has obviously become slower and slower.
Ideally I'd want to store balances rather than calculate them every time, but currently the system doesn't do that, and it would be a very large change to implement, so for now I'm just attempting a band-aid fix.
The fix I'm attempting to implement is to simply not do the Sum calculations (there are several, example above just has one) for people that aren't interested in them.
My first attempt was simply to use ternary conditional operators to determine whether or not to do the calculation:-
from c in db.Clients
where {where clauses}
select new
{
ClientID = c.ClientID,
ClientName = c.ClientName,
CurrentBalance = ClientSearchExcludeCurrentBalance ? 0m :
c.ClientTransactions.Select(ct => ct.fAmount).DefaultIfEmpty(0m).Sum(),
}).ToList();
The problem with this, it turns out, is that regardless of the value of the condition (ClientSearchExcludeCurrentBalance) both sides are still calculated, and then the ternary decides which one to use. So even setting the condition to false, the Sum still gets processed and the query takes too long.
Commenting out the sum, as below...
from c in db.Clients
where {where clauses}
select new
{
ClientID = c.ClientID,
ClientName = c.ClientName,
CurrentBalance = ClientSearchExcludeCurrentBalance ? 0m : 0m,
//c.ClientTransactions.Select(ct => ct.fAmount).DefaultIfEmpty(0m).Sum(),
}).ToList();
... is now nice and fast, so the ternary is definitely running it even when it's not used.
So, with that idea out the window, I tried using an expression instead:-
Expression<Func<Client, Decimal>> currentBalance = c => 0m;
if (!ClientSearchExcludeCurrentBalance)
{
currentBalance = c => c.ClientTransactions
.Select(ct => ct.Amount).DefaultIfEmpty(0m).Sum();
}
from c in db.Clients
where {where clauses}
select new
{
ClientID = c.ClientID,
ClientName = c.ClientName,
CurrentBalance = currentBalance.Invoke(c),
}).ToList();
This fell over with an unknown expression error:-
LINQ to Entities does not recognize the method 'System.Decimal Invoke[Client,Decimal](System.Linq.Expressions.Expression`1[System.Func`2[RPM.DAO.UI.Client,System.Decimal]], RPM.DAO.UI.Client)' method, and this method cannot be translated into a store expression
I also tried using Expand()
CurrentBalance = currentBalance.Expand().Invoke(c)
but still got the unknown expression error.
Just to see, I tried it with defaulting the Sum values to 0m, and then in the loop that assigns the results to the DTO Collection doing the sum there if needed
foreach (var client in Clients)
{
if (!ClientSearchExcludeCurrentBalance) {
var c = db.Clients.FirstOrDefault(cl => cl.ClientID == client.ClientID);
client.CurrentBalance = c.ClientTransactions.Select(ct => ct.fAmount)
.DefaultIfEmpty(0m).Sum();
}
}
This works, in that it only does the sum if told to, but doing it outside the main select means the entire query now takes twice as long as it used to, so is clearly not viable.
So, my questions are:-
Does anyone know if it's possible to make Entity Framework only run the parts of a ternary conditional operator that will be used?
Does anyone know if it's possible to use an Expression to return a value in Entity Framework?
Or, alternatively, how to add an IF statement into an Entity Framework query?
For (non-working) example:-
from c in db.Clients
where {where clauses}
select new
{
ClientID = c.ClientID,
ClientName = c.ClientName,
CurrentBalance = if (ClientSearchExcludeCurrentBalance)
return 0m;
else
return c.ClientTransactions.Select(tf => tf.fAmount)
.DefaultIfEmpty(0m).Sum(),
}).ToList();
Thanks!
Edit:
I tried Barr J's solution:-
from c in db.Clients
let currentBalance = ClientSearchExcludeCurrentBalance ? 0m :
c.ClientTransactions.Select(ct => ct.Amount).DefaultIfEmpty(0m).Sum()
where {where clauses}
select new
{
ClientID = c.ClientID,
ClientName = c.ClientName,
CurrentBalance = currentBalance
}).ToList();
I get a null reference exception:
System.NullReferenceException: 'Object reference not set to an instance of an object.'
Edit #2: The cut-down version above doesn't give the null exception error, but the full version (with identical code) does... weird!
Anyway, with the working cut-down version above I tried it with the setting set to true and fall, and both took the same time, so it still does the Sum evaluation either way

Linq will evaluate the operand from both sides regardless of the ternary operator, because it is being evaluated at run-time.
You will have to evaluate the operands outside of your linq statement and then use it.
for example:
var tst = from p in products join i in info on p.id equals i.pid
let status = p.type = "home" ? homestatus.Select(s=>s.status) :
p.type = "offshore" ? offshorestatus.Select(s=>s.status) :
p.type = "internal" ? internalestatus.Select(s=>s.status) : null
select new {
name = p.name,
status = status != null ? status.StatusText : string.Empty;
}
or:
var tst = from p in products join i in info on p.id equals i.pid
let status = (p.type = "home" ? homestatus.Select(s=>s.status.StatusText) :
p.type = "offshore" ? offshorestatus.Select(s=>s.status.StatusText) :
p.type = "internal" ? internalestatus.Select(s=>s.status.StatusText) : null) ?? string.Empty
select new {
name = p.name,
status = status;
}

first of all: #Barr 's answer ist not correct. Not the evaluation at runtime is the problem (in the end, it is not evaluated at all for Linq To Entities!), but what the underlying provider of Linq2Entities tries to do:
Run though the whole expresion tree and build some valid SQL out of it. And of course, Find a SQL equivalent of "Invoke". Well there is nothing it can use so it throws the exception LINQ to Entities does not recognize the method
You have to avoid everything within such linq2entity statements that MUST be evaluated at runtime. E.g. access to DateTimeOffset.Now will also not work.
Currently I am not able to test your query so I can not tell you why the ternary operator does not work as expected. It may depend on how the SQL looks like.
I can give you two advices:
take a look at the query outcome. To do this, install SQL profiler (distributed with SQL Server installation), debug into your application until your linq2entities statement is executed. Hopefully you know that this will not happen unless you call ToList(), Any(), First() or something else onto the query.
If you have no profiler, you should also be able to store the whole linq query in a variable (without calling toList()) and call ToString() onto it. This should you also give the query.
Have you thought about checking the execution plan of the query? This sounds like a missing index on ClientIdof the table Transaction. Maybe you can provide us with the SQL statement and/or the execution plan, so we will be able to help you more.
Additional hint: After retriving the query, you can execute it in SQL Management Studio. Please let you show the real execution plan. If you do this and there are some indices missing and SQL Server detects this missing index, it will suggest you what you can do to speed up the query.

Error converting SQL with join to LINQ

I have a SQL query that I'm trying to convert to LINQ and am having trouble understanding the obscure error messages when the query is enumerated.
The SQL query (which works as intended), is:
select a.TestGuid, MIN(a.StartTime) as StartTime, COUNT(b.TestCaseId) as NumTests, COUNT(DINSTINCT a.Id) as NumScenarios
from LoadTestSummary as a
join LoadTestTestSummaryData as b
on a.LoadTestRunid = b.LoadTestRunId
where
a.TargetStack = env and
a.TestGuid IS NOT NULL AND
a.StartTime IS NOT NULL AND
a.LoadTestRunId IS NOT NULL
group by a.TestGuid
Converting to LINQ, I get the following:
var q = from a in _context.LoadTestSummary
where
a.TargetStack == env &&
a.TestGuid != null &&
a.StartTime != null &&
a.LoadTestRunId != null
join b in _context.LoadTestTestSummaryData on new
{
LoadTestRunId = Convert.ToInt32(a.LoadTestRunId)
} equals new
{
LoadTestRunId = b.LoadTestRunId
}
group new { a, b } by new
{
a.TestGuid
}
into g
select new
{
DateCreated = g.Min(p => p.a.StartTime),
NumScenarios = g.Count(),
TestGuid = g.Key.TestGuid
NumTests = // ???
};
Two problems I have:
1) When the query is enumerated I get a run-time error that I'm having trouble deciphering. The query works fine in Linqpad, but gives me a run-time error in my program. I am not sure what would cause this. Just staring at this makes my head hurt:
ArgumentException: Expression of type 'System.Func``2[Microsoft.Data.Entity.Query.EntityQueryModelVisitor+TransparentIdentifier``2[PerfPortal.Models.LoadTestSummary,PerfPortal.Models.LoadTestTestSummaryData],<>f__AnonymousType7``1[System.String]]' cannot be used for parameter of type 'System.Func``2[<>f__AnonymousType5``2[PerfPortal.Models.LoadTestSummary,PerfPortal.Models.LoadTestTestSummaryData],<>f__AnonymousType7``1[System.String]]' of method 'System.Collections.Generic.IEnumerable``1[System.Linq.IGrouping``2[<>f__AnonymousType7``1[System.String],<>f__AnonymousType5``2[PerfPortal.Models.LoadTestSummary,PerfPortal.Models.LoadTestTestSummaryData]]] _GroupBy[<>f__AnonymousType5``2,<>f__AnonymousType7``1,<>f__AnonymousType5``2](System.Collections.Generic.IEnumerable``1[<>f__AnonymousType5``2[PerfPortal.Models.LoadTestSummary,PerfPortal.Models.LoadTestTestSummaryData]], System.Func``2[<>f__AnonymousType5``2[PerfPortal.Models.LoadTestSummary,PerfPortal.Models.LoadTestTestSummaryData],<>f__AnonymousType7``1[System.String]], System.Func``2[<>f__AnonymousType5``2[PerfPortal.Models.LoadTestSummary,PerfPortal.Models.LoadTestTestSummaryData],<>f__AnonymousType5``2[PerfPortal.Models.LoadTestSummary,PerfPortal.Models.LoadTestTestSummaryData]])'
2) I am not quite sure how to get the COUNT(DISTINCT a.Id) into the NumTests field. It looks like this isn't supported in LINQ but it looks like other people have asked this question to so I may be able to figure it out once #1 is resolved.
Any thoughts on what's wrong here? I am not even sure exactly what the error is telling me.
All help is appreciated!

Looking just at the SQL query and your LINQ code, I came up with something like this:
from a in LoadTestSummary
join b in LoadTestTestSummaryData
on a.LoadTestRunId equals b.LoadTestRunId
where
a.TargetStack == env &&
a.TestGuid != null &&
a.StartTime != null &&
a.LoadTestRunId != null
group new { a, b } by a.TestGuid into g
select new
{
TestGuid = g.Key,
DateCreated = g.Min(el => el.a.StartTime),
NumTests = g.Select(el => el.b.TestCaseId).Count(),
NumScenarios = g.Select(el => el.a.Id).Distinct().Count()
};
Note, that you don't need to convert LoadTestRunId to int, you may just use standard string comparision.
That horrendous error is most likely caused by grouping and comparing using anonimous objects, thou I prefer not to read that error too much as it's an eldritch abomination not ment to be seen nor comprehend by mere mortals, it seems.

comparison operator not supported for type int[] - linq to sql

Here is the problematic line:
var originalSummaryCandidates =
(from a in masterDB.tbl_thirty_second_summaries_multi_variant_associations
join d in masterDB.tbl_thirty_second_summaries_multi_variants on a.ThirtySecSummaryId equals d.ThirtySecondSummaryId_this
where d.DrugId == drugId &&
variantGenotypeIds.Contains(new int[] {a.VariantId, a.GenotypeId})
select d.ThirtySecondSummaryId_this)
.Distinct()
.ToList();
variantGeotpeIds is of type List<int[]>. Both a.VariantId and a.GenotypeId are of type int.
I cannot figure out why it why it will not do the comparison. Is this a deferred execution issue? It doesn't seem like it should be...
Thanks in advance.

List<T>.Contains only takes a single parameter of type T. In your case, T is Int32 but you're passing in a Int32[].
If you want to check that both values are in the list, you have to break the calls apart:
where d.DrugId == drugId &&
variantGenotypeIds.Contains(a.VariantId) &&
variantGenotypeIds.Contains(a.GenotypeId)
EDIT
If variantGenotypeIds is actually a List<Int32[]>, then there's another issue. LINQ to SQL will try to convert your query into its SQL equivalent. In this case, there's no way to translate your query into SQL so LINQ to SQL will throw an Exception.
If you really need to query this way, you'll have to read the records into memory first and then query using LINQ to Objects (which may or may not be a big deal depending on how many rows you are reading):
var query =
from a in masterDB.tbl_thirty_second_summaries_multi_variant_associations
join d in masterDB.tbl_thirty_second_summaries_multi_variants
on a.ThirtySecSummaryId equals d.ThirtySecondSummaryId_this
where d.DrugId == drugId
select new { a, d }
var originalSummaryCandidates =
(from q in query.AsEnumerable()
where variantGenotypeIds.Contains(new [] { q.a.VariantId, q.a.GenotypeId})
select d.ThirtySecondSummaryId_this)
.Distinct()
.ToList();

Array comparison uses reference equality by default. It's possible that linq-to-sql just tries to translate that into SQL that compares the values, but you'd have to look at the generated SQL to be sure. Another option would be to use Any instead:
where d.DrugId == drugId &&
variantGenotypeIds.Any(v => v[0] == a.VariantId && v[1] == a.GenotypeId)
but I'm not sure if Linq-to-Sql will be able to translate that to the correct SQL either. Another option would be to project the List` to a > and then do a string comparison:
variantGenotypeStrings = variantGenotypeIds.Select(v => string.Format("{0}|{1}", v[0],v[1]);
var originalSummaryCandidates =
(from a in masterDB.tbl_thirty_second_summaries_multi_variant_associations
join d in masterDB.tbl_thirty_second_summaries_multi_variants on a.ThirtySecSummaryId equals d.ThirtySecondSummaryId_this
where d.DrugId == drugId &&
variantGenotypeStrings.Contains(string.Format("{0}|{1}", a.VariantId, a.GenotypeId))
select d.ThirtySecondSummaryId_this)
.Distinct()
.ToList();

The cast to value type 'Int32' failed because the materialized value is null

I have the following code. I'm getting error:
"The cast to value type 'Int32' failed because the materialized value is null. Either the result type's generic parameter or the query must use a nullable type."
when CreditHistory table has no records.
var creditsSum = (from u in context.User
join ch in context.CreditHistory on u.ID equals ch.UserID
where u.ID == userID
select ch.Amount).Sum();
How can I modify the query to accept null values?

A linq-to-sql query isn't executed as code, but rather translated into SQL. Sometimes this is a "leaky abstraction" that yields unexpected behaviour.
One such case is null handling, where there can be unexpected nulls in different places. ...DefaultIfEmpty(0).Sum(0) can help in this (quite simple) case, where there might be no elements and sql's SUM returns null whereas c# expect 0.
A more general approach is to use ?? which will be translated to COALESCE whenever there is a risk that the generated SQL returns an unexpected null:
var creditsSum = (from u in context.User
join ch in context.CreditHistory on u.ID equals ch.UserID
where u.ID == userID
select (int?)ch.Amount).Sum() ?? 0;
This first casts to int? to tell the C# compiler that this expression can indeed return null, even though Sum() returns an int. Then we use the normal ?? operator to handle the null case.
Based on this answer, I wrote a blog post with details for both LINQ to SQL and LINQ to Entities.

To allow a nullable Amount field, just use the null coalescing operator to convert nulls to 0.
var creditsSum = (from u in context.User
join ch in context.CreditHistory on u.ID equals ch.UserID
where u.ID == userID
select ch.Amount ?? 0).Sum();

Had this error message when I was trying to select from a view.
The problem was the view recently had gained some new null rows (in SubscriberId column), and it had not been updated in EDMX (EF database first).
The column had to be Nullable type for it to work.
var dealer = Context.Dealers.Where(x => x.dealerCode == dealerCode).FirstOrDefault();
Before view refresh:
public int SubscriberId { get; set; }
After view refresh:
public Nullable<int> SubscriberId { get; set; }
Deleting and adding the view back in EDMX worked.
Hope it helps someone.

I have used this code and it responds correctly, only the output value is nullable.
var packesCount = await botContext.Sales.Where(s => s.CustomerId == cust.CustomerId && s.Validated)
.SumAsync(s => (int?)s.PackesCount);
if(packesCount != null)
{
// your code
}
else
{
// your code
}

You are using aggregate function which not getting the items to perform action , you must verify linq query is giving some result as below:
var maxOrderLevel =sdv.Any()? sdv.Max(s => s.nOrderLevel):0

I see that this question is already answered. But if you want it to be split into two statements, following may be considered.
var credits = from u in context.User
join ch in context.CreditHistory
on u.ID equals ch.UserID
where u.ID == userID
select ch;
var creditSum= credits.Sum(x => (int?)x.Amount) ?? 0;

Got this error in Entity Framework 6 with this code at runtime:
var fileEventsSum = db.ImportInformations.Sum(x => x.FileEvents)
Update from LeandroSoares:
Use this for single execution:
var fileEventsSum = db.ImportInformations.Sum(x => (int?)x.FileEvents) ?? 0
Original:
Changed to this and then it worked:
var fileEventsSum = db.ImportInformations.Any() ? db.ImportInformations.Sum(x => x.FileEvents) : 0;

I was also facing the same problem and solved through making column as nullable using "?" operator.
Sequnce = db.mstquestionbanks.Where(x => x.IsDeleted == false && x.OrignalFormID == OriginalFormIDint).Select(x=><b>(int?)x.Sequence</b>).Max().ToString();
Sometimes null is returned.

"IN" Operator in Linq

I am trying to convert an old raw Sql query in Linq with Entity Framework here.
It was using the IN operator with a collection of items. The query was something like that:
SELECT Members.Name
FROM Members
WHERE Members.ID IN ( SELECT DISTINCT ManufacturerID FROM Products WHERE Active = 1)
ORDER BY Members.Name ASC
Since the return of the subquery is not a single string but a collection of strings I can't use the String.Contains() method.
I thought about doing something like :
var activeProducts = (
from products in db.ProductSet
where product.Active == true
select product.ManufacturerID);
and then
var activeMembers = (
from member in db.ContactSet
where member.ID.ToString().Contains(activeProducts));
but it stops at the contains saying it has invalid arguments ... I can't select activeProducts.ManufacturerID because obviously the proprety is not there since it returns an IQueryable...
Bottom line what I'm trying to do here is to return a list of members who have at least one active product.
Any hint ?
[edit]
Here's the full query code ... I tried with the contains on the second expression, Linq didn't seem to like it :
Server Error in '/' Application.
LINQ to Entities does not recognize the method 'Boolean Contains[String](System.Linq.IQueryable``1[System.String], System.String)' method, and this method cannot be translated into a store expression.
var activeProduct =(from product in Master.DataContext.ProductSet
where product.Active == true
&& product.ShowOnWebSite == true
&& product.AvailableDate <= DateTime.Today
&& ( product.DiscontinuationDate == null || product.DiscontinuationDate >= DateTime.Today )
select product.ManufacturerID.ToString() );
var activeArtists = from artist in Master.DataContext.ContactSet
where activeProduct.Contains(artist.ID.ToString())
select artist;
NumberOfArtists = activeArtists.Count();
artistsRepeater.DataSource = activeArtists;
artistsRepeater.DataBind();
[More details]
ManufacturerID is a nullable GUID apparently...
For some reason the ContactSet class do not contain any reference to the products I guess I will have to do a join query, no clues here.

var activeMembers = (
from member in db.ContactSet
where activeProducts.Select(x=>x.ID).Contains(member.ID));

Try where activeProducts.Contains(member.ID).
EDIT: Did you try it without any ToStrings?

You can do it in one query:
var q = from member in db.ContactSet
where member.Products.Any(p => p.IsActive)
select member;

Try the solution posted by Colin Meek at: http://social.msdn.microsoft.com/forums/en-US/adodotnetentityframework/thread/095745fe-dcf0-4142-b684-b7e4a1ab59f0/. It worked for me.

What about this:
from m in members
where products.FirstOrDefault(prod => prod.IsActive == 1 && prod.Id == m.Id) != null
select m;
you can chain any number of conditions required in the where clause using &&
Ash..

from m in members
where products.Any(p => p.Active && p.ManufacturerID == m.ID)
select m
or
from m in members
join p in products on m.ID equals p.ManufacturerID
where p.Active
select m

Instead of this:
var activeMembers = (
from member in db.ContactSet
where member.ID.ToString().Contains(activeProducts));
Try this:
var activeMembers = (
from member in db.ContactSet
where activeProducts.Contains(member.ID));

What if you swap the statement (untested)?
where activeProducts.Contains(member.ID)

How about this...
var activeProducts = (
from products in db.ProductSet
where product.Active == true
select product.ManufacturerID);
var activeMembers = (
from member in db.ContactSet
where activeProducts.Contains(member.ID.ToString()));

A helper or extension method will work fine when querying against objects in memory. But against an SQL database, your LINQ code will be compiled into an expression tree, analysed and translated into an SQL command. This functionality has no concept of custom-made extension methods or methods of other objects like .Contains(...).
It could be easily implemented into the standard LINQ-To-SQL functionality by Microsoft though. But as long as they don't want, we're helpless as long it's not an open source functionality.
All you can do is create your own QueryProvider that goes against an SQL database. But it will be hard and it would be only for that one in feature alone that you're missing.
However, if you really wanna go that route, have fun: LINQ: BUILDING AN IQUERYABLE PROVIDER SERIES

Finally I managed to code something really ugly, but that actually works! (lol)
var activeProduct =(from product in Master.DataContext.ProductSet
where product.Active == true
&& product.ShowOnWebSite == true
&& product.AvailableDate <= DateTime.Today
&& ( product.DiscontinuationDate == null || product.DiscontinuationDate >= DateTime.Today )
select product.ManufacturerID ).Distinct();
var artists = from artist in Master.DataContext.ContactSet
select artist;
List<Evolution.API.Contact> activeArtists = new List<Evolution.API.Contact>();
foreach (var artist in artists)
{
foreach(var product in activeProduct)
{
if (product.HasValue && product.Value == artist.ID)
activeArtists.Add(artist);
}
}
NumberOfArtists = activeArtists.Count();
artistsRepeater.DataSource = activeArtists;
artistsRepeater.DataBind();

I have already posted about the same at
http://www.codeproject.com/Tips/336253/Filtering-records-from-List-based-similar-to-Sql-I

var q = (from p in db.DOCAuditTrails
where p.ActionUser == "MyUserID"
&& p.ActionTaken == "Not Actioned"
&& p.ActionDate > DateTime.Parse("2011-09-13")
select p.RequisitionId).Distinct();
var DocAuditResults = db.DOCAuditTrails.Where(p
=> q.ToArray().Contains(p.RequisitionId));

Without know the exact mappings it is hard to tell what can be done and what can't. I will assume that there isn't any casting involved. Firstly you have to remember that everything in the Linq Expression tree must have an equivalent in SQL. As some others have noted, you have a object.ToString() in your Linq Statements.
However it seems that what people have neglected to mention is that you have TWO usages of object.ToSting(), both of which must be removed.
I would also make an extra variable to change the closure's capture type to be explicitly of DataContext (since the Linq statement is like a lambda, and delayed evaluated. It will need to take the whole of the Master variable. Earlier I stated that everything in your Linq must have an equivalent in SQL. Given that Master can't possibly exist in SQL, there is no DataContext property/column/mapping for the type of Master).
var context = Master.DataContext;
var activeProduct = from product in context.ProductSet
where product.Active == true
&& product.ShowOnWebSite == true
&& product.AvailableDate <= DateTime.Today
&& ( product.DiscontinuationDate == null || product.DiscontinuationDate >= DateTime.Today )
select product.ManufacturerID;
var activeArtists = from artist in context.ContactSet
where activeProduct.Contains(artist.ID)
select artist;
I hope the above changes work for you.
In many cases issues with Linq to ORMs can be traced back to your Linq Expression capturing a non primative (DateTime, int, string etc) and non ORM based class (DataContext/EntityObject etc). The other major gotcha is usage of functions and operators that aren't exposed by the ORM (it is possible to map user defined functions to .net function through the ORM, but I would not recommend it due to indexing issues).