Develop Reference

Entity Framework "SELECT IN" not using parameters

Why is Entity Framework putting literal values in generated SQL rather than using parameters when using a "SELECT IN":
using (var context = new TestContext())
{
var values = new int[] { 1, 2, 3 };
var query = context.Things.Where(x => values.Contains(x.Id));
Console.WriteLine(query.ToString());
}
This produces the following SQL:
SELECT
[Extent1].[Id] AS [Id]
FROM [dbo].[PaymentConfigurations] AS [Extent1]
WHERE [Extent1].[Id] IN (1, 2, 3)
I see a lot of cached query plans in SQL Server. Is there anyway of making EF to put parameters rather than harcoded values, or activating parameter sniffing is the only option?
It happens also in EF Core.

I can't say why EF (Core) designers decided to use constants instead of variables when translating Enumerable.Contains. As #Gert Arnold pointed out in the comments, could be related to SQL query parameter count limit.
Interestingly, both EF (6.2) and EF Core (2.1.2) generate IN with parameters when you use the equivalent || expression like:
var values = new int[] { 1, 2, 3 };
var value0 = values[0];
var value1 = values[1];
var value2 = values[2];
var query = context.Things.Where(x =>
x.Id == value0 ||
x.Id == value1 ||
x.Id == value2);
EF6.2 generated query is
SELECT
[Extent1].[Id] AS [Id]
FROM [dbo].[Things] AS [Extent1]
WHERE [Extent1].[Id] IN (#p__linq__0,#p__linq__1,#p__linq__2)
EF Core 2.1 does something similar.
So the solution is to convert the Contains expression to || based expression. It has to be dynamically using Expression class methods. And to make it easier to use, could be encapsulated in a custom extension method, which internally user ExpressionVisitor to perform the conversion.
Something like this:
public static partial class EfQueryableExtensions
{
public static IQueryable<T> Parameterize<T>(this IQueryable<T> source)
{
var expression = new ContainsConverter().Visit(source.Expression);
if (expression == source) return source;
return source.Provider.CreateQuery<T>(expression);
}
class ContainsConverter : ExpressionVisitor
{
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.DeclaringType == typeof(Enumerable) &&
node.Method.Name == nameof(Enumerable.Contains) &&
node.Arguments.Count == 2 &&
CanEvaluate(node.Arguments[0]))
{
var values = Expression.Lambda<Func<IEnumerable>>(node.Arguments[0]).Compile().Invoke();
var left = Visit(node.Arguments[1]);
Expression result = null;
foreach (var value in values)
{
// var variable = new Tuple<TValue>(value);
var variable = Activator.CreateInstance(typeof(Tuple<>).MakeGenericType(left.Type), value);
// var right = variable.Item1;
var right = Expression.Property(Expression.Constant(variable), nameof(Tuple<int>.Item1));
var match = Expression.Equal(left, right);
result = result != null ? Expression.OrElse(result, match) : match;
}
return result ?? Expression.Constant(false);
}
return base.VisitMethodCall(node);
}
static bool CanEvaluate(Expression e)
{
if (e == null) return true;
if (e.NodeType == ExpressionType.Convert)
return CanEvaluate(((UnaryExpression)e).Operand);
if (e.NodeType == ExpressionType.MemberAccess)
return CanEvaluate(((MemberExpression)e).Expression);
return e.NodeType == ExpressionType.Constant;
}
}
}
Applying it to the sample query
var values = new int[] { 1, 2, 3 };
var query = context.Things
.Where(x => values.Contains(x.Id))
.Parameterize();
produces the desired translation.

Parameterised queries with IN can be done, though it's a little round-about. You will need to use a direct SQL query, and generate the parameterised SQL manually, something like the following:
var values = new object[] { 1, 2, 3 };
var idx = 0;
var query = context.Things.SqlQuery($#"
SELECT
[Extent1].[Id] AS [Id]
FROM [dbo].[PaymentConfigurations] AS [Extent1]
WHERE [Extent1].[Id] IN ({string.Join(",", values.Select(i => $"#p{idx++}"))})",
values);
The generated list of parameter names are directly embedded into the SQL used in the query, and are provided values by the values parameter. Note that you may need to ensure that your values array is an object[] and not int[] to ensure it's unpacked into the SqlQuery params. This approach is not so easy to maintain as a LINQ query, however sometimes we have to make these compromises for the sake of efficiency.

Nullable filters in XRM where clause

I'm using the XRM (early bound) types in a WCF project so I have access to the CRM model and can use LINQ queries. But I've been running into a problem described here, it's the limitations on the where clause specific to XRM LINQ:
where [clause limitations]
The left side of the clause must be an attribute name and the right side of
the clause must be a value. You cannot set the left side to a constant. Both
the sides of the clause cannot be constants.
Supports the String functions Contains, StartsWith, EndsWith, and Equals.
One requirement that keeps popping up is when a parameter is null, all entities should be returned otherwise filter by the parameter. But I can't think of a way to do this without breaking the requirements above, or writing multiple queries to handle the scenario when it's null.
this is an example of one of my queries, the typeFilter == null is the problem here is I've used a constant on the LHS. In my real code there's a guard clause that points typeFilter == null to another query but I now have to add a start/end date filter (both nullable) and I cannot express how much I don't want to write a query for every combination of nullables.
private IQueryable<EventInfo> getAllEvents( DataContext context, EventType? typeFilter )
{
return (
from evt in context.new_eventSet
where
( evt.statecode == new_eventState.Active ) &&
( typeFilter == null || evt.new_EventType.Value == (int) typeFilter.Value )
select new EventInfo()
{
ID = evt.Id,
EventType = (EventType) evt.new_EventType.Value
...
} );
}

How about:
if (typeFilter == null)
{
return (
from evt in context.new_eventSet
where
( evt.statecode == new_eventState.Active )
select new EventInfo()
{
ID = evt.Id,
EventType = (EventType) evt.new_EventType.Value
...
} );
}
else
{
return (
from evt in context.new_eventSet
where
( evt.statecode == new_eventState.Active ) &&
evt.new_EventType.Value == (int) typeFilter.Value )
select new EventInfo()
{
ID = evt.Id,
EventType = (EventType) evt.new_EventType.Value
...
} );
}

I've answered my own question! sometimes you just need a place to vent your problems before you get it.
the trick was to not use LINQ syntax:
private IQueryable<EventInfo> getAllEvents( DataContext context, EventType? typeFilter, DateTime? startDateFilter, DateTime? endDateFilter )
{
var result = context.new_eventSet
// active records
.Where( evt => evt.statecode == new_eventState.Active )
// is publish-able
.Where( ..etc.. );
if ( typeFilter != null )
{
// filter by type
result = result.Where( evt => evt.new_EventType.Value == (int) typeFilter.Value );
}
if( startDateFilter != null)
{
// filter by startDate
result = result.Where(evt => evt.new_StartDate > startDateFilter.Value);
}
if( endDateFilter != null)
{
// filter by endDate
result = result.Where(evt => evt.new_StartDate < endDateFilter.Value);
}
return result.Select( evt => new EventInfo()
{
ID = evt.Id,
EventType = (EventType) evt.new_EventType.Value,
...
} );
}

If you want to use the Linq syntax, it is possible to construct a query dynamically using LinqKit.
I have used it for this purpose on the Dynamics CRM project which I'm currently working on, and it does the job very well.
Please refer to the following answer, which is where I got the idea: https://stackoverflow.com/a/5152946/344988

linq function to handle : both fields equal or both null or empty

Hi I have a linq query where I compare an object with all other entity stored in my database. Firstname and lastname are mandatory fields so I don't have to check for null. But in the case of street I have to do these checks.
I want to match if both fields are null or empty strings or that they are the same. The below linq query is working fine.
But I was wandering isn't there any way to make it more readable. For example with a custom function like Bool FieldsAreEqualOrBothNullOrEmpty(r.street, request.street)
Can't figure how to do that and if it's possible.
var same =
from r in db.Requests
where r.firstName.ToLower() == request.firstName.ToLower()
&& r.lastName.ToLower() == request.lastName.ToLower()
//Seems long to just compare two fields
&& ( string.IsNullOrEmpty(r.street) && string.IsNullOrEmpty(request.street) )
|| r.street.ToLower() == request.street.ToLower()
select r;
return same;

I would simplify:
var same =
from r in db.Requests
where r.firstName.ToLower() == request.firstName.ToLower()
&& r.lastName.ToLower() == request.lastName.ToLower()
select r;
if(string.IsNullOrEmpty(request.street)) {
same = same.Where(r => string.IsNullOrEmpty(r.street));
} else {
string street = request.street.ToLower();
same = same.Where(r => r.street.ToLower() == street);
}
The nice thing about the this is that it keeps the query simple in each case.
You could use similar logic for the first two if you choose, and it could also be moved to an expression-based utility method. Untested:
public static IQueryable<T> FieldsAreEqualOrBothNullOrEmpty<T>(
this IQueryable<T> source,
Expression<Func<T, string>> member, string value)
{
Expression body;
if (string.IsNullOrEmpty(value))
{
body = Expression.Call(
typeof(string), "IsNullOrEmpty", null, member.Body);
}
else
{
body = Expression.Equal(
Expression.Call(member.Body, "ToLower", null),
Expression.Constant(value.ToLower(), typeof(string)));
}
return source.Where(Expression.Lambda<Func<T,bool>>(
body, member.Parameters));
}
then:
var same = db.Requests
.FieldsAreEqualOrBothNullOrEmpty(x => x.firstName, request.firstName)
.FieldsAreEqualOrBothNullOrEmpty(x => x.lastName, request.lastName)
.FieldsAreEqualOrBothNullOrEmpty(x => x.street, request.street);

You can just call String.Equals but ignore case, using an appropriate StringComparison.
string.equals(r.street, request.street, StringComparison.OrdinalIgnoreCase);
It's bad practice to convert strings to lower-case for comparison purposes, as it won't reliably work in all languages, and is slower than using a comparison which ignores case. See, e.g., this answer

How about creating extension method on string IsEqualOrBothNullOrEmpty -
public static class Extensions
{
public static bool IsEqualOrBothNullOrEmpty(this string firstValue,
string secondValue)
{
return string.IsNullOrEmpty(firstValue) &&
string.IsNullOrEmpty(secondValue)
|| firstValue.ToLower() == secondValue.ToLower();
}
}
And use in your query like this -
var same =
from r in db.Requests
where r.firstName.ToLower() == request.firstName.ToLower()
&& r.lastName.ToLower() == request.lastName.ToLower()
&& r.street.IsEqualOrBothNullOrEmpty(request.Street)
select r;
return same;

Linq to SQL Strange SQL Translation

I have a simple query that is generating some odd SQL translations, which is blowing up my code when the object is saturated.
from x in DataContext.MyEntities
select new
{
IsTypeCDA = x.EntityType == "CDA"
}
I would expect this query should translate to:
SELECT (CASE WHEN [t0].[EntityType] = #p1 THEN 1 ELSE 0 END) as [IsTypeCDA]
...
Instead I get this :
SELECT
(CASE
WHEN #p1 = [t0].[EntityType] THEN 1
WHEN NOT (#p1 = [t0].[EntityType]) THEN 0
ELSE NULL
END) AS [IsTypeCDA]
...
Since I'm saturating a POCO where IsTypeCDA is a bool, it blows up stating I can't assign null to bool.
Any thoughts?
Edit: fixed the property names so they make sense...

from x in DataContext.MyEntities
select new
{
IsTypeCDA = x.EntityType == null
}
c# interpretation (false) or sql interpretation (null)?
This runs in sql so sql interpretation. That's why the funky translation - the operation does return a nullable bool.
Use this query to punt that nullable bool into a plain old bool.
from x in DataContext.MyEntities
select new
{
IsTypeCDA = ((bool?)(x.EntityType == "CDA")) ?? false
}

Linq to SQL does the "strange" comparison because database values can be NULL. Unfortunately, it doesn't seem to do well translating the tri-valued comparison result. At first blush, I'd wonder why any of your entities have NULL for their EntityType. Modifying your schema to disallow NULL values would be the most straightforward solution.
However, assuming the table is defined this way for legitimate business reasons you can work around it in a couple ways.
First off, you could coalesce the NULLs into a placeholder value.
from x in DataContext.MyEntities
select new
{
IsTypeCDA = (x.EntityType ?? "") == "CDA"
}
Alternately, using String.Equals generates a more thorough comparison that handles NULLs.
from x in DataContext.MyEntities
select new
{
IsTypeCDA = string.Equals(x.EntityType, "CDA")
}
Neither of these will generate the simple SQL you were expecting, but they'll both get the job done.

I would have IsTypeCDA as a get only property and select into that class:
public class SomeName
{
public string EntityType { get; set; }
public bool IsTypeCDA { get { return EntityType == EntityType.CDA; } }
}
...
from x in DataContext.MyEntities
select new SomeName
{
EntityType = x.EntityType
}

Compare nullable types in Linq to Sql

I have a Category entity which has a Nullable ParentId field. When the method below is executing and the categoryId is null, the result seems null however there are categories which has null ParentId value.
What is the problem in here, what am I missing?
public IEnumerable<ICategory> GetSubCategories(long? categoryId)
{
var subCategories = this.Repository.Categories.Where(c => c.ParentId == categoryId)
.ToList().Cast<ICategory>();
return subCategories;
}
By the way, when I change the condition to (c.ParentId == null), result seems normal.

Other way:
Where object.Equals(c.ParentId, categoryId)
or
Where (categoryId == null ? c.ParentId == null : c.ParentId == categoryId)

The first thing to do is to put on logging, to see what TSQL was generated; for example:
ctx.Log = Console.Out;
LINQ-to-SQL seems to treat nulls a little inconsistently (depending on literal vs value):
using(var ctx = new DataClasses2DataContext())
{
ctx.Log = Console.Out;
int? mgr = (int?)null; // redundant int? for comparison...
// 23 rows:
var bosses1 = ctx.Employees.Where(x => x.ReportsTo == (int?)null).ToList();
// 0 rows:
var bosses2 = ctx.Employees.Where(x => x.ReportsTo == mgr).ToList();
}
So all I can suggest is use the top form with nulls!
i.e.
Expression<Func<Category,bool>> predicate;
if(categoryId == null) {
predicate = c=>c.ParentId == null;
} else {
predicate = c=>c.ParentId == categoryId;
}
var subCategories = this.Repository.Categories
.Where(predicate).ToList().Cast<ICategory>();
Update - I got it working "properly" using a custom Expression:
static void Main()
{
ShowEmps(29); // 4 rows
ShowEmps(null); // 23 rows
}
static void ShowEmps(int? manager)
{
using (var ctx = new DataClasses2DataContext())
{
ctx.Log = Console.Out;
var emps = ctx.Employees.Where(x => x.ReportsTo, manager).ToList();
Console.WriteLine(emps.Count);
}
}
static IQueryable<T> Where<T, TValue>(
this IQueryable<T> source,
Expression<Func<T, TValue?>> selector,
TValue? value) where TValue : struct
{
var param = Expression.Parameter(typeof (T), "x");
var member = Expression.Invoke(selector, param);
var body = Expression.Equal(
member, Expression.Constant(value, typeof (TValue?)));
var lambda = Expression.Lambda<Func<T,bool>>(body, param);
return source.Where(lambda);
}

My guess is that it's due to a rather common attribute of DBMS's - Just because two things are both null does not mean they are equal.
To elaborate a bit, try executing these two queries:
SELECT * FROM TABLE WHERE field = NULL
SELECT * FROM TABLE WHERE field IS NULL
The reason for the "IS NULL" construct is that in the DBMS world, NULL != NULL since the meaning of NULL is that the value is undefined. Since NULL means undefined, you can't say that two null values are equal, since by definition you don't know what they are.
When you explicitly check for "field == NULL", LINQ probably converts that to "field IS NULL". But when you use a variable, I'm guessing that LINQ doesn't automatically do that conversion.
Here's an MSDN forum post with more info about this issue.
Looks like a good "cheat" is to change your lambda to look like this:
c => c.ParentId.Equals(categoryId)

You need to use operator Equals:
var subCategories = this.Repository.Categories.Where(c => c.ParentId.Equals(categoryId))
.ToList().Cast<ICategory>();
Equals fot nullable types returns true if:
The HasValue property is false, and the other parameter is null. That is, two null values are equal by definition.
The HasValue property is true, and the value returned by the Value property is equal to the other parameter.
and returns false if:
The HasValue property for the current Nullable structure is true, and the other parameter is null.
The HasValue property for the current Nullable structure is false, and the other parameter is not null.
The HasValue property for the current Nullable structure is true, and the value returned by the Value property is not equal to the other parameter.
More info here Nullable<.T>.Equals Method

Or you can simply use this. It will also translate to a nicer sql query
Where((!categoryId.hasValue && !c.ParentId.HasValue) || c.ParentId == categoryId)

What about something simpler like this?
public IEnumerable<ICategory> GetSubCategories(long? categoryId)
{
var subCategories = this.Repository.Categories.Where(c => (!categoryId.HasValue && c.ParentId == null) || c.ParentId == categoryId)
.ToList().Cast<ICategory>();
return subCategories;
}

Linq to Entities supports Null Coelescing (??) so just convert the null on the fly to a default value.
Where(c => c.ParentId == categoryId ?? 0)