Related
given a list of ids, I can query all relevant rows by:
context.Table.Where(q => listOfIds.Contains(q.Id));
But how do you achieve the same functionality when the Table has a composite key?
This is a nasty problem for which I don't know any elegant solution.
Suppose you have these key combinations, and you only want to select the marked ones (*).
Id1 Id2
--- ---
1 2 *
1 3
1 6
2 2 *
2 3 *
... (many more)
How to do this is a way that Entity Framework is happy? Let's look at some possible solutions and see if they're any good.
Solution 1: Join (or Contains) with pairs
The best solution would be to create a list of the pairs you want, for instance Tuples, (List<Tuple<int,int>>) and join the database data with this list:
from entity in db.Table // db is a DbContext
join pair in Tuples on new { entity.Id1, entity.Id2 }
equals new { Id1 = pair.Item1, Id2 = pair.Item2 }
select entity
In LINQ to objects this would be perfect, but, too bad, EF will throw an exception like
Unable to create a constant value of type 'System.Tuple`2 (...) Only primitive types or enumeration types are supported in this context.
which is a rather clumsy way to tell you that it can't translate this statement into SQL, because Tuples is not a list of primitive values (like int or string). For the same reason a similar statement using Contains (or any other LINQ statement) would fail.
Solution 2: In-memory
Of course we could turn the problem into simple LINQ to objects like so:
from entity in db.Table.AsEnumerable() // fetch db.Table into memory first
join pair Tuples on new { entity.Id1, entity.Id2 }
equals new { Id1 = pair.Item1, Id2 = pair.Item2 }
select entity
Needless to say that this is not a good solution. db.Table could contain millions of records.
Solution 3: Two Contains statements (incorrect)
So let's offer EF two lists of primitive values, [1,2] for Id1 and [2,3] for Id2. We don't want to use join, so let's use Contains:
from entity in db.Table
where ids1.Contains(entity.Id1) && ids2.Contains(entity.Id2)
select entity
But now the results also contains entity {1,3}! Well, of course, this entity perfectly matches the two predicates. But let's keep in mind that we're getting closer. In stead of pulling millions of entities into memory, we now only get four of them.
Solution 4: One Contains with computed values
Solution 3 failed because the two separate Contains statements don't only filter the combinations of their values. What if we create a list of combinations first and try to match these combinations? We know from solution 1 that this list should contain primitive values. For instance:
var computed = ids1.Zip(ids2, (i1,i2) => i1 * i2); // [2,6]
and the LINQ statement:
from entity in db.Table
where computed.Contains(entity.Id1 * entity.Id2)
select entity
There are some problems with this approach. First, you'll see that this also returns entity {1,6}. The combination function (a*b) does not produce values that uniquely identify a pair in the database. Now we could create a list of strings like ["Id1=1,Id2=2","Id1=2,Id2=3]" and do
from entity in db.Table
where computed.Contains("Id1=" + entity.Id1 + "," + "Id2=" + entity.Id2)
select entity
(This would work in EF6, not in earlier versions).
This is getting pretty messy. But a more important problem is that this solution is not sargable, which means: it bypasses any database indexes on Id1 and Id2 that could have been used otherwise. This will perform very very poorly.
Solution 5: Best of 2 and 3
So the most viable solution I can think of is a combination of Contains and a join in memory: First do the contains statement as in solution 3. Remember, it got us very close to what we wanted. Then refine the query result by joining the result as an in-memory list:
var rawSelection = from entity in db.Table
where ids1.Contains(entity.Id1) && ids2.Contains(entity.Id2)
select entity;
var refined = from entity in rawSelection.AsEnumerable()
join pair in Tuples on new { entity.Id1, entity.Id2 }
equals new { Id1 = pair.Item1, Id2 = pair.Item2 }
select entity;
It's not elegant, messy all the same maybe, but so far it's the only scalable1 solution to this problem I found, and applied in my own code.
Solution 6: Build a query with OR clauses
Using a Predicate builder like Linqkit or alternatives, you can build a query that contains an OR clause for each element in the list of combinations. This could be a viable option for really short lists. With a couple of hundreds of elements, the query will start performing very poorly. So I don't consider this a good solution unless you can be 100% sure that there will always be a small number of elements. One elaboration of this option can be found here.
Solution 7: Unions
There's also a solution using UNIONs that I posted later here.
1As far as the Contains statement is scalable: Scalable Contains method for LINQ against a SQL backend
Solution for Entity Framework Core with SQL Server
🎉 NEW! QueryableValues EF6 Edition has arrived!
The following solution makes use of QueryableValues. This is a library that I wrote to primarily solve the problem of query plan cache pollution in SQL Server caused by queries that compose local values using the Contains LINQ method. It also allows you to compose values of complex types in your queries in a performant way, which will achieve what's being asked in this question.
First you will need to install and set up the library, after doing that you can use any of the following patterns that will allow you to query your entities using a composite key:
// Required to make the AsQueryableValues method available on the DbContext.
using BlazarTech.QueryableValues;
// Local data that will be used to query by the composite key
// of the fictitious OrderProduct table.
var values = new[]
{
new { OrderId = 1, ProductId = 10 },
new { OrderId = 2, ProductId = 20 },
new { OrderId = 3, ProductId = 30 }
};
// Optional helper variable (needed by the second example due to CS0854)
var queryableValues = dbContext.AsQueryableValues(values);
// Example 1 - Using a Join (preferred).
var example1Results = dbContext
.OrderProduct
.Join(
queryableValues,
e => new { e.OrderId, e.ProductId },
v => new { v.OrderId, v.ProductId },
(e, v) => e
)
.ToList();
// Example 2 - Using Any (similar behavior as Contains).
var example2Results = dbContext
.OrderProduct
.Where(e => queryableValues
.Where(v =>
v.OrderId == e.OrderId &&
v.ProductId == e.ProductId
)
.Any()
)
.ToList();
Useful Links
Nuget Package
GitHub Repository
Benchmarks
QueryableValues is distributed under the MIT license.
You can use Union for each composite primary key:
var compositeKeys = new List<CK>
{
new CK { id1 = 1, id2 = 2 },
new CK { id1 = 1, id2 = 3 },
new CK { id1 = 2, id2 = 4 }
};
IQuerable<CK> query = null;
foreach(var ck in compositeKeys)
{
var temp = context.Table.Where(x => x.id1 == ck.id1 && x.id2 == ck.id2);
query = query == null ? temp : query.Union(temp);
}
var result = query.ToList();
You can create a collection of strings with both keys like this (I am assuming that your keys are int type):
var id1id2Strings = listOfIds.Select(p => p.Id1+ "-" + p.Id2);
Then you can just use "Contains" on your db:
using (dbEntities context = new dbEntities())
{
var rec = await context.Table1.Where(entity => id1id2Strings .Contains(entity.Id1+ "-" + entity.Id2));
return rec.ToList();
}
You need a set of objects representing the keys you want to query.
class Key
{
int Id1 {get;set;}
int Id2 {get;set;}
If you have two lists and you simply check that each value appears in their respective list then you are getting the cartesian product of the lists - which is likely not what you want. Instead you need to query the specific combinations required
List<Key> keys = // get keys;
context.Table.Where(q => keys.Any(k => k.Id1 == q.Id1 && k.Id2 == q.Id2));
I'm not completely sure that this is valid use of Entity Framework; you may have issues with sending the Key type to the database. If that happens then you can be creative:
var composites = keys.Select(k => p1 * k.Id1 + p2 * k.Id2).ToList();
context.Table.Where(q => composites.Contains(p1 * q.Id1 + p2 * q.Id2));
You can create an isomorphic function (prime numbers are good for this), something like a hashcode, which you can use to compare the pair of values. As long as the multiplicative factors are co-prime this pattern will be isomorphic (one-to-one) - i.e. the result of p1*Id1 + p2*Id2 will uniquely identify the values of Id1 and Id2 as long as the prime numbers are correctly chosen.
But then you end up in a situation where you're implementing complex concepts and someone is going to have to support this. Probably better to write a stored procedure which takes the valid key objects.
Ran into this issue as well and needed a solution that both did not perform a table scan and also provided exact matches.
This can be achieved by combining Solution 3 and Solution 4 from Gert Arnold's Answer
var firstIds = results.Select(r => r.FirstId);
var secondIds = results.Select(r => r.SecondId);
var compositeIds = results.Select(r => $"{r.FirstId}:{r.SecondId}");
var query = from e in dbContext.Table
//first check the indexes to avoid a table scan
where firstIds.Contains(e.FirstId) && secondIds.Contains(e.SecondId))
//then compare the compositeId for an exact match
//ToString() must be called unless using EF Core 5+
where compositeIds.Contains(e.FirstId.ToString() + ":" + e.SecondId.ToString()))
select e;
var entities = await query.ToListAsync();
For EF Core I use a slightly modified version of the bucketized IN method by EricEJ to map composite keys as tuples. It performs pretty well for small sets of data.
Sample usage
List<(int Id, int Id2)> listOfIds = ...
context.Table.In(listOfIds, q => q.Id, q => q.Id2);
Implementation
public static IQueryable<TQuery> In<TKey1, TKey2, TQuery>(
this IQueryable<TQuery> queryable,
IEnumerable<(TKey1, TKey2)> values,
Expression<Func<TQuery, TKey1>> key1Selector,
Expression<Func<TQuery, TKey2>> key2Selector)
{
if (values is null)
{
throw new ArgumentNullException(nameof(values));
}
if (key1Selector is null)
{
throw new ArgumentNullException(nameof(key1Selector));
}
if (key2Selector is null)
{
throw new ArgumentNullException(nameof(key2Selector));
}
if (!values.Any())
{
return queryable.Take(0);
}
var distinctValues = Bucketize(values);
if (distinctValues.Length > 1024)
{
throw new ArgumentException("Too many parameters for SQL Server, reduce the number of parameters", nameof(values));
}
var predicates = distinctValues
.Select(v =>
{
// Create an expression that captures the variable so EF can turn this into a parameterized SQL query
Expression<Func<TKey1>> value1AsExpression = () => v.Item1;
Expression<Func<TKey2>> value2AsExpression = () => v.Item2;
var firstEqual = Expression.Equal(key1Selector.Body, value1AsExpression.Body);
var visitor = new ReplaceParameterVisitor(key2Selector.Parameters[0], key1Selector.Parameters[0]);
var secondEqual = Expression.Equal(visitor.Visit(key2Selector.Body), value2AsExpression.Body);
return Expression.AndAlso(firstEqual, secondEqual);
})
.ToList();
while (predicates.Count > 1)
{
predicates = PairWise(predicates).Select(p => Expression.OrElse(p.Item1, p.Item2)).ToList();
}
var body = predicates.Single();
var clause = Expression.Lambda<Func<TQuery, bool>>(body, key1Selector.Parameters[0]);
return queryable.Where(clause);
}
class ReplaceParameterVisitor : ExpressionVisitor
{
private ParameterExpression _oldParameter;
private ParameterExpression _newParameter;
public ReplaceParameterVisitor(ParameterExpression oldParameter, ParameterExpression newParameter)
{
_oldParameter = oldParameter;
_newParameter = newParameter;
}
protected override Expression VisitParameter(ParameterExpression node)
{
if (ReferenceEquals(node, _oldParameter))
return _newParameter;
return base.VisitParameter(node);
}
}
/// <summary>
/// Break a list of items tuples of pairs.
/// </summary>
private static IEnumerable<(T, T)> PairWise<T>(this IEnumerable<T> source)
{
var sourceEnumerator = source.GetEnumerator();
while (sourceEnumerator.MoveNext())
{
var a = sourceEnumerator.Current;
sourceEnumerator.MoveNext();
var b = sourceEnumerator.Current;
yield return (a, b);
}
}
private static TKey[] Bucketize<TKey>(IEnumerable<TKey> values)
{
var distinctValueList = values.Distinct().ToList();
// Calculate bucket size as 1,2,4,8,16,32,64,...
var bucket = 1;
while (distinctValueList.Count > bucket)
{
bucket *= 2;
}
// Fill all slots.
var lastValue = distinctValueList.Last();
for (var index = distinctValueList.Count; index < bucket; index++)
{
distinctValueList.Add(lastValue);
}
var distinctValues = distinctValueList.ToArray();
return distinctValues;
}
In the absence of a general solution, I think there are two things to consider:
Avoid multi-column primary keys (will make unit testing easier too).
But if you have to, chances are that one of them will reduce the
query result size to O(n) where n is the size of the ideal query
result. From here, its Solution 5 from Gerd Arnold above.
For example, the problem leading me to this question was querying order lines, where the key is order id + order line number + order type, and the source had the order type being implicit. That is, the order type was a constant, order ID would reduce the query set to order lines of relevant orders, and there would usually be 5 or less of these per order.
To rephrase: If you have a composite key, changes are that one of them have very few duplicates. Apply Solution 5 from above with that.
I tried this solution and it worked with me and the output query was perfect without any parameters
using LinqKit; // nuget
var customField_Ids = customFields?.Select(t => new CustomFieldKey { Id = t.Id, TicketId = t.TicketId }).ToList();
var uniqueIds1 = customField_Ids.Select(cf => cf.Id).Distinct().ToList();
var uniqueIds2 = customField_Ids.Select(cf => cf.TicketId).Distinct().ToList();
var predicate = PredicateBuilder.New<CustomFieldKey>(false); //LinqKit
var lambdas = new List<Expression<Func<CustomFieldKey, bool>>>();
foreach (var cfKey in customField_Ids)
{
var id = uniqueIds1.Where(uid => uid == cfKey.Id).Take(1).ToList();
var ticketId = uniqueIds2.Where(uid => uid == cfKey.TicketId).Take(1).ToList();
lambdas.Add(t => id.Contains(t.Id) && ticketId.Contains(t.TicketId));
}
predicate = AggregateExtensions.AggregateBalanced(lambdas.ToArray(), (expr1, expr2) =>
{
var invokedExpr = Expression.Invoke(expr2, expr1.Parameters.Cast<Expression>());
return Expression.Lambda<Func<CustomFieldKey, bool>>
(Expression.OrElse(expr1.Body, invokedExpr), expr1.Parameters);
});
var modifiedCustomField_Ids = repository.GetTable<CustomFieldLocal>()
.Select(cf => new CustomFieldKey() { Id = cf.Id, TicketId = cf.TicketId }).Where(predicate).ToArray();
I ended up writing a helper for this problem that relies on System.Linq.Dynamic.Core;
Its a lot of code and don't have time to refactor at the moment but input / suggestions appreciated.
public static IQueryable<TEntity> WhereIsOneOf<TEntity, TSource>(this IQueryable<TEntity> dbSet,
IEnumerable<TSource> source,
Expression<Func<TEntity, TSource,bool>> predicate) where TEntity : class
{
var (where, pDict) = GetEntityPredicate(predicate, source);
return dbSet.Where(where, pDict);
(string WhereStr, IDictionary<string, object> paramDict) GetEntityPredicate(Expression<Func<TEntity, TSource, bool>> func, IEnumerable<TSource> source)
{
var firstP = func.Parameters[0];
var binaryExpressions = RecurseBinaryExpressions((BinaryExpression)func.Body);
var i = 0;
var paramDict = new Dictionary<string, object>();
var res = new List<string>();
foreach (var sourceItem in source)
{
var innerRes = new List<string>();
foreach (var bExp in binaryExpressions)
{
var emp = ToEMemberPredicate(firstP, bExp);
var val = emp.GetKeyValue(sourceItem);
var pName = $"#{i++}";
paramDict.Add(pName, val);
var str = $"{emp.EntityMemberName} {emp.SQLOperator} {pName}";
innerRes.Add(str);
}
res.Add( "(" + string.Join(" and ", innerRes) + ")");
}
var sRes = string.Join(" || ", res);
return (sRes, paramDict);
}
EMemberPredicate ToEMemberPredicate(ParameterExpression firstP, BinaryExpression bExp)
{
var lMember = (MemberExpression)bExp.Left;
var rMember = (MemberExpression)bExp.Right;
var entityMember = lMember.Expression == firstP ? lMember : rMember;
var keyMember = entityMember == lMember ? rMember : lMember;
return new EMemberPredicate(entityMember, keyMember, bExp.NodeType);
}
List<BinaryExpression> RecurseBinaryExpressions(BinaryExpression e, List<BinaryExpression> runningList = null)
{
if (runningList == null) runningList = new List<BinaryExpression>();
if (e.Left is BinaryExpression lbe)
{
var additions = RecurseBinaryExpressions(lbe);
runningList.AddRange(additions);
}
if (e.Right is BinaryExpression rbe)
{
var additions = RecurseBinaryExpressions(rbe);
runningList.AddRange(additions);
}
if (e.Left is MemberExpression && e.Right is MemberExpression)
{
runningList.Add(e);
}
return runningList;
}
}
Helper class:
public class EMemberPredicate
{
public readonly MemberExpression EntityMember;
public readonly MemberExpression KeyMember;
public readonly PropertyInfo KeyMemberPropInfo;
public readonly string EntityMemberName;
public readonly string SQLOperator;
public EMemberPredicate(MemberExpression entityMember, MemberExpression keyMember, ExpressionType eType)
{
EntityMember = entityMember;
KeyMember = keyMember;
KeyMemberPropInfo = (PropertyInfo)keyMember.Member;
EntityMemberName = entityMember.Member.Name;
SQLOperator = BinaryExpressionToMSSQLOperator(eType);
}
public object GetKeyValue(object o)
{
return KeyMemberPropInfo.GetValue(o, null);
}
private string BinaryExpressionToMSSQLOperator(ExpressionType eType)
{
switch (eType)
{
case ExpressionType.Equal:
return "==";
case ExpressionType.GreaterThan:
return ">";
case ExpressionType.GreaterThanOrEqual:
return ">=";
case ExpressionType.LessThan:
return "<";
case ExpressionType.LessThanOrEqual:
return "<=";
case ExpressionType.NotEqual:
return "<>";
default:
throw new ArgumentException($"{eType} is not a handled Expression Type.");
}
}
}
Use Like so:
// This can be a Tuple or whatever.. If Tuple, then y below would be .Item1, etc.
// This data structure is up to you but is what I use.
[FromBody] List<CustomerAddressPk> cKeys
var res = await dbCtx.CustomerAddress
.WhereIsOneOf(cKeys, (x, y) => y.CustomerId == x.CustomerId
&& x.AddressId == y.AddressId)
.ToListAsync();
Hope this helps others.
in Case of composite key you can use another idlist and add a condition for that in your code
context.Table.Where(q => listOfIds.Contains(q.Id) && listOfIds2.Contains(q.Id2));
or you can use one another trick create a list of your keys by adding them
listofid.add(id+id1+......)
context.Table.Where(q => listOfIds.Contains(q.Id+q.id1+.......));
I tried this on EF Core 5.0.3 with the Postgres provider.
context.Table
.Select(entity => new
{
Entity = entity,
CompositeKey = entity.Id1 + entity.Id2,
})
.Where(x => compositeKeys.Contains(x.CompositeKey))
.Select(x => x.Entity);
This produced SQL like:
SELECT *
FROM table AS t
WHERE t.Id1 + t.Id2 IN (#__compositeKeys_0)),
Caveats
this should only be used where the combination of Id1 and Id2 will always produce a unique result (e.g., they're both UUIDs)
this cannot use indexes, though you could save the composite key to the db with an index
My users should be able to configure a filter to get a result from the database.
I decided to use an expression tree, to be flexible with the query. But I have problems with the expression tree for filtering by ReferenceCode.
Entities:
PumpStartUp
Duration (TimeSpan)
DutDbId (Guid/FK)
TimeStamp (DateTime)
DUT (DeviceUnderTest)
DeviceUnderTest
AdminId (string/unique)
DbId (Guid/PK)
ReferenceCode (string)
StartUps (List)
Here is a part of the filter in equivalent linq. But linq I can't use, because I don't know how many ReferenceCodes will be defined by the user.:
//-------------------reference code filter---------------------------
var query = context.PumpStartUps.Where((p => p.DUT.ReferenceCode == "HKR566" ||
p.DUT.ReferenceCode == "HSH967" ||
.
.
.));
startUps = query.ToList();
For filtering by DeviceUnderTest part of the filter, my solution is:
// --------------------duts filter-----------------------------------
Expression dutsExpression = null;
Expression psuExpression = Expression.Parameter(typeof(PumpStartUp), "psu");
Expression psuDutIdExpression = Expression.Property(psuExpression, "DutDbId");
foreach (var dut in filter.Duts)
{
DeviceUnderTest deviceUnderTest = context.DevicesUnderTest.Where(d => d.AdminId == dut.Id).Single();
Expression dutIdExpression = Expression.Constant(deviceUnderTest.DbId);
Expression dutExpression = Expression.Equal(pumpStartUpDutIdExpression, dutIdExpression);
if (dutsExpression == null)
{
dutsExpression = dutExpression;
}
else
{
dutsExpression = Expression.Or(dutsExpression, dutExpression);
}
}
How can I filter by ReferenceCode in that manner:
Use this:
var dutExpression = Expression.Property(psuExpression, "DUT");
var referenceCodeExp = = Expression.Property(dutExpression, "ReferenceCode ");
var constExpr = Expression.Constant("HKR566");
var eqExp = Expression.Equal(referenceCodeExp , constExpr);
dutsExpression = Expression.Or(dutsExpression, eqExp);
If you have a limited amount of codes, you can always say
var query = context.PumpStartUps.Where(p => codes.Contains(p.DUT.ReferenceCode))
This works up until about 2000 parameters. If you need more, then you should probably send the codes somehow to a temp table (or rather a function returning a table, since ef does not support temp tables), and join on that since constructing an expression with more than 2000 ors is not gonna perform well.
I have a paging API that returns rows a user requests, but only so many at one time, not the entire collection. The API works as designed, but I do have to calculate the total number of records that are available (for proper page calculations). Within the API, I use Linq2Sql and I work a lot with the IQueryable before i finally make my requests. When I go to get the count, I call something like: totalRecordCount = queryable.Count();
The resulting SQL is interesting none the less, but it also adds an unnecessary Order By which makes the query very expensive.
exec sp_executesql N'SELECT COUNT(*) AS [value]
FROM (
SELECT TOP (1) NULL AS [EMPTY]
FROM [dbo].[JournalEventsView] AS [t0]
WHERE [t0].[DataOwnerID] = #p0
ORDER BY [t0].[DataTimeStamp] DESC
) AS [t1]',N'#p0 int',#p0=1
Because I am using the IQueryable, I can manipulate the IQueryable prior to it making it to the SQL server.
My question is, if I already have an IQueryable with a OrderBy in it, is it possible to remove that OrderBy before I call the Count()?
like: totalRecordCount = queryable.NoOrder.Count();
If not, no biggie. I see many questions how to OrderBy, but not any involving removing an OrderBy from the Linq expression.
Thanks!
So, the below code is a spike against an in-memory array. There may be some hurdles to get this working with Entity Framework (or some other arbitrary IQueryProvider implementation). Basically, what we are going to do is visit the expression tree and look for any Ordering method call and simply remove it from the tree. Hope this points you in the right direction.
class Program
{
static void Main(string[] args)
{
var seq = new[] { 1, 3, 5, 7, 9, 2, 4, 6, 8 };
var query = seq.OrderBy(x => x);
Console.WriteLine("Print out in reverse order.");
foreach (var item in query)
{
Console.WriteLine(item);
}
Console.WriteLine("Prints out in original order");
var queryExpression = seq.AsQueryable().OrderBy(x => x).ThenByDescending(x => x).Expression;
var queryDelegate = Expression.Lambda<Func<IEnumerable<int>>>(new OrderByRemover().Visit(queryExpression)).Compile();
foreach (var item in queryDelegate())
{
Console.WriteLine(item);
}
Console.ReadLine();
}
}
public class OrderByRemover : ExpressionVisitor
{
protected override Expression VisitMethodCall(MethodCallExpression node)
{
if (node.Method.DeclaringType != typeof(Enumerable) && node.Method.DeclaringType != typeof(Queryable))
return base.VisitMethodCall(node);
if (node.Method.Name != "OrderBy" && node.Method.Name != "OrderByDescending" && node.Method.Name != "ThenBy" && node.Method.Name != "ThenByDescending")
return base.VisitMethodCall(node);
//eliminate the method call from the expression tree by returning the object of the call.
return base.Visit(node.Arguments[0]);
}
}
There isn't just an unneeded ORDER BY, there's also a spurious TOP(1).
SELECT TOP (1) NULL AS [EMPTY] ...
That subselect will only return 0 or 1 rows. In fact without the TOP there it wouldn't be legal to have an ORDER BY in a subselect.
The ORDER BY clause is invalid in views, inline functions, derived tables, subqueries, and common table expressions, unless TOP or FOR XML is also specified.: SELECT COUNT(*) FROM ( SELECT * FROM Table1 ORDER BY foo )
sqlfiddle
I think you have probably done something wrong in your LINQ. Are you sure you haven't written .Take(1) or similar somewhere in your query, before calling .Count()?
This is wrong:
IQueryable<Foo> foo = (...).OrderBy(x => x.Foo).Take(1);
int count = foo.Count();
You should do this instead:
IQueryable<Foo> foo = (...);
Iqueryable<Foo> topOne = foo.OrderBy(x => x.Foo).Take(1);
int count = foo.Count();
I am afraid there is no easy way to remove the OrderBy operator from queryable.
What you can do, however, is to re-create the IQueryable based on the new expression obtained from rewriting queryable.Expression(see here) omitting the OrderBy call.
If you can't eliminate the root cause, here is a workaround:
totalRecordCount = queryable.OrderBy(x => 0).Count();
SQL Server's query optimizer will remove this useless ordering. It won't have runtime cost.
I think you have implemented you paging code wrongly. You actually need to query the database twice, once for the paged datasource and once for the total row count. This is how the setup should look.
public IList<MyObj> GetPagedData(string filter, string sort, int skip, int take)
{
using(var db = new DataContext())
{
var q = GetDataInternal(db);
if(!String.IsNullOrEmpty(filter))
q = q.Where(filter); //Using Dynamic linq
if(!String.IsNullOrEmpty(sort))
q = q.OrderBy(sort); //And here
return q.Skip(skip).Take(take).ToList();
}
}
public int GetTotalCount(string filter)
{
using(var db = new DataContext())
{
var q = GetDataInternal(db);
if(!String.IsNullOrEmpty(filter))
q = q.Where(filter); //Using Dynamic linq
return q.Count(); //Without ordering and paging.
}
}
private static IQuerable<MyObj> GetDataInternal(DataContext db)
{
return
from x in db.JournalEventsView
where ...
select new ...;
}
The filtering and sorting is done using the Dynamic linq library
I know it is not quite what you are looking for, but index on [DataOwnerID] with inclusion of DataTimeStamp could make your query less expensive.
I know there are a lot of examples of this on the web, but I can't seem to get this to work.
Let me try to set this up, I have a list of custom objects that I need to have limited on a range of values.
I have a sort variable that changes based on some action on the UI, and I need to process the object differently based on that.
Here is my object:
MyObject.ID - Just an identifier
MyObject.Cost - The cost of the object.
MyObject.Name - The name of the object.
Now I need to filter this based on a range in the cost, so I will have something similar to this, considering that I could be limiting by Either of my bottom two properties.
var product = from mo in myobject
where mo.Cost <= 10000
or
var product = from mo in myobject
where mo.Name equals strName
Now I have the dynamic linq in my project, but I'm not figuring out how to get it to actually work, as when I do some of the examples I am only getting:
Func<Tsourse>bool> predicate
as an option.
Update:
I am trying to find a solution that helps me Objectify my code, as right now it is a lot of copy and paste for my linq queries.
Update 2:
Is there an obvious performance difference between:
var product = from mo in myobject
... a few joins ...
where mo.Cost <= 10000
and
var product = (from mo in myobject
... a few joins ...)
.AsQueryable()
.Where("Cost > 1000")
Maybe not directly answering your question, but DynamicQuery is unnecessary here. You can write this query as:
public IEnumerable<MyObject> GetMyObjects(int? maxCost, string name)
{
var query = context.MyObjects;
if (maxCost != null)
{
query = query.Where(mo => mo.Cost <= (int)maxCost);
}
if (!string.IsNullOrEmpty(name))
{
query = query.Where(mo => mo.Name == name);
}
return query;
}
If the conditions are mutually exclusive then just change the second if into an else if.
I use this pattern all the time. What "Dynamic Query" really means is combining pure SQL with Linq; it doesn't really help you that much with generating conditions on the fly.
using System.Linq;
var products = mo.Where(x => x.Name == "xyz");
var products = mo.Where(x => x.Cost <= 1000);
var products = mo.Where(x => x.Name == "xyz" || x.Cost <= 1000);
Read this great post on DLINQ by ScottGu
Dynamic LINQ (Part 1: Using the LINQ Dynamic Query Library)
You would need something like
var product = myobject.Where("Cost <= 10000");
var product = myobject.Where("Name = #0", strName);
If you downloaded the samples you need to find the Dynamic.cs file in the sample. You need to copy this file into your project and then add
using System.Linq.Dynamic; to the class you are trying to use Dynamic Linq in.
EDIT: To answer your edit. Yes, there is of course a performance difference. If you know the variations of filters beforehand then I would suggest writing them out without using DLINQ.
You can create your own Extension Method like so.
public static class FilterExtensions
{
public static IEnumerable<T> AddFilter<T,T1>(this IEnumerable<T> list, Func<T,T1, bool> filter, T1 argument )
{
return list.Where(foo => filter(foo, argument) );
}
}
Then create your filter methods.
public bool FilterById(Foo obj, int id)
{
return obj.id == id;
}
public bool FilterByName(Foo obj, string name)
{
return obj.name == name;
}
Now you can use this on an IEnumerable<Foo> very easily.
List<Foo> foos = new List<Foo>();
foos.Add(new Foo() { id = 1, name = "test" });
foos.Add(new Foo() { id = 1, name = "test1" });
foos.Add(new Foo() { id = 2, name = "test2" });
//Example 1
//get all Foos's by Id == 1
var list1 = foos.AddFilter(FilterById, 1);
//Example 2
//get all Foo's by name == "test1"
var list2 = foos.AddFilter(FilterByName, "test1");
//Example 3
//get all Foo's by Id and Name
var list1 = foos.AddFilter(FilterById, 1).AddFilter(FilterByName, "test1");
What is the best way to assemble a dynamic WHERE clause to a LINQ statement?
I have several dozen checkboxes on a form and am passing them back as: Dictionary<string, List<string>> (Dictionary<fieldName,List<values>>) to my LINQ query.
public IOrderedQueryable<ProductDetail> GetProductList(string productGroupName, string productTypeName, Dictionary<string,List<string>> filterDictionary)
{
var q = from c in db.ProductDetail
where c.ProductGroupName == productGroupName && c.ProductTypeName == productTypeName
// insert dynamic filter here
orderby c.ProductTypeName
select c;
return q;
}
(source: scottgu.com)
You need something like this? Use the Linq Dynamic Query Library (download includes examples).
Check out ScottGu's blog for more examples.
I have similar scenario where I need to add filters based on the user input and I chain the where clause.
Here is the sample code.
var votes = db.Votes.Where(r => r.SurveyID == surveyId);
if (fromDate != null)
{
votes = votes.Where(r => r.VoteDate.Value >= fromDate);
}
if (toDate != null)
{
votes = votes.Where(r => r.VoteDate.Value <= toDate);
}
votes = votes.Take(LimitRows).OrderByDescending(r => r.VoteDate);
You can also use the PredicateBuilder from LinqKit to chain multiple typesafe lambda expressions using Or or And.
http://www.albahari.com/nutshell/predicatebuilder.aspx
A simple Approach can be if your Columns are of Simple Type like String
public static IEnumerable<MyObject> WhereQuery(IEnumerable<MyObject> source, string columnName, string propertyValue)
{
return source.Where(m => { return m.GetType().GetProperty(columnName).GetValue(m, null).ToString().StartsWith(propertyValue); });
}
It seems much simpler and simpler to use the ternary operator to decide dynamically if a condition is included
List productList = new List();
productList =
db.ProductDetail.Where(p => p.ProductDetailID > 0 //Example prop
&& (String.IsNullOrEmpty(iproductGroupName) ? (true):(p.iproductGroupName.Equals(iproductGroupName)) ) //use ternary operator to make the condition dynamic
&& (ID == 0 ? (true) : (p.ID == IDParam))
).ToList();
I came up with a solution that even I can understand... by using the 'Contains' method you can chain as many WHERE's as you like. If the WHERE is an empty string, it's ignored (or evaluated as a select all). Here is my example of joining 2 tables in LINQ, applying multiple where clauses and populating a model class to be returned to the view. (this is a select all).
public ActionResult Index()
{
string AssetGroupCode = "";
string StatusCode = "";
string SearchString = "";
var mdl = from a in _db.Assets
join t in _db.Tags on a.ASSETID equals t.ASSETID
where a.ASSETGROUPCODE.Contains(AssetGroupCode)
&& a.STATUSCODE.Contains(StatusCode)
&& (
a.PO.Contains(SearchString)
|| a.MODEL.Contains(SearchString)
|| a.USERNAME.Contains(SearchString)
|| a.LOCATION.Contains(SearchString)
|| t.TAGNUMBER.Contains(SearchString)
|| t.SERIALNUMBER.Contains(SearchString)
)
select new AssetListView
{
AssetId = a.ASSETID,
TagId = t.TAGID,
PO = a.PO,
Model = a.MODEL,
UserName = a.USERNAME,
Location = a.LOCATION,
Tag = t.TAGNUMBER,
SerialNum = t.SERIALNUMBER
};
return View(mdl);
}
Just to share my idea for this case.
Another approach by solution is:
public IOrderedQueryable GetProductList(string productGroupName, string productTypeName, Dictionary> filterDictionary)
{
return db.ProductDetail
.where
(
p =>
(
(String.IsNullOrEmpty(productGroupName) || c.ProductGroupName.Contains(productGroupName))
&& (String.IsNullOrEmpty(productTypeName) || c.ProductTypeName.Contains(productTypeName))
// Apply similar logic to filterDictionary parameter here !!!
)
);
}
This approach is very flexible and allow with any parameter to be nullable.
You could use the Any() extension method. The following seems to work for me.
XStreamingElement root = new XStreamingElement("Results",
from el in StreamProductItem(file)
where fieldsToSearch.Any(s => el.Element(s) != null && el.Element(s).Value.Contains(searchTerm))
select fieldsToReturn.Select(r => (r == "product") ? el : el.Element(r))
);
Console.WriteLine(root.ToString());
Where 'fieldsToSearch' and 'fieldsToReturn' are both List objects.
This is the solution I came up with if anyone is interested.
https://kellyschronicles.wordpress.com/2017/12/16/dynamic-predicate-for-a-linq-query/
First we identify the single element type we need to use ( Of TRow As DataRow) and then identify the “source” we are using and tie the identifier to that source ((source As TypedTableBase(Of TRow)). Then we must specify the predicate, or the WHERE clause that is going to be passed (predicate As Func(Of TRow, Boolean)) which will either be returned as true or false. Then we identify how we want the returned information ordered (OrderByField As String). Our function will then return a EnumerableRowCollection(Of TRow), our collection of datarows that have met the conditions of our predicate(EnumerableRowCollection(Of TRow)). This is a basic example. Of course you must make sure your order field doesn’t contain nulls, or have handled that situation properly and make sure your column names (if you are using a strongly typed datasource never mind this, it will rename the columns for you) are standard.
System.Linq.Dynamic might help you build LINQ expressions at runtime.
The dynamic query library relies on a simple expression language for formulating expressions and queries in strings.
It provides you with string-based extension methods that you can pass any string expression into instead of using language operators or type-safe lambda extension methods.
It is simple and easy to use and is particularly useful in scenarios where queries are entirely dynamic, and you want to provide an end-user UI to help build them.
Source: Overview in Dynamic LINQ
The library lets you create LINQ expressions from plain strings, therefore, giving you the possibility to dynamically build a LINQ expression concatenating strings as you require.
Here's an example of what can be achieved:
var resultDynamic = context.Customers
.Where("City == #0 and Age > #1", "Paris", 50)
.ToList();