Related
given a list of ids, I can query all relevant rows by:
context.Table.Where(q => listOfIds.Contains(q.Id));
But how do you achieve the same functionality when the Table has a composite key?
This is a nasty problem for which I don't know any elegant solution.
Suppose you have these key combinations, and you only want to select the marked ones (*).
Id1 Id2
--- ---
1 2 *
1 3
1 6
2 2 *
2 3 *
... (many more)
How to do this is a way that Entity Framework is happy? Let's look at some possible solutions and see if they're any good.
Solution 1: Join (or Contains) with pairs
The best solution would be to create a list of the pairs you want, for instance Tuples, (List<Tuple<int,int>>) and join the database data with this list:
from entity in db.Table // db is a DbContext
join pair in Tuples on new { entity.Id1, entity.Id2 }
equals new { Id1 = pair.Item1, Id2 = pair.Item2 }
select entity
In LINQ to objects this would be perfect, but, too bad, EF will throw an exception like
Unable to create a constant value of type 'System.Tuple`2 (...) Only primitive types or enumeration types are supported in this context.
which is a rather clumsy way to tell you that it can't translate this statement into SQL, because Tuples is not a list of primitive values (like int or string). For the same reason a similar statement using Contains (or any other LINQ statement) would fail.
Solution 2: In-memory
Of course we could turn the problem into simple LINQ to objects like so:
from entity in db.Table.AsEnumerable() // fetch db.Table into memory first
join pair Tuples on new { entity.Id1, entity.Id2 }
equals new { Id1 = pair.Item1, Id2 = pair.Item2 }
select entity
Needless to say that this is not a good solution. db.Table could contain millions of records.
Solution 3: Two Contains statements (incorrect)
So let's offer EF two lists of primitive values, [1,2] for Id1 and [2,3] for Id2. We don't want to use join, so let's use Contains:
from entity in db.Table
where ids1.Contains(entity.Id1) && ids2.Contains(entity.Id2)
select entity
But now the results also contains entity {1,3}! Well, of course, this entity perfectly matches the two predicates. But let's keep in mind that we're getting closer. In stead of pulling millions of entities into memory, we now only get four of them.
Solution 4: One Contains with computed values
Solution 3 failed because the two separate Contains statements don't only filter the combinations of their values. What if we create a list of combinations first and try to match these combinations? We know from solution 1 that this list should contain primitive values. For instance:
var computed = ids1.Zip(ids2, (i1,i2) => i1 * i2); // [2,6]
and the LINQ statement:
from entity in db.Table
where computed.Contains(entity.Id1 * entity.Id2)
select entity
There are some problems with this approach. First, you'll see that this also returns entity {1,6}. The combination function (a*b) does not produce values that uniquely identify a pair in the database. Now we could create a list of strings like ["Id1=1,Id2=2","Id1=2,Id2=3]" and do
from entity in db.Table
where computed.Contains("Id1=" + entity.Id1 + "," + "Id2=" + entity.Id2)
select entity
(This would work in EF6, not in earlier versions).
This is getting pretty messy. But a more important problem is that this solution is not sargable, which means: it bypasses any database indexes on Id1 and Id2 that could have been used otherwise. This will perform very very poorly.
Solution 5: Best of 2 and 3
So the most viable solution I can think of is a combination of Contains and a join in memory: First do the contains statement as in solution 3. Remember, it got us very close to what we wanted. Then refine the query result by joining the result as an in-memory list:
var rawSelection = from entity in db.Table
where ids1.Contains(entity.Id1) && ids2.Contains(entity.Id2)
select entity;
var refined = from entity in rawSelection.AsEnumerable()
join pair in Tuples on new { entity.Id1, entity.Id2 }
equals new { Id1 = pair.Item1, Id2 = pair.Item2 }
select entity;
It's not elegant, messy all the same maybe, but so far it's the only scalable1 solution to this problem I found, and applied in my own code.
Solution 6: Build a query with OR clauses
Using a Predicate builder like Linqkit or alternatives, you can build a query that contains an OR clause for each element in the list of combinations. This could be a viable option for really short lists. With a couple of hundreds of elements, the query will start performing very poorly. So I don't consider this a good solution unless you can be 100% sure that there will always be a small number of elements. One elaboration of this option can be found here.
Solution 7: Unions
There's also a solution using UNIONs that I posted later here.
1As far as the Contains statement is scalable: Scalable Contains method for LINQ against a SQL backend
Solution for Entity Framework Core with SQL Server
🎉 NEW! QueryableValues EF6 Edition has arrived!
The following solution makes use of QueryableValues. This is a library that I wrote to primarily solve the problem of query plan cache pollution in SQL Server caused by queries that compose local values using the Contains LINQ method. It also allows you to compose values of complex types in your queries in a performant way, which will achieve what's being asked in this question.
First you will need to install and set up the library, after doing that you can use any of the following patterns that will allow you to query your entities using a composite key:
// Required to make the AsQueryableValues method available on the DbContext.
using BlazarTech.QueryableValues;
// Local data that will be used to query by the composite key
// of the fictitious OrderProduct table.
var values = new[]
{
new { OrderId = 1, ProductId = 10 },
new { OrderId = 2, ProductId = 20 },
new { OrderId = 3, ProductId = 30 }
};
// Optional helper variable (needed by the second example due to CS0854)
var queryableValues = dbContext.AsQueryableValues(values);
// Example 1 - Using a Join (preferred).
var example1Results = dbContext
.OrderProduct
.Join(
queryableValues,
e => new { e.OrderId, e.ProductId },
v => new { v.OrderId, v.ProductId },
(e, v) => e
)
.ToList();
// Example 2 - Using Any (similar behavior as Contains).
var example2Results = dbContext
.OrderProduct
.Where(e => queryableValues
.Where(v =>
v.OrderId == e.OrderId &&
v.ProductId == e.ProductId
)
.Any()
)
.ToList();
Useful Links
Nuget Package
GitHub Repository
Benchmarks
QueryableValues is distributed under the MIT license.
You can use Union for each composite primary key:
var compositeKeys = new List<CK>
{
new CK { id1 = 1, id2 = 2 },
new CK { id1 = 1, id2 = 3 },
new CK { id1 = 2, id2 = 4 }
};
IQuerable<CK> query = null;
foreach(var ck in compositeKeys)
{
var temp = context.Table.Where(x => x.id1 == ck.id1 && x.id2 == ck.id2);
query = query == null ? temp : query.Union(temp);
}
var result = query.ToList();
You can create a collection of strings with both keys like this (I am assuming that your keys are int type):
var id1id2Strings = listOfIds.Select(p => p.Id1+ "-" + p.Id2);
Then you can just use "Contains" on your db:
using (dbEntities context = new dbEntities())
{
var rec = await context.Table1.Where(entity => id1id2Strings .Contains(entity.Id1+ "-" + entity.Id2));
return rec.ToList();
}
You need a set of objects representing the keys you want to query.
class Key
{
int Id1 {get;set;}
int Id2 {get;set;}
If you have two lists and you simply check that each value appears in their respective list then you are getting the cartesian product of the lists - which is likely not what you want. Instead you need to query the specific combinations required
List<Key> keys = // get keys;
context.Table.Where(q => keys.Any(k => k.Id1 == q.Id1 && k.Id2 == q.Id2));
I'm not completely sure that this is valid use of Entity Framework; you may have issues with sending the Key type to the database. If that happens then you can be creative:
var composites = keys.Select(k => p1 * k.Id1 + p2 * k.Id2).ToList();
context.Table.Where(q => composites.Contains(p1 * q.Id1 + p2 * q.Id2));
You can create an isomorphic function (prime numbers are good for this), something like a hashcode, which you can use to compare the pair of values. As long as the multiplicative factors are co-prime this pattern will be isomorphic (one-to-one) - i.e. the result of p1*Id1 + p2*Id2 will uniquely identify the values of Id1 and Id2 as long as the prime numbers are correctly chosen.
But then you end up in a situation where you're implementing complex concepts and someone is going to have to support this. Probably better to write a stored procedure which takes the valid key objects.
Ran into this issue as well and needed a solution that both did not perform a table scan and also provided exact matches.
This can be achieved by combining Solution 3 and Solution 4 from Gert Arnold's Answer
var firstIds = results.Select(r => r.FirstId);
var secondIds = results.Select(r => r.SecondId);
var compositeIds = results.Select(r => $"{r.FirstId}:{r.SecondId}");
var query = from e in dbContext.Table
//first check the indexes to avoid a table scan
where firstIds.Contains(e.FirstId) && secondIds.Contains(e.SecondId))
//then compare the compositeId for an exact match
//ToString() must be called unless using EF Core 5+
where compositeIds.Contains(e.FirstId.ToString() + ":" + e.SecondId.ToString()))
select e;
var entities = await query.ToListAsync();
For EF Core I use a slightly modified version of the bucketized IN method by EricEJ to map composite keys as tuples. It performs pretty well for small sets of data.
Sample usage
List<(int Id, int Id2)> listOfIds = ...
context.Table.In(listOfIds, q => q.Id, q => q.Id2);
Implementation
public static IQueryable<TQuery> In<TKey1, TKey2, TQuery>(
this IQueryable<TQuery> queryable,
IEnumerable<(TKey1, TKey2)> values,
Expression<Func<TQuery, TKey1>> key1Selector,
Expression<Func<TQuery, TKey2>> key2Selector)
{
if (values is null)
{
throw new ArgumentNullException(nameof(values));
}
if (key1Selector is null)
{
throw new ArgumentNullException(nameof(key1Selector));
}
if (key2Selector is null)
{
throw new ArgumentNullException(nameof(key2Selector));
}
if (!values.Any())
{
return queryable.Take(0);
}
var distinctValues = Bucketize(values);
if (distinctValues.Length > 1024)
{
throw new ArgumentException("Too many parameters for SQL Server, reduce the number of parameters", nameof(values));
}
var predicates = distinctValues
.Select(v =>
{
// Create an expression that captures the variable so EF can turn this into a parameterized SQL query
Expression<Func<TKey1>> value1AsExpression = () => v.Item1;
Expression<Func<TKey2>> value2AsExpression = () => v.Item2;
var firstEqual = Expression.Equal(key1Selector.Body, value1AsExpression.Body);
var visitor = new ReplaceParameterVisitor(key2Selector.Parameters[0], key1Selector.Parameters[0]);
var secondEqual = Expression.Equal(visitor.Visit(key2Selector.Body), value2AsExpression.Body);
return Expression.AndAlso(firstEqual, secondEqual);
})
.ToList();
while (predicates.Count > 1)
{
predicates = PairWise(predicates).Select(p => Expression.OrElse(p.Item1, p.Item2)).ToList();
}
var body = predicates.Single();
var clause = Expression.Lambda<Func<TQuery, bool>>(body, key1Selector.Parameters[0]);
return queryable.Where(clause);
}
class ReplaceParameterVisitor : ExpressionVisitor
{
private ParameterExpression _oldParameter;
private ParameterExpression _newParameter;
public ReplaceParameterVisitor(ParameterExpression oldParameter, ParameterExpression newParameter)
{
_oldParameter = oldParameter;
_newParameter = newParameter;
}
protected override Expression VisitParameter(ParameterExpression node)
{
if (ReferenceEquals(node, _oldParameter))
return _newParameter;
return base.VisitParameter(node);
}
}
/// <summary>
/// Break a list of items tuples of pairs.
/// </summary>
private static IEnumerable<(T, T)> PairWise<T>(this IEnumerable<T> source)
{
var sourceEnumerator = source.GetEnumerator();
while (sourceEnumerator.MoveNext())
{
var a = sourceEnumerator.Current;
sourceEnumerator.MoveNext();
var b = sourceEnumerator.Current;
yield return (a, b);
}
}
private static TKey[] Bucketize<TKey>(IEnumerable<TKey> values)
{
var distinctValueList = values.Distinct().ToList();
// Calculate bucket size as 1,2,4,8,16,32,64,...
var bucket = 1;
while (distinctValueList.Count > bucket)
{
bucket *= 2;
}
// Fill all slots.
var lastValue = distinctValueList.Last();
for (var index = distinctValueList.Count; index < bucket; index++)
{
distinctValueList.Add(lastValue);
}
var distinctValues = distinctValueList.ToArray();
return distinctValues;
}
In the absence of a general solution, I think there are two things to consider:
Avoid multi-column primary keys (will make unit testing easier too).
But if you have to, chances are that one of them will reduce the
query result size to O(n) where n is the size of the ideal query
result. From here, its Solution 5 from Gerd Arnold above.
For example, the problem leading me to this question was querying order lines, where the key is order id + order line number + order type, and the source had the order type being implicit. That is, the order type was a constant, order ID would reduce the query set to order lines of relevant orders, and there would usually be 5 or less of these per order.
To rephrase: If you have a composite key, changes are that one of them have very few duplicates. Apply Solution 5 from above with that.
I tried this solution and it worked with me and the output query was perfect without any parameters
using LinqKit; // nuget
var customField_Ids = customFields?.Select(t => new CustomFieldKey { Id = t.Id, TicketId = t.TicketId }).ToList();
var uniqueIds1 = customField_Ids.Select(cf => cf.Id).Distinct().ToList();
var uniqueIds2 = customField_Ids.Select(cf => cf.TicketId).Distinct().ToList();
var predicate = PredicateBuilder.New<CustomFieldKey>(false); //LinqKit
var lambdas = new List<Expression<Func<CustomFieldKey, bool>>>();
foreach (var cfKey in customField_Ids)
{
var id = uniqueIds1.Where(uid => uid == cfKey.Id).Take(1).ToList();
var ticketId = uniqueIds2.Where(uid => uid == cfKey.TicketId).Take(1).ToList();
lambdas.Add(t => id.Contains(t.Id) && ticketId.Contains(t.TicketId));
}
predicate = AggregateExtensions.AggregateBalanced(lambdas.ToArray(), (expr1, expr2) =>
{
var invokedExpr = Expression.Invoke(expr2, expr1.Parameters.Cast<Expression>());
return Expression.Lambda<Func<CustomFieldKey, bool>>
(Expression.OrElse(expr1.Body, invokedExpr), expr1.Parameters);
});
var modifiedCustomField_Ids = repository.GetTable<CustomFieldLocal>()
.Select(cf => new CustomFieldKey() { Id = cf.Id, TicketId = cf.TicketId }).Where(predicate).ToArray();
I ended up writing a helper for this problem that relies on System.Linq.Dynamic.Core;
Its a lot of code and don't have time to refactor at the moment but input / suggestions appreciated.
public static IQueryable<TEntity> WhereIsOneOf<TEntity, TSource>(this IQueryable<TEntity> dbSet,
IEnumerable<TSource> source,
Expression<Func<TEntity, TSource,bool>> predicate) where TEntity : class
{
var (where, pDict) = GetEntityPredicate(predicate, source);
return dbSet.Where(where, pDict);
(string WhereStr, IDictionary<string, object> paramDict) GetEntityPredicate(Expression<Func<TEntity, TSource, bool>> func, IEnumerable<TSource> source)
{
var firstP = func.Parameters[0];
var binaryExpressions = RecurseBinaryExpressions((BinaryExpression)func.Body);
var i = 0;
var paramDict = new Dictionary<string, object>();
var res = new List<string>();
foreach (var sourceItem in source)
{
var innerRes = new List<string>();
foreach (var bExp in binaryExpressions)
{
var emp = ToEMemberPredicate(firstP, bExp);
var val = emp.GetKeyValue(sourceItem);
var pName = $"#{i++}";
paramDict.Add(pName, val);
var str = $"{emp.EntityMemberName} {emp.SQLOperator} {pName}";
innerRes.Add(str);
}
res.Add( "(" + string.Join(" and ", innerRes) + ")");
}
var sRes = string.Join(" || ", res);
return (sRes, paramDict);
}
EMemberPredicate ToEMemberPredicate(ParameterExpression firstP, BinaryExpression bExp)
{
var lMember = (MemberExpression)bExp.Left;
var rMember = (MemberExpression)bExp.Right;
var entityMember = lMember.Expression == firstP ? lMember : rMember;
var keyMember = entityMember == lMember ? rMember : lMember;
return new EMemberPredicate(entityMember, keyMember, bExp.NodeType);
}
List<BinaryExpression> RecurseBinaryExpressions(BinaryExpression e, List<BinaryExpression> runningList = null)
{
if (runningList == null) runningList = new List<BinaryExpression>();
if (e.Left is BinaryExpression lbe)
{
var additions = RecurseBinaryExpressions(lbe);
runningList.AddRange(additions);
}
if (e.Right is BinaryExpression rbe)
{
var additions = RecurseBinaryExpressions(rbe);
runningList.AddRange(additions);
}
if (e.Left is MemberExpression && e.Right is MemberExpression)
{
runningList.Add(e);
}
return runningList;
}
}
Helper class:
public class EMemberPredicate
{
public readonly MemberExpression EntityMember;
public readonly MemberExpression KeyMember;
public readonly PropertyInfo KeyMemberPropInfo;
public readonly string EntityMemberName;
public readonly string SQLOperator;
public EMemberPredicate(MemberExpression entityMember, MemberExpression keyMember, ExpressionType eType)
{
EntityMember = entityMember;
KeyMember = keyMember;
KeyMemberPropInfo = (PropertyInfo)keyMember.Member;
EntityMemberName = entityMember.Member.Name;
SQLOperator = BinaryExpressionToMSSQLOperator(eType);
}
public object GetKeyValue(object o)
{
return KeyMemberPropInfo.GetValue(o, null);
}
private string BinaryExpressionToMSSQLOperator(ExpressionType eType)
{
switch (eType)
{
case ExpressionType.Equal:
return "==";
case ExpressionType.GreaterThan:
return ">";
case ExpressionType.GreaterThanOrEqual:
return ">=";
case ExpressionType.LessThan:
return "<";
case ExpressionType.LessThanOrEqual:
return "<=";
case ExpressionType.NotEqual:
return "<>";
default:
throw new ArgumentException($"{eType} is not a handled Expression Type.");
}
}
}
Use Like so:
// This can be a Tuple or whatever.. If Tuple, then y below would be .Item1, etc.
// This data structure is up to you but is what I use.
[FromBody] List<CustomerAddressPk> cKeys
var res = await dbCtx.CustomerAddress
.WhereIsOneOf(cKeys, (x, y) => y.CustomerId == x.CustomerId
&& x.AddressId == y.AddressId)
.ToListAsync();
Hope this helps others.
in Case of composite key you can use another idlist and add a condition for that in your code
context.Table.Where(q => listOfIds.Contains(q.Id) && listOfIds2.Contains(q.Id2));
or you can use one another trick create a list of your keys by adding them
listofid.add(id+id1+......)
context.Table.Where(q => listOfIds.Contains(q.Id+q.id1+.......));
I tried this on EF Core 5.0.3 with the Postgres provider.
context.Table
.Select(entity => new
{
Entity = entity,
CompositeKey = entity.Id1 + entity.Id2,
})
.Where(x => compositeKeys.Contains(x.CompositeKey))
.Select(x => x.Entity);
This produced SQL like:
SELECT *
FROM table AS t
WHERE t.Id1 + t.Id2 IN (#__compositeKeys_0)),
Caveats
this should only be used where the combination of Id1 and Id2 will always produce a unique result (e.g., they're both UUIDs)
this cannot use indexes, though you could save the composite key to the db with an index
I have been looking at many posts here and on the web but none of them seem to be helping.
I have a table with about 2 million records, it has over 200 columns.
A simple web service allow the user to pull a specific number of columns out of the table, the user has the option to choose which column to pull.
The result needs to be string of comma separated values, so my query needs to pull the requested columns and return a concatenate string.
I have done this using ADO.NET and pure SQL queries it works fine however I was asked to do it in Entity Framework.
Here is what I have and have done already.
I get the requested columns list as an array of strings.
The following is my query, not sure if it's the best solution or idea hence I'm asking for help here.
var valueList2 = ctx.mytable.Where(x => x.pcds == comValue).Select(x => new{temp = x.column1 +", "+ x.column2}).Select(x => x.temp).ToList();
The above gives me string of two columns separated by commas, I just need to somehow push my array of column names into the lambda part of it.
I did the following but then realised that it only works with a specific type of a class not anonymous, also I can't figure out how I can use it for a multiple columns and not make it so complex.
var createdType = typeof(mytable);
var Param = Expression.Parameter(typeof(string), "pr");
var obj = Expression.New(createdType);
var ValueProperty = createdType.GetProperty("long");
var ValueAssignment = Expression.Bind(ValueProperty, Param);
var memberInit = Expression.MemberInit(obj, ValueAssignment);
var lm = Expression.Lambda<Func<string, mytable>>(memberInit, Param);
Thank you
I'm using Dynamic Linq (source code). Sadly there is little documentation about how to use it :-) In a fun boomerang effect, there is an "evolved" version. The boomerang effect is because the code for generating the dynamic class is based on one of my responses :-) The remaining code seems to be very beautiful... And there is a full suit of unit tests with code samples!!! Note that this second library is a superset of the first library, so you can probably apply many examples to the first one! :-)
I'm adding some static methods to translate the result of a a Dynamic Linq query to a IEnumerable<object[]>.... Example code:
using (var ctx = new Model1())
{
var result = ctx.MyTable
.Take(100)
.SimpleSelect(new[] { "ID", "Col1", "Col2" })
.ToObjectArray();
foreach (var row in result)
{
Console.WriteLine(string.Join(", ", row));
}
}
More complex example:
var columnsNames = new[] { "SomeNullableInt32", "SomeNonNullableDateTimeColumn" };
// One for each column!
var formatters = new Func<object, string>[]
{
x => x != null ? x.ToString() : null,
x => ((DateTime)x).ToShortDateString()
};
var result = ctx.MyTable.Take(100).SimpleSelect(columnsNames).ToObjectArray();
foreach (var row in result)
{
var stringRow = new string[row.Length];
for (int i = 0; i < row.Length; i++)
{
stringRow[i] = formatters[i](row[i]);
}
Console.WriteLine(string.Join(", ", stringRow));
}
And the classes... One (SimpleSelect) produces the Dynamic SQL Select, and "anonymizes" the field names. I do this because for each type of return the Dynamic Linq will generate at runtime a class. This class won't be unloaded until the program ends. By anonymizing the columns (I rename them to Item1, Item2, Item3...) I increase the possibility that the same class will be reused. Note that different type of columns will generate different classes! (int Item1, string Item2 will be a different class from int Item1, DateTime Item2), the other (ToObjectArray) returns a IEnumerable<object[]>, something easier to parse.
public static class DynamicLinqTools
{
private static ConcurrentDictionary<Type, Func<object, object[]>> Converters = new ConcurrentDictionary<Type, Func<object, object[]>>();
public static IQueryable SimpleSelect(this IQueryable query, string[] fields)
{
// With a little luck, "anonymizing" the field names we should
// reduce the number of types created!
// new (field1 as Item1, field2 as Item2)
return query.Select(string.Format("new ({0})", string.Join(", ", fields.Select((x, ix) => string.Format("{0} as Item{1}", x, ix + 1)))));
}
public static IEnumerable<object[]> ToObjectArray(this IQueryable query)
{
Func<object, object[]> converter;
Converters.TryGetValue(query.ElementType, out converter);
if (converter == null)
{
var row = Expression.Parameter(typeof(object), "row");
// ElementType row2;
var row2 = Expression.Variable(query.ElementType, "row2");
// (ElementType)row;
var cast = Expression.Convert(row, query.ElementType);
// row2 = (ElementType)row;
var assign = Expression.Assign(row2, cast);
var properties = query.ElementType.GetProperties(BindingFlags.Public | BindingFlags.Instance)
.Where(x => x.CanRead && x.GetIndexParameters().Length == 0)
.ToArray();
// (object)row2.Item1, (object)row2.Item2, ...
var properties2 = Array.ConvertAll(properties, x => Expression.Convert(Expression.Property(row2, x), typeof(object)));
// new object[] { row2.Item1, row2.Item2 }
var array = Expression.NewArrayInit(typeof(object), properties2);
// row2 = (ElementType)row; (return) new object[] { row2.Item1, row2.Item2 }
var body = Expression.Block(typeof(object[]), new[] { row2 }, assign, array);
var exp = Expression.Lambda<Func<object, object[]>>(body, row);
converter = exp.Compile();
Converters.TryAdd(query.ElementType, converter);
}
foreach (var row in query)
{
yield return converter(row);
}
}
}
This is a short and easy answer for whoever needs a different answer, but per our discussion with #xanatos, it's not the best as it also returns all the columns which need to be cut off before adding to a list of strings.
List<string> valueList = new List<string>();
using (var ctx = new DataEntities1())
{
var query = ctx.myTable.Where(x => x.pcds == scode).SingleOrDefault();
foreach (var item in columnsArray)
{
valueList.Add(typeof(myTable).GetProperty(onsColumns[Convert.ToInt32(item)]).GetValue(query).ToString());
}
}
I am new to EF. I have a table with a list of projects. I have found a query in my software that finds all projects .
public Project[] FindAll()
{
var projects = new List<Project>();
using (var db = new ProjetDbConext())
{
var qProjects = from project in db.ProjectSet
where project.CreateDateTime != null
select project;
projects = qProjects.ToList();
}
return projects.ToArray();
}
This seems to be fine , but I am not sure how to parametrize it. I need this because I am implementing a search feature trying to re use some query logic from EF.
This takes a List of tuples . Each tuple basically has an attribute and a list of search terms.
eg. Tuple(FirstName , { Prasaanth ,Bill } ; Tuple( LastName , { Neelakandan , Gates } ;
This means I need to write a select query where I search projects where FirstName is Prasaanth or Bill . If the list has only one term.
eg. Tuple( Company , { Microsoft} ; then i need to search only one where condition in my query.
public Project[] LoadSearchProjects(List<System.Tuple<string, List<string>>> searchTerms)
{
var projects = new List<Project>();
using (var db = new ProjetDbConext())
{
foreach (System.Tuple<string, List<string>> pair in searchTerms)
{
string attribute = pair.Item1;
List<string> terms = pair.Item2;
/// logic here
}
}
return projects.ToArray();
}
I can always write an if condition where I do :
if(attribute.equals("FirstName"){
// query project.FirstName in the where conditon
}
But I have too many attributes to search on.
I know the ADO.NET way of doing this :
mycommands = new SqlCommand(" select projects from from Persons where '"+attibute+"' = some search terms ...
I don't know how to do something like this in my EF query.
1 ) Is there a way EF allows me to do the search on dynamic attributes ? or parametrize using '"+attribute+"' ??
2) Is there a better data structure I could use to simplify my structure instead of using List<Tuple<string, List<string>> ?
3) I was recommend to use 3rd party LINQKit or dynamic linq but am not sure how to integrate that to EF querying.
My apologies if much of this sounds like collegeboy code. Please let me know if any additional details needed.
Regards,
Prasaanth
UPDATE :
Working method as per Andriy's answer. My question here is this doesnt work if any particular entry in my database say Name is Null.
private static Expression<Func<TEntity, bool>> BuildStringFilter<TEntity, TProp>(
Tuple<string, List<string>> filter)
{
// entity is the Project table
var entity = Expression.Parameter(typeof (TEntity));
var prop = Expression.Property(entity, filter.Item1);
//check if contains returns true
var body = filter.Item2
.Select(v => Expression.Equal(Expression.Call(prop,
typeof (String).GetMethod("Contains"),
new Expression[] { Expression.Constant(v) }), Expression.Constant(true)))
.Aggregate(Expression.Or);
var result = (Expression<Func<TEntity, bool>>) Expression.Lambda(body, entity);
return result;
}
Any way I can modify the expression so that the Contains method :
prop,
typeof (String).GetMethod("Contains"),
new Expression[] { Expression.Constant(v)
works if the value of the attribute (prop) is null ?
You can build filter expression using snippet:
public static Expression<Func<TEntity, bool>> BuildFilter<TEntity, TProp>(
KeyValuePair<string, IEnumerable<TProp>> filter)
{
var entity = Expression.Parameter(typeof(TEntity));
var prop = Expression.Property(entity, filter.Key);
var body = filter.Value
.Select(v => Expression.Equal(prop, Expression.Constant(v)))
.Aggregate((curr, next) => Expression.Or(curr, next));
var result = (Expression<Func<TEntity, bool>>)Expression.Lambda(body, entity);
return result;
}
And call it like:
var filter = new KeyValuePair<string, IEnumerable<string>> (
"FirstName",
new [] {"Alice", "Bob"}
);
var predicate = BuildFilter<Item, string>(filter);
var result = ctx.Items.Where(predicate);
Also, see How to: Use Expression Trees to Build Dynamic Queries.
I'm spitballing here, but I think something like this would be simpler:
public Project[] Find(Expression<Func<Project, bool> filter = null)
{
using (var db = new ProjetDbConext())
{
var query = db.ProjectSet.Where(p => p.CreateDateTime != null);
if(filter != null)
query = query.Where(filter);
return query.ToArray();
}
}
Use it like:
var projects = repo.Find(p => p.id > 100);
Ingredient class:
class Ingredient
{
public String Name { get; set; }
public Double Amount { get; set; }
}
List of Ingredients:
var ingredientsList = new List<Ingredient>();
Database layout of my "Ingredients" table:
[Ingredients] (
[IngredientsID] [int] IDENTITY(1,1) NOT NULL,
[RecipeID] [int] NOT NULL,
[IngredientsName] [nvarchar](512) NOT NULL,
[IngredientsAmount] [float] NOT NULL
)
Am I able to query my ingredientsList against my "Ingredients" table, doing a where-clause which goes something like this (pseudo code alert!):
SELECT * FROM Ingredients WHERE
IngredientsName = ["Name" property on entities in my ingredientsList] AND
IngredientsAmount <= ["Amount" property on entities in my ingredientsList]
I of course want this to be done with LINQ, and not using dynamically generated SQL queries.
LINQ is composable, but to do this without using UNION you'd have to roll your own Expression. Basically, we (presumably) want to create TSQL of the form:
SELECT *
FROM [table]
WHERE (Name = #name1 AND Amount <= #amount1)
OR (Name = #name2 AND Amount <= #amount2)
OR (Name = #name3 AND Amount <= #amount3)
...
where the name/amount pairs are determined at runtime. There is easy way of phrasing that in LINQ; if it was "AND" each time, we could use .Where(...) repeatedly. Union is a candidate, but I've seen repeated people have problems with that. What we want to do is emulate us writing a LINQ query like:
var qry = from i in db.Ingredients
where ( (i.Name == name1 && i.Amount <= amount1)
|| (i.Name == name2 && i.Amount <= amount2)
... )
select i;
This is done by crafting an Expression, using Expression.OrElse to combine each - so we will need to iterate over our name/amount pairs, making a richer Expression.
Writing Expression code by hand is a bit of a black art, but I have a very similar example up my sleeve (from a presentation I give); it uses some custom extension methods; usage via:
IQueryable query = db.Ingredients.WhereTrueForAny(
localIngredient => dbIngredient =>
dbIngredient.Name == localIngredient.Name
&& dbIngredient.Amount <= localIngredient.Amount
, args);
where args is your array of test ingredients. What this does is: for each localIngredient in args (our local array of test ingredients), it asks us to provide an Expression (for that localIngredient) that is the test to apply at the database. It then combines these (in turn) with Expression.OrElse:
public static IQueryable<TSource> WhereTrueForAny<TSource, TValue>(
this IQueryable<TSource> source,
Func<TValue, Expression<Func<TSource, bool>>> selector,
params TValue[] values)
{
return source.Where(BuildTrueForAny(selector, values));
}
public static Expression<Func<TSource, bool>> BuildTrueForAny<TSource, TValue>(
Func<TValue, Expression<Func<TSource, bool>>> selector,
params TValue[] values)
{
if (selector == null) throw new ArgumentNullException("selector");
if (values == null) throw new ArgumentNullException("values");
// if there are no filters, return nothing
if (values.Length == 0) return x => false;
// if there is 1 filter, use it directly
if (values.Length == 1) return selector(values[0]);
var param = Expression.Parameter(typeof(TSource), "x");
// start with the first filter
Expression body = Expression.Invoke(selector(values[0]), param);
for (int i = 1; i < values.Length; i++)
{ // for 2nd, 3rd, etc - use OrElse for that filter
body = Expression.OrElse(body,
Expression.Invoke(selector(values[i]), param));
}
return Expression.Lambda<Func<TSource, bool>>(body, param);
}
The only extent to which you can use a local collection in a LINQ 2 SQL query is with the Contains() function, which is basically a translation to the SQL in clause. For example...
var ingredientsList = new List<Ingredient>();
... add your ingredients
var myQuery = (from ingredient in context.Ingredients where ingredientsList.Select(i => i.Name).Contains(ingredient.Name) select ingredient);
This would generate SQL equivalent to "...where ingredients.Name in (...)"
Unfortunately I don't think that's going to work for you, as you'd have to join each column atomically.
And just as an aside, using LINQ 2 SQL is a dynamically generated SQL query.
You could, of course, do the joining on the client side, but that would require bringing back the entire Ingredients table, which could be performance-prohibitive, and is definitely bad practice.
I think you'll either have to use multiple queries, or copy your ingredients list into a temporary table and do the database query in that way.
I mean, you could have a SQL statement of:
SELECT * FROM Ingredients WHERE
(IngredientsName = 'Flour' AND IngredientsAmount < 10) OR
(IngredientsName = 'Water' AND IngredientsAmount <= 5) OR
(IngredientsName = 'Eggs' AND IngredientsAmount <= 20)
but it get ugly pretty quickly.
Personally I suspect that the temporary table solution is going to be the neatest - but I don't know whether LINQ to SQL has much support for them.
List<string> ingredientNames = ingredientsList
.Select( i => i.Name).ToList();
Dictionary<string, Double> ingredientValues = ingredientsList
.ToDictionary(i => i.Name, i => i.Amount);
//database hit
List<Ingredient> queryResults = db.Ingredients
.Where(i => ingredientNames.Contains(i.Name))
.ToList();
//continue filtering locally - TODO: handle case-sensitivity
List<Ingredient> filteredResults = queryResults
.Where(i => i.Amount <= ingredientValues[i.Name])
.ToList();
I was messing around with this solution in LINQPad, if you have it, you can see the dump outputs. Not sure if it is what you need, but from what I understand it is. I used it against my Users table, but you could replaced that for Ingredients and "UserList" for "IngredientList" and "Username" for "Ingredient Name". You can add further "OR" filtering expressions inside the if statement. It is important you set an ID though.
So final note the "Dump()" method is specific to LINQPad and is not required.
var userList = new List<User>();
userList.Add(new User() { ID = 1, Username = "goneale" });
userList.Add(new User() { ID = 2, Username = "Test" });
List<int> IDs = new List<int>();
// vv ingredients from db context
IQueryable<User> users = Users;
foreach(var user in userList)
{
if (users.Any(x => x.Username == user.Username))
IDs.Add(user.ID);
}
IDs.Dump();
userList.Dump();
users.Dump();
users = users.Where(x => IDs.Contains(x.ID));
users.Dump();
I am using Union to concatinate results of each subquery:
public static IQueryable<TSource> WhereTrueForAny<TSource, TValue>(this IQueryable<TSource> source, Func<TValue, Expression<Func<TSource, bool>>> selector, params TValue[] values)
{
// code is based on Marc Gravells answer
if (selector == null) throw new ArgumentNullException("selector");
if (values == null) throw new ArgumentNullException("values");
// if there are no filters, return nothing
if (values.Length == 0) return source.Where(x => false);
// if there is 1 filter, use it directly
if (values.Length == 1) return source.Where(selector(values[0]));
var lockingUpArray = values;
var p = lockingUpArray.First();
IQueryable<TSource> query = source.Where(selector(p));
foreach (var param in lockingUpArray.Skip(1))
{
query = query.Union(source.Where(selector(param)));
}
return query;
}
How do I write a dynamic query for Linq, if I have say Customer class which holds the fields:
string name
string address
int phoneno
I have to query based on information given similar to
query = string.Empty;
if(!string.IsNullorEmpty(name))
{
query += "#name = name";
}
if(!string.IsNullorEmpty(address))
{
query += "#address = address";
}
if(!string.IsNullorEmpty(phoneno))
{
query += "#phoneno = phoneno";
}
var result = from condition in customer
where(query)
select condition;
Edit #1:
the items are changeable at run time like
private Customer[] GetCustomers(Dictionary<string,string> attributes)
{
here the attribute may be, name alone, or name and address, or name address and phoneno
foreach(string field in attributes.key)
{
query += field == attributes[key];
}
Customers[] =ExecuteQuery(query);
}
Is this kind of query supported by LINQ?
Edit #2:
Hi Mouk,
As I am new to C#, I am still struggling, this is not working for me.
var query = _ConfigFile.ConnectionMasterSection;
for(int i = 0; i < filter.count; i++)
{
query = result.Where(p => typeof(ConnectionMaster).GetProperty(filter[i].Attribute).Name == filter[i].Value);
}
This yeilds Empty, where as i used this
var query = _ConfigFile.ConnectionMasterSection;
//Hard coded
res.Where(q => q.category == filter[0].Value);
And it worked as I expected.
Hi Bryan Watts,
I tried your code also and I getting this error: "Lambda Parameter not in scope".
for(int i = 0; i < filter.count; i++)
{
Field item = filter[i];
MemberExpression param = Expression.MakeMemberAccess(Expression.Parameter(typeof(Connection), "p"), typeof(Connection).GetProperty(item.Attribute));
MemberExpression constant = Expression.MakeMemberAccess(Expression.Constant(item), typeof(Field).GetProperty("Value"));
}
try
{
var myquery = Queryable.Where(coll, Expression.Lambda<Func<Connection, bool>>(
Expression.Equal(param, constant), Expression.Parameter(typeof(Connection),"p")));
}
What is the mistake here?
Check out this http://www.albahari.com/nutshell/predicatebuilder.aspx, it allows for strongly typed predicate building, it can be really nice. If you want actually dynamic string built predicates than you can use the LINQ Dynamic Query Library provided by ScottGu.
Both will accomplish what you want although I would recommend the first option before the second.
Allowing you to do:
var predicate = PredicateBuilder.True<MyLinqType>();
if(!string.IsNullOrEmpty(name))
predicate = predicate.And(p => p.name == name);
...
var myResults = Context.MyLinTypeQueryTable.Where(predicate);
And more.
Here you go:
var result = from customer in Customers
where string.IsNullOrEmpty(phoneNo) || customer.PhoneNo == phoneNo
where string.IsNullOrEmpty(address) || customer.Address == address
select customer;
If you're concerned that this generate the optimal SQL query underneath, as always you should attach a SQL Query Analyzer and check. But I believe the expression parser in Linq To Sql will collapse down the where clauses as appropriate based on the value of the arguments.
You can use the fluent interface and add a new Where clause fpr each condition. Something like:
var result = from cus in customers select cus;
if(!string.IsNullOrEmpty(name))
result= result.Where(p => p.Name == name);
EDIT upon hte comment:
if you are querying over a collection in memory, you could retrieve the properties using reflection.
private Customer[] GetCustomers(Dictionary<string,string> attributes)
{
var result = from cus in customers select cus;
foreach(string key in attributes.Keys)
result= result.Where(p => GetProperty(p, key )== attributes[key]);
return result.ToList();
}
Supposing GetProperty retrieve the property by reflection.
Using Linq2Sql this method will result in retrieving all record an then iterating over them using reflection.
I've had good experience with Dynamic LINQ.
I used it for a rich HTML table that could be filtered and sorted server side. The server receives a request containing a request parameter where the key is the name of the property (for example 'Lastname') and the value is the value that the property needs to be sorted on (for example 'Smith'). Using that information I built a query string that I passed to the Dynamic LINQ's Where method.
Roughly, you could think of something like the following:
public static IQueryable<T> Filter<T>(this IQueryable<T> query, Dictionary<string, string> dictionary)
{
Type t = typeof(T);
StringBuilder sb = new StringBuilder();
PropertyInfo[] properties = t.GetProperties();
foreach(string key in dictionary.Keys)
{
PropertyInfo property = properties.Where(p => p.Name == key).SingleOrDefault();
if(property != null)
{
if (sb.Length > 0) sb.Append(" && ");
string value = dictionary[key];
sb.Append(string.Format(#"{0}.ToString().Contains(""{1}"")", key, value));
}
}
if (sb.Length > 0)
return query.Where(sb.ToString());
else
return query;
}
The code is out of the top of my head and thus untested.
Of course, this is the most basic version: it does a simple string comparison. If you want to have numerical comparison (meaning you want for example the User where UserID is exactly 100, not where the UserID.ToString().Contains("100")), or query nested Properties (Customer.Company.CompanyAddress for example), or query Collections this gets more complicated. You should also think about security: while Dynamic LINQ is not vulnerable to SQL injection, you shouldn't let it blindly parse all user input.
It sounds like you need to dynamically compose queries.
See my answer to this question.
It explains how queries against an IQueryable<T> are composed by the compiler, and what you can do to add dynamic elements.
Edit
Here is an example of how you would dynamically build a set of Where conditions on top of an IQueryable<Customer>:
// This method ANDs equality expressions for each property, like so:
//
// customers.Where(c => c.Property1 == value1 && c.Property2 == value2 && ...);
private IQueryable<Customer> FilterQuery(IQueryable<Customer> customers, IDictionary<string, string> filter)
{
var parameter = Expression.Parameter(typeof(Customer), "c");
Expression filterExpression = null;
foreach(var filterItem in filter)
{
var property = typeof(Customer).GetProperty(filterItem.Key);
var propertyAccess = Expression.MakeMemberAccess(parameter, property);
var equality = Expression.Equal(propertyAccess, Expression.Constant(filterItem.Value));
if(filterExpression == null)
{
filterExpression = equality;
}
else
{
filterExpression = Expression.And(filterExpression, equality);
}
}
if(filterExpression != null)
{
var whereBody = Expression.Lambda<Func<Customer, bool>>(filterExpression, parameter);
customers = customers.Where(whereBody);
}
return customers;
}