Develop Reference

Efficiently make iterative LINQ to SQL queries

I am trying to implement the following algorithm using LINQ-to-SQL:
Given a list of strings L, return every row R in the DB for which every string in L is a substring of one of the column values in R.
The question is how do I do this iteratively for every string in L? I don't know how I can slickly put it all into one Linq-To-SQL statement. Note that I have no problem writing code along the lines of:
field1.contains(...) || field2.contains(...) || ...
as there are not that many columns.
E.g., if the input is
["Charlie", "Doctor", "Kor"]
we would output all rows that have a field with "Charlie" as a substring, a field with "Doctor" as a substring, and a field with "Kor" as a substring.
One approach I thought of was to make separate SQL queries for each input value and to take the intersection of all of those.
Another approach is to pick just one of the strings from the input, make a SQL query on that, convert it to a list, and filter out the rest of the strings one at a time using just LINQ in C#.
Any thoughts on an optimal way to do this?

I would try All extension method (EF6 supports it, not sure about LINQ to SQL):
List<string> values = new List<string> { "Charlie", "Doctor", "Kor" };
var query = db.Table
.Where(r => values.All(v => r.Field1.Contains(v) || r.Field2.Contains(v) || ...));
Update: Well, the assumption was wrong - as mentioned in the comments, unfortunately LINQ to SQL does not support the above construct (shame on them).
As usual in such cases, I would build dynamically a corresponding predicate expression.
In this particular case we need something like this (for N fields and M values):
r => (r.Field1.Contains(value1) || r.Field2.Contains(value1) ... || r.FieldN.Contains(value1))
&& (r.Field1.Contains(value2) || r.Field2.Contains(value2) ... || r.FieldN.Contains(value2))
...
&& (r.Field1.Contains(valueM) || r.Field2.Contains(valueM) ... || r.FieldN.Contains(valueM));
And here is a custom extension method which does that:
public static class QueryableExtensions
{
public static IQueryable<T> WhereContainsAll<T>(
this IQueryable<T> source,
IEnumerable<string> values,
params Expression<Func<T, string>>[] members)
{
var parameter = Expression.Parameter(typeof(T), "r");
var body = values
.Select(value => members
.Select(member => (Expression)Expression.Call(
Expression.MakeMemberAccess(parameter, ((MemberExpression)member.Body).Member),
"Contains", Type.EmptyTypes, Expression.Constant(value)))
.Aggregate(Expression.OrElse))
.Aggregate(Expression.AndAlso);
var predicate = Expression.Lambda<Func<T, bool>>(body, parameter);
return source.Where(predicate);
}
}
and the sample usage would be
List<string> values = new List<string> { "Charlie", "Doctor", "Kor" };
var query = db.Table.WhereContainsAll(values,
r => r.Field1, r => r.Field2, r => r.Field3, ...);
which should lead to a single SQL query which IMO should be optimal because the heavy work will be done by the database engine. Of course the query most likely will cause full table scan, but the same will happen even with single Contains (SQL LIKE) criteria.

Try this (I made an example using Lists):
var dbValues = new List<string> {"hello", "how", "are", "you"};
var substrings = new List<string> {"ello", "re"};
var result = dbValues.Where(i => substrings.Any(l => i.Contains(l))).ToList();
Result will contain {"hello","are"}
Example with database:
using (var db = new MyDatabase())
{
var substrings = new List<string> { "ello", "re" };
var result = db.MyTable.Where(i => substrings.Any(l => i.Value.Contains(l))).ToList();
}

Not able to Select after using Expression Tree in Group By

I have been referring this post to group by using expression tree. Here is my code:
String[] fields = { "DepartmentID", "SkillID" };
var groupLambda = GroupByExpression<Person>(fields);
var query = dbContext.People.GroupBy(groupLambda.Compile());
var queryResult = query.ToList();
Here is the method GroupByExpression which uses solution given in aforesaid post (Thanks Daniel!):
public static Expression<Func<TItem, object>> GroupByExpression<TItem>(string[] propertyNames)
{
var properties = propertyNames.Select(name => typeof(TItem).GetProperty(name)).ToArray();
var propertyTypes = properties.Select(p => p.PropertyType).ToArray();
var tupleTypeDefinition = typeof(Tuple).Assembly.GetType("System.Tuple`" + properties.Length);
var tupleType = tupleTypeDefinition.MakeGenericType(propertyTypes);
var constructor = tupleType.GetConstructor(propertyTypes);
var param = Expression.Parameter(typeof(TItem), "x");
var body = Expression.New(constructor, properties.Select(p => Expression.Property(param, p)));
var expr = Expression.Lambda<Func<TItem, object>>(body, param);
return expr;
}
I want to be able to identify fields in the group by keys with strong names in select part like query.Select(x => new { x.Key.DepartmentID, x.Key.SkillID });
How do I do this?

Now... I won't give you the solution to the question you asked, but I'll try to help you :-)
If you want to do dynamic queries, you should probably use DynamicLinq
With DynamicLinq you can do things like:
IQueryable query = context.YourTable;
var groups = query.GroupBy("new (Field1, Field2)");
I'm rereading your question...
I want to be able to identify fields in the group by keys with strong names in select part like query.Select(x => new { x.Key.DepartmentID, x.Key.SkillID });
You can't. GroupBy in general will return a IGrouping<TKey, TSource>. TKey is dynamic (because you build it based on strings), so you can't "extract" it and pass it to the compiler, so you can't do the select with strong names.
There is a single exception: if you know the types and numbers of the GroupBy TKey then something can be done. So, you gave us:
String[] fields = { "DepartmentID", "SkillID" };
If you always have two int then you can cast your query with:
.Cast<IGrouping<Tuple<int, int>, Person>>()
.Select(x => new { x.Key.DepartmentID, x.Key.SkillID });
Note that, as I've written in a comment, your GroupBy will be executed client-side, and everything after the GroupBy will be executed client-side (where client-side == where your program is vs sql-side == where your sql server is)!
DynamicLinq will solve the problem of executing the query sql-side instead of client-side, but won't solve the problem of strong vs weak naming (after a DynamicLinq you can: A) use .Cast<>() method or B) return a dynamic object/IEnumerable<dynamic>)

The syntax you're using new { x.Key.DepartmentID, x.Key.SkillID } constructs an anonymous class at compile time. If you want to create an anonymous class at runtime, see here. However, that won't allow you to "identify fields in the group by keys with strong names". If you want to construct an anonymous class at runtime, but be able to use those names at compile time, I'm afraid that's impossible.

Using LINQ with a WHERE to ask for multiple possible results without redundant "||" commands

This is a cosmetics issue, it is syntactical sugar only, just because I want to do it. But basically I have a lot of code that reads like this ... I am using a dynamic for the sheer purpose of time saving and example.
var terms = new List<string> {"alpha", "beta", "gamma", "delta", "epsilon", "rho"};
var list = new List<dynamic> {
new {
Foo = new List<string> {
"alpha",
"gamma"
}
}
};
var result = list
.Where(
n => n.Foo.Contains(terms[0]) ||
n.Foo.Contains(terms[1]) ||
n.Foo.Contains(terms[2]) ||
n.Foo.Contains(terms[3]) ||
n.Foo.Contains(terms[4]) ||
n.Foo.Contains(terms[5]))
.ToList();
Obviously this is a ludicrous hyperbole for the sheer sake of example, more accurate code is ...
Baseline =
Math.Round(Mutations.Where(n => n.Format.Is("Numeric"))
.Where(n => n.Sources.Contains("baseline") || n.Sources.Contains("items"))
.Sum(n => n.Measurement), 3);
But the basic point is that I have plenty of places where I want to check to see if a List<T> (usually List<string>, but there may at times be other objects. string is my present focus though) contains any of the items from another List<T>.
I thought that using .Any() would work, but I actually haven't been able to get that to function as expected. So far, only excessive "||" is all that yields the correct results.
This is functionally fine, but it is annoying to write. I wanted to know if there is a simpler way to write this out - an extension method, or a LINQ method that perhaps I've not understood.
Update
I am really aware that this may be a duplicate, but I am having a hard time figuring out the wording of the question to a level of accuracy that I can find duplicates. Any help is much appreciated.
Solution
Thank you very much for all of the help. This is my completed solution. I am using a dynamic here just to save time on example classes, but several of your proposed solutions worked.
var terms = new List<string> {"alpha", "beta", "gamma", "delta", "epsilon", "rho"};
var list = new List<dynamic> {
new { // should match
Foo = new List<string> {
"alpha",
"gamma"
},
Index = 0
},
new { // should match
Foo = new List<string> {
"zeta",
"beta"
},
Index = 1
},
new { // should not match
Foo = new List<string> {
"omega",
"psi"
},
Index = 2
},
new { // should match
Foo = new List<string> {
"kappa",
"epsilon"
},
Index = 3
},
new { // should not match
Foo = new List<string> {
"sigma"
},
Index = 4
}
};
var results = list.Where(n => terms.Any(t => n.Foo.Contains(t))).ToList();
// expected output - [0][1][3]
results.ForEach(result => {
Console.WriteLine(result.Index);
});

I thought that using .Any() would work, but I actually haven't been able to get that to function as expected.
You should be able to make it work by applying Any() to terms, like this:
var result = list
.Where(n => terms.Any(t => n.Foo.Contains(t)))
.ToList();

You could try this one:
var result = list.Where(terms.Contains(n.Foo))
.ToList();

I assume n.Foo is a string rather than a collection, in which case:
var terms = new List<string> { "alpha", "beta", "gamma", "delta", "epsilon", "rho" };
var list = (new List<string> { "alphabet", "rhododendron" })
.Select(x => new { Foo = x });
var result = list.Where(x => terms.Any(y => x.Foo.Contains(y)));

You want to find out if Any element in one collection exists within another collection, so Intersect should work nicely here.
You can modify your second code snippet accordingly:
var sources = new List<string> { "baseline", "items" };
Baseline =
Math.Round(Mutations.Where(n => n.Format.Is("Numeric"))
.Where(n => sources.Intersect(n.Sources).Any())
.Sum(n => n.Measurement), 3);
Regarding the part of the docs you quoted for "Intersect":
Produces the set intersection of two sequences by using the default equality comparer to compare values.
Every object can be compared to an object of the same type, to determine whether they're equal. If it's a custom class you created, you can implement IEqualityComparer and then you get to decide what makes two instances of your class "equal".
In this case, however, we're just comparing strings, nothing special. "Foo" = "Foo", but "Foo" ≠ "Bar"
So in the above code snippet, we intersect the two collections by comparing all the strings in the first collection to all the strings in the second collection. Whichever strings are "equal" in both collections end up in a resulting third collection.
Then we call "Any()" to determine if there are any elements in that third collection, which tells us there was at least one match between the two original collections.

If performance is an issue when using Any() you can use a regular expression instead. Obviously, you should probably measure to make sure that regular expressions performs faster:
var terms = new List<string> { "alpha", "beta", "gamma", "delta", "epsilon", "rho" };
var regex = new Regex(string.Join("|", terms));
var result = list
.Where(n => regex.Match(n.Foo).Success);
This assumes that joining the terms to a list creates a valid regular expression but with simple words that should not be a problem.
One advantage of using a regular expression is that you can require that the terms are surrounded by word boundaries. Also, the predicate in the Where clause may be easier to understand when compared to a solution using Contains inside Any.

Parameterise LINQ GroupBy

The following C# code takes a large datatable with many columns and an array of 2 column names. It will give a new datatable with two rows where there are duplicate rows for the two fields supplied staff no & skill.
This is too specific and I need to supply any number of fields as the groupby.
can someone help me?
string[] excelField = new string[0]; // contains a list of field name for uniquness
excelField[0] = "staff No";
excelField[1] = "skill";
DataTable dataTableDuplicateRows = new DataTable();
dataTableDuplicateRows.Clear();
dataTableDuplicateRows.Columns.Clear();
foreach (string fieldName in excelField)
{
dataTableDuplicateRows.Columns.Add(fieldName);
}
var duplicateValues = dataTableCheck.AsEnumerable()
.GroupBy(row => new { Field0 = row[excelField[0]], Field1 = row[excelField[1]] })
.Where(group => group.Count() > 1)
.Select(g => g.Key);
foreach (var duplicateValuesRow in duplicateValues)
{
dataTableDuplicateRows.Rows.Add(duplicateValuesRow.Field0, duplicateValuesRow.Field1);
}

I think what you require is something make the linq more dynamic, even though you could achieve it by using expression tree, the DynamicLinq library would appear to solve your issue in an easier way.
For you case, with the library, just use the GroupBy extension method with a string value.
More info about DynamicLinq library:
Scott Gu's blog

How can I convert IEnumerable<T> to List<T> in C#?

I am using LINQ to query a generic dictionary and then use the result as the datasource for my ListView (WebForms).
Simplified code:
Dictionary<Guid, Record> dict = GetAllRecords();
myListView.DataSource = dict.Values.Where(rec => rec.Name == "foo");
myListView.DataBind();
I thought that would work but in fact it throws a System.InvalidOperationException:
ListView with id 'myListView' must
have a data source that either
implements ICollection or can perform
data source paging if AllowPaging is
true.
In order to get it working I have had to resort to the following:
Dictionary<Guid, Record> dict = GetAllRecords();
List<Record> searchResults = new List<Record>();
var matches = dict.Values.Where(rec => rec.Name == "foo");
foreach (Record rec in matches)
searchResults.Add(rec);
myListView.DataSource = searchResults;
myListView.DataBind();
Is there a small gotcha in the first example to make it work?
(Wasn't sure what to use as the question title for this one, feel free to edit to something more appropriate)

Try this:
var matches = dict.Values.Where(rec => rec.Name == "foo").ToList();
Be aware that that will essentially be creating a new list from the original Values collection, and so any changes to your dictionary won't automatically be reflected in your bound control.

I tend to prefer using the new Linq syntax:
myListView.DataSource = (
from rec in GetAllRecords().Values
where rec.Name == "foo"
select rec ).ToList();
myListView.DataBind();
Why are you getting a dictionary when you don't use the key? You're paying for that overhead.

You might also try:
var matches = new List<Record>(dict.Values.Where(rec => rec.Name == "foo"));
Basically generic collections are very difficult to cast directly, so you really have little choice but to create a new object.

myListView.DataSource = (List<Record>) dict.Values.Where(rec => rec.Name == "foo");

Just adding knowledge the next sentence doesn´t recover any data from de db. Just only create the query (for that it is iqueryable type). For launching this query you must to add .ToList() or .First() at the end.
dict.Values.Where(rec => rec.Name == "foo")