I have been looking at many posts here and on the web but none of them seem to be helping.
I have a table with about 2 million records, it has over 200 columns.
A simple web service allow the user to pull a specific number of columns out of the table, the user has the option to choose which column to pull.
The result needs to be string of comma separated values, so my query needs to pull the requested columns and return a concatenate string.
I have done this using ADO.NET and pure SQL queries it works fine however I was asked to do it in Entity Framework.
Here is what I have and have done already.
I get the requested columns list as an array of strings.
The following is my query, not sure if it's the best solution or idea hence I'm asking for help here.
var valueList2 = ctx.mytable.Where(x => x.pcds == comValue).Select(x => new{temp = x.column1 +", "+ x.column2}).Select(x => x.temp).ToList();
The above gives me string of two columns separated by commas, I just need to somehow push my array of column names into the lambda part of it.
I did the following but then realised that it only works with a specific type of a class not anonymous, also I can't figure out how I can use it for a multiple columns and not make it so complex.
var createdType = typeof(mytable);
var Param = Expression.Parameter(typeof(string), "pr");
var obj = Expression.New(createdType);
var ValueProperty = createdType.GetProperty("long");
var ValueAssignment = Expression.Bind(ValueProperty, Param);
var memberInit = Expression.MemberInit(obj, ValueAssignment);
var lm = Expression.Lambda<Func<string, mytable>>(memberInit, Param);
Thank you
I'm using Dynamic Linq (source code). Sadly there is little documentation about how to use it :-) In a fun boomerang effect, there is an "evolved" version. The boomerang effect is because the code for generating the dynamic class is based on one of my responses :-) The remaining code seems to be very beautiful... And there is a full suit of unit tests with code samples!!! Note that this second library is a superset of the first library, so you can probably apply many examples to the first one! :-)
I'm adding some static methods to translate the result of a a Dynamic Linq query to a IEnumerable<object[]>.... Example code:
using (var ctx = new Model1())
{
var result = ctx.MyTable
.Take(100)
.SimpleSelect(new[] { "ID", "Col1", "Col2" })
.ToObjectArray();
foreach (var row in result)
{
Console.WriteLine(string.Join(", ", row));
}
}
More complex example:
var columnsNames = new[] { "SomeNullableInt32", "SomeNonNullableDateTimeColumn" };
// One for each column!
var formatters = new Func<object, string>[]
{
x => x != null ? x.ToString() : null,
x => ((DateTime)x).ToShortDateString()
};
var result = ctx.MyTable.Take(100).SimpleSelect(columnsNames).ToObjectArray();
foreach (var row in result)
{
var stringRow = new string[row.Length];
for (int i = 0; i < row.Length; i++)
{
stringRow[i] = formatters[i](row[i]);
}
Console.WriteLine(string.Join(", ", stringRow));
}
And the classes... One (SimpleSelect) produces the Dynamic SQL Select, and "anonymizes" the field names. I do this because for each type of return the Dynamic Linq will generate at runtime a class. This class won't be unloaded until the program ends. By anonymizing the columns (I rename them to Item1, Item2, Item3...) I increase the possibility that the same class will be reused. Note that different type of columns will generate different classes! (int Item1, string Item2 will be a different class from int Item1, DateTime Item2), the other (ToObjectArray) returns a IEnumerable<object[]>, something easier to parse.
public static class DynamicLinqTools
{
private static ConcurrentDictionary<Type, Func<object, object[]>> Converters = new ConcurrentDictionary<Type, Func<object, object[]>>();
public static IQueryable SimpleSelect(this IQueryable query, string[] fields)
{
// With a little luck, "anonymizing" the field names we should
// reduce the number of types created!
// new (field1 as Item1, field2 as Item2)
return query.Select(string.Format("new ({0})", string.Join(", ", fields.Select((x, ix) => string.Format("{0} as Item{1}", x, ix + 1)))));
}
public static IEnumerable<object[]> ToObjectArray(this IQueryable query)
{
Func<object, object[]> converter;
Converters.TryGetValue(query.ElementType, out converter);
if (converter == null)
{
var row = Expression.Parameter(typeof(object), "row");
// ElementType row2;
var row2 = Expression.Variable(query.ElementType, "row2");
// (ElementType)row;
var cast = Expression.Convert(row, query.ElementType);
// row2 = (ElementType)row;
var assign = Expression.Assign(row2, cast);
var properties = query.ElementType.GetProperties(BindingFlags.Public | BindingFlags.Instance)
.Where(x => x.CanRead && x.GetIndexParameters().Length == 0)
.ToArray();
// (object)row2.Item1, (object)row2.Item2, ...
var properties2 = Array.ConvertAll(properties, x => Expression.Convert(Expression.Property(row2, x), typeof(object)));
// new object[] { row2.Item1, row2.Item2 }
var array = Expression.NewArrayInit(typeof(object), properties2);
// row2 = (ElementType)row; (return) new object[] { row2.Item1, row2.Item2 }
var body = Expression.Block(typeof(object[]), new[] { row2 }, assign, array);
var exp = Expression.Lambda<Func<object, object[]>>(body, row);
converter = exp.Compile();
Converters.TryAdd(query.ElementType, converter);
}
foreach (var row in query)
{
yield return converter(row);
}
}
}
This is a short and easy answer for whoever needs a different answer, but per our discussion with #xanatos, it's not the best as it also returns all the columns which need to be cut off before adding to a list of strings.
List<string> valueList = new List<string>();
using (var ctx = new DataEntities1())
{
var query = ctx.myTable.Where(x => x.pcds == scode).SingleOrDefault();
foreach (var item in columnsArray)
{
valueList.Add(typeof(myTable).GetProperty(onsColumns[Convert.ToInt32(item)]).GetValue(query).ToString());
}
}
Related
I have a Generic function that contains an IQueryable<T> where each row contains an instance of a class with a set of properties.
I have another class (MyClass) that has some of the same properties as the class T above... i.e. same name and datatypes.
I also have a List of Strings containing the Property Names shared between the two classes.
I want to be able to create a new IQueryable<myClass> where the myClass instances properties are populated with the name-sake proprties in the original IQueryable<T>
Does that make sense? Please let me know if I can supply any more info or make anything more clear.
EDIT
I will try an add some code to illustrate better. I know there are countless faults here including Adding to an IQueryable - but this is for illustration:
IQueryable<T> qry = this.GetSomeDataIntoIQueryable();
// Just getting a list of the Shared Property Names between the two classes
List<string> sharedProprtyNames = new List<string>();
foreach (var item in ListofSharedPropertyNames)
{
sharedProprtyNames .Add(item.SharedPropertyName);
}
IQueryable<myClass> myClassIQ;
foreach(var classItem in qry)
{
myClass x = new myClass();
foreach(var sharedProperty in sharedProprtyNames )
{
myClass[sharedProperty] = classItem[sharedProperty];
}
myClassIQ.Add(myClass);
}
Something like:
static IQueryable<TTo> Select<TFrom, TTo>(
this IQueryable<TFrom> source,
params string[] members)
{
var p = Expression.Parameter(typeof(TFrom));
var body = Expression.MemberInit(
Expression.New(typeof(TTo)),
members.Select(member => Expression.Bind(
typeof(TTo).GetMember(member).Single(),
Expression.PropertyOrField(p, member))));
var lambda = Expression.Lambda<Func<TFrom, TTo>>(body, p);
return source.Select(lambda);
}
?
This creates an IQueryable<T>-friendly projection from TFrom to TTo, respecting all the members from members.
In your example, it would be:
IQueryable<myClass> myClassIQ = qry.Select<T, myClass>(ListofSharedPropertyNames);
(adjust between array / list for the members parameter to suit your convenience - because we use Select, it'll work with either)
Using System.Linq.Dynamic.Core you can:
IQueryable<Table1> query1 = ...
var res = query.Select<SubTable1>("new(" + string.Join(",", new[] { "Col1", "Col2" }) + ")").ToArray();
Where query1 is your query, Table1 is the T of your question, SubTable1 is MyClass, "Col1", "Col2" are the columsn that must be selected.
You could do everything without usig the System.Linq.Dynamic.Core library and simply building an expression tree... But it is a pain :-)
My variant of the code of Hans Passant:
public static IQueryable<TResult> Select<TSource, TResult>(this IQueryable<TSource> source, IEnumerable<string> columns)
{
// the x in x => ...
var par = Expression.Parameter(typeof(TSource), "x");
// "Bindings" (the Col1 = x.Col1 inside the x => new { Col1 = x.Col1 })
var binds = columns.Select(x => Expression.Bind((MemberInfo)typeof(TResult).GetProperty(x) ?? typeof(TResult).GetField(x), Expression.PropertyOrField(par, x)));
// new TResult
var new1 = Expression.New(typeof(TResult));
// new TResult { Bindings }
var mi = Expression.MemberInit(new1, binds);
// x => new TResult { Bindings }
var lambda = Expression.Lambda<Func<TSource, TResult>>(mi, par);
// Select(x => new TResult { Bindings })
return source.Select(lambda);
}
(nearly totally equivalent... Only difference is that he uses GetMember() while I used GetProperty() + GetField())
I currently use the ClosedXml to open up an Excel file, and Im wanting to convert the rows/cells into a json string which looks like this (or this format):
{"listOfRows":
[
{"column0":"test", "column1":"test1", "column2":"test1", },
{"column0":"test", "column1":"test1"}
],
"rowNumber":1}
The column number needs to increment and thats where im struggling. I currently have the following linq query:
var test = from row in workSheet.Rows()
select new
{
listOfRows = from cell in row.Cells()
select new
{
column = cell.Value,
},
rowNumber = row.RowNumber()
};
which as you will work out doesnt increment my column. Can anyone help with the curly one?
Is this more like you expected it?
Linqs .Select() method also supports an indexer:
var test = workSheet.Rows().Select((row, i) =>
new
{
listOfRows = row.Cells().Select((cell, j) => {
var obj = new ExpandoObject() as IDictionary<string, Object>;
obj.Add($"column{j}", cell.Value);
return obj;
}),
rowNumber = i
});
Now to the expando object (let me explain a bit):
It is only available through the DLR (dynamic language runtime) and allows you to add properties at runtime.
The first iterations most inner code is about equal to:
dynamic obj = new ExpandoObject();
obj.column0 = cell.Value;
return obj;
However, then you can't control the property name with the indexer, hence using the obj.Add($"column{j}", cell.Value); syntax in my answer.
For the sake of completness, here's the MSDN link.
I would break the problem out by first writing an extension method ...
public static dynamic ToObject(this IDictionary<string, object> source)
{
dynamic result = new ExpandoObject();
source.ForEach(i => result[i.Key] = i.Value);
return result;
}
If you then build each row out to a dictionary then ToObject() each dictionary a simple json convert on the final test object should suffice ...
var test = new {
listOfRows = workSheet.Rows()
.Select(r => {
var row = r.Cells().Select((c,i) => new { Key = $"Column{i}", Value = c.Value })
.ToDictionary(x => x.Key, x => x.Value)
.ToObject();
row.rowNumber = r.RowNumber();
return row;
};
I have a LINQ statement that I need to do a "contains" with, but also need some sort of loop.
The format of the data is as follows:
x.Product_Name = "product[x], product[y], product[z]"
As user selects multiple items from a list to search on.
I need to find anything within Product_Name that was selected from the user.
var names = JsonConvert.DeserializeObject<IEnumerable<string>>
(criteria.value).ToArray();
This line gets the items a user selected from the list and stores them in an array.
query = query.Where(x => names.contains(x.Product_Name))
Doesn't work because Product_Name is a flattened out version of products, so I can't do this.
What I need is something like the following:
foreach (string s in names)
{
projectsQuery = projectsQuery.Where(x => x.Product_Name.Contains(s));
}
But when the SQL is created for the above, it uses an AND conditional instead of an OR conditional. I need to find any instances where string s is contained within the Product_Name.
You can achieve it by creating Expression tree manually. Although it is kind of hard to manage code.
var containsMethod = typeof(string).GetMethod("Contains", new[] { typeof(string) });
var xParameter = Expression.Parameter(typeof(x), "x");
var searchexpression = new stack<expression>();
foreach (string s in names)
{
var containsmethodexp = expression.call(Expression.Property(xParameter, "Product_Name", containsMethod, expression.constant(s));
if (searchexpression.count == 0)
{
searchexpression.push(containsmethodexp);
}
else
{
searchexpression.push(expression.orelse(containsmethodexp, searchexpression.pop()));
}
}
var finalResult = projectsQuery.Where(Expression.Lambda<Func<x, bool>>(searchexpression.pop(), new ParameterExpression[] { xParameter }));
here x is your Entity Name
I am new to EF. I have a table with a list of projects. I have found a query in my software that finds all projects .
public Project[] FindAll()
{
var projects = new List<Project>();
using (var db = new ProjetDbConext())
{
var qProjects = from project in db.ProjectSet
where project.CreateDateTime != null
select project;
projects = qProjects.ToList();
}
return projects.ToArray();
}
This seems to be fine , but I am not sure how to parametrize it. I need this because I am implementing a search feature trying to re use some query logic from EF.
This takes a List of tuples . Each tuple basically has an attribute and a list of search terms.
eg. Tuple(FirstName , { Prasaanth ,Bill } ; Tuple( LastName , { Neelakandan , Gates } ;
This means I need to write a select query where I search projects where FirstName is Prasaanth or Bill . If the list has only one term.
eg. Tuple( Company , { Microsoft} ; then i need to search only one where condition in my query.
public Project[] LoadSearchProjects(List<System.Tuple<string, List<string>>> searchTerms)
{
var projects = new List<Project>();
using (var db = new ProjetDbConext())
{
foreach (System.Tuple<string, List<string>> pair in searchTerms)
{
string attribute = pair.Item1;
List<string> terms = pair.Item2;
/// logic here
}
}
return projects.ToArray();
}
I can always write an if condition where I do :
if(attribute.equals("FirstName"){
// query project.FirstName in the where conditon
}
But I have too many attributes to search on.
I know the ADO.NET way of doing this :
mycommands = new SqlCommand(" select projects from from Persons where '"+attibute+"' = some search terms ...
I don't know how to do something like this in my EF query.
1 ) Is there a way EF allows me to do the search on dynamic attributes ? or parametrize using '"+attribute+"' ??
2) Is there a better data structure I could use to simplify my structure instead of using List<Tuple<string, List<string>> ?
3) I was recommend to use 3rd party LINQKit or dynamic linq but am not sure how to integrate that to EF querying.
My apologies if much of this sounds like collegeboy code. Please let me know if any additional details needed.
Regards,
Prasaanth
UPDATE :
Working method as per Andriy's answer. My question here is this doesnt work if any particular entry in my database say Name is Null.
private static Expression<Func<TEntity, bool>> BuildStringFilter<TEntity, TProp>(
Tuple<string, List<string>> filter)
{
// entity is the Project table
var entity = Expression.Parameter(typeof (TEntity));
var prop = Expression.Property(entity, filter.Item1);
//check if contains returns true
var body = filter.Item2
.Select(v => Expression.Equal(Expression.Call(prop,
typeof (String).GetMethod("Contains"),
new Expression[] { Expression.Constant(v) }), Expression.Constant(true)))
.Aggregate(Expression.Or);
var result = (Expression<Func<TEntity, bool>>) Expression.Lambda(body, entity);
return result;
}
Any way I can modify the expression so that the Contains method :
prop,
typeof (String).GetMethod("Contains"),
new Expression[] { Expression.Constant(v)
works if the value of the attribute (prop) is null ?
You can build filter expression using snippet:
public static Expression<Func<TEntity, bool>> BuildFilter<TEntity, TProp>(
KeyValuePair<string, IEnumerable<TProp>> filter)
{
var entity = Expression.Parameter(typeof(TEntity));
var prop = Expression.Property(entity, filter.Key);
var body = filter.Value
.Select(v => Expression.Equal(prop, Expression.Constant(v)))
.Aggregate((curr, next) => Expression.Or(curr, next));
var result = (Expression<Func<TEntity, bool>>)Expression.Lambda(body, entity);
return result;
}
And call it like:
var filter = new KeyValuePair<string, IEnumerable<string>> (
"FirstName",
new [] {"Alice", "Bob"}
);
var predicate = BuildFilter<Item, string>(filter);
var result = ctx.Items.Where(predicate);
Also, see How to: Use Expression Trees to Build Dynamic Queries.
I'm spitballing here, but I think something like this would be simpler:
public Project[] Find(Expression<Func<Project, bool> filter = null)
{
using (var db = new ProjetDbConext())
{
var query = db.ProjectSet.Where(p => p.CreateDateTime != null);
if(filter != null)
query = query.Where(filter);
return query.ToArray();
}
}
Use it like:
var projects = repo.Find(p => p.id > 100);
I have two lists I would like to compare them for updated/modified columns.
Compare 2 Lists of the same class and show the different values in a new list
I would like to do this using linq. The only problem is I am dealing with a lot of columns, over excess of 30 columns in each. Any suggestions would be of great help...
//In Dal
List<PartnerAndPartnerPositionEntity> GetAllPartnerAndPartnerPositionOldDB(int modelId);
List<PartnerAndPartnerPositionEntity> GetAllPartnerAndPartnerPosition(int modelId);
//BL
//get from new db
var list1= _partnerDAL.GetAllPartnerAndPartnerPosition(modelId);
//get from old db
var list2= _partnerDAL.GetAllPartnerAndPartnerPositionOldDB(modelId);
Let's assume that:
PartnerAndPartnerPositionEntity class contains a property named Id that represents the unique key of an item
Given the above you can:
Get all properties of your type
var properties = typeof(PartnerAndPartnerPositionEntity).GetProperties();
Join the two lists on the Id property and iterate through the properties to see which one has changed:
var list = list1.Join(list2,
x => x.Id,
y => y.Id,
(x, y) => Tuple.Create(x, y))
.ToList();
list.Foreach(tuple =>
{
foreach(var propertyInfo in properties)
{
var value1 = propertyInfo.GetValue(tuple.Item1, null);
var value2 = propertyInfo.GetValue(tuple.Item2, null);
if(value1 != value2)
Console.WriteLine("Item with id {0} has different values for property {1}.",
tuple.Item1,Id, propertyInfo.Name);
}
});
Well, if you want to avoid doing it the boring and tedious way, you need to use reflection to dynamically get the class members then get their values for each instance.
See C# Reflection - Get field values from a simple class for the code.
There...this will generate new IL!
public static class EqualityHelper
{
private ConcurrentDictionary<Type, object> _cache = new ConcurrentDictionary<Type, object>();
public bool AreEqual<T>(T left, T right)
{
var equality = (Func<T,T,bool>)_cache.GetOrAdd(typeof(T), CreateEquality<T>());
return equality(left, right);
}
private Func<T, T, bool> CreateEquality<T>()
{
var left = Expression.Parameter(typeof(T));
var right = Expression.Parameter(typeof(T));
var properties = from x in typeof(T).GetProperties()
where x.GetIndexParameters().Any() == false
select x;
var expressions = from p in properties
select Expression.Equal(
Expression.Property(left, p),
Expression.Property(right, p));
var body = expressions.Aggregate(Expression.AndAlso);
var lambda = Expression.Lambda<Func<T,T,bool>>(body, new [] {left, right});
return lambda.Compile();
}
}