I want to use a pre-loaded lookup data from a list within a query. I need the query to return as IQueryable because it is used in a grid & being paged (not included here). I need the Labels to load from the lookup to avoid joins (more are in the actual code).
I know I could do a ToList() and post-process it, but I need the IQueryable. Here is the code:
// Run in intialization other code...
var contactLookups = new ContactsDbContext();
List<StatusType> _statusTypes = contactLookups.StatusTypes.ToList();
public IQueryable GetQuery()
{
var contactsCtx = new ContactsDbContext();
IQueryable query = contactsCtx.Select(contact => new
{
Id = contact.Id,
FullName = contact.FirstName + " " + contact.LastName,
CompanyName = contact.CompanyName,
Status = _statusTypes.FirstOrDefault(l => contact.StatusId == l.Id).Label
});
return query;
}
The code above throws an error because EF/LINQ can't form the in-memory list into SQL (for obvious reasons) - unless something is added/changed.
I want to somehow tell EF to apply the lookup after the SQL or something to that effect. I had read about doing something similar with EF Helper code and Expressions, but I can't find that article anymore.
Note, I'm flexible on the lookup itself. The only non-negotiable is the "contact.StatusId" (int), but the rest of the structure "_statusTypes.FirstOrDefault(l => contact.StatusId == l.Id)", like the List type, is open.
Any help is appreciated.
You can wrap EF's query into your own intercepting implementation of IQueryable, in which you can inject values of in-memory lookups prior to returning objects to application.
It may sound complex, but actually is not that hard to implement. The following needs to be done:
Mark the Status property in your entity as non-mapped (using Ignore() with Fluent API or [NotMapped] attribute on the property).
Write InterceptingQueryable<T> (let's name it so) implementation of IOrderedQueryable<T>, which wraps an IQueryable<T> object from the EF (returned by the Select method in your example).
Write InterceptingQueryProvider<T> implementation of IQueryProvider<T>, which in turn wraps query provider obtained from the EF.
Write InterceptingEnumerator<T> implementation of IEnumerator<T>, which relays to enumerator object returned by the EF. This enumerator will inject the value of Status property (can easily be generalized to populate any lookup property this way), right after it performs MoveNext, so that object returned by Current is fully populated.
The above chain is connected as follows:
InterceptingQueryable relays to EF's query object, passed in the constructor.
InterceptingQueryable.Provider property returns InterceptingQueryProvider.
InterceptingQueryable.GetEnumerator method returns InterceptingEnumerator.
InterceptingQueryProvider.CreateQuery methods obtain query object from the EF query provider, then return it wrapped in another instance of InterceptingQueryable.
InterceptingQueryProvider.Execute invokes Execute on the EF query provider, then in the case it gets an entity which is subject to lookup injection, it injects the lookup values in the same way as InterceptingEnumerator does (extract a method for reuse).
UPDATE
Here is the code:
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using System.Text;
using System.Threading.Tasks;
namespace Examples.Queryables
{
public class InterceptingQueryable<T> : IOrderedQueryable<T>
{
private readonly Action<T> _interceptor;
private readonly IQueryable<T> _underlyingQuery;
private InterceptingQueryProvider _provider;
public InterceptingQueryable(Action<T> interceptor, IQueryable<T> underlyingQuery)
{
_interceptor = interceptor;
_underlyingQuery = underlyingQuery;
_provider = null;
}
public IEnumerator<T> GetEnumerator()
{
return new InterceptingEnumerator(_interceptor, _underlyingQuery.GetEnumerator());
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
public Expression Expression
{
get { return _underlyingQuery.Expression; }
}
public Type ElementType
{
get { return _underlyingQuery.ElementType; }
}
public IQueryProvider Provider
{
get
{
if ( _provider == null )
{
_provider = new InterceptingQueryProvider(_interceptor, _underlyingQuery.Provider);
}
return _provider;
}
}
private class InterceptingQueryProvider : IQueryProvider
{
private readonly Action<T> _interceptor;
private readonly IQueryProvider _underlyingQueryProvider;
public InterceptingQueryProvider(Action<T> interceptor, IQueryProvider underlyingQueryProvider)
{
_interceptor = interceptor;
_underlyingQueryProvider = underlyingQueryProvider;
}
public IQueryable CreateQuery(Expression expression)
{
throw new NotImplementedException();
}
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
{
var query = _underlyingQueryProvider.CreateQuery<TElement>(expression);
if ( typeof(T).IsAssignableFrom(typeof(TElement)) )
{
return new InterceptingQueryable<TElement>((Action<TElement>)(object)_interceptor, query);
}
else
{
return query;
}
}
public object Execute(Expression expression)
{
throw new NotImplementedException();
}
public TResult Execute<TResult>(Expression expression)
{
var result = _underlyingQueryProvider.Execute<TResult>(expression);
if ( result is T )
{
_interceptor((T)(object)result);
}
return result;
}
}
private class InterceptingEnumerator : IEnumerator<T>
{
private readonly Action<T> _interceptor;
private readonly IEnumerator<T> _underlyingEnumerator;
private bool _hasCurrent;
public InterceptingEnumerator(Action<T> interceptor, IEnumerator<T> underlyingEnumerator)
{
_interceptor = interceptor;
_underlyingEnumerator = underlyingEnumerator;
_hasCurrent = false;
}
public void Dispose()
{
_underlyingEnumerator.Dispose();
}
public bool MoveNext()
{
_hasCurrent = _underlyingEnumerator.MoveNext();
if ( _hasCurrent )
{
_interceptor(_underlyingEnumerator.Current);
}
return _hasCurrent;
}
public void Reset()
{
_underlyingEnumerator.Reset();
}
public T Current
{
get
{
return _underlyingEnumerator.Current;
}
}
object IEnumerator.Current
{
get { return Current; }
}
}
}
public static class QueryableExtensions
{
public static IOrderedQueryable<T> InterceptWith<T>(this IQueryable<T> query, Action<T> interceptor)
{
return new InterceptingQueryable<T>(interceptor, query);
}
}
}
And here is test case/example. First, we should not forget to add non-mapped Status property to Contact entity:
public partial class Contact
{
[NotMapped]
public StatusType Status { get; set; }
}
Then, we can use the interceptor mechanism as follows:
var contactLookups = contactsCtx.StatusTypes.ToList();
Action<Contact> interceptor = contact => {
contact.Status = contactLookups.FirstOrDefault(l => contact.StatusId == l.Id);
};
// note that we add InterceptWith(...) to entity set
var someContacts =
from c in contactsCtx.Contacts.InterceptWith(interceptor)
where c.FullName.StartsWith("Jo")
orderby c.FullName, c.CompanyName
select c;
Console.WriteLine("--- SOME CONTACTS ---");
foreach ( var c in someContacts )
{
Console.WriteLine(
"{0}: {1}, {2}, {3}={4}",
c.Id, c.FullName, c.CompanyName, c.StatusId, c.Status.Name);
}
which prints:
--- SOME CONTACTS ---
1: John Smith, Acme Corp, 3=Status Three
3: Jon Snow, The Wall, 2=Status Two
and the query gets translated into:
SELECT
[Extent1].[Id] AS [Id],
[Extent1].[FullName] AS [FullName],
[Extent1].[CompanyName] AS [CompanyName],
[Extent1].[StatusId] AS [StatusId]
FROM [dbo].[Contacts] AS [Extent1]
WHERE [Extent1].[FullName] LIKE 'Jo%'
ORDER BY [Extent1].[FullName] ASC, [Extent1].[CompanyName] ASC
I'm not sure what is the benefit of avoiding the joins compared to the obvious drawback of not being able to process the whole query at the database side, but what you are asking for can be achieved by doing as much as possible (filtering, ordering, grouping, projection) with Linq to Entities, then turn it into IEnumerable and do the rest with Linq To Objects. You can always use Enumerable.AsQueryable to switch to IQueryable implementation over IEnumerable. Something like this
public IQueryable GetQuery()
{
var db = new ContactsDbContext();
var query = db.Contacts.Select(contact => new
{
Id = contact.Id,
FullName = contact.FirstName + " " + contact.LastName,
CompanyName = contact.CompanyName,
StatusId = contact.StatusId
})
.AsEnumerable()
.Select(contact => new
{
Id = contact.Id,
FullName = contact.FullName,
CompanyName = contact.CompanyName,
Status = _statusTypes.FirstOrDefault(l => contact.StatusId == l.Id).Label
})
.AsQueryable();
return query;
}
Related
I develop simple app with Business layer and Data access layer (using EF). On my BL I define interfase IRepository.
public interface IRepository<T> where T : class
{
public Task<T> GetAsync(object id);
public Task<T> GetFirstByAsync(Func<T, bool> predicate);
public Task<IEnumerable<T>> GetAllAsync();
public Task<int> CountAsync();
public Task<T> AddAsync(T #object);
public Task AddRangeAsync(IEnumerable<T> objects);
public Task DeleteAllAsync();
}
Also on BL I have Parameter model
public class Parameter : IEquatable<Parameter>
{
private readonly double _tolerance;
public int Number { get; }
public string Name { get; }
public double LowerLimit { get; }
public double UpperLimit { get; }
public string Unit { get; }
public Parameter(int number, string name, double lowerLimit, double upperLimit, string unit)
{
if (number < 0)
{
throw new ArgumentException($"Parameter number cannot be less than 0");
}
if (string.IsNullOrWhiteSpace(name))
{
throw new ArgumentNullException(nameof(name));
}
if (lowerLimit >= upperLimit)
{
throw new ArgumentException($"Parameter {nameof(lowerLimit)} cannot be >= {nameof(upperLimit)}");
}
Number = number;
Name = name;
LowerLimit = lowerLimit;
UpperLimit = upperLimit;
Unit = unit;
_tolerance = (upperLimit - lowerLimit) / 1e10;
}
public bool Equals(Parameter other)
{
if (other is null)
{
return false;
}
if (ReferenceEquals(this, other))
{
return true;
}
return Number == other.Number && Name == other.Name && Unit == other.Unit;
}
public override int GetHashCode()
{
return HashCode.Combine(Number, Name, Unit);
}
public bool StrongEquals(Parameter other)
{
return Equals(other) &&
Math.Abs(LowerLimit - other.LowerLimit) < _tolerance &&
Math.Abs(UpperLimit - other.UpperLimit) < _tolerance;
}
}
On DAL I have Parameter entity
public class Parameter : BaseEntity
{
public int Number { get; set; }
public string Name { get; set; }
public double LowerLimit { get; set; }
public double UpperLimit { get; set; }
public string Unit { get; set; }
public virtual ICollection<Measurement> Measurements { get; set; } //navigation
}
public class BaseEntity
{
public int Id { get; set; }
}
So then I start to develop ParameterReporitory class on DAL which implements IRepository. In such case I decided to close interface type with a parameter 'Parameter' from BL because from BL perspective it's not neccessary to work with the entity classes themselves. My BL and DAL models are basically small so it's not neccessary for me to use some mapping library such Automapper. I just wrote extensions methods ToBusinessModel()/ ToDbEntity() to convert BL model to DAL entity and backward. I can implement all interface methods in ParameterRepository besides GetFirstBy(Func<ParameterModel, bool> predicate) because I doubted whether it was right to try to do such a transformation Func<ParameterModel, bool> to Func<ParameterEntity, bool> like I did it in code. When I try to execute this method I get an exception
The LINQ expression 'DbSet()
.Where(p => Invoke(__predicate_0, p.ToBusinessModel())
)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by
inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or
'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for
more information.
What can I do to prevent this exception and implement this method in repository class corectly? Perhaps, I don't want to execute this expression on client side because it leads to get all entities from DB in memory and then filter them. So, what can I do other than that? Please pay attention to these definitions in the code
using ParameterEntity = DatabaseAccess.Models.Parameter;
using ParameterModel = BusinessLayer.Models.Parameter;
using DatabaseAccess.Extensions;
using Microsoft.EntityFrameworkCore;
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using ParameterEntity = DatabaseAccess.Models.Parameter;
using ParameterModel = BusinessLayer.Models.Parameter;
using System.Linq;
using BusinessLayer.Interfaces;
namespace DatabaseAccess
{
public class ParameterRepository : IRepository<ParameterModel>
{
private readonly MeasurementContext _context;
private readonly DbSet<ParameterEntity> _dbSet;
public ParameterRepository(MeasurementContext context)
{
_context = context ?? throw new ArgumentNullException(nameof(context));
_dbSet = _context.Parameters;
}
public async Task<ParameterModel> GetAsync(object id) => (await _dbSet.FindAsync(id)).ToBusinessModel();
public async Task<ParameterModel> GetFirstByAsync(Func<ParameterModel, bool> predicate)
{
Expression<Func<ParameterEntity, bool>> entityPredicate = x => predicate(x.ToBusinessModel());
return (await _dbSet.Where(entityPredicate).FirstOrDefaultAsync()).ToBusinessModel();
}
public async Task<ParameterModel> AddAsync(ParameterModel parameterModel)
{
if (!await IsExistAsync(parameterModel))
{
_dbSet.Add(parameterModel.ToDbEntity());
await _context.SaveChangesAsync();
}
return parameterModel;
}
public async Task DeleteAllAsync()
{
_dbSet.RemoveRange(_dbSet);
await _context.SaveChangesAsync();
}
public async Task AddRangeAsync(IEnumerable<ParameterModel> parameterModels)
{
var uniqueParameters = parameterModels.Distinct();
foreach (var parameter in uniqueParameters)
{
if (!await IsExistAsync(parameter))
{
_dbSet.Add(parameter.ToDbEntity());
}
}
await _context.SaveChangesAsync();
}
public async Task<IEnumerable<ParameterModel>> GetAllAsync() => (await _dbSet.ToListAsync()).ToBusinessModels();
public async Task<int> CountAsync() => await _dbSet.CountAsync();
public async Task<ParameterModel> FindByNumberAsync(int number) => (await _dbSet.FirstOrDefaultAsync(p => p.Number == number)).ToBusinessModel();
public async Task<bool> IsExistAsync(ParameterModel parameterModel) => (await FindByNumberAsync(parameterModel.Number)) != null;
}
}
My extension methods:
public static class MappingExtensions
{
public static ParameterModel ToBusinessModel(this ParameterEntity parameter)
{
if (parameter is null)
{
return null;
}
return new ParameterModel(
parameter.Number,
parameter.Name,
parameter.LowerLimit,
parameter.UpperLimit,
parameter.Unit);
}
public static ParameterEntity ToDbEntity(this ParameterModel parameter)
{
if (parameter is null)
{
return null;
}
return new ParameterEntity
{
Number = parameter.Number,
Name = parameter.Name,
LowerLimit = parameter.LowerLimit,
UpperLimit = parameter.UpperLimit,
Unit = parameter.Unit
};
}
}
I also wanted to know if this is a normal approach to create a repository class for an entity from BL (ParameterModel). Maybe it would be more correct to make a repository that works with ParameterEntity and then make something like a decorator as a repository that will already work with the ParameterModel (from BL). This decorator repository will call methods of decoratee repository. Or perhaps you can advise me on the best approach.
First, don't misleading Func<T, bool> and Expression<Func<T, bool>>.
Func<T, bool>
It's a delegate to invoke a method.
Expression<Func<T, bool>>
A expression can be parsed. For example to be translated to SQL like EF Core do.
What can I do to prevent this exception and implement this method in repository class corectly?
EF Core translate expression to SQL to query the DB. For example, the expression :
Expression<Func<ParameterEntity, bool>> predicateExp = x => x.Name.Contains("foo");
is translated to [name] like '%foo%'. But how do you want it translate this?
ParameterModel m = x.ToBusinessModel();
bool predicateResult = predicate(m);
return predicateResult;
EF Core don't know and throw the error you see. To avoid the error, you can use client evaluation :
public async Task<ParameterModel> GetFirstByAsync(Func<ParameterModel, bool> predicate)
{
Expression<Func<ParameterEntity, bool>> entityPredicate = x => predicate(x.ToBusinessModel());
return (await _dbSet.ToList().Where(entityPredicate).FirstOrDefaultAsync()).ToBusinessModel();
}
With ToList(), EF Core will load the full table in memory and it's the C# program that will apply the predicate. If the table has a lot of row, this can be a big memory pressure... don't do this.
Perhaps, I don't want to execute this expression on client side because it leads to get all entities from DB in memory and then filter them. So, what can I do other than that?
EF Core can't translate Expression<Func<ParameterModel, bool>> to Expression<Func<ParameterEntity, bool>>. But you can manually parse Expression<Func<ParameterModel, bool>> to generate Expression<Func<ParameterEntity, bool>>.
It's can be funny and challenging, but need a lot of work and sound error prone... I don't recommend.
Or perhaps you can advise me on the best approach.
My favorite advise about repository pattern with EF Core is don't use the repository pattern. Use directly the DbContext, DbContext is the DAL.
The rare case when the repository pattern is useful, it's to encapsulate all access to the DB. It's help to have a good view on db access and optimize the DB.
In the OP code, the repository allow to filter on all columns... you need to inspect all code to find that index is necessary in the DB.
But if the repository allow only to filter on ID and name, then you only need to add the index on this corresponding columns.
I want to filter an IEnumerable object by a specific property of whatever object it is collecting. I want the option to filter by one or more property value but how many values (and what values) to filter by is only known at runtime.
Ok, so to give an example, the collected objects could be the following struct:
public struct Person
{
public string Name { get; set; }
public string Profession{ get; set; }
}
This struct could then be used by the following list, which I have populated with some arbitrary values:
List<Person> people= new List<Person>;
people.Add(new Person(){Name = "Mickey", Profession="Tinker"};
people.Add(new Person(){Name = "Donald", Profession="Tailor"};
people.Add(new Person(){Name = "Goofy", Profession="Soldier"};
people.Add(new Person(){Name = "Pluto", Profession="Spy"};
This is then put into an IEnumerable (all of them are transferred to it first)
var wantedPeople = from n in this.people select n;
So say a user was only interested in the "Tailor" and "Spy" professions, and via some sort of gui trickery the following collection was created:
List<string> wantedProfessions = new List<string>();
wantedProfessions.Add("Tailor");
wantedProfessions.Add("Spy");
Now what Linq statement can I use to filer my wantedPeople so it only includes the tailor and spy entries?
I know I could use a where clause but I don't know how to tailor it to get what I want (and doing the following is not what I want as it only works with the wantedProfessions collection above (e.g. this collection will change at runtime):
wantedPeople = from n in wantedPeople
where n.Profession == wantedProffessions[0] || n.Profession == wantedProffessions[1]
select n;
If you want to check any wanted profession from given list:
wantedPeople = from n in wantedPeople
where wantedProffessions.Contains(n.Profession)
select n;
Or you can build query with lambda syntax by applying filters one by one:
var query = people.AsEnumerable();
if (!String.IsNullOrEmpty(name))
query = query.Where(p => p.Name == name);
if (wantedProfessions.Any())
query = query.Where(p => wantedProfessions.Contains(p.Profession));
If you wanted to create more complex filters, like some name, and several professions, you can use Specification pattern. Specification can be defined by this simple interface:
public interface ISpecification<T>
{
bool Satisfied(T entity);
}
It just checks whether given entity (person) satisfies specification. Specification also look very simple:
public class PersonNameSpecification : ISpecification<Person>
{
private string _name;
public PersonNameSpecification(string name)
{
_name = name;
}
public bool Satisfied(Person person)
{
return person.Name == _name;
}
}
Profession specification:
public class PersonProfessionSpecification : ISpecification<Person>
{
private string[] _professions;
public PersonProfessionSpecification(params string[] professions)
{
_professions = professions;
}
public bool Satisfied(Person person)
{
return _professions.Contains(person.Profession);
}
}
You can create specifications which implement boolean logic, like OrSpecification or AndSpecification:
public class AndSpecification<T> : ISpecification<T>
{
private ISpecification<T> _specA;
private ISpecification<T> _specB;
public AndSpecification(ISpecification<T> specA, ISpecification<T> specB)
{
_specA = specA;
_specB = specB;
}
public bool Satisfied(T entity)
{
return _specA.Satisfied(entity) && _specB.Satisfied(entity);
}
}
public static class SpecificationExtensions
{
public static ISpecification<T> And<T>(
this ISpecification<T> specA, ISpecification<T> specB)
{
return new AndSpecification<T>(specA, specB);
}
}
Now you can create complex specification which describes people you want to get:
var professionSpec = new PersonProfessionSpecification("Tailor", "Spy");
var nameSpec = new PersonNameSpecification("Pluto");
var spec = professionSpec.And(nameSpec);
And get required people:
var result = people.Where(spec.Satisfied);
Sergey B's Solution is the correct one for your example.
Assuming that you weren't using a collection that had the Contains() method, you could also do the following:
var wantedPeople = from n in people
from p in wantedProffessions
where n.Profession.Equals(p)
select n;
for access control purposes in a intensive DB use system I had to implement an objectset wrapper, where the AC will be checked.
The main objective is make this change preserving the existing code for database access, that is implemented with linq to entities all over the classes (there is no centralized layer for database).
The ObjectSetWrapper created is like that:
public class ObjectSetWrapper<TEntity> : IQueryable<TEntity> where TEntity : EntityObject
{
private IQueryable<TEntity> QueryableModel;
private ObjectSet<TEntity> ObjectSet;
public ObjectSetWrapper(ObjectSet<TEntity> objectSetModels)
{
this.QueryableModel = objectSetModels;
this.ObjectSet = objectSetModels;
}
public ObjectQuery<TEntity> Include(string path)
{
return this.ObjectSet.Include(path);
}
public void DeleteObject(TEntity #object)
{
this.ObjectSet.DeleteObject(#object);
}
public void AddObject(TEntity #object)
{
this.ObjectSet.AddObject(#object);
}
public IEnumerator<TEntity> GetEnumerator()
{
return QueryableModel.GetEnumerator();
}
public Type ElementType
{
get { return typeof(TEntity); }
}
public System.Linq.Expressions.Expression Expression
{
get { return this.QueryableModel.Expression; }
}
public IQueryProvider Provider
{
get { return this.QueryableModel.Provider; }
}
public void Attach(TEntity entity)
{
this.ObjectSet.Attach(entity);
}
public void Detach(TEntity entity)
{
this.ObjectSet.Detach(entity);
}
IEnumerator IEnumerable.GetEnumerator()
{
return this.QueryableModel.GetEnumerator();
}
}
It's really simple and works for simple queries, like that:
//db.Product is ObjectSetWrapper<Product>
var query = (from item in db.Product where item.Quantity > 0 select new { item.Id, item.Name, item.Value });
var itensList = query.Take(10).ToList();
But when I have subqueries like that:
//db.Product is ObjectSetWrapper<Product>
var query = (from item in db.Product
select new
{
Id = item.Id,
Name = item.Name,
SalesQuantity = (from sale in db.Sale where sale.ProductId == item.Id select sale.Id).Count()
}).OrderByDescending(x => x.SalesQuantity);
var productsList = query.Take(10).ToList();
I get NotSupportedException, saying I can't create a constant value of my inner query entity type:
Unable to create a constant value of type 'MyNamespace.Model.Sale'.
Only primitive types or enumeration types are supported in this
context.
How can I get my queries working? I don't really need to make my wrapper an ObjectSet type, I just need to use it in queries.
Updated
I have changed my class signature. Now it's also implementing IObjectSet<>, but I'm getting the same NotSupportedException:
public class ObjectSetWrapper<TEntity> : IQueryable<TEntity>, IObjectSet<TEntity> where TEntity : EntityObject
EDIT:
The problem is that the following LINQ construction is translated into LINQ expression containing your custom class inside (ObjectSetWrapper).
var query = (from item in db.Product
select new
{
Id = item.Id,
Name = item.Name,
SalesQuantity = (from sale in db.Sale where sale.ProductId == item.Id select sale.Id).Count()
}).OrderByDescending(x => x.SalesQuantity);
LINQ to Entities tries to convert this expression into SQL statement, but it has no idea how to deal with the custom classes (as well as custom methods).
The solution in such cases is to replace IQueryProvider with the custom one, which should intercept the query execution and translate LINQ expression, containing custom classes/methods into valid LINQ to Entities expression (which operates with entities and object sets).
Expression conversion is performed using the class, derived from ExpressionVisitor, which performs expression tree traversal, replacing relevant nodes, to the nodes which can be accepted by LINQ to Entities
Part 1 - IQueryWrapper
// Query wrapper interface - holds and underlying query
interface IQueryWrapper
{
IQueryable UnderlyingQueryable { get; }
}
Part 2 - Abstract QueryWrapperBase (not generic)
abstract class QueryWrapperBase : IQueryProvider, IQueryWrapper
{
public IQueryable UnderlyingQueryable { get; private set; }
class ObjectWrapperReplacer : ExpressionVisitor
{
public override Expression Visit(Expression node)
{
if (node == null || !typeof(IQueryWrapper).IsAssignableFrom(node.Type)) return base.Visit(node);
var wrapper = EvaluateExpression<IQueryWrapper>(node);
return Expression.Constant(wrapper.UnderlyingQueryable);
}
public static Expression FixExpression(Expression expression)
{
var replacer = new ObjectWrapperReplacer();
return replacer.Visit(expression);
}
private T EvaluateExpression<T>(Expression expression)
{
if (expression is ConstantExpression) return (T)((ConstantExpression)expression).Value;
var lambda = Expression.Lambda(expression);
return (T)lambda.Compile().DynamicInvoke();
}
}
protected QueryWrapperBase(IQueryable underlyingQueryable)
{
UnderlyingQueryable = underlyingQueryable;
}
public abstract IQueryable<TElement> CreateQuery<TElement>(Expression expression);
public abstract IQueryable CreateQuery(Expression expression);
public TResult Execute<TResult>(Expression expression)
{
return (TResult)Execute(expression);
}
public object Execute(Expression expression)
{
expression = ObjectWrapperReplacer.FixExpression(expression);
return typeof(IQueryable).IsAssignableFrom(expression.Type)
? ExecuteQueryable(expression)
: ExecuteNonQueryable(expression);
}
protected object ExecuteNonQueryable(Expression expression)
{
return UnderlyingQueryable.Provider.Execute(expression);
}
protected IQueryable ExecuteQueryable(Expression expression)
{
return UnderlyingQueryable.Provider.CreateQuery(expression);
}
}
Part 3 - Generic QueryWrapper<TElement>
class QueryWrapper<TElement> : QueryWrapperBase, IOrderedQueryable<TElement>
{
private static readonly MethodInfo MethodCreateQueryDef = GetMethodDefinition(q => q.CreateQuery<object>(null));
public QueryWrapper(IQueryable<TElement> underlyingQueryable) : this(null, underlyingQueryable)
{
}
protected QueryWrapper(Expression expression, IQueryable underlyingQueryable) : base(underlyingQueryable)
{
Expression = expression ?? Expression.Constant(this);
}
public virtual IEnumerator<TElement> GetEnumerator()
{
return ((IEnumerable<TElement>)Execute<IEnumerable>(Expression)).GetEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
public Expression Expression { get; private set; }
public Type ElementType
{
get { return typeof(TElement); }
}
public IQueryProvider Provider
{
get { return this; }
}
public override IQueryable CreateQuery(Expression expression)
{
var expressionType = expression.Type;
var elementType = expressionType
.GetInterfaces()
.Single(t => t.IsGenericType && t.GetGenericTypeDefinition() == typeof(IEnumerable<>))
.GetGenericArguments()
.Single();
var createQueryMethod = MethodCreateQueryDef.MakeGenericMethod(elementType);
return (IQueryable)createQueryMethod.Invoke(this, new object[] { expression });
}
public override IQueryable<TNewElement> CreateQuery<TNewElement>(Expression expression)
{
return new QueryWrapper<TNewElement>(expression, UnderlyingQueryable);
}
private static MethodInfo GetMethodDefinition(Expression<Action<QueryWrapper<TElement>>> methodSelector)
{
var methodCallExpression = (MethodCallExpression)methodSelector.Body;
return methodCallExpression.Method.GetGenericMethodDefinition();
}
}
Part 4 - finally your ObjectSetWrapper
public class ObjectSetWrapper<TEntity> : IQueryable<TEntity>, IQueryWrapper where TEntity : class
{
private IQueryable<TEntity> QueryableModel;
private ObjectSet<TEntity> ObjectSet;
public ObjectSetWrapper(ObjectSet<TEntity> objectSetModels)
{
this.QueryableModel = new QueryWrapper<TEntity>(objectSetModels);
this.ObjectSet = objectSetModels;
}
public ObjectQuery<TEntity> Include(string path)
{
return this.ObjectSet.Include(path);
}
public void DeleteObject(TEntity #object)
{
this.ObjectSet.DeleteObject(#object);
}
public void AddObject(TEntity #object)
{
this.ObjectSet.AddObject(#object);
}
public IEnumerator<TEntity> GetEnumerator()
{
return QueryableModel.GetEnumerator();
}
public Type ElementType
{
get { return typeof(TEntity); }
}
public System.Linq.Expressions.Expression Expression
{
get { return this.QueryableModel.Expression; }
}
public IQueryProvider Provider
{
get { return this.QueryableModel.Provider; }
}
public void Attach(TEntity entity)
{
this.ObjectSet.Attach(entity);
}
public void Detach(TEntity entity)
{
this.ObjectSet.Detach(entity);
}
IEnumerator IEnumerable.GetEnumerator()
{
return this.QueryableModel.GetEnumerator();
}
IQueryable IQueryWrapper.UnderlyingQueryable
{
get { return this.ObjectSet; }
}
}
Your inner query fails because you are referencing another dataset when you should be traversing foreign keys:
SalesQuantity = item.Sales.Count()
I can't get the QueryObjectGraph to add INCLUDE child tables if my life depended on it...what am I missing? Stuck for third day on something that should be simple :-/
DAL:
public abstract class RepositoryBase<T> where T : class
{
private MyLPL2Context dataContext;
private readonly IDbSet<T> dbset;
protected RepositoryBase(IDatabaseFactory databaseFactory)
{
DatabaseFactory = databaseFactory;
dbset = DataContext.Set<T>();
DataContext.Configuration.LazyLoadingEnabled = true;
}
protected IDatabaseFactory DatabaseFactory
{
get;
private set;
}
protected MyLPL2Context DataContext
{
get { return dataContext ?? (dataContext = DatabaseFactory.Get()); }
}
public IQueryable<T> QueryObjectGraph(Expression<Func<T, bool>> filter,
params string[] children)
{
foreach (var child in children)
{
dbset.Include(child);
}
return dbset.Where(filter);
}
...
DAL repositories
public interface IBreed_TranslatedSqlRepository : ISqlRepository<Breed_Translated>
{
}
public class Breed_TranslatedSqlRepository : RepositoryBase<Breed_Translated>,
IBreed_TranslatedSqlRepository
{
public Breed_TranslatedSqlRepository(IDatabaseFactory databaseFactory)
: base(databaseFactory)
{}
}
BLL Repo:
public IQueryable<Breed_Translated>
QueryObjectGraph(Expression<Func<Breed_Translated, bool>> filter,
params string[] children)
{
return _r.QueryObjectGraph(filter, children);
}
Controller:
var breeds1 = _breedTranslatedRepository
.QueryObjectGraph(b => b.Culture == culture, new string[] { "AnimalType_Breed" })
.ToList();
I can't get to Breed.AnimalType_Breed.AnimalTypeId ..I can drill as far as Breed.AnimalType_Breed then the intelisense expects an expression?
Cues if any, DB Tables: bold is many-to-many
Breed, Breed_Translated, AnimalType_Breed, AnimalType, ...
AnimalBreed_Type represents many to many relation so the AnimalBreed_Type property in Breed_Translated entity is collection! The collection type doesn't have your table properties. You must use First or Single to get single related entity from this collection and check it's AnimalTypeId.
If you look at Include it has a return type. Linq is mostly functional so it will NOT alter any objects but rather return new ones. You need to store this new object instead.
Try:
var query = dbset.Where(filter);
foreach (var child in children)
{
query = query.Include(child);
}
return query;
OTHER NOTE: You can simplify this:
var breeds1 = _breedTranslatedRepository
.QueryObjectGraph(b => b.Culture == culture, new string[] { "AnimalType_Breed" })
.ToList();
To
var breeds1 = _breedTranslatedRepository
.QueryObjectGraph(b => b.Culture == culture, "AnimalType_Breed")
.ToList();
When you methods takes a params string[] argument
My masters thesis is about discovering bad database design by analyzing metadata and the data stored. We do this by extracting a metadata model from a given DBMS and then running a set of rules on this metadata.
To extend this process with data analysis, we need to allow rules to query the database directly, but we must retain DBMS independence, such that queries can be applied to PostgreSQL, MSSQL and MySQL.
We have discussed a sort of functional construction of queries such as:
new Query(new Select(columnID), new From(tableID), new Where(new Equality(columnID1, columnID2)))
And then using a DBMS-specific serializer.
Another approach is to let rules handle it all by themselves:
public Query QueryDatabase(DBMS dbms)
{
if (dbms == PostgreSQL) { return "select count(1) from Users"}
if (dbms == MSSQL) {return ....}
}
Are we missing something? Does all this in fact exist in a nice library somewhere? And yes, we have looked at Entity frameworks, but they seem to rely on a statically types model of the database, which for obvious reasons cannot be created.
I should mention that we maintain an extensible rule architecture, allowing end users to implement their own rules.
To clarify what we want to achieve, look at the following query (mssql), it needs two parameters, the name of the table (#table) and the name of the column (#column):
DECLARE #TotalCount FLOAT;
SELECT #TotalCount = COUNT(1) FROM [#table];
SELECT SUM(pcount * LOG10(#TotalCount / pcount)) / (LOG10(2) * #TotalCount)
FROM (SELECT (Count([#column])) as pcount
FROM [#table]
GROUP BY [#column]) as exp1
The query measures the amount of information stored in a given attribute, by estimating the entropy. It needs to access all rows in the table. To avoid extracting all rows from the database and transferring them over a slow network connection it is better to express them in SQL an only transfer a single number.
NOTE: We DO have all the metadata we need. This question is only for accessing data!
I was not very sure of whether to add this to my already long question, edit an existing answer or what todo. Please feel free to advise. ;)
Building on mrnye answer:
new Query()
.Variable(varname => FLOAT)
.Set(varname => new Query().Count(1).From(table) )
.Select(new Aggregate().Sum(varname => "pcount * LOG10(varname / pcount)"))
.From(
new Query()
.Select(pcount => new Aggregate().Count(column)
.From(table)
.GroupBy(column)
)
Syntax errors and misuse of lambda statements aside, i played with the idea of using some extension methods for building queries. It does seem as a fairly complex approach. How would you think about such an approach?
Building on the LINQ answer:
let totalCount = Table.Count
from uv un from r in Table
group r by r["attr"]
select r.Count
select r.Count * Log2((totalCount / r.Count))
Seems fairly nice, but a helluva lot to implement...
You could achieve the same by implementing a custom LINQ provider infrastructure. The queries are generic, but the AST tree visitors that generate the SQL queries can be made pluggable. You can even mock a database using a in memory data store and translating your custom LINQ query to a LINQ to objects query!
You would need to create a provider that would know how to extract the column name from the object's indexer. Here is a basic framework that you can extend:
// Runs in LinqPad!
public class TableQueryObject
{
private readonly Dictionary<string, object> _data = new Dictionary<string, object>();
public string TableName { get; set; }
public object this[string column]
{
get { return _data.ContainsKey(column) ? _data[column] : null; }
set { if (_data.ContainsKey(column)) _data[column] = value; else _data.Add(column, value); }
}
}
public interface ITableQuery : IEnumerable<TableQueryObject>
{
string TableName { get; }
string ConnectionString { get; }
Expression Expression { get; }
ITableQueryProvider Provider { get; }
}
public interface ITableQueryProvider
{
ITableQuery Query { get; }
IEnumerable<TableQueryObject> Execute(Expression expression);
}
public interface ITableQueryFactory
{
ITableQuery Query(string tableName);
}
public static class ExtensionMethods
{
class TableQueryContext : ITableQuery
{
private readonly ITableQueryProvider _queryProvider;
private readonly Expression _expression;
public TableQueryContext(ITableQueryProvider queryProvider, Expression expression)
{
_queryProvider = queryProvider;
_expression = expression;
}
public string TableName { get { return _queryProvider.Query.TableName; } }
public string ConnectionString { get { return _queryProvider.Query.ConnectionString; } }
public Expression Expression { get { return _expression; } }
public ITableQueryProvider Provider { get { return _queryProvider; } }
public IEnumerator<TableQueryObject> GetEnumerator() { return Provider.Execute(Expression).GetEnumerator(); }
IEnumerator IEnumerable.GetEnumerator() { return GetEnumerator(); }
}
public static MethodInfo MakeGeneric(MethodBase method, params Type[] parameters)
{
return ((MethodInfo)method).MakeGenericMethod(parameters);
}
public static Expression StaticCall(MethodInfo method, params Expression[] expressions)
{
return Expression.Call(null, method, expressions);
}
public static ITableQuery CreateQuery(this ITableQueryProvider source, Expression expression)
{
return new TableQueryContext(source, expression);
}
public static IEnumerable<TableQueryObject> Select<TSource>(this ITableQuery source, Expression<Func<TSource, TableQueryObject>> selector)
{
return source.Provider.CreateQuery(StaticCall(MakeGeneric(MethodBase.GetCurrentMethod(), typeof(TSource)), source.Expression, Expression.Quote(selector)));
}
public static ITableQuery Where(this ITableQuery source, Expression<Func<TableQueryObject, bool>> predicate)
{
return source.Provider.CreateQuery(StaticCall((MethodInfo)MethodBase.GetCurrentMethod(), source.Expression, Expression.Quote(predicate)));
}
}
class SqlTableQueryFactory : ITableQueryFactory
{
class SqlTableQuery : ITableQuery
{
private readonly string _tableName;
private readonly string _connectionString;
private readonly ITableQueryProvider _provider;
private readonly Expression _expression;
public SqlTableQuery(string tableName, string connectionString)
{
_connectionString = connectionString;
_tableName = tableName;
_provider = new SqlTableQueryProvider(this);
_expression = Expression.Constant(this);
}
public IEnumerator<TableQueryObject> GetEnumerator() { return Provider.Execute(Expression).GetEnumerator(); }
IEnumerator IEnumerable.GetEnumerator() { return GetEnumerator(); }
public string TableName { get { return _tableName; } }
public string ConnectionString { get { return _connectionString; } }
public Expression Expression { get { return _expression; } }
public ITableQueryProvider Provider { get { return _provider; } }
}
class SqlTableQueryProvider : ITableQueryProvider
{
private readonly ITableQuery _query;
public ITableQuery Query { get { return _query; } }
public SqlTableQueryProvider(ITableQuery query) { _query = query; }
public IEnumerable<TableQueryObject> Execute(Expression expression)
{
//var connecitonString = _query.ConnectionString;
//var tableName = _query.TableName;
// TODO visit expression AST (generate any sql dialect you want) and execute resulting sql
// NOTE of course the query can be easily parameterized!
// NOTE here the fun begins, just return some dummy data for now :)
for (int i = 0; i < 100; i++)
{
var obj = new TableQueryObject();
obj["a"] = i;
obj["b"] = "blah " + i;
yield return obj;
}
}
}
private readonly string _connectionString;
public SqlTableQueryFactory(string connectionString) { _connectionString = connectionString; }
public ITableQuery Query(string tableName)
{
return new SqlTableQuery(tableName, _connectionString);
}
}
static void Main()
{
ITableQueryFactory database = new SqlTableQueryFactory("SomeConnectionString");
var result = from row in database.Query("myTbl")
where row["someColumn"] == "1" && row["otherColumn"] == "2"
where row["thirdColumn"] == "2" && row["otherColumn"] == "4"
select row["a"]; // NOTE select executes as linq to objects! FTW
foreach(var a in result)
{
Console.WriteLine(a);
}
}
I think the LINQ route is the way to go, but for fun I tried to think of a solution. It needs some work, but the general idea is to have the query interface fluent and hide the implementation logic behind interfaces. Just throwing it out there as food for thought...
public interface IDBImplementation
{
public void ProcessQuery(Select query);
}
public class SqlServerImplementation : IDBImplementation
{
public void ProcessQuery(Select query)
{
string sqlQuery = "SELECT " + String.Join(", ", query.Columns)
+ " FROM " + query.TableName + " WHERE " + String.Join(" AND ", query.Conditions);
// execute query...
}
}
public class Select
{
public Select(params string[] columns)
{
Columns = columns;
}
public string[] Columns { get; set; }
public string TableName { get; set; }
public string[] Conditions { get; set; }
}
public static class Extensions
{
public static Select From(this Select select, string tableName)
{
select.TableName = tableName;
return select;
}
public static Select Where(this Select select, params string[] conditions)
{
select.Conditions = conditions;
return select;
}
}
public static class Main
{
public static void Example()
{
IDBImplementation database = new SqlServerImplementation();
var query = new Select("a", "b", "c").From("test").Where("c>5", "b<10");
database.ProcessQuery(query);
}
}
The most DBMS-independent way to retrieve information about a database is iNFORMATION_SCHEMA. See MySQL, SQL Server, PostgreSQL.
I'm kind of curious what type of rules you're thinking of. :)
What about NHibernate? http://community.jboss.org/wiki/NHibernateforNET