I have a couple of areas in an application I am building where it looks like I may have to violate the living daylights out of the DRY (Don't Repeat Yourself) principle. I'd really like to stay dry and not get hosed and wondered if someone might be able to offer me a poncho. For background, I am using C#/.NET 3.51 SP1, Sql Server 2008, and Linq-to-Sql.
Basically, my situations revolve around the following scenario. I need to be able to retrieve either a filtered list of items from virtually any table in my database or I need to be able to retrieve a single item from any table in my database given the id of the primary key. I am pretty sure that the best solutions to these problems will involve a good dose of generics and/or reflection.
Here are the two challenges in a little more depth. (Please forgive the verbosity.)
Given a table name (or perhaps a pluralized table name), I would like to be able to retrieve a filtered list of elements in the table. Specifically, this functionality will be used with lookup tables. (There are approximately 50 lookup tables in this database. Additional tables will frequently be added and/or removed.) The current lookup tables all implement an interface (mine) called IReferenceData and have fields of ID (PK), Title, Description, and IsActive.
For each of these lookup tables, I need to sometimes return a list of all records. Other times I need to only return the active records. Any Linq-to-Sql data context automatically contains a List property for each and every TableName. Unfortunately, I don't believe I can use this in it's raw form because it is unfiltered, and I need to apply a filter on the IsActive property.
One option is to write code similar to the following for all 50 tables. Yuk!!!
public List<AAA> GetListAAA(bool activeOnly)
{
return AAAs.Where(b => b.IsActive == true || b.IsActive == activeOnly).OrderBy(c => c.Title).ToList();
}
This would not be terribly hard, but it does add a burden to maintenance.
Note: It is important that when the list is returned that I maintain the underlying data type. The records in these lookup tables may be modified, and I have to apply the updates appropriately.
For each of my 150 tables, I need to be able to retrieve an individual record (FirstOrDefault or SingleOrDefault) by its primary key id. Again, I would prefer not to write this same code many times. I would prefer to have one method that could be used for all of my tables.
I am not really sure what the best approach would be here. Some possibilities that crossed my mind included the following. (I don't have specific ideas for their implementation. I am simply listing them as food for thought.)
A. Have a method like GetTableNameItemByID (Guid id) on the data context. (Good)
B. Have an extension method like GetItem(this, string tableName, Guid id) on the data context. (Better)
C. Have a Generic method or extension method like GetItem (this, Table, Guid id). (I don't even know if this possible but it would be the cleanest to use.) (Best)
Additional Notes
For a variety of reasons, I have already created a partial class for my data context. It would certainly be acceptable if the methods were included in that partial class either as normal methods or in a separate static class for extension methods.
Since you already have a partial implementation of your data context, you could add:
public IQueryable<T> GetList<T>( bool activeOnly ) where T : class, IReferenceData
{
return this.GetTable<T>()
.Where( b => !activeOnly || b.isActive )
.OrderBy( c => c.Title );
}
Retaining the IQueryable character of the data will defer the execution of the query until you are ready to materialize it. Note that you may want to omit the default ordering or have separate methods with and without ordering to allow you to apply different orderings if you desire. If you leave it as an IQueryable, this is probably more valuable since you can use it with paging to reduce the amount of data actually returned (per query) if you desire.
There's a design pattern for your needs called "Generic Repository" .Using this pattern you'll get an IQueryable instead of a real list of your entities which lets you do some other stuff with your query as you go.The point is to let the business layer gets whatever it needs whenever it needs it in a generic approach.
You can find an example here.
Have you considered using a code generation tool? Have a look at CodeSmith. Using a tool like that or T4 will allow you to generate your filter functions automatically and should make them fairly easy to maintain.
I'm not sure the best link to provide for T4, but you could start with this video.
Would this meet your needs?
public static IEnumerable<T> GetList<T>(this IEnumerable<IReferenceData> items, bool activeOnly)
{
return items.Where(b => b.IsActive == true || b.IsActive == activeOnly).OrderBy(c => c.Title).Cast<T>().ToList();
}
You could use it like this:
IEnumerable<IReferenceData> yourList;
List<DerivedClass> filtered = yourList.GetList<DerivedClass>(true);
To do something like this without demanding interfaces etc, you can use dynamic Expressions; something like:
public static IList<T> GetList<T>(
this DataContext context, bool activeOnly )
where T : class
{
IQueryable<T> query = context.GetTable<T>();
var param = Expression.Parameter(typeof(T), "row");
if(activeOnly)
{
var predicate = Expression.Lambda<Func<T, bool>>(
Expression.Equal(
Expression.PropertyOrField(param, "IsActive"),
Expression.Constant(true,typeof(bool))
), param);
query = query.Where(predicate);
}
var selector = Expression.Lambda<Func<T, string>>(
Expression.PropertyOrField(param, "Title"), param);
return query.OrderBy(selector).ToList();
}
Related
I need to dynamically filter data from the particular table depending on user's permissions. E.g. "normal" user can see only records assigned to him but admin can see all. I'm using ninject to create DB context per request and create context by passing additional user info to a constructor. Then I apply dynamic filtering (EF6) from EntityFramework-Plus extensions:
public MyDbContext(bool isAdmin, string userId) : this()
{
if (!isAdmin)
{
this.Filter<MyTable>(table => table.Where(...));
}
}
This solution works as expected i.e. calling methods like:
ctx.MyTable.Where(...)
Results in extra join declared in filter.
But behaves oddly when I'm using method Find(). I'm using SqlServer profiler to see what happens under the hood:
I Create context as restricted (non-admin user) - calling Find() will result in extra WHERE statement corresponding to filter's lambda expression
Then I create the context as admin (separate request) - calling Find() will result in the same SQL expression (i expect no extra SQL clauses).
AFAIK this has something to do with query caching since adding extra line to constructor seems to solve the problem:
public MyDbContext(bool isAdmin, string userId) : this()
{
// this "solves" the problem
QueryFilterManager.ClearQueryCache(this);
if (!isAdmin)
{
this.Filter<MyTable>(table => table.Where(...));
}
}
That looks like a big overkill and it doesn't bring me any closer to understanding the problem. So here are my questions:
Why this problem does not affect Where() but affects Find()?
Is there any cleaner way to solve this issue? I've read about Dynamic Filters library but it's no good for me as it works only in code first model (DB first here).
Are there better concepts of filtering data basing on per-request data (like userId in my example)?
UPDATE
This is what my lambda expression looks like:
private Func<IQueryable<MyTable>, IQueryable<MyTable>> GetFilter(string userId)
{
return t => t
.Where(c.DataScopes.Any(
x => x.AspNetGroups.Any(
ang => ang.AspNetUsers.Any(
anu => anu.Id == userId))));
}
AspNetGroups is my custom table to group users. Data persmissions are assigned to users group.
I have the following method in a data access class which uses entity framework:
public static IEnumerable<entityType> GetWhere(Func<entityType, bool> wherePredicate)
{
using (DataEntities db = new DataEntities())
{
var query = (wherePredicate != null)
? db.Set<entityType>().Where(wherePredicate).ToList()
: db.Set<entityType>().ToList();
return query;
}
}
This works fine when I use the entities across all layers... however I am trying to move to using a DTO class and I would like to do something like the following:
public static IEnumerable<EntityTypeDTO> GetWhere(Func<EntityTypeDTO, bool> wherePredicate)
{
//call a method here which will convert Func<EntityTypeDTO,bool> to
// Func<EntityType,bool>
using (DataEntities db = new DataEntities())
{
var query = new List<EntityType>();
if (wherePredicate == null)
{
query = db.Set<EntityType>().ToList();
}
else
{
query = (wherePredicate != null)
? db.Set<EntityType>().Where(wherePredicate).AsQueryable<EntityType>().ToList()
: db.Set<EntityType>().ToList();
}
List<EntityTypeDTO> result = new List<EntityTypeDTO>();
foreach(EntityType item in query)
{
result.Add(item.ToDTO());
}
return result;
}
}
Essentially I want a method which will convert Func to Func.
I think I have to break down the Func into an expression tree and then rebuild it somehow in the entityType?
I want to do this to allow the Presentation Layer to just pass the Expression queries?
Am I missing something basic or is there an easier design pattern that can pass a query from a DTO to a data access class without knowing the details of the query?
I have tried making the DTO inherit from the entity which doesn't seem to work either?
If there is a better design pattern that I am missing I would love a pointer and I can investigate from there...
Firstly I would suggest that you put a querying layer of your own in front of Entity Framework rather than allowing any arbitrary Func to be passed in because it will be very easy in the future to pass a Func that Entity Framework can not translate into a SQL statement (it can only translate some expressions - the basics are fine but if your expression calls a C# method, for example, then Entity Framework will probably fail).
So your search layer could have classes that you build up as criteria (eg. a "ContainsName" search class or a "ProductHasId" class) that are then translated into expressions in your search layer. This separates your app entirely from the ORM, which means that ORM details (like the entities or like the limitations of what Funcs can and can't be translated) don't leak out. There's lots out there that's been written about this some of arrangement.
One final note, though, if you are working close to the ORM layer, Entity Framework is very clever and you could probably get a long way without trying to translate your Func<dto, bool> to a Func<entity, bool>. For example, in the below code, accessing "context.Products" returns a "DbSet" and calling Select on it returns an IQueryable and calling Where on that also returns an IQueryable. Entity Framework will translate all of that into a single SQL statement so it won't pull all other Products into memory and then filter the ID on that memory set, it will actually perform the filtering in SQL even though the filter is operating on a projected type (which is equivalent to the DTO in your case) and not the Entity Framework entity -
var results = context.Products
.Select(p => new { ID = p.ProductID, Name = p.ProductName })
.Where(p => p.ID < 10)
.ToList();
The SQL executed is:
SELECT
[Extent1].[ProductID] AS [ProductID],
[Extent1].[ProductName] AS [ProductName]
FROM [dbo].[Products] AS [Extent1]
WHERE [Extent1].[ProductID] < 10
So, if you changed your code to get something like..
return context.Products
.Map<Product, ProductDTO()>()
.Where(productDtoWherePredicate)
.ToList();
.. then you might be just fine with the Funcs that you already have. I presume that you already have some sort of mapping functions to get from EF entities to DTOs (but if not then you might want to look into AutoMapper to help you out - which has support for "projections", which are basically IQueryable maps).
I am going to put this up as an answer.Thanks to Dan for the quick answer. Looking at what you are saying I can write a query/filter set of classes. for example, take the following code:
GetProducts().GetProductsInCategory().GetProductsWithinPriceRange(minPrice, maxPrice);
This code would run like so: Get Products would get all products in the table and the remaining functions would filter the results. if all queries run like this it may put a significant load on the Data Access Layer/ DB Server Connections... not sure.
or
An Alternate I will work on also is:
If each function creates a Linq expression, I could combine them like this: How do I combine multiple linq queries into one results set?
this may allow me to do this in a manner where I can return the filtered results set from the database.
Either way I am marking this as answered. I will update when I have more details.
For a few days now, I have been struggling with retrieving my entities from a repository (DbContext).
I am trying to save all the entities in an atomic action. Thus, different entities together represent something of value to me. If all the entities are 'valid', then I can save them all to the database. Entity 'a' is already stored in my repository, and needs to be retrieved to 'validate' entity 'b'.
That's where the problem arises. My repository relies on the DbSet<TEntity> class which works great with Linq2Sql (Include() navigation properties e.g.). But, the DbSet<TEntity> does not contain entities that are in the 'added' state.
So I have (as far as I know) two options:
Use the ChangeTracker to see which entities are available and query them into a set based on their EntityState.
Use the DbSet<TEntity>.Local property.
The ChangeTracker seems to involve some extra hard work to get it working in a way such that I can use Linq2Sql to Include() navigation properties e.g.
The DbSet<TEntity>.Local seems a bit weird to me. It might just be the name. I just read something that it is not performing very well (slower than DbSet<> itself). Not sure if that is a false statement.
Could somebody with significant EntityFramework experience shine some light on this? What's the 'wise' path to follow? Or am I seeing ghosts and should I always use the .Local property?
Update with code examples:
An example of what goes wrong
public void AddAndRetrieveUncommittedTenant()
{
_tenantRepository = new TenantRepository(new TenantApplicationTestContext());
const string tenantName = "testtenant";
// Create the tenant, but not call `SaveChanges` yet until all entities are validated
_tenantRepository.Create(tenantName);
//
// Some other code
//
var tenant = _tenantRepository.GetTenants().FirstOrDefault(entity => entity.Name.Equals(tenantName));
// The tenant will be null, because I did not call save changes yet,
// and the implementation of the Repository uses a DbSet<TEntity>
// instead of the DbSet<TEntity>.Local.
Assert.IsNotNull(tenant);
// Can I safely use DbSet<TEntity>.Local ? Or should I play
// around with DbContext.ChangeTracker instead?
}
An example of how I want to use my Repository
In my Repository I have this method:
public IQueryable<TEntity> GetAll()
{
return Context.Set<TEntity>().AsQueryable();
}
Which I use in business code in this fashion:
public List<Case> GetCasesForUser(User user)
{
return _repository.GetAll().
Where(#case => #case.Owner.EmailAddress.Equals(user.EmailAddress)).
Include(#case => #case.Type).
Include(#case => #case.Owner).
ToList();
}
That is mainly the reason why I prefer to stick to DbSet like variables. I need the flexibility to Include navigation properties. If I use the ChangeTracker I retrieve the entities in a List, which does not allow me to lazy load related entities at a later point in time.
If this is close to incomprehensible bullsh*t, then please let me know so that I can improve the question. I desperately need an answer.
Thx a lot in advance!
If you want to be able to 'easily' issue a query against the DbSet and have it find newly created items, then you will need to call SaveChanges() after each entity is created. If you are using a 'unit of work' style approach to working with persistent entities, this is actually not problematic because you can have the unit of work wrap all actions within the UoW as a DB transaction (i.e. create a new TransactionScope when the UoW is created, and call Commit() on it when the UoW completed). With this structure, the changes are sent to the DB, and will be visible to DbSet, but not visible to other UoWs (modulo whatever isolation level you use).
If you don't want the overhead of this, then you need to modify your code to make use of Local at appropriate times (which may involve looking at Local, and then issuing a query against the DbSet if you didn't find what you were looking for). The Find() method on DbSet can also be quite helpful in these situations. It will find an entity by primary key in either Local or the DB. So if you only need to locate items by primary key, this is pretty convenient (and has performance advantages as well).
As mentioned by Terry Coatta, the best approach if you don't want to save the records first would be checking both sources.
For example:
public Person LookupPerson(string emailAddress, DateTime effectiveDate)
{
Expression<Func<Person, bool>> criteria =
p =>
p.EmailAddress == emailAddress &&
p.EffectiveDate == effectiveDate;
return LookupPerson(_context.ObjectSet<Person>.Local.AsQueryable(), criteria) ?? // Search local
LookupPerson(_context.ObjectSet<Person>.AsQueryable(), criteria); // Search database
}
private Person LookupPerson(IQueryable<Person> source, Expression<Func<Person, bool>> predicate)
{
return source.FirstOrDefault(predicate);
}
For those who come after, I ran into some similar issues and decided to give the .Concat method a try. I have not done extensive performance testing so someone with more knowledge than I should feel free to chime in.
Essentially, in order to properly break up functionality into smaller chunks, I ended up with a situation in which I had a method that didn't know about consecutive or previous calls to that same method in the current UoW. So I did this:
var context = new MyDbContextClass();
var emp = context.Employees.Concat(context.Employees.Local).FirstOrDefault(e => e.Name.Contains("some name"));
This may only apply to EF Core, but every time you reference .Local of a DbSet, you're silently triggering change detection on the context, which can be a performance hit, depending on how complex your model is, and how many entries are currently being tracked.
If this is a concern, you'll want to use (fore EFCore) dbContext.ChangeTracker.Entries<T>() to get the locally tracked entities, which will not trigger change detection, but does require manual filtering of the DB state, as it will include deleted and detached entities.
There's a similar version of this in EF6, but in EFCore the Entries is a list of EntityEntries which you'll have to select out the entry.Entity to get out the same data the DbSet would give you.
I am looking for a library that would accept a collection of objects and return an indexed data structure that would be optimised for fast querying.
This is probably better illustrated by an example:
public class MyClass
{
public sting Name {get;set;}
public double Number {get;set;}
public ... (Many more fields)
}
var dataStore = Indexer.Parse(myClassCollection).Index(x => x.Name).Index(x => x.Number).Index( x => x.SomeOtherProperty);
var queryResult = dataStore.Where( x => x.Name == "ABC").Where(x => x.Number == 23).Where( x => x.SomeOtherProperty == dateTimeValue);
The idea is that the query on the dataStore will be very fast, of the order of O(log n).
Using dictionaries of dictionaries starts getting complicated when you have more than 2 or 3 fields you want to index.
Is there a library that already exists that does something like this?
What about an object oriented database.
Sterling is a recommended option. It supports LINQ to Object so don't worry about queries and we have used it for a couple of medium projects with good results (it's pretty fast).
You should take a look at RaptorDB as well. Several versions, including a fully embedded version, can be found on CodeProject here.
You could use Lucene.NET which can also run fully in memory (though I'm not sure that's what you'd want). It supports lightning fast retrieval of documents based on field criteria.
So that actually gives you a document database. If you take that one step further, you end up with something like RavenDB (commercial).
I am wondering whether we could achieve this by creating a SortedDictionary for each of the indexed properties.
SortedDictionary<property, List<MyClass>>
Then parsing the Linq expression tree to find out which properties are being queried. We can retrieve the valid keys of the sortedDictionaries, and then loop through these keys to get a List for each sorted dictionary and then use Set operations such as Union() and Intersect() depending on whether the expression tree has OR or AND directives.
Then return the a List matching the search criteria.
If the query includes a property that is not indexed, execute the query with indexed properties first and then use normal Linq to finish it off.
The interesting bit then becomes parsing the expression tree.
Any thoughts on this approach?
I have a LINQ to SQL class, we'll call it Test, and I want to be able to access properties with LINQ queries but I get the famed "No Supported Translation to SQL" runtime error. I'm interested in the conceptual problem. Here is my simplified class:
public class Test
{
public int ID {get; set;} // Stored in Database
public int NonForeignKeyValue {get; set;} // Stored in Database
}
Here is sort of an example of what I'm trying to accomplish, but I don't want the overhead of always explicitly writing the join in LINQ:
var db = (new DataContext()).GetTable<Test>();
var q = (from t in db.GetTable<Test>()
join o in db.GetTable<OtherTable>() on o.ID equals t.ID
where t.OtherStuff
select t)
I'd like to be able to add a property to Test that tells me if there are any rows in OtherTable that could be joined with Test:
public bool IsInOtherTable
{
get
{
return (new DataContext())
.GetTable<OtherTabke>()
.Any(x => x.NonForeignKeyValue == this.NonForeignKeyValue));
}
}
Ultimately this is what I want my code to look like, but it errors. I basically want to return all entries that contain some database computed value:
using (DataContext db = new DataContext())
{
var q = db.GetTable<Test>()
.Where(x => x.IsInOtherTable && x.OtherStuff); //Error
}
I'm basically trying to save myself from writing this code every single time I want to check if Test has certain information in another table. I'm not that interested in the exact problem I described, I'm more interested in how to conceptually add the join part to the SQL and still use LINQ. I'm guessing I use Linq.Expression, but I really don't know and I'm not aware of how to do it.
As an aside, I could just write the actual SQL, as its not that complicated, but I'd like to know how to get around this and still use LINQ.
Edit: I tried this property, but I get the same error. Its more complicated that just changing the return type to Expression...
public System.Linq.Expressions.Expression<Func<Article3, bool>> Exists
{
get
{
using (DataContext db = new DataContext())
{
return i => db.GetTable<OtherTable>()
.Any(x => x.NonForeignKeyValue == i.NonForeignKeyValue));
}
}
}
Each time the linq generator is to translate a code into a query, it has to process an expression tree.
In your examples, you are not passing around expression but rather - properties, delegates, i.e. stuff which the expression visitor is unable to "step into".
In general, try to rethink your conditions so that instead of bool you have Expression<T, bool> etc.
http://netpl.blogspot.com/2008/02/linq-to-object-vs-linq-to-sql-and.html
Firstly, I think you may be overestimating "the overhead of always explicitly writing the join in LINQ". It's an extra line of code which has the advantage of being relatively self-documenting as to just what you are doing (always a nice thing), and any other approach is going to be turned first into SQL and then into a query plan that will be at least as expensive to execute, possibly more expensive (SQLServer is good a joins!)
Still, there are two alternatives I can thinkof.
One is to have an EntityRef property on the class that defines this relationship with the other table. You can then test if it is null in your query (or EntitySet if it's on the other side of a one-to-many relationship).
The other is to define a SQL function that returns a bit result indicating whether an id refers to a row that does or doesn't relate to the other table.
Then define a protected method on your DataContext-derived class that matches the signature in C# terms, and has a Function attribute mapping it to that SQL function. (Since this isn't something that you can give a sensible non-db-using version of in the C# version, you can implement the C# function by calling ExecuteMethodCall).
Then you can use that method instead of the join.
Still, this is likely less self-explanatory in the code and at risk of being less efficient than just keeping the join.