First, I apologise if this is a dupe, finding the right search terms seemed impossible...
We are trying to adopt some best practice and looking at refactoring duplicate code in our projects. On a number of occasions we have something like;
public List<EventModel> GetEvents(bool showInactive, bool showPastEvents)
{
return eventRepository
.GetEvents(_customerId, showInactive, showPastEvents)
.Select(e => New EventModel() { Id = e.EventId, Name = e.EventName, Capacity = e.EventCapacity, Active = e.EventActive })
.ToList();
}
So we tried doing something like this instead;
public List<EventModel> GetEvents(bool showInactive, bool showPastEvents)
{
return eventRepository
.GetEvents(_customerId, showInactive, showPastEvents)
.Select(e => ConvertPocoToModel(e))
.ToList();
}
private EventModel ConvertPocoToModel(TsrEvent tsrEvent)
{
EventModel eventModel = new EventModel()
{
Id = tsrEvent.EventId,
Name = tsrEvent.EventName,
Capacity = tsrEvent.EventCapacity,
Active = tsrEvent.EventActive
};
return eventModel;
}
Sometimes this works, but intermittently we get;
System.NotSupportedException: 'LINQ to Entities does not recognize the
method 'Bll.Models.EventModel ConvertPocoToModel(Dal.Pocos.TsrEvent)'
method, and this method cannot be translated into a store expression.'
I am aware we could add .ToList() or similar to force the conversion to happen in C# but I believe that means SQL will execute SELECT * instead of SELECT EVentId, EventName, EventCapacity, EventActive
Can anyone explain;
Why EF is having issues trying to understand how to handle this simple mapping?
why it work intermittently?
How we should be doing it?
Entity framework doesnt know how to translate your method. You have to use method which returns Expression<Func<TsrEvent,EventModel>> or an property which stores it.
public List<EventModel> GetEvents(bool showInactive, bool showPastEvents)
{
return eventRepository
.GetEvents(_customerId, showInactive, showPastEvents)
.Select(ConvertPocoToModelExpr)
.ToList();
}
private static Expression<Func<TsrEvent,EventModel>> ConvertPocoToModelExpr => (x)=>new EventModel()
{
Id = x.EventId,
Name = x.EventName,
Capacity = x.EventCapacity,
Active = x.EventActive
};
You have to be aware about the differences between an IEnumerable and an IQueryable.
An IEnumerable object holds everything to enumerate over the sequence. You can ask for the first element, and once you've got an element you can ask for the next one, as long as there is a next one. The IEnumerable is meant to be processes locally by your process.
Enumeration at its lowest level is done by asking for the Enumerator and repeatedly calling MoveNext, until you don't need anymore elements. Like this:
IEnumerable<Student> students = ...
IEnumerator<Student> studentEnumerator = students.GetEnumerator();
while (studentEnumerator.MoveNext())
{
// there is still a Student to process:
Student student = studentEnumerator.Current;
ProcessStudent(student);
}
You can do this explicitly, or call it implicitly using foreach or one of the LINQ functions.
On the other hand, an IQueryable is meant to be processed by a different process, usually a database management system. The IQueryable holds an Expression and a Provider. The Expression expresses the query that must be performed in some generic format. The Provider knows who must execute the query (usually a database management system), and the language that this process uses (usually something SQL like).
As soon as you start enumerating by calling GetEnumerator, the Expression is sent to the Provider, who tries to translate the Expression into SQL and executes the query. The fetched data is put into an enumerable sequence, and the enumerator is returned.
Back to your question
The problem is, that SQL does not know ConvertPocoToModel. Hence your provider can't convert the Expression. The compiler can't detect this, because it does not know how smart your Provider is. That is why you don't get this error until you call GetEnumerator, in your case by calling ToList.
Solution
The solution is to make a function that changes the expression. The easiest method would be an extension function. See extension methods demystified. This way you can use it like any other LINQ method:
public static IQueryable<EventModel> ToEventModels(this IQueryable<TsrEvent> tsrEvents)
{
return tsrEvent.Select(tsrEvent => new EventModel
{
Id = tsrEvent.EventId,
Name = tsrEvent.EventName,
Capacity = tsrEvent.EventCapacity,
Active = tsrEvent.EventActive
};
}
Note that I omit the () in the constructor: SQL can't call constructors!
Usage:
var result = dbContext.TsrEvents
.Where(tsrEvent => tsrEvent.Active && tsrEvent.Date == Today)
.ToEventModels()
.GroupBy(...)
... etc
Or, if your GetEvents returns an IQueryable<TsrEvents>
return eventRepository.GetEvents(_customerId, showInactive, showPastEvents)
.ToEventModels();
Final Remark
It is better to let your data-fetch-functions return IQueryable<...> and IEnumerable<...> as long as possible. Let only the end-user materialize the query. It would be a waste of processing power if you do the ToList() and your caller only wants to do FirstOrDefault()
Related
I was writing unit tests to compare an original response to a filtered response using a request object as a parameter. In doing so I noticed that if I change the request object after getting a response the IEnumerable list will change - As I type this, my thinking is that because it is an IEnumerable with LINQ, the request.Filter property is a reference in the LINQ query, which is what causes this behavior. If I converted this to a list instead of an IEnumerable, I suspect the behavior would go away because the .ToList() will evaluate the LINQ expressions instead of deferring. Is that the case?
public class VendorResponse {
public IEnumerable<string> Vendors { get; set; }
}
var request = new VendorRequest() {
Filter = ""
};
var response = await _service.GetVendors(request);
int vendorCount = response.Vendors.Count(); // 20
request.Filter = "at&t";
int newCount = response.Vendors.Count(); // 17
public async Task<VendorResponse> GetVendors(VendorRequest request)
{
var vendors = await _dataService.GetVendors();
return new VendorResponse {
Vendors = vendors.Where(v => v.IndexOf(request.Filter) >= 0)
}
}
If deferred execution is preferable, you can capture the current state of request.Filter with a local variable and use that in the Where predicate
public async Task<VendorResponse> GetVendors(VendorRequest request)
{
var filter = request.Filter;
var vendors = await _dataService.GetVendors();
return new VendorResponse {
Vendors = vendors.Where(v => v.IndexOf(filter) >= 0)
}
}
Yes!
This is an example of deferred execution of an IEnumerable, which just encapsulates a query on some data without encapsulating the result of that query.
An IEnumerable can be enumerated (via its IEnumerator), and "knows" how to enumerate the query it encapsulates, but this will not actually happen until something executes the enumeration.
In your case the enumeration is executed by the call to .Count() which needs to know how many items are in the result of the query. The enumeration occurs every time you call .Count(), so changing the filter between the two invocations leads to you getting two different results.
As you have correctly deduced, calling .ToList() and capturing the result in a variable before performing any further operations would lead to you capturing the resulting data rather than the query, and so lead to both counts having the same value.
Try this out yourself. In future, be sure to force the evaluation of the enumerable before passing to other queries, or returning out to unknown code, otherwise you or your users will encounter unexpected behaviour and possible performance issues.
Hope this helps :)
Edit 1:
As Moho has pointed out, and you have also alluded to in your original post, this is also a result of the request.Filter being captured by the IEnumerable as a reference type. If you can capture the value and pass this in instead, the result of the IEnumerable will no longer be modified by changing the filter.
Code:
public static IEnumerable<TableRowModel> GetRows()
{
var to_ret = db.TableRows.select(x=> new TableRowModel(){
TableRowId = x.TableRowId,
Type = x.Type,
Name = x.Name,
CreatedAt = x.CreatedAt,
ModifiedAt = x.ModifiedAt,
Enums = x.Enums.Select(y => y.ToEnumModel()),
});
return to_ret;
}
public static EnumModel ToEnumModel(this Enum x)
{
var to_ret = new EnumModel()
{
CFPId = x.CFPId,
CreatedAt = x.CreatedAt,
ModifiedAt = x.ModifiedAt,
};
return to_ret;
}
I get the following error when using the GetRows method:
LINQ to Entities does not recognize the method
Given the error, it's understood that LINQ To Entities is not able to recognize the extension method ToEnumModel.
I would like to know if there is a way around this?
So that I would not be repeating ToEnumModel code again in GetRows extension.
Under normal circumstances, when performing an operation like a Where in EntityFramework, it isn't actually executed in memory like when you operate on an enumerable (like a List). Instead, it converted into SQL which is then executed by the Db provider. This means that doing certain things, such as using extension methods on objects, is not an option, as the converter cannot turn such things into SQL.
The quickest and easiest solution would be to load your set in-memory, where it can be operated on like any other enumerable. You can do this like so:
var result = myDbSet.AsEnumerable().Etc()
Etc() here represents all other operations you want to run. Be advised however, that this will load all data into memory, which may be prohibitively slow and expensive in some scenarios. One way to alleviate this is to put AsEnumerable() right before you need to use the extension method, thus offloading as many operations as possible to the provider. For example this:
var result = myDbSet.Where([condition]).AsEnumerable().Select(item => item.ExtensionMethod());
Might be much quicker and lighter than this:
var result = myDbSet.AsEnumerable().Where([condition]).Select(item => item.ExtensionMethod());
Your ability to use extension methods in EF is limited, but you can still extend your IQueryables
public static class Extensions
{
public static IQueryable<MyModelVM> SelectVMs(this IQueryable<MyModel> models)
{
return models.Select(x => new MyModelVM { MyModelId = x.MyModelId });
}
}
I tried to create a generic CreateOrUpdate function for EntityFramework where my classes/tables always have an ID field. Since that data is sent to a webpage and posted back to the server I am totally disconnected from the context and have to use the standard (set.Any / set.FirstOrDefault) way of checking to see if my object already exists.
The method works perfectly in terms of doing its job and creating/updating as required. But what I have found is that the db.Set<T>().FirstOrDefault(whereFunction); brings back all data from the DB and then performs the FirstOrDefault in memory. I would much prefer this to happen in SQL, but no matter what I have tried I can't get it to work.
Do you have any suggestions on how I might make the FirstOrDefault function translate to proper SQL so that I don't retrieve too much from the DB?
Also, I have tried First, Any & Count, all of which get all of the rows from the DB.
public void CreateOrUpdateEntity<T>(T entity) where T : class
{
using (var db = new ProjectContext())
{
Func<T, bool> whereFunction = m => m.As<dynamic>().ID == entity.As<dynamic>().ID;
var firstValue = db.Set<T>().FirstOrDefault(whereFunction);
if (firstValue == null)
{
db.Set<T>().Attach(entity);
db.ChangeTracker.Entries<T>().First(e => e.Entity == entity).State = EntityState.Added;
}
else
{
db.ChangeTracker.Entries<T>().First(e => e.Entity == entity).State = EntityState.Modified;
}
}
}
You should use expression instead of Func<T, bool> delegate. Usage of delegate invokes Enumerable.FirstOrDefault(Func<T,TResult>) method which is executed in memory (and requires all data loaded to client), instead of Queryable.FirstOrDefault(Expression<Func<T, TResult>>) which is translated into SQL and executed on server side.
NOTE: I have doubts about whether your delegate can be translated into SQL.
BTW: In order to get entity by id you can use method Find:
var firstValue = db.Set<T>().Find(entity.As<dynamic>().ID);
I'm busy with a LINQ to SQL project that basically creates multiple threads for each entity type in my database, which constantly queries information from the DB in a thread.
Here's a pseudo example:
streamer.DefineDataExpression<Contacts>(x => x.FirstName == "Bob");
while(true)
{
List<Contacts> MyContactsResult = streamer.ResultList;
// do whatever with MyContactsResult
}
The above code doesn't exist, but this is what I have so far for the 'streamer' class (it obviously doesn't work, but you can see what I'm trying to achieve above):
public void DefineExpression(System.Linq.Expressions.Expression<System.Func<T, bool>> expression)
{
using (var db = new LINQDataContext())
{
ResultList = db.GetTable<T>().Where(expression);
}
}
How do I go about creating a method like 'DefineExpression' that will allow me to query a LINQ type dynamically?
Why not use the Dynamic LINQ provider, as mentioned by Scott Guthrie. I think that would give you everything you are looking for, because you can define the query as a string. Therefore, you can more easily build a string representation of your query, and execute on the fly.
I'm trying to implement encrypted columns in EF4 and using the CTP5 features to allow simple use of POCO's to query the database. Sorry that this is a lot of words, but I hope the below gives enough to explain the need and the problem!
So, bit of background, and my progress so far:
The intention is that if you query the tables without using our DAL then the data is rubbish, but I don't want the developers to worry about if/when/how the data is encrypted.
For simplicity, at this stage I'm working on the assumption any string column will be encrypted.
Now, I have successfully implemented this for returning the data using the Objectmaterialized event, and for data commits using the SavingChanges event.
So given the following class:
public class Thing
{
public int ID { get; set; }
[Required]
public string Name { get; set; }
public DateTime Date { get; set; }
public string OtherString { get; set; }
}
The below query returns all the required values and the POCO materialized has clear data in it.
var things = from t in myDbContext.Things
select t;
where myDbContext.Things is a DbSet<Thing>
Likewise, passing an instance of Thing to Things.Add()
(with clear string data in the Name and/or OtherString values)
and then calling myDbContext.SaveChanges() encrypts the strings before it gets to the data store.
Now, the problem I have is in this query:
var things = from t in myDbContext.Things
where t.Name == "Hairbrush"
select t;
This results in the unencrypted value being compared to the encrypted value in the DB. Obviously I don't want to get all the records from the database, materialize them, and then filter the results based on any supplied Where clause... so what I need to do is: intercept that query and rewrite it by encrypting any strings in the Where clause.
So I've looked at:
writing a query provider, but that doesn't seem like the right solution... (is it?)
writing my own IQueryable wrapper for the DbSet which will capture the expression, run over it using an expression tree visitor and then forward the new expression to the DbSet...
Attempts at both have left me somewhat lost! I prefer the second solution i think since it feels a bit neater, and is probably clearer to other developers in future. But I'm happy to go with either or another better option!!
The main thing I am struggling with is when/how the LINQ expression is applied to the object... I think i've got myself a bit confused as to where the expression executes in the IQueryable object thus I'm not sure which method I need to implement in my wrapper to then grab and manipulate the expression being passed in...
I'm sure I'm missing something fairly obvious here and I'm waiting for that light bulb moment... but its not coming!!
Any help will be very gratefully received!
Thought I'd let you know what my final solution was.
In the end I have gone a wrapper class which implements a Where method, but without going to the extent of implementing IQueryable entirely. LINQ will still execute against the class (at least to the extent that I want/need it to) and will call the Where method with the expression from the LINQ.
I then traverse this ExpressionTree and replace my strings with encrypted values before forwarding the new expressiontree to the internal DbSet. and then returning the result.
Its pretty crude, and has its limitation, but works for our particular circumstance without problem.
Thanks,
Ben
you should use the QueryInterceptor attribute, search here in SO or in google and you find examples on how to use it.
a snippet:
[QueryInterceptor("Orders")]
public Expression<Func<Order, bool>> FilterOrders()
{
return o => o.Customer.Name == /* Current principal name. */;
}
// Insures that the user accessing the customer(s) has the appropriate
// rights as defined in the QueryRules object to access the customer
// resource(s).
[QueryInterceptor ("Customers")]
public Expression<Func<Customer, bool>> FilterCustomers()
{
return c => c.Name == /* Current principal name. */ &&
this.CurrentDataSource.QueryRules.Contains(
rule => rule.Name == c.Name &&
rule.CustomerAllowedToQuery == true
);
}
You can use David Fowler's Query Interceptor:
https://github.com/davidfowl/QueryInterceptor
One example of its use:
IQueryable q = ...;
IQueryable modifed = q.InterceptWith(new MyInterceptor());
And on class MyInterceptor:
protected override Expression VisitBinary(BinaryExpression node) {
if (node.NodeType == ExpressionType.Equal) {
// Change == to !=
return Expression.NotEqual(node.Left, node.Right);
}
return base.VisitBinary(node);
}