I tried to create a generic CreateOrUpdate function for EntityFramework where my classes/tables always have an ID field. Since that data is sent to a webpage and posted back to the server I am totally disconnected from the context and have to use the standard (set.Any / set.FirstOrDefault) way of checking to see if my object already exists.
The method works perfectly in terms of doing its job and creating/updating as required. But what I have found is that the db.Set<T>().FirstOrDefault(whereFunction); brings back all data from the DB and then performs the FirstOrDefault in memory. I would much prefer this to happen in SQL, but no matter what I have tried I can't get it to work.
Do you have any suggestions on how I might make the FirstOrDefault function translate to proper SQL so that I don't retrieve too much from the DB?
Also, I have tried First, Any & Count, all of which get all of the rows from the DB.
public void CreateOrUpdateEntity<T>(T entity) where T : class
{
using (var db = new ProjectContext())
{
Func<T, bool> whereFunction = m => m.As<dynamic>().ID == entity.As<dynamic>().ID;
var firstValue = db.Set<T>().FirstOrDefault(whereFunction);
if (firstValue == null)
{
db.Set<T>().Attach(entity);
db.ChangeTracker.Entries<T>().First(e => e.Entity == entity).State = EntityState.Added;
}
else
{
db.ChangeTracker.Entries<T>().First(e => e.Entity == entity).State = EntityState.Modified;
}
}
}
You should use expression instead of Func<T, bool> delegate. Usage of delegate invokes Enumerable.FirstOrDefault(Func<T,TResult>) method which is executed in memory (and requires all data loaded to client), instead of Queryable.FirstOrDefault(Expression<Func<T, TResult>>) which is translated into SQL and executed on server side.
NOTE: I have doubts about whether your delegate can be translated into SQL.
BTW: In order to get entity by id you can use method Find:
var firstValue = db.Set<T>().Find(entity.As<dynamic>().ID);
Related
I only need it to work for SQL Server. This is an example. The question is about a general approach.
There is a nice extension method from https://entityframework-extensions.net called WhereBulkContains. It is, sort of, great, except that the code of the methods in this library is obfuscated and they do not produce valid SQL when .ToQueryString() is called on IQueryable<T> with these extension methods applied.
Subsequently, I can't use such methods in production code as I am not "allowed" to trust such code due to business reasons. Sure, I can write tons of tests to ensure that WhereBulkContains works as expected, except that there are some complicated cases where the performance of WhereBulkContains is well below stellar, whereas properly written SQL works in a blink of an eye. And (read above), since the code of this library is obfuscated, there is no way to figure out what's wrong there without spending a large amount of time. We would've bought the license (as this is not a freeware) if the library weren't obfuscated. All together that basically kills the library for our purposes.
This is where it gets interesting. I can easily create and populate a temporary table, e.g. (I have a table called EFAgents with an int PK called AgentId in the database):
private string GetTmpAgentSql(IEnumerable<int> agentIds) => #$"
drop table if exists #tmp_Agents;
create table #tmp_Agents (AgentId int not null, primary key clustered (AgentId asc));
{(agentIds
.Chunk(1_000)
.Select(e => $#"
insert into #tmp_Agents (AgentId)
values
({e.JoinStrings("), (")});
")
.JoinStrings(""))}
select 0 as Result
";
private const string AgentSql = #"
select a.* from EFAgents a inner join #tmp_Agents t on a.AgentID = t.AgentId";
where GetContext returns EF Core database context and JoinStrings comes from Unity.Interception.Utilities and then use it as follows:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
var tmpSql = GetTmpAgentSql(agentIds);
using var ctx = GetContext();
// This creates a temporary table and populates it with the ids.
// This is a proprietary port of EF SqlQuery code, but I can post the whole thing if necessary.
var _ = await ctx.GetDatabase().SqlQuery<int>(tmpSql).FirstOrDefaultAsync();
// There is a DbSet<EFAgent> called Agents.
var query = ctx.Agents
.FromSqlRaw(AgentSql)
.Join(ctx.Agents, t => t.AgentId, a => a.AgentId, (t, a) => a);
var sql = query.ToQueryString() + Environment.NewLine;
// This should provide a valid SQL; https://entityframework-extensions.net does NOT!
// WriteLine - writes to console or as requested. This is irrelevant to the question.
WriteLine(sql);
var result = await query.ToListAsync();
return result;
}
So, basically, I can do what I need in two steps:
using var ctx = GetContext();
// 1. Create a temp table and populate it - call GetTmpAgentSql.
...
// 2. Build the join starting from `FromSqlRaw` as in example above.
This is doable, half-manual, and it is going to work.
The question is how to do that in one step, e.g., call:
.WhereMyBulkContains(aListOfIdConstraints, whateverElseIsneeded, ...)
and that's all.
I am fine if I need to pass more than one parameter in each case in order to specify the constraints.
To clarify the reasons why do I need to go into all these troubles. We have to interact with a third party database. We don't have any control of the schema and data there. The database is large and poorly designed. That resulted in some ugly EFC LINQ queries. To remedy that, some of that ugliness was encapsulated into a method, which takes IQueryable<T> (and some more parameters) and returns IQueryable<T>. Under the hood this method calls WhereBulkContains. I need to replace this WhereBulkContains by, call it, WhereMyBulkContains, which would be able to provide correct ToQueryString representation (for debugging purposes) and be performant. The latter means that SQL should not contain in clause with hundreds (and even sometimes thousands) of elements. Using inner join with a [temp] table with a PK and having an index on the FK field seem to do the trick if I do that in pure SQL. But, ... I need to do that in C# and effectively in between two LINQ method calls. Refactoring everything is also not an option because that method is used in many places.
Thanks a lot!
I think you really want to use a Table Valued Parameter.
Creating an SqlParameter from an enumeration is a little fiddly, but not too difficult to get right;
CREATE TYPE [IntValue] AS TABLE (
Id int NULL
)
private IEnumerable<SqlDataRecord> FromValues(IEnumerable<int> values)
{
var meta = new SqlMetaData(
"Id",
SqlDbType.Int
);
foreach(var value in values)
{
var record = new SqlDataRecord(
meta
);
record.SetInt32(0, value);
yield return record;
}
}
public SqlParameter ToIntTVP(IEnumerable<int> values){
return new SqlParameter()
{
TypeName = "IntValue",
SqlDbType = SqlDbType.Structured,
Value = FromValues(values)
};
}
Personally I would define a query type in EF Core to represent the TVP. Then you can use raw sql to return an IQueryable.
public class IntValue
{
public int Id { get; set; }
}
modelBuilder.Entity<IntValue>(e =>
{
e.HasNoKey();
e.ToView("IntValue");
});
IQueryable<IntValue> ToIntQueryable(DbContext ctx, IEnumerable<int> values)
{
return ctx.Set<IntValue>()
.FromSqlInterpolated($"select * from {ToIntTVP(values)}");
}
Now you can compose the rest of your query using Linq.
var ids = ToIntQueryable(ctx, agentIds);
var query = ctx.Agents
.Where(a => ids.Any(i => i.Id == a.Id));
I would propose to use linq2db.EntityFrameworkCore (note that I'm one of the creators). It has built-in temporary tables support.
We can create simple and reusable function which filters records of any type:
public static class HelperMethods
{
private class KeyHolder<T>
{
[PrimaryKey]
public T Key { get; set; } = default!;
}
public static async Task<List<TEntity>> GetRecordsByIds<TEntity, TKey>(this IQueryable<TEntity> query, IEnumerable<TKey> ids, Expression<Func<TEntity, TKey>> keyFunc)
{
var ctx = LinqToDBForEFTools.GetCurrentContext(query) ??
throw new InvalidOperationException("Query should be EF Core query");
// based on DbContext options, extension retrieves connection information
using var db = ctx.CreateLinqToDbConnection();
// create temporary table and BulkCopy records into that table
using var tempTable = await db.CreateTempTableAsync(ids.Select(id => new KeyHolder<TKey> { Key = id }), tableName: "temporaryIds");
var resultQuery = query.Join(tempTable, keyFunc, t => t.Key, (q, t) => q);
// we use ToListAsyncLinqToDB to avoid collission with EF Core async methods.
return await resultQuery.ToListAsyncLinqToDB();
}
}
Then we can rewrite your function GetAgents to the following:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
using var ctx = GetContext();
var result = await ctx.Agents.GetRecordsByIds(agentIds, a => a.AgentId);
return result;
}
First, I apologise if this is a dupe, finding the right search terms seemed impossible...
We are trying to adopt some best practice and looking at refactoring duplicate code in our projects. On a number of occasions we have something like;
public List<EventModel> GetEvents(bool showInactive, bool showPastEvents)
{
return eventRepository
.GetEvents(_customerId, showInactive, showPastEvents)
.Select(e => New EventModel() { Id = e.EventId, Name = e.EventName, Capacity = e.EventCapacity, Active = e.EventActive })
.ToList();
}
So we tried doing something like this instead;
public List<EventModel> GetEvents(bool showInactive, bool showPastEvents)
{
return eventRepository
.GetEvents(_customerId, showInactive, showPastEvents)
.Select(e => ConvertPocoToModel(e))
.ToList();
}
private EventModel ConvertPocoToModel(TsrEvent tsrEvent)
{
EventModel eventModel = new EventModel()
{
Id = tsrEvent.EventId,
Name = tsrEvent.EventName,
Capacity = tsrEvent.EventCapacity,
Active = tsrEvent.EventActive
};
return eventModel;
}
Sometimes this works, but intermittently we get;
System.NotSupportedException: 'LINQ to Entities does not recognize the
method 'Bll.Models.EventModel ConvertPocoToModel(Dal.Pocos.TsrEvent)'
method, and this method cannot be translated into a store expression.'
I am aware we could add .ToList() or similar to force the conversion to happen in C# but I believe that means SQL will execute SELECT * instead of SELECT EVentId, EventName, EventCapacity, EventActive
Can anyone explain;
Why EF is having issues trying to understand how to handle this simple mapping?
why it work intermittently?
How we should be doing it?
Entity framework doesnt know how to translate your method. You have to use method which returns Expression<Func<TsrEvent,EventModel>> or an property which stores it.
public List<EventModel> GetEvents(bool showInactive, bool showPastEvents)
{
return eventRepository
.GetEvents(_customerId, showInactive, showPastEvents)
.Select(ConvertPocoToModelExpr)
.ToList();
}
private static Expression<Func<TsrEvent,EventModel>> ConvertPocoToModelExpr => (x)=>new EventModel()
{
Id = x.EventId,
Name = x.EventName,
Capacity = x.EventCapacity,
Active = x.EventActive
};
You have to be aware about the differences between an IEnumerable and an IQueryable.
An IEnumerable object holds everything to enumerate over the sequence. You can ask for the first element, and once you've got an element you can ask for the next one, as long as there is a next one. The IEnumerable is meant to be processes locally by your process.
Enumeration at its lowest level is done by asking for the Enumerator and repeatedly calling MoveNext, until you don't need anymore elements. Like this:
IEnumerable<Student> students = ...
IEnumerator<Student> studentEnumerator = students.GetEnumerator();
while (studentEnumerator.MoveNext())
{
// there is still a Student to process:
Student student = studentEnumerator.Current;
ProcessStudent(student);
}
You can do this explicitly, or call it implicitly using foreach or one of the LINQ functions.
On the other hand, an IQueryable is meant to be processed by a different process, usually a database management system. The IQueryable holds an Expression and a Provider. The Expression expresses the query that must be performed in some generic format. The Provider knows who must execute the query (usually a database management system), and the language that this process uses (usually something SQL like).
As soon as you start enumerating by calling GetEnumerator, the Expression is sent to the Provider, who tries to translate the Expression into SQL and executes the query. The fetched data is put into an enumerable sequence, and the enumerator is returned.
Back to your question
The problem is, that SQL does not know ConvertPocoToModel. Hence your provider can't convert the Expression. The compiler can't detect this, because it does not know how smart your Provider is. That is why you don't get this error until you call GetEnumerator, in your case by calling ToList.
Solution
The solution is to make a function that changes the expression. The easiest method would be an extension function. See extension methods demystified. This way you can use it like any other LINQ method:
public static IQueryable<EventModel> ToEventModels(this IQueryable<TsrEvent> tsrEvents)
{
return tsrEvent.Select(tsrEvent => new EventModel
{
Id = tsrEvent.EventId,
Name = tsrEvent.EventName,
Capacity = tsrEvent.EventCapacity,
Active = tsrEvent.EventActive
};
}
Note that I omit the () in the constructor: SQL can't call constructors!
Usage:
var result = dbContext.TsrEvents
.Where(tsrEvent => tsrEvent.Active && tsrEvent.Date == Today)
.ToEventModels()
.GroupBy(...)
... etc
Or, if your GetEvents returns an IQueryable<TsrEvents>
return eventRepository.GetEvents(_customerId, showInactive, showPastEvents)
.ToEventModels();
Final Remark
It is better to let your data-fetch-functions return IQueryable<...> and IEnumerable<...> as long as possible. Let only the end-user materialize the query. It would be a waste of processing power if you do the ToList() and your caller only wants to do FirstOrDefault()
I am working on a project to connect to PostgreSQL database using NpGsql EntityFramework 6. I am getting the exception in question heading, when I try to execute the query in GetAdminUsersCount:
public class GenieRepository : IDisposable
{
GenieDbContext db = new GenieDbContext();
public IEnumerable<User> GetUsers()
{
return db.Users;
}
}
public int GetAdminUsersCount()
{
return repo.GetUsers().Where(u => u.Role.RoleName == "Administrator").Count();
}
What is the reason for this error and how to resolve it?
Use a ToList<T> right after the LINQ query like this:
using (ElisContext db = new ElisContext()) {
var q = from a in db.aktie select a;
List<aktie> akties = q.ToList<aktie>();
foreach (aktie a in akties) {
Console.WriteLine("aktie: id {0}, name {1}, market name {2}"
, a.id, a.name, a.marked.name);
}
}
Note the q.ToList<T> which does the trick. .NET delays the execution of the linq statement to the latest moment, which may be part of the problem. I have tried to use q in the foreach without success.
The issue is caused by the return type of the GetUsers() method. Since it is IEnumerable<User>, LINQ-to-SQL will load the result into memory (row by row) and subsequent Where, OrderBy, Select, Count, etc. calls will be executed in memory. When the Where condition is evaluated, the original result set is still being iterated over, but the relation User.Role needs to be resolved on the DB (that's where the "An operation is already in progess" error message comes from).
If the return type is changed to IQueryable<User>, the query won't be executed until the Count method is called, and furthermore, the whole query will be translated into SQL returning only the count without ever loading any User records into memory.
See also this related question/answer: Returning IEnumerable<T> vs. IQueryable<T>
The answer suggesting to call ToList() on the query, will load the entire result set into memory, which makes your error go away, but depending on the size of the result set, could also be very inefficient.
I'm busy with a LINQ to SQL project that basically creates multiple threads for each entity type in my database, which constantly queries information from the DB in a thread.
Here's a pseudo example:
streamer.DefineDataExpression<Contacts>(x => x.FirstName == "Bob");
while(true)
{
List<Contacts> MyContactsResult = streamer.ResultList;
// do whatever with MyContactsResult
}
The above code doesn't exist, but this is what I have so far for the 'streamer' class (it obviously doesn't work, but you can see what I'm trying to achieve above):
public void DefineExpression(System.Linq.Expressions.Expression<System.Func<T, bool>> expression)
{
using (var db = new LINQDataContext())
{
ResultList = db.GetTable<T>().Where(expression);
}
}
How do I go about creating a method like 'DefineExpression' that will allow me to query a LINQ type dynamically?
Why not use the Dynamic LINQ provider, as mentioned by Scott Guthrie. I think that would give you everything you are looking for, because you can define the query as a string. Therefore, you can more easily build a string representation of your query, and execute on the fly.
I'm trying to implement encrypted columns in EF4 and using the CTP5 features to allow simple use of POCO's to query the database. Sorry that this is a lot of words, but I hope the below gives enough to explain the need and the problem!
So, bit of background, and my progress so far:
The intention is that if you query the tables without using our DAL then the data is rubbish, but I don't want the developers to worry about if/when/how the data is encrypted.
For simplicity, at this stage I'm working on the assumption any string column will be encrypted.
Now, I have successfully implemented this for returning the data using the Objectmaterialized event, and for data commits using the SavingChanges event.
So given the following class:
public class Thing
{
public int ID { get; set; }
[Required]
public string Name { get; set; }
public DateTime Date { get; set; }
public string OtherString { get; set; }
}
The below query returns all the required values and the POCO materialized has clear data in it.
var things = from t in myDbContext.Things
select t;
where myDbContext.Things is a DbSet<Thing>
Likewise, passing an instance of Thing to Things.Add()
(with clear string data in the Name and/or OtherString values)
and then calling myDbContext.SaveChanges() encrypts the strings before it gets to the data store.
Now, the problem I have is in this query:
var things = from t in myDbContext.Things
where t.Name == "Hairbrush"
select t;
This results in the unencrypted value being compared to the encrypted value in the DB. Obviously I don't want to get all the records from the database, materialize them, and then filter the results based on any supplied Where clause... so what I need to do is: intercept that query and rewrite it by encrypting any strings in the Where clause.
So I've looked at:
writing a query provider, but that doesn't seem like the right solution... (is it?)
writing my own IQueryable wrapper for the DbSet which will capture the expression, run over it using an expression tree visitor and then forward the new expression to the DbSet...
Attempts at both have left me somewhat lost! I prefer the second solution i think since it feels a bit neater, and is probably clearer to other developers in future. But I'm happy to go with either or another better option!!
The main thing I am struggling with is when/how the LINQ expression is applied to the object... I think i've got myself a bit confused as to where the expression executes in the IQueryable object thus I'm not sure which method I need to implement in my wrapper to then grab and manipulate the expression being passed in...
I'm sure I'm missing something fairly obvious here and I'm waiting for that light bulb moment... but its not coming!!
Any help will be very gratefully received!
Thought I'd let you know what my final solution was.
In the end I have gone a wrapper class which implements a Where method, but without going to the extent of implementing IQueryable entirely. LINQ will still execute against the class (at least to the extent that I want/need it to) and will call the Where method with the expression from the LINQ.
I then traverse this ExpressionTree and replace my strings with encrypted values before forwarding the new expressiontree to the internal DbSet. and then returning the result.
Its pretty crude, and has its limitation, but works for our particular circumstance without problem.
Thanks,
Ben
you should use the QueryInterceptor attribute, search here in SO or in google and you find examples on how to use it.
a snippet:
[QueryInterceptor("Orders")]
public Expression<Func<Order, bool>> FilterOrders()
{
return o => o.Customer.Name == /* Current principal name. */;
}
// Insures that the user accessing the customer(s) has the appropriate
// rights as defined in the QueryRules object to access the customer
// resource(s).
[QueryInterceptor ("Customers")]
public Expression<Func<Customer, bool>> FilterCustomers()
{
return c => c.Name == /* Current principal name. */ &&
this.CurrentDataSource.QueryRules.Contains(
rule => rule.Name == c.Name &&
rule.CustomerAllowedToQuery == true
);
}
You can use David Fowler's Query Interceptor:
https://github.com/davidfowl/QueryInterceptor
One example of its use:
IQueryable q = ...;
IQueryable modifed = q.InterceptWith(new MyInterceptor());
And on class MyInterceptor:
protected override Expression VisitBinary(BinaryExpression node) {
if (node.NodeType == ExpressionType.Equal) {
// Change == to !=
return Expression.NotEqual(node.Left, node.Right);
}
return base.VisitBinary(node);
}