Develop Reference

Execute custom SQL before running FromSqlRaw in Entity Framework Core 6 or above in SQL Server

I only need it to work for SQL Server. This is an example. The question is about a general approach.
There is a nice extension method from https://entityframework-extensions.net called WhereBulkContains. It is, sort of, great, except that the code of the methods in this library is obfuscated and they do not produce valid SQL when .ToQueryString() is called on IQueryable<T> with these extension methods applied.
Subsequently, I can't use such methods in production code as I am not "allowed" to trust such code due to business reasons. Sure, I can write tons of tests to ensure that WhereBulkContains works as expected, except that there are some complicated cases where the performance of WhereBulkContains is well below stellar, whereas properly written SQL works in a blink of an eye. And (read above), since the code of this library is obfuscated, there is no way to figure out what's wrong there without spending a large amount of time. We would've bought the license (as this is not a freeware) if the library weren't obfuscated. All together that basically kills the library for our purposes.
This is where it gets interesting. I can easily create and populate a temporary table, e.g. (I have a table called EFAgents with an int PK called AgentId in the database):
private string GetTmpAgentSql(IEnumerable<int> agentIds) => #$"
drop table if exists #tmp_Agents;
create table #tmp_Agents (AgentId int not null, primary key clustered (AgentId asc));
{(agentIds
.Chunk(1_000)
.Select(e => $#"
insert into #tmp_Agents (AgentId)
values
({e.JoinStrings("), (")});
")
.JoinStrings(""))}
select 0 as Result
";
private const string AgentSql = #"
select a.* from EFAgents a inner join #tmp_Agents t on a.AgentID = t.AgentId";
where GetContext returns EF Core database context and JoinStrings comes from Unity.Interception.Utilities and then use it as follows:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
var tmpSql = GetTmpAgentSql(agentIds);
using var ctx = GetContext();
// This creates a temporary table and populates it with the ids.
// This is a proprietary port of EF SqlQuery code, but I can post the whole thing if necessary.
var _ = await ctx.GetDatabase().SqlQuery<int>(tmpSql).FirstOrDefaultAsync();
// There is a DbSet<EFAgent> called Agents.
var query = ctx.Agents
.FromSqlRaw(AgentSql)
.Join(ctx.Agents, t => t.AgentId, a => a.AgentId, (t, a) => a);
var sql = query.ToQueryString() + Environment.NewLine;
// This should provide a valid SQL; https://entityframework-extensions.net does NOT!
// WriteLine - writes to console or as requested. This is irrelevant to the question.
WriteLine(sql);
var result = await query.ToListAsync();
return result;
}
So, basically, I can do what I need in two steps:
using var ctx = GetContext();
// 1. Create a temp table and populate it - call GetTmpAgentSql.
...
// 2. Build the join starting from `FromSqlRaw` as in example above.
This is doable, half-manual, and it is going to work.
The question is how to do that in one step, e.g., call:
.WhereMyBulkContains(aListOfIdConstraints, whateverElseIsneeded, ...)
and that's all.
I am fine if I need to pass more than one parameter in each case in order to specify the constraints.
To clarify the reasons why do I need to go into all these troubles. We have to interact with a third party database. We don't have any control of the schema and data there. The database is large and poorly designed. That resulted in some ugly EFC LINQ queries. To remedy that, some of that ugliness was encapsulated into a method, which takes IQueryable<T> (and some more parameters) and returns IQueryable<T>. Under the hood this method calls WhereBulkContains. I need to replace this WhereBulkContains by, call it, WhereMyBulkContains, which would be able to provide correct ToQueryString representation (for debugging purposes) and be performant. The latter means that SQL should not contain in clause with hundreds (and even sometimes thousands) of elements. Using inner join with a [temp] table with a PK and having an index on the FK field seem to do the trick if I do that in pure SQL. But, ... I need to do that in C# and effectively in between two LINQ method calls. Refactoring everything is also not an option because that method is used in many places.
Thanks a lot!

I think you really want to use a Table Valued Parameter.
Creating an SqlParameter from an enumeration is a little fiddly, but not too difficult to get right;
CREATE TYPE [IntValue] AS TABLE (
Id int NULL
)
private IEnumerable<SqlDataRecord> FromValues(IEnumerable<int> values)
{
var meta = new SqlMetaData(
"Id",
SqlDbType.Int
);
foreach(var value in values)
{
var record = new SqlDataRecord(
meta
);
record.SetInt32(0, value);
yield return record;
}
}
public SqlParameter ToIntTVP(IEnumerable<int> values){
return new SqlParameter()
{
TypeName = "IntValue",
SqlDbType = SqlDbType.Structured,
Value = FromValues(values)
};
}
Personally I would define a query type in EF Core to represent the TVP. Then you can use raw sql to return an IQueryable.
public class IntValue
{
public int Id { get; set; }
}
modelBuilder.Entity<IntValue>(e =>
{
e.HasNoKey();
e.ToView("IntValue");
});
IQueryable<IntValue> ToIntQueryable(DbContext ctx, IEnumerable<int> values)
{
return ctx.Set<IntValue>()
.FromSqlInterpolated($"select * from {ToIntTVP(values)}");
}
Now you can compose the rest of your query using Linq.
var ids = ToIntQueryable(ctx, agentIds);
var query = ctx.Agents
.Where(a => ids.Any(i => i.Id == a.Id));

I would propose to use linq2db.EntityFrameworkCore (note that I'm one of the creators). It has built-in temporary tables support.
We can create simple and reusable function which filters records of any type:
public static class HelperMethods
{
private class KeyHolder<T>
{
[PrimaryKey]
public T Key { get; set; } = default!;
}
public static async Task<List<TEntity>> GetRecordsByIds<TEntity, TKey>(this IQueryable<TEntity> query, IEnumerable<TKey> ids, Expression<Func<TEntity, TKey>> keyFunc)
{
var ctx = LinqToDBForEFTools.GetCurrentContext(query) ??
throw new InvalidOperationException("Query should be EF Core query");
// based on DbContext options, extension retrieves connection information
using var db = ctx.CreateLinqToDbConnection();
// create temporary table and BulkCopy records into that table
using var tempTable = await db.CreateTempTableAsync(ids.Select(id => new KeyHolder<TKey> { Key = id }), tableName: "temporaryIds");
var resultQuery = query.Join(tempTable, keyFunc, t => t.Key, (q, t) => q);
// we use ToListAsyncLinqToDB to avoid collission with EF Core async methods.
return await resultQuery.ToListAsyncLinqToDB();
}
}
Then we can rewrite your function GetAgents to the following:
private async Task<List<EFAgent>> GetAgents(List<int> agentIds)
{
using var ctx = GetContext();
var result = await ctx.Agents.GetRecordsByIds(agentIds, a => a.AgentId);
return result;
}

how to insert records to unrelated tables using EF

I have a Comment table which can be linked to many different entities that have comments, but for reasons, I have not linked those tables. Instead Comment contains TableReferenceId and EntryReferenceId. TableReferenceId is just an int that we can check in the app layer as to which entity/table that comment refers to, and EntryReferenceId is an int that refers to a particular entry in said entity/table to which the comment belongs.
Querying such comments by table and entry reference would be fine, but when inserting bulk data, I am drawing a blank. For example if I have Vehicle entity and a Vehicle can have many comments, when inserting the data, how would I link them since I don't have a VehicleId yet? Is this doable or is it better to just go many-to-many route for each of the tables that link to comments?

If you can avoid this situation, then you should try to, or you should try to avoid supporting a bulk insert. If you must do this though, then either of the following patterns may work for you.
Perform the Bulk Insert in 2 stages, before the normal import, maintain a map or dictionary of records and the comments that they are linked to, then after the first call to SaveChanges() the IDs will be available to insert.
You could store the mapped comments inside an unbound collection on the entity, after SaveChanges() if there are any entries in this collection, they should be inserted using the new record's Id.
Lets look at the first option:
var mappedComments = new Dictionary<Vehicle,Comment[]>();
// bulk processing, however you choose to do it
// importantly for each item, capture the record reference and the comments
foreach(var item in source)
{
Vehicle newItem;
... construct/parse the new Entity object
List<Comment> newComments = new List<Comment>();
... parse the comments records
// store the map
mappedComments.Add(newItem, newComments.ToArray());
// Add the entity to the context?
db.AddToVehicles(newItem);
}
db.SaveChanges();
foreach(var mapEntry in mappedComments)
{
var newVehicle = mapEntry.Key;
// replace this with your actual logic of course...
int vehicleTableReferenceId = db.TableReferences.Single(x => x.TableName == nameof(Vehicle));
foreach(var comment in mappEntry.Value)
{
comment.TableReferenceId = vehicleTableReferenceId;
comment.EntityReferenceId = newVehicle.Id; // the Id that is now populated
db.AddToComments(comment);
}
}
db.SaveChanges();
If you have a lot Entity types that exhibit this linking behaviour, then you could build this functionality into the Entities themselves, by embedding the mapped comments within the entity itself.
Define an Interface that describes an object that has a weak reference to these Comments
public interface ICommentsToInsert
{
// Only necessary if your convention is NOT to use a common name for the PK
int Id { get; }
ICollection<Comment> CommentsToInsert { get;set;}
}
Implement this interface and add an unmapped collection property to the entities to store the Comment Entries to insert against each record.
partial class Vehicle : ICommentsToInsert
{
[NotMapped]
int ICommentsToInsert.Id { get => Vehicle_Id; }
[NotMapped]
public ICollection<Comment> CommentsToInsert { get;set; } = new HashSet<Comment>();
}
In your bulk logic, add the Comment records into the Vehicle.CommentsToInsert collection, I'll leave that to you...
Override SaveChanges() to detect entities that have comments and re-process them after the save operation.
In this example I am storing the EntityState for all modified entries before the save, this is overkill for this particular example, but you only lose this state information during the save, keeping a record of it becomes useful for a whole range of other applications for post-processing logic.
public override int SaveChanges()
{
var beforeStates = BeforeSaveChanges();
int result = base.SaveChanges();
if (AfterSaveChanges(beforeStates);
result += base.SaveChanges();
return results;
}
private Dictionary<DbEntityEntry, EntityState> BeforeSaveChanges()
{
var beforeSaveChanges = new Dictionary<DbEntityEntry, EntityState>();
foreach( var entry in this.ChangeTracker.Entries())
{
//skip unchanged entries!
if (entry.State == EntityState.Unchanged)
continue;
// Today, only cache the ICommentsToInsert records...
if (entry.Entity is ICommentsToInsert)
beforeSaveChanges.Add(entry, entry.State);
}
return beforeSaveChanges;
}
private bool AfterSaveChanges(Dictionary<DbEntityEntry, EntityState> statesBeforeSaveChanges)
{
bool moreChanges = false;
foreach (var entry in statesBeforeChanges)
{
if (entry.Key.Entity is ICommentsToInsert hasComments)
{
if(hasComments.CommentsToInsert.Any())
{
moreChanges = true;
// Get the Id to the TableReference, based on the name of the Entity type
// you would normally cache this type of lookup, rather than hitting the DB every time
int tableReferenceId = db.TableReferences.Single(x =
> x.TableName == entry.Key.Entity.GetType().Name);
foreach (var comment in hasComments.CommentsToInsert)
{
comment.TableReferenceId = tableReferenceId;
comment.EntityReferenceId = hasComments.Id;
db.AddToComments(comment);
}
}
}
}
return moreChanges;
}
You can further evolve this by implementing DbTransaction scopes to rollback the whole lot if things fail, this code itself is para-phrased from my common routines that I use in production code, so whilst it may not work as is, the concept has served me well in many projects.

Entity framework: Nesting same context says connection already open

I have a piece of code that needs to return me a delimited string on basis of data from a DB, all runs fine except for the line against which comment is mentioned.
I know the fact that single DbContext can not be used for multiple QUERIES at single instance of time.
private string FetchNewEmployees(DateTime addedAfter)
{
StringBuilder newEmployees = new StringBuilder();
using (MyDbContext dbContext = new MyDbContext())
{
var employees = dbContext.Student.Where(p => p.lastupdatedon > addedAfter);
foreach (var employee in employees)
{
newEmployees.Append(string.Format("{0}|{1}|{2}|{3}|{4}{5}",
employee.Name,
employee.DOB,
employee.department.DepartmentName, // This line throws error saying connection already open | commenting this makes it work like charm
employee.EMailAddress,
employee.SchoolName,
System.Environment.NewLine));
}
}
return newEmployees.ToString();
}
The problem id, "department" is another table, hence a foreign key for "employee"...
In case I am unclear, let me know.
Any help right now will be like winning two worlds for me :)

First workaround:
var employees = dbContext.Student.Where(p => p.lastupdatedon > addedAfter).ToList();
...
This closes the connection to the student table but will generate additional queries to lazy load departments.
Another option:
var employees = dbContext.Student.Include( s => s.department ).Where(p => p.lastupdatedon > addedAfter);
...
This causes a single query to be generated which joins both tables.

bulk delete in entity framework

I'd like to bulk delete records from a table using linq.
There's a post that describes how to do it:
Bulk-deleting in LINQ to Entities
var query = from c in ctx.Customers
where c.SalesPerson.Email == "..."
select c;
query.Delete();
But the function "Delete" doesn't exist in my var variable.
Furthermore, the function "SubmitChanges" doesn't exist on my context.

There is an interesting NuGet package that lets you do batch deletes and updates:

There is no currently supported bulk delete baked into Entity Framework. Its actually one of the features being discussed on codeplex now EF is open-source.
EntityFramework.Extended provides batch delete support (you can find this in nuget) however my experience is that it has some performance issues.

This code adds a simple extension method to any DbContext that will bulk delete all data in any table referenced within the entity framework query you provide. It works by simply extracting all table names involved in the query, and attempting to delete the data by issuing a "DELETE FROM tablename" SQL query, which is common across most types of database.
To use, simply do this:
myContext.BulkDelete(x => x.Things);
which will delete everything in the table linked to the Things entity store.
The code:
using System.Linq;
using System.Text.RegularExpressions;
namespace System.Data.Entity {
public static class DbContextExtensions {
/// <summary>
/// Efficiently deletes all data from any database tables used by the specified entity framework query.
/// </summary>
/// <typeparam name="TContext">The DbContext Type on which to perform the delete.</typeparam>
/// <typeparam name="TEntity">The Entity Type to which the query resolves.</typeparam>
/// <param name="ctx">The DbContext on which to perform the delete.</param>
/// <param name="deleteQuery">The query that references the tables you want to delete.</param>
public static void BulkDelete<TContext, TEntity>(this TContext ctx, Func<TContext, IQueryable<TEntity>> deleteQuery) where TContext : DbContext {
var findTables = new Regex(#"(?:FROM|JOIN)\s+(\[\w+\]\.\[\w+\])\s+AS");
var qry = deleteQuery(ctx).ToString();
// Get list of all tables mentioned in the query
var tables = findTables.Matches(qry).Cast<Match>().Select(m => m.Result("$1")).Distinct().ToList();
// Loop through all the tables, attempting to delete each one in turn
var max = 30;
var exception = (Exception)null;
while (tables.Any() && max-- > 0) {
// Get the next table
var table = tables.First();
try {
// Attempt the delete
ctx.Database.ExecuteSqlCommand(string.Format("DELETE FROM {0}", table));
// Success, so remove table from the list
tables.Remove(table);
} catch (Exception ex) {
// Error, probably due to dependent constraint, save exception for later if needed.
exception = ex;
// Push the table to the back of the queue
tables.Remove(table);
tables.Add(table);
}
}
// Error error has occurred, and cannot be resolved by deleting in a different
// order, then rethrow the exception and give up.
if (max <= 0 && exception != null) throw exception;
}
}
}

I do it like this which seems to work fine. Please let know if there is a reason why this is bad practice in any way.
var customersToDelete = await ctx.Customers.Where(c => c.Email == "...").ToListAsync();
foreach (var customerToDelete in customersToDelete)
{
ctx.Customers.Remove(customerToDelete);
}
await ctx.SaveChangesAsync();

I was experiencing the same problem with EF executing thousands of DELETE queries after SaveChanges call. I wasn't sure that EntityFramework.Extensions commercial library would help me so I decided to implement bulk DELETE myself and came up with something similar to BG100's solution!
public async Task<List<TK>> BulkDeleteAsync(List<TK> ids)
{
if (ids.Count < 1) {
return new List<TK>();
}
// NOTE: DbContext here is a property of Repository Class
// SOURCE: https://stackoverflow.com/a/9775744
var tableName = DbContext.GetTableName<T>();
var sql = $#"
DELETE FROM {tableName}
OUTPUT Deleted.Id
// NOTE: Be aware that 'Id' column naming depends on your project conventions
WHERE Id IN({string.Join(", ", ids)});
";
return await #this.Database.SqlQuery<TK>(sql).ToListAsync();
}
If you have something like generic repository that should work for you just fine. At least you could try to fit it into your EF infrastructure.
I also tweaked it a bit more and was able to execute queries on multiple chunks of entities. It would help you if there are any restrictions of query size in your DB.
const int ChunkSize = 1024;
public async Task<List<TK>> BulkDeleteAsync(List<TK> ids)
{
if (ids.Count < 1) {
return new List<TK>();
}
// SOURCE: https://stackoverflow.com/a/20953521/11344051
var chunks = ids.Chunks(ChunkSize).Select(c => c.ToList()).ToList();
var tableName = DbContext.GetTableName<T>();
var deletedIds = new List<TK>();
foreach (var chunk in chunks) {
var sql = $#"
DELETE FROM {tableName}
OUTPUT Deleted.Id
WHERE Id IN({string.Join(", ", chunk)});
";
deletedIds.AddRange(DbContext.Database.SqlQuery<TK>(sql).ToListAsync());
}
return deletedIds;
}

How can I correctly attach record object to data context in Entity Framework?

I need to switch data context for some records. So basically I have db context A and B, I fetch records using A, then I switch to B, alter records, and save them.
When I call Attach for B, I get exception that records are using by multiple data context, when I add Detach for A, I get exception, that records are not attached to A.
So how can I switch the data context?
Example
db_creator is creator of db context. Here I fetch the data (corrected version):
using (var db = db_creator.Create())
{
var coll = db.Mailing.Where(it => !it.mail_IsSent).ToList(); // (*)
coll.ForEach(it => db.Detach(it));
return coll;
}
(*) the mistake was caused by refactoring this piece, I created extra data context, and then later I tried to detach records from another one.
Now I would like to switch data context to new one, do some computation and modifications and save the records. coll is List of the records:
using (var db = db_creator.Create())
{
coll.ForEach(it => db.Mailing.Attach(it));
...
db.SaveChanges();
}

I recommend change your design and have ONE context at a time. (Based on your project type this could vary. Usually in web apps it's one context per http request.)
For example in a web application, you can do this like below:
protected MyContext Context
{
get
{
var context = HttpContext.Current.Items["MyContext"];
if (context == null)
{
context = new MyContext();
HttpContext.Current.Items.Add("MyContext", context);
}
return context as MyContext;
}
}
And dispose it in your Application_EndRequest:
app.EndRequest += (sender, args) =>
{
HttpContext.Current.Items.Remove("MyContext");
}
If you have multiple project types, then consider using an Ioc.
But if you still want to use two contexts, you can do as below(myEntity is your object you want to detach/attach):
if (context1.Entry(myEntity).State != EntityState.Detached);
{
((IObjectContextAdapter)context1).ObjectContext.Detach(myEntity);
}
context2.MyEntities.Attach(myEntity);

I have come to the conclusion that it's best (i.e. easier to avoid problems) to use ApplyCurrentValues instead of attaching. That is because when you call Attach there are several things going on that we don't know about, but which may surface in one way or the other through an exception. I prefer to do things the way I have control over what is done.
var myMailings = db_creator.Create().Mailing.Where(it => !it.mail_IsSent).ToList();
... // make modifications and retrieve coll a collection of Mailing objects
using (var db = db_creator.Create()) {
... // if you want to further modify the objects in coll you should do this before writing it to the context
foreach (Mailing it in coll) {
if (it.EntityKey != null) db.GetObjectByKey(it.EntityKey); // load entity
else db.Mailing.Single(x => x.YourId == it.YourId); // load the entity when EntityKey is not available, e.g. because it's a POCO
db.Mailing.ApplyCurrentValues(it); // sets the entity state to Modified
}
db.SaveChanges();
}
EDIT:
I tested the performance of this vs using Attach. It should be noted that for a simple table with an integer primary key, an int, a float and a string column for updating 1000 entries: the difference was 2.6s vs 0.27s, so this is significantly slower.
EDIT2:
A similar question was raised here. There the answer warned about using ApplyCurrentValues in conjunction with timestamp columns.
I also compared performance when loading the entity with db.GetObjectByKey(it.EntityKey) and there the performance difference is much smaller. ApplyCurrentValues then just takes 0.44s.