SQLite with Entity Framework Core is slow - c#

Inserting a record to SQLite DB via Entity Framework Core in C# in .NET Core is extremely slow. It is 10 times slower than my expectation. Is there a magic to improve performance?
public class MyRecord
{
[Key]
public int ID { get; set; }
public int Value { get; set; }
}
public class MyDatabase : DbContext
{
public DbSet<MyRecord> Records { get; set; }
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseSqlite("Data source=mydb.db");
#if DEBUG
optionsBuilder.EnableSensitiveDataLogging(true);
#endif
}
}
public class MyDatabaseTests
{
public MyDatabaseTests()
{
var db = new MyDatabase();
db.Database.EnsureDeleted();
db.Database.EnsureCreated();
}
[TestMethod]
public void EntityFramework1000Transactions()
{
for (int i = 0; i < 1000; i++)
{
using var db = new MyDatabase();
var rec = new MyRecord
{
ID = i+1,
Value = 123456789
};
var tx = db.Database.BeginTransaction();
db.Records.Add(rec);
db.SaveChanges();
tx.Commit();
}
}
}
EntityFramework1000Transactions takes 10 seconds on my PC with SATA-SSD drive. Replacing to eNVM is not my option because I am writing an application for a sort of embedded system which has equivalent storage.
I tried using db.ChangeTracker.AutoDetectChangesEnabled = false;, or
db.BulkSaveChanges(); (with Z.EntityFramework.Extensions.EFCore)
And they didn't help me.
My environment:
Visual Studio 2019 Version 16.11.2
.NET Core 3.1
C#
Microsoft.EntityFrameworkCore.Sqlite 5.0.11
Windows 10
Edit
The loop was a simulation of a process in the application. They are not single transaction. The loop is intended to simulate 1000 transactions. Therefore they cannot be in the same transaction.
It also cannot remove transaction. The transactions in the application is actually more complicate and can update two or more tables/records at the same time. It must be handled in a transaction. therefore removing transaction is not my option.

Limiting the number of connection hits(inside for loop) to database can resolve the problem as mentioned in this thread.

Related

How to properly write a wrapper around Database.BeginTransaction/BeginTransactionAsync() in EF Core 6? [duplicate]

I am already using transactions inside my repository functions in some cases because I sometimes need to insert data into two tables at once and I want the whole operation to fail if one of the inserts fails.
Now I ran into a situation where I had to wrap calls to multiple repositories / functions in another transaction, but when one of those functions already uses a transaction internally I will get the error The connection is already in a transaction and cannot participate in another transaction.
I do not want to remove the transaction from the repository function because this would mean that I have to know for which repository functions a transaction is required which I would then have to implement in the service layer. On the other hand, it seems like I cannot use repository functions in a transaction when they already use a transaction internally. Here is an example for where I am facing this problem:
// Reverse engineered classes
public partial class TblProject
{
public TblProject()
{
TblProjectStepSequences = new HashSet<TblProjectStepSequence>();
}
public int ProjectId { get; set; }
public virtual ICollection<TblProjectStepSequence> TblProjectStepSequences { get; set; }
}
public partial class TblProjectTranslation
{
public int ProjectId { get; set; }
public string Language { get; set; }
public string ProjectName { get; set; }
public virtual TblProject Project { get; set; }
}
public partial class TblProjectStepSequence
{
public int SequenceId { get; set; }
public int ProjectId { get; set; }
public int StepId { get; set; }
public int SequencePosition { get; set; }
public virtual TblStep Step { get; set; }
public virtual TblProject Project { get; set; }
}
// Creating a project in the ProjectRepository
public async Task<int> CreateProjectAsync(TblProject project, ...)
{
using (var transaction = this.Context.Database.BeginTransaction())
{
await this.Context.TblProjects.AddAsync(project);
await this.Context.SaveChangesAsync();
// Insert translations... (project Id is required for this)
await this.Context.SaveChangesAsync();
transaction.Commit();
return entity.ProjectId;
}
}
// Creating the steps for a project in the StepRepository
public async Task<IEnumerable<int>> CreateProjectStepsAsync(int projectId, IEnumerable<TblProjectStepSequence> steps)
{
await this.Context.TblProjectStepSequences.AddRangeAsync(steps);
await this.Context.SaveChangesAsync();
return steps.Select(step =>
{
return step.SequenceId;
}
);
}
// Creating a project with its steps in the service layer
public async Task<int> CreateProjectWithStepsAsync(TblProject project, IEnumerable<TblProjectStepSequence> steps)
{
// This is basically a wrapper around Database.BeginTransaction() and IDbContextTransaction
using (Transaction transaction = await transactionService.BeginTransactionAsync())
{
int projectId = await projectRepository.CreateProjectAsync(project);
await stepRepository.CreateProjectStepsAsync(projectId, steps);
return projectId;
}
}
Is there a way how I can nest multiple transactions inside each other without already knowing in the inner transactions that there could be an outer transaction?
I know that it might not be possible to actually nest those transactions from a technical perspective but I still need a solution which either uses the internal transaction of the repository or the outer one (if one exists) so there is no way how I could accidentally forget to use a transaction for repository functions which require one.
You could check the CurrentTransaction property and do something like this:
var transaction = Database.CurrentTransaction ?? Database.BeginTransaction()
If there is already a transaction use that, otherwise start a new one...
Edit: Removed the Using block, see comments. More logic is needed for Committing/Rollback the transcaction though...
I am answering the question you asked "How to nest transactions in EF Core 6?"
Please note that this is just a direct answer, but not an evaluation what is best practice and what not. There was a lot of discussion going around best practices, which is valid to question what fits best for your use case but not an answer to the question (keep in mind that Stack overflow is just a Q+A site where people want to have direct answers).
Having said that, let's continue with the topic:
Try to use this helper function for creating a new transaction:
public CommittableTransaction CreateTransaction()
=> new System.Transactions.CommittableTransaction(new TransactionOptions()
{
IsolationLevel = System.Transactions.IsolationLevel.ReadCommitted
});
Using the Northwind database as example database, you can use it like:
public async Task<int?> CreateCategoryAsync(Categories category)
{
if (category?.CategoryName == null) return null;
using(var trans = CreateTransaction())
{
await this.Context.Categories.AddAsync(category);
await this.Context.SaveChangesAsync();
trans.Commit();
return category?.CategoryID;
}
}
And then you can call it from another function like:
/// <summary>Create or use existing category with associated products</summary>
/// <returns>Returns null if transaction was rolled back, else CategoryID</returns>
public async Task<int?> CreateProjectWithStepsAsync(Categories category)
{
using var trans = CreateTransaction();
int? catId = GetCategoryId(category.CategoryName)
?? await CreateCategoryAsync(category);
if (!catId.HasValue || string.IsNullOrWhiteSpace(category.CategoryName))
{
trans.Rollback(); return null;
}
var product1 = new Products()
{
ProductName = "Product A1", CategoryID = catId
};
await this.Context.Products.AddAsync(product1);
var product2 = new Products()
{
ProductName = "Product A2", CategoryID = catId
};
await this.Context.Products.AddAsync(product2);
await this.Context.SaveChangesAsync();
trans.Commit();
return catId;
}
To run this with LinqPad you need an entry point (and of course, add the NUGET package EntityFramework 6.x via F4, then create an EntityFramework Core connection):
// Main method required for LinqPad
UserQuery Context;
async Task Main()
{
Context = this;
var category = new Categories()
{
CategoryName = "Category A1"
// CategoryName = ""
};
var catId = await CreateProjectWithStepsAsync(category);
Console.WriteLine((catId == null)
? "Transaction was aborted."
: "Transaction successful.");
}
This is just a simple example - it does not check if there are any product(s) with the same name existing, it will just create a new one. You can implement that easily, I have shown it in the function CreateProjectWithStepsAsync for the categories:
int? catId = GetCategoryId(category.CategoryName)
?? await CreateCategoryAsync(category);
First it queries the categories by name (via GetCategoryId(...)), and if the result is null it will create a new category (via CreateCategoryAsync(...)).
Also, you need to consider the isolation level: Check out System.Transactions.IsolationLevel to see if the one used here (ReadCommitted) is the right one for you (it is the default setting).
What it does is creating a transaction explicitly, and notice that here we have a transaction within a transaction.
Note:
I have used both ways of using - the old one and the new one. Pick the one you like more.
Just don't call SaveChanges multiple times.
The problem is caused by calling SaveChanges multiple times to commit changes made to the DbContext instead of calling it just once at the end. It's simply not needed. A DbContext is a multi-entity Unit-of-Work. It doesn't even keep an open connection to the database. This allows 100-1000 times better throughput for the entire application by eliminating cross-connection blocking.
A DbContext tracks all modifications made to the objects it tracks and persists/commits them when SaveChanges is called using an internal transaction. To discard the changes, simply dispose the DbContext. That's why all examples show using a DbContext in a using block - that's actually the scope of the Unit-of-Work "transaction".
There's no need to "save" parent objects first. EF Core will take care of this itself inside SaveChanges.
Using the Blog/Posts example in the EF Core documentation tutorial :
public class BloggingContext : DbContext
{
public DbSet<Blog> Blogs { get; set; }
public DbSet<Post> Posts { get; set; }
public string DbPath { get; }
// The following configures EF to create a Sqlite database file in the
// special "local" folder for your platform.
protected override void OnConfiguring(DbContextOptionsBuilder options)
=> options.UseSqlServer($"Data Source=.;Initial Catalog=tests;Trusted_Connection=True; Trust Server Certificate=Yes");
}
public class Blog
{
public int BlogId { get; set; }
public string Url { get; set; }
public List<Post> Posts { get; } = new();
}
public class Post
{
public int PostId { get; set; }
public string Title { get; set; }
public string Content { get; set; }
public int BlogId { get; set; }
public Blog Blog { get; set; }
}
The following Program.cs will add a Blog with 5 posts but only call SaveChanges once at the end :
using (var db = new BloggingContext())
{
Blog blog = new Blog { Url = "http://blogs.msdn.com/adonet" };
IEnumerable<Post> posts = Enumerable.Range(0, 5)
.Select(i => new Post {
Title = $"Hello World {i}",
Content = "I wrote an app using EF Core!"
});
blog.Posts.AddRange(posts);
db.Blogs.Add(blog);
await db.SaveChangesAsync();
}
The code never specifies or retrieves the IDs. Add is an in-memory operation so there's no reason to use AddAsync. Add starts tracking both the blog and the related Posts in the Inserted state.
The contents of the tables after this are :
select * from blogs
select * from posts;
-----------------------
BlogId Url
1 http://blogs.msdn.com/adonet
PostId Title Content BlogId
1 Hello World 0 I wrote an app using EF Core! 1
2 Hello World 1 I wrote an app using EF Core! 1
3 Hello World 2 I wrote an app using EF Core! 1
4 Hello World 3 I wrote an app using EF Core! 1
5 Hello World 4 I wrote an app using EF Core! 1
Executing the code twice will add another blog with another 5 posts.
PostId Title Content BlogId
1 Hello World 0 I wrote an app using EF Core! 1
2 Hello World 1 I wrote an app using EF Core! 1
3 Hello World 2 I wrote an app using EF Core! 1
4 Hello World 3 I wrote an app using EF Core! 1
5 Hello World 4 I wrote an app using EF Core! 1
6 Hello World 0 I wrote an app using EF Core! 2
7 Hello World 1 I wrote an app using EF Core! 2
8 Hello World 2 I wrote an app using EF Core! 2
9 Hello World 3 I wrote an app using EF Core! 2
10 Hello World 4 I wrote an app using EF Core! 2
Using SQL Server XEvents Profiler shows that these SQL calls are made:
exec sp_executesql N'SET NOCOUNT ON;
INSERT INTO [Blogs] ([Url])
VALUES (#p0);
SELECT [BlogId]
FROM [Blogs]
WHERE ##ROWCOUNT = 1 AND [BlogId] = scope_identity();
',N'#p0 nvarchar(4000)',#p0=N'http://blogs.msdn.com/adonet'
exec sp_executesql N'SET NOCOUNT ON;
DECLARE #inserted0 TABLE ([PostId] int, [_Position] [int]);
MERGE [Posts] USING (
VALUES (#p1, #p2, #p3, 0),
(#p4, #p5, #p6, 1),
(#p7, #p8, #p9, 2),
(#p10, #p11, #p12, 3),
(#p13, #p14, #p15, 4)) AS i ([BlogId], [Content], [Title], _Position) ON 1=0
WHEN NOT MATCHED THEN
INSERT ([BlogId], [Content], [Title])
VALUES (i.[BlogId], i.[Content], i.[Title])
OUTPUT INSERTED.[PostId], i._Position
INTO #inserted0;
SELECT [i].[PostId] FROM #inserted0 i
ORDER BY [i].[_Position];
',N'#p1 int,#p2 nvarchar(4000),#p3 nvarchar(4000),#p4 int,#p5 nvarchar(4000),#p6 nvarchar(4000),#p7 int,#p8 nvarchar(4000),#p9 nvarchar(4000),#p10 int,#p11 nvarchar(4000),#p12 nvarchar(4000),#p13 int,#p14 nvarchar(4000),#p15 nvarchar(4000)',#p1=3,#p2=N'I wrote an app using EF Core!',#p3=N'Hello World 0',#p4=3,#p5=N'I wrote an app using EF Core!',#p6=N'Hello World 1',#p7=3,#p8=N'I wrote an app using EF Core!',#p9=N'Hello World 2',#p10=3,#p11=N'I wrote an app using EF Core!',#p12=N'Hello World 3',#p13=3,#p14=N'I wrote an app using EF Core!',#p15=N'Hello World 4'
The unusual SELECT and MERGE are used to ensure IDENTITY values are returned in the order the objects were inserted, so EF Core can assign them to the object properties. After calling SaveChanges all Blog and Post objects will have the correct database-generated IDs

How to nest transactions in EF Core 6?

I am already using transactions inside my repository functions in some cases because I sometimes need to insert data into two tables at once and I want the whole operation to fail if one of the inserts fails.
Now I ran into a situation where I had to wrap calls to multiple repositories / functions in another transaction, but when one of those functions already uses a transaction internally I will get the error The connection is already in a transaction and cannot participate in another transaction.
I do not want to remove the transaction from the repository function because this would mean that I have to know for which repository functions a transaction is required which I would then have to implement in the service layer. On the other hand, it seems like I cannot use repository functions in a transaction when they already use a transaction internally. Here is an example for where I am facing this problem:
// Reverse engineered classes
public partial class TblProject
{
public TblProject()
{
TblProjectStepSequences = new HashSet<TblProjectStepSequence>();
}
public int ProjectId { get; set; }
public virtual ICollection<TblProjectStepSequence> TblProjectStepSequences { get; set; }
}
public partial class TblProjectTranslation
{
public int ProjectId { get; set; }
public string Language { get; set; }
public string ProjectName { get; set; }
public virtual TblProject Project { get; set; }
}
public partial class TblProjectStepSequence
{
public int SequenceId { get; set; }
public int ProjectId { get; set; }
public int StepId { get; set; }
public int SequencePosition { get; set; }
public virtual TblStep Step { get; set; }
public virtual TblProject Project { get; set; }
}
// Creating a project in the ProjectRepository
public async Task<int> CreateProjectAsync(TblProject project, ...)
{
using (var transaction = this.Context.Database.BeginTransaction())
{
await this.Context.TblProjects.AddAsync(project);
await this.Context.SaveChangesAsync();
// Insert translations... (project Id is required for this)
await this.Context.SaveChangesAsync();
transaction.Commit();
return entity.ProjectId;
}
}
// Creating the steps for a project in the StepRepository
public async Task<IEnumerable<int>> CreateProjectStepsAsync(int projectId, IEnumerable<TblProjectStepSequence> steps)
{
await this.Context.TblProjectStepSequences.AddRangeAsync(steps);
await this.Context.SaveChangesAsync();
return steps.Select(step =>
{
return step.SequenceId;
}
);
}
// Creating a project with its steps in the service layer
public async Task<int> CreateProjectWithStepsAsync(TblProject project, IEnumerable<TblProjectStepSequence> steps)
{
// This is basically a wrapper around Database.BeginTransaction() and IDbContextTransaction
using (Transaction transaction = await transactionService.BeginTransactionAsync())
{
int projectId = await projectRepository.CreateProjectAsync(project);
await stepRepository.CreateProjectStepsAsync(projectId, steps);
return projectId;
}
}
Is there a way how I can nest multiple transactions inside each other without already knowing in the inner transactions that there could be an outer transaction?
I know that it might not be possible to actually nest those transactions from a technical perspective but I still need a solution which either uses the internal transaction of the repository or the outer one (if one exists) so there is no way how I could accidentally forget to use a transaction for repository functions which require one.
You could check the CurrentTransaction property and do something like this:
var transaction = Database.CurrentTransaction ?? Database.BeginTransaction()
If there is already a transaction use that, otherwise start a new one...
Edit: Removed the Using block, see comments. More logic is needed for Committing/Rollback the transcaction though...
I am answering the question you asked "How to nest transactions in EF Core 6?"
Please note that this is just a direct answer, but not an evaluation what is best practice and what not. There was a lot of discussion going around best practices, which is valid to question what fits best for your use case but not an answer to the question (keep in mind that Stack overflow is just a Q+A site where people want to have direct answers).
Having said that, let's continue with the topic:
Try to use this helper function for creating a new transaction:
public CommittableTransaction CreateTransaction()
=> new System.Transactions.CommittableTransaction(new TransactionOptions()
{
IsolationLevel = System.Transactions.IsolationLevel.ReadCommitted
});
Using the Northwind database as example database, you can use it like:
public async Task<int?> CreateCategoryAsync(Categories category)
{
if (category?.CategoryName == null) return null;
using(var trans = CreateTransaction())
{
await this.Context.Categories.AddAsync(category);
await this.Context.SaveChangesAsync();
trans.Commit();
return category?.CategoryID;
}
}
And then you can call it from another function like:
/// <summary>Create or use existing category with associated products</summary>
/// <returns>Returns null if transaction was rolled back, else CategoryID</returns>
public async Task<int?> CreateProjectWithStepsAsync(Categories category)
{
using var trans = CreateTransaction();
int? catId = GetCategoryId(category.CategoryName)
?? await CreateCategoryAsync(category);
if (!catId.HasValue || string.IsNullOrWhiteSpace(category.CategoryName))
{
trans.Rollback(); return null;
}
var product1 = new Products()
{
ProductName = "Product A1", CategoryID = catId
};
await this.Context.Products.AddAsync(product1);
var product2 = new Products()
{
ProductName = "Product A2", CategoryID = catId
};
await this.Context.Products.AddAsync(product2);
await this.Context.SaveChangesAsync();
trans.Commit();
return catId;
}
To run this with LinqPad you need an entry point (and of course, add the NUGET package EntityFramework 6.x via F4, then create an EntityFramework Core connection):
// Main method required for LinqPad
UserQuery Context;
async Task Main()
{
Context = this;
var category = new Categories()
{
CategoryName = "Category A1"
// CategoryName = ""
};
var catId = await CreateProjectWithStepsAsync(category);
Console.WriteLine((catId == null)
? "Transaction was aborted."
: "Transaction successful.");
}
This is just a simple example - it does not check if there are any product(s) with the same name existing, it will just create a new one. You can implement that easily, I have shown it in the function CreateProjectWithStepsAsync for the categories:
int? catId = GetCategoryId(category.CategoryName)
?? await CreateCategoryAsync(category);
First it queries the categories by name (via GetCategoryId(...)), and if the result is null it will create a new category (via CreateCategoryAsync(...)).
Also, you need to consider the isolation level: Check out System.Transactions.IsolationLevel to see if the one used here (ReadCommitted) is the right one for you (it is the default setting).
What it does is creating a transaction explicitly, and notice that here we have a transaction within a transaction.
Note:
I have used both ways of using - the old one and the new one. Pick the one you like more.
Just don't call SaveChanges multiple times.
The problem is caused by calling SaveChanges multiple times to commit changes made to the DbContext instead of calling it just once at the end. It's simply not needed. A DbContext is a multi-entity Unit-of-Work. It doesn't even keep an open connection to the database. This allows 100-1000 times better throughput for the entire application by eliminating cross-connection blocking.
A DbContext tracks all modifications made to the objects it tracks and persists/commits them when SaveChanges is called using an internal transaction. To discard the changes, simply dispose the DbContext. That's why all examples show using a DbContext in a using block - that's actually the scope of the Unit-of-Work "transaction".
There's no need to "save" parent objects first. EF Core will take care of this itself inside SaveChanges.
Using the Blog/Posts example in the EF Core documentation tutorial :
public class BloggingContext : DbContext
{
public DbSet<Blog> Blogs { get; set; }
public DbSet<Post> Posts { get; set; }
public string DbPath { get; }
// The following configures EF to create a Sqlite database file in the
// special "local" folder for your platform.
protected override void OnConfiguring(DbContextOptionsBuilder options)
=> options.UseSqlServer($"Data Source=.;Initial Catalog=tests;Trusted_Connection=True; Trust Server Certificate=Yes");
}
public class Blog
{
public int BlogId { get; set; }
public string Url { get; set; }
public List<Post> Posts { get; } = new();
}
public class Post
{
public int PostId { get; set; }
public string Title { get; set; }
public string Content { get; set; }
public int BlogId { get; set; }
public Blog Blog { get; set; }
}
The following Program.cs will add a Blog with 5 posts but only call SaveChanges once at the end :
using (var db = new BloggingContext())
{
Blog blog = new Blog { Url = "http://blogs.msdn.com/adonet" };
IEnumerable<Post> posts = Enumerable.Range(0, 5)
.Select(i => new Post {
Title = $"Hello World {i}",
Content = "I wrote an app using EF Core!"
});
blog.Posts.AddRange(posts);
db.Blogs.Add(blog);
await db.SaveChangesAsync();
}
The code never specifies or retrieves the IDs. Add is an in-memory operation so there's no reason to use AddAsync. Add starts tracking both the blog and the related Posts in the Inserted state.
The contents of the tables after this are :
select * from blogs
select * from posts;
-----------------------
BlogId Url
1 http://blogs.msdn.com/adonet
PostId Title Content BlogId
1 Hello World 0 I wrote an app using EF Core! 1
2 Hello World 1 I wrote an app using EF Core! 1
3 Hello World 2 I wrote an app using EF Core! 1
4 Hello World 3 I wrote an app using EF Core! 1
5 Hello World 4 I wrote an app using EF Core! 1
Executing the code twice will add another blog with another 5 posts.
PostId Title Content BlogId
1 Hello World 0 I wrote an app using EF Core! 1
2 Hello World 1 I wrote an app using EF Core! 1
3 Hello World 2 I wrote an app using EF Core! 1
4 Hello World 3 I wrote an app using EF Core! 1
5 Hello World 4 I wrote an app using EF Core! 1
6 Hello World 0 I wrote an app using EF Core! 2
7 Hello World 1 I wrote an app using EF Core! 2
8 Hello World 2 I wrote an app using EF Core! 2
9 Hello World 3 I wrote an app using EF Core! 2
10 Hello World 4 I wrote an app using EF Core! 2
Using SQL Server XEvents Profiler shows that these SQL calls are made:
exec sp_executesql N'SET NOCOUNT ON;
INSERT INTO [Blogs] ([Url])
VALUES (#p0);
SELECT [BlogId]
FROM [Blogs]
WHERE ##ROWCOUNT = 1 AND [BlogId] = scope_identity();
',N'#p0 nvarchar(4000)',#p0=N'http://blogs.msdn.com/adonet'
exec sp_executesql N'SET NOCOUNT ON;
DECLARE #inserted0 TABLE ([PostId] int, [_Position] [int]);
MERGE [Posts] USING (
VALUES (#p1, #p2, #p3, 0),
(#p4, #p5, #p6, 1),
(#p7, #p8, #p9, 2),
(#p10, #p11, #p12, 3),
(#p13, #p14, #p15, 4)) AS i ([BlogId], [Content], [Title], _Position) ON 1=0
WHEN NOT MATCHED THEN
INSERT ([BlogId], [Content], [Title])
VALUES (i.[BlogId], i.[Content], i.[Title])
OUTPUT INSERTED.[PostId], i._Position
INTO #inserted0;
SELECT [i].[PostId] FROM #inserted0 i
ORDER BY [i].[_Position];
',N'#p1 int,#p2 nvarchar(4000),#p3 nvarchar(4000),#p4 int,#p5 nvarchar(4000),#p6 nvarchar(4000),#p7 int,#p8 nvarchar(4000),#p9 nvarchar(4000),#p10 int,#p11 nvarchar(4000),#p12 nvarchar(4000),#p13 int,#p14 nvarchar(4000),#p15 nvarchar(4000)',#p1=3,#p2=N'I wrote an app using EF Core!',#p3=N'Hello World 0',#p4=3,#p5=N'I wrote an app using EF Core!',#p6=N'Hello World 1',#p7=3,#p8=N'I wrote an app using EF Core!',#p9=N'Hello World 2',#p10=3,#p11=N'I wrote an app using EF Core!',#p12=N'Hello World 3',#p13=3,#p14=N'I wrote an app using EF Core!',#p15=N'Hello World 4'
The unusual SELECT and MERGE are used to ensure IDENTITY values are returned in the order the objects were inserted, so EF Core can assign them to the object properties. After calling SaveChanges all Blog and Post objects will have the correct database-generated IDs

EntityFramework6 memory usage with large amount of table due to InitializedDatabases list

In our application there are a large amount of tables (around 50k) - all of those tables are actually used and this results in a high memory consumption in entity framework.
After some memory profiling I noticed that DbCompiledModel classes were being kept in memory so after some searching tracked it down to the LazyInternalContext class that keeps a list of "InitializedDatabases".
https://github.com/dotnet/ef6/blob/master/src/EntityFramework/Internal/LazyInternalContext.cs#L670
Is there a way to prevent entity framework from doing this?, it's not a code first setup, database setup and migration are not done in this app if that is what the "InitializeDatabaseAction" implies.
Setting a "return null" or setting "InitializerDisabled" to true makes everything work but would rather not run a custom entity build plus don't know what the impact would be to just 'change' the source.
Most tables have the same definition so also tried the solution I found here:
Change table name at runtime
When trying this I'm getting an error "An open data reader exists for this command", using postgres and MARS isn't supported there (no idea why I'd need it, this just changes the sql that's run)
The solution was given in a comment bu Ivan Stoev and works.
There is no way to turn this off without using reflection, setting the "InternalContext.InitializerDisabled" property to true will make this skip the dictionary.
So:
Use a DbContext constructor that provides the DbCachedModel
Use Database.SetInitializer(null);
Set InternalContext.InitializerDisabled = true using reflection
Code from the sample I used to test this, as a test setup I had 1 main table with 30k partitions, the partitions themselves are queried because postgres (especialy 9.x) does not scale well with high number of partitions:
public class PartContext : DbContext {
private static readonly string _ConnectionString = new NpgsqlConnectionStringBuilder {
Host = "localhost",
Port = 5432,
Database = "postgres",
Username = "postgres",
Password = "password"
}.ConnectionString;
public readonly string Table;
public readonly string Partition;
public PartContext(string pMainTable, string pPartition) : base(
new NpgsqlConnection() { ConnectionString = _ConnectionString },
PartDbModelBuilder.Get(_ConnectionString, pPartition),
true
) {
Table = pMainTable;
Partition = pPartition;
Database.SetInitializer<PartContext>(null);
/**
* Disable database initialization so that the DbCachedModels are not kept internally in Entity
* This causes high memory usage when having a lot of tables
* In EF 6.4.2 there was no way to 'manage' that Dictionary externally
*/
try {
var InternalContext = typeof(PartContext).BaseType.GetProperty("InternalContext", BindingFlags.NonPublic | BindingFlags.Instance).GetValue(this, null);
InternalContext.GetType().GetProperty("InitializerDisabled").SetValue(InternalContext, true);
} catch(Exception) { }
}
public DbSet<MyPart> Parts { get; set; }
protected override void OnModelCreating(DbModelBuilder modelBuilder) {
modelBuilder.HasDefaultSchema("public");
modelBuilder.Conventions.Remove<PluralizingTableNameConvention>();
}
}
This provides the DbCachedModels:
I recommend adding some custom caching code etc, this is just from a sample
class PartDbModelBuilder {
public static DbCompiledModel Get(string pConnectionString, string pTable) {
DbModelBuilder builder = new DbModelBuilder();
builder.Entity<MyPart>().ToTable(pTable, "public");
using (var connection = new NpgsqlConnection() { ConnectionString = pConnectionString }) {
var obj = builder.Build(connection).Compile();
return obj;
}
}
}
This is the entity I used as a test:
public class MyPart {
public int id { get; set; }
public string name { get; set; }
public string value { get; set; }
}
Class I used to run the test:
class EFTest {
public void Run(int tableCount) {
int done = 0;
Parallel.For(0, tableCount, new ParallelOptions { MaxDegreeOfParallelism = 5 }, (i) => {
string id = i.ToString().PadLeft(5, '0');
using (var context = new PartContext("mypart", "mypart_" + id)) {
var objResult = context.Parts.First();
Console.WriteLine(objResult.name);
}
done++;
Console.WriteLine(done + " DONE");
});
}
}
Table definition:
CREATE TABLE IF NOT EXISTS mypart (
id SERIAL,
name text,
value text
) partition by list (name);
CREATE TABLE IF NOT EXISTS part partition of mypart_00000 for values in ('mypart00000');
CREATE TABLE IF NOT EXISTS part partition of mypart_00001 for values in ('mypart00001');
CREATE TABLE IF NOT EXISTS part partition of mypart_00002 for values in ('mypart00002');
...
Postgres 9:
CREATE TABLE IF NOT EXISTS mypart (
id SERIAL,
name text,
value text
);
CREATE TABLE IF NOT EXISTS ".$name."( CHECK ( name = 'mypart00000')) INHERITS (mypart);
CREATE TABLE IF NOT EXISTS ".$name."( CHECK ( name = 'mypart00001')) INHERITS (mypart);
CREATE TABLE IF NOT EXISTS ".$name."( CHECK ( name = 'mypart00002')) INHERITS (mypart);
...

Read MemoryCache Or InMemory Automatically for Lookup Tables in Database

Instead of reading database lookup tables directly, team wants to apply (a) MemoryCache or (b) Entity Framework InMemoryDatabase/SQLite to reduce database usage and increase memory performance.
How to automatically forward existing Lookup Tables DbContext Linq queries to read from Memory? Lookup tables only change few times a year, are mostly static, AddressCategory, CustomerStatus, ProductType . The database is composed of Large Transaction tables which should be read from Database, and lookup tables to be read from Memory.
Note: Company has existing Linq queries, and want to auto forward existing code to Cache.
Method 1: EF InMemory / SQLite Proposed Solution:
Copy/Paste Original DbContext (StoreContext) into new DbContext copy called StoryMemoryContext. In Startup.cs
services.AddDbContext<StoreContext>(options => options.UseSqlServer(#"Server=localhost;Database=StoreDatabase";));
services.AddDbContext<StoreMemoryContext>(options => options.UseInMemoryDatabase(databaseName: "StoreMemoryContext"));
services.AddDbContext<CustomDbContext>(options => options.UseSqlServer(#"Server=localhost;Database=StoreDatabase")));
Save Lookup Mini lookup tables into InMemoryDatabase 'StoryMemoryContext' (around 5 Mb max).
Have CustomDbContext inherit from original, where Original StoreContext get accessors forward to StoryMemoryContext.
Loop code all lookup Tables with T4 or Powershell automation
public class CustomDBContext : StoreContext
{
public StoreMemoryContext _storeMemoryContext; = new StoreMemoryContext();
public CustomDBContext()
{
}
public override DbSet<ProductType> ProductType
{
set
{
base.ProductType = value;
_storeMemoryContext.ProductType = value;
}
get
{
return _storeMemoryContext.ProductType;
}
}
Questions:
1) Would this Get method work? Any issues to research for? Feel free to revise/edit in answer. Open to any solution not only this,
Get accessors seems to be working.
If executing _customDbContext.ProductType.ToList() , it reads InMemory EF.
_customDbContext.CustomerTransaction.Include(c => c.ProductType).ToListAsync() will read database, matching intended behavior. Don't want to read large non-lookup table customer transaction from InMemory Database.
2) Set Accessors only partially works, since many ways exist to modify DBSet and DBContext: Add, AddRange, Remove, Add Entity Graph (is challenging to track) Physical lookup tables change only couple times a year. Thought about ChangeTracker, however to place If statement after every SaveChanges() to validate if lookup table changed/then update InMemory database, may slow application, as we have 500 transactions/per second. (this is what I was told? willing to hear more opinions about ChangeTracker and effect on performance/speed)
Note: Physical and InMemory Lookup tables, can have 30 min sync period between two, per customer requirement, as lookup tables rarely change. However near instantaneous data sync between two is goal solution for question(< 5 sec).
Method 2: MemoryCache Does not Seem to Work:
Similar create Inherited/custom DbContext where read overrides forward lookup tables to MemoryCache. Storing DBContext in MemoryCache only stores IQueryable, not actual Materialized values. Going from controller to controller refreshes DbContext, and clears out cache, feel free to revise/edit in answer.
public class CustomDBContext : EcommerceContext
{
public CustomDBContext(IMemoryCache memoryCache)
{
_memoryCache = memoryCache;
}
public override DbSet<ProductType> ProductType
{
get
{
return _memoryCache.Get<DbSet<ProductType>>("ProductTypeMemoryCache");
}
}
Update:
Answer from David Browne will not work for question requirements, as has
Usage like
var pt2 = db.Cache.LK_ProductTypes; //from cache
Company has many existing queries, and want to forward existing code to Cache. Additionally, programmers may forget to apply Cache, so code below should automatically forward
var pt1 = db.LK_ProductTypes.ToList() ;
We have mix of large transactional tables which should not be cached and small lookup tables.
DBA does not allow use SQL Server memory-optimized tables due to some limitations. Memory Optimized restrictions
Resources:
Does DbContext need MemoryCache or Redis in Net Core?
How to make Entity Framework Data Context Readonly
Net Core: Automatically Update Lookup Table Cache after Entity Framework Save and UnitOfWorkPattern
Using EF Core 2.2,
Currently thinking of creating own dbcontext where read overrides can forward LK_ tables to MemoryCache. Would this work?
No. That would break the DbContext, and prevent you from writing queries that joins the lookup items with non-cached items. So you'll need a seperate cache. You can try a pattern like this:
public class Db : DbContext
{
private IMemoryCache _memoryCache;
public Db(IMemoryCache memoryCache)
{
_memoryCache = memoryCache;
}
public class EntityCache
{
private Db db;
public EntityCache(Db db)
{
this.db = db;
}
public IList<LK_ProductType> LK_ProductTypes => db._memoryCache.GetOrCreate<IList<LK_ProductType>>("LK_ProductTypeMemoryCache",f => db.LK_ProductTypes.AsNoTracking().ToList());
}
public EntityCache Cache => new EntityCache(this);
public DbSet<LK_ProductType> LK_ProductTypes { get; set; }
}
Usage would be like
using var db = new Db(memoryCache);
var pt1 = db.LK_ProductTypes.ToList(); //from database
var pt2 = db.Cache.LK_ProductTypes; //from cache
EF Core 3 introduces a new query interception framework, which should make it possible to introduce a query results cache in your DbContext. Here's a really rough example of how this would work.
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Diagnostics;
using Microsoft.Extensions.Logging;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Data;
using System.Data.Common;
using System.Linq;
using System.Threading;
namespace EfCore3Test
{
class Customer
{
public int Id { get; set; }
public string Name { get; set; }
}
class Db : DbContext
{
string constr;
private static QueryResultsCache cache { get; } = new QueryResultsCache();
private QueryResultsCache Cache { get; } = cache;
public IList<T> CacheQueryResults<T>(IQueryable<T> query)
{
return Cache.ReadThrough(query);
}
public Db() : this("server=.;database=EfCore3Test;Integrated Security=true")
{ }
public Db(string constr)
{
this.constr = constr;
}
static readonly ILoggerFactory loggerFactory = LoggerFactory.Create(builder =>
{
builder.AddFilter((category, level) =>
category == DbLoggerCategory.Database.Command.Name
&& level == LogLevel.Information).AddConsole();
}
);
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseSqlServer(constr, a => a.UseRelationalNulls(true));
optionsBuilder.AddInterceptors(cache);
optionsBuilder.UseLoggerFactory(loggerFactory);
base.OnConfiguring(optionsBuilder);
}
public DbSet<Customer> Customers { get; set; }
}
public class QueryResultsCache : DbCommandInterceptor
{
class CacheEntry
{
public CacheEntry(DataTable dt)
{
this.Data = dt;
this.LastRefresh = DateTime.Now;
}
public DateTime LastRefresh { get; set; }
public DataTable Data { get; set; }
}
private ConcurrentDictionary<string, CacheEntry> resultCache = new ConcurrentDictionary<string, CacheEntry>();
AsyncLocal<bool> cacheEntry = new AsyncLocal<bool>();
public IList<T> ReadThrough<T>(IQueryable<T> query)
{
cacheEntry.Value = true;
var results = query.ToList();
cacheEntry.Value = false;
return results;
}
public override InterceptionResult<DbDataReader> ReaderExecuting(
DbCommand command,
CommandEventData eventData,
InterceptionResult<DbDataReader> result)
{
if (resultCache.ContainsKey(command.CommandText))
{
Console.WriteLine("Query Result from Cache");
return InterceptionResult<DbDataReader>.SuppressWithResult(resultCache[command.CommandText].Data.CreateDataReader());
}
if (cacheEntry.Value)
{
using (var rdr = command.ExecuteReader())
{
var dt = new DataTable();
dt.Load(rdr);
resultCache.AddOrUpdate(command.CommandText, s => new CacheEntry(dt), (s, d) => d);
Console.WriteLine("Cached Result Created");
return InterceptionResult<DbDataReader>.SuppressWithResult(dt.CreateDataReader());
}
}
return result;
}
}
class Program
{
static void Main(string[] args)
{
using var db = new Db();
db.Database.EnsureCreated();
var c = db.CacheQueryResults( db.Customers );
for (int i = 0; i < 1000; i++)
{
var c2 = db.Customers.ToList();
}
Console.WriteLine(c);
}
}
}
You already have a working Data Access Layer (DAL) which does a DB lookup for you. Leave it untouched.
For caching, create another abstraction over the current DAL. Let's call this Cache Access Layer (CAL). This CAL is what your application should be using now, rather than the DAL.
The CAL would now be responsible for:
Replying to Data Requests;
Sending data from cache when there's a hit;
Populating the data in cache from the DB, when there's a miss;
Writing the data to DB when it is updated in the cache.
Your application architecture would look like this before and after your changes:
Here is how it is usually done in Cloud. The use case may be different but the principles are the same, viz:
Deciding when to cache;
Determining how to cache effectively;
Caching highly dynamic data;
Managing data expiration in cache.

Entity Framework 7 and SQLite Tables not creating

I've been trying for awhile to figure out how to use a single DBContext to create multiple tables in a Code First fashion without any luck. I'm sure it's just my unfamiliarity with the framework but I'm not sure what I'm missing. Here's a simple example with entities and the DBContext.
[Table("MyEntity")]
public class MyEntity
{
[Key, DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public Guid Id { get; set; }
public string MyColumn { get; set; }
public int MyNumber { get; set; }
}
[Table("MySecondEntity")]
public class MySecondEntity
{
[Key, DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public Guid Id { get; set; }
public string MyColumn { get; set; }
public int MyNumber { get; set; }
}
public class MyContext : DbContext
{
public DbSet<MyEntity> MyTable { get; set; }
public DbSet<MySecondEntity> MyTable2 { get; set; }
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
var connectionStringBuilder = new SqliteConnectionStringBuilder {DataSource = "test.db"};
var connectionString = connectionStringBuilder.ToString();
var connection = new SqliteConnection(connectionString);
optionsBuilder.UseSqlite(connection);
}
}
It looks to me like it should work, but when I call it in the below code it blows up with a 'no such table: MyEntity' Sqlite exception when hitting the first foreach loop.
static void Main(string[] args)
{
using (var db = new MyContext())
{
MyEntity testEntity1 = new MyEntity();
MySecondEntity entity1 = new MySecondEntity();
testEntity1.MyColumn = "Test Data 1";
testEntity1.MyNumber = 12345;
db.MyTable.Add(testEntity1);
db.Database.Migrate();
entity1.MyColumn = "New Data 1";
entity1.MyNumber = 2;
db.MyTable2.Add(entity1);
db.Database.Migrate();
Console.WriteLine("Inserting Data...");
Console.WriteLine("Data in the Database");
foreach (var entity in db.MyTable)
{
Console.WriteLine("Id: " + entity.Id);
Console.WriteLine("Column Data: " + entity.MyColumn);
Console.WriteLine("Number: " + entity.MyNumber);
}
foreach (var entity in db.MyTable2)
{
Console.WriteLine("Id: " + entity.Id);
Console.WriteLine("Column Data: " + entity.MyColumn);
Console.WriteLine("Number: " + entity.MyNumber);
}
}
Console.WriteLine("Examples run finished,press Enter to continue...");
Console.ReadLine();
}
I can almost guarantee it's something simple I'm missing but I just can't seem to find it, and there aren't any examples I can find in their documentation. There seems to be a similar issue submitted on GitHub here https://github.com/aspnet/EntityFramework/issues/2874 but that's for multiple contexts. So maybe this is another piece that just hasn't quite made it to release yet?
Solution
By following the tutorial posted on http://ef.readthedocs.org/en/latest/getting-started/uwp.html as suggested by #natemcmaster and the solution recommended by #lukas-kabrt I was able to get it to work as desired. By running the below commands I was able to get the tables created and insert/select data from them.
Install-Package EntityFramework.Commands –Pre
Add-Migration MyFirstMigration
Update-Database
Check out Getting Started on UWP - EF 7 in the official docs. The following notes are from that document.
The default path on UWP is not writable. Your DB file needs to be in ApplicationData.Current.LocalFolder
options.UseSqlite("Data Source=" + Path.Combine(ApplicationData.Current.LocalFolder.Path, "blogging.db"))
Also, take note that on UWP you cannot run migrations from commands. You need to run them in the app.
using (var db = new BloggingContext())
{
db.Database.Migrate();
}
The DbContext class contains configuration of your database - tables, relationsships, etc. To use the DbContext you need to create a database that matches your DbContext. In the Code-first world, it is done by database migrations.
For ASP.NET 5 you need to add this configuration to your project.json file
"commands": {
"ef": "EntityFramework.Commands"
}
and then add a migration to your project and apply this migration to the DB by running following commands
dnx ef migrations add MyFirstMigration
dnx ef database update
For Full .NET you need to run following commands in the Package Manager Console (Tools ‣ NuGet Package Manager ‣ Package Manager Console)
Install-Package EntityFramework.Commands –Pre
Add-Migration MyFirstMigration
Update-Database

Categories

Resources