Develop Reference

Entity Framework query throws 'async error' after many requests

In my project using .NET framework 4.6.1, EF 6.1.4 and IdentityServer3, I set the following DbContext:
public class ValueContext : DbContext
{
public IValueContext(bool lazyLoadingEnabled = false) : base("MyConnectionString")
{
Database.SetInitializer<IValueContext>(null);
Configuration.LazyLoadingEnabled = lazyLoadingEnabled;
}
public DbSet<NetworkUser> NetworkUser { get; set; }
public DbSet<User> User { get; set; }
[...]
And my Entity model User:
[Table("shared.tb_usuarios")]
public class NetworkUser
{
[Column("id")]
[Key()]
public int Id { get; set; }
[Required]
[StringLength(255)]
[Column("email")]
public string Email { get; set; }
[...]
public virtual Office Office { get; set; }
[...]
So far I think its all good.
Then I set this following query in my UserRepository (using DI)
protected readonly ValueContext Db;
public RepositoryBase(ValueContext db)
{
Db = db;
}
public async Task<ImobUser> GetUser(string email)
{
//sometimes I get some error here
return await Db.User.AsNoTracking()
.Include(im => im.Office)
.Include(off => off.Office.Agency)
.Where(u => u.Email == email &&
u.Office.Agency.Active)
.FirstOrDefaultAsync();
}
And everything runs well, until it starts to get many sequential requests, then I begin to get these type of errors, randomly in any function that uses my ValueContext as data source:
System.NotSupportedException: 'A second operation started on this context before a previous asynchronous operation completed. Use 'await' to ensure that any asynchronous operations have completed before calling another method on this context. Any instance members are not guaranteed to be thread safe.'
This is my last hope, as I tried a bunch of different things. Some of them work, and some dont, like:
Convert dbContext to use DI: no difference.
Use context lifetime to run the queries: works, but isnt the solution I want.
Remove asyncronous from requests: works, but also I feel is not the correct way to do.
What Im doing wrong?
EDIT 1
This is how I set up DI in Startup.cs:
private void AddAuth()
{
Builder.Map("/identity", app =>
{
var factory = new IdentityServerServiceFactory()
{
//here I implemented the IdentityServer services to work
ClientStore = new Registration<IClientStore>(typeof(ClientStore)),
[...]
};
AddDependencyInjector(factory);
}
[...]
}
private void AddDependencyInjector(IdentityServerServiceFactory factory)
{
//here I inject all the services I need, as my DbContext
factory.Register(new Registration<ValueContext>(typeof(ValueContext)));
[...]
}
And this is how my UserService is working:
public class UserService : IUserService
{
[Service injection goes here]
//this is a identityServer method using my dbContext implementation on UserRepository
public async Task AuthenticateLocalAsync(LocalAuthenticationContext context)
{
SystemType clientId;
Enum.TryParse(context.SignInMessage.ClientId, true, out clientId);
switch (clientId)
{
case 2:
result = await _userService.GetUser(context.UserName);
break;
case 3:
//also using async/await correctly
result = await _userService.Authenticate(context.UserName, context.Password);
break;
default:
result = false;
break;
}
if (result)
context.AuthenticateResult = new AuthenticateResult(context.UserName, context.UserName);
}

Update - After code posted
When using ASP.Net DI and IdentityServer DI together, we have to be careful to make sure that both the IdentityServer and the underlying DbContext are scoped to the OWIN request context, we do that by Injecting the DbContext into the IdentityServer context. this answer has some useful background: https://stackoverflow.com/a/42586456/1690217
I suspect all you need to do is resolve the DbContext, instead of explicitly instantiating it:
private void AddDependencyInjector(IdentityServerServiceFactory factory)
{
//here I inject all the services I need, as my DbContext
factory.Register(new Registration<ValueContext>(resolver => new ValueContext()));
[...]
}
Supporting dicussion, largely irrelevant now...
With EF it is important to make sure that there are no concurrent queries against the same DbContext instance at the same time. Even though you have specified AsNoTracking() for this endpoint there is no indication that this endpoint is actually the culprit. The reason for synchronicity is so that the context can manage the original state, there are many internals that are simply not designed for multiple concurrent queries, including the way the database connection and transactions are managed.
(under the hood the DbContext will pool and re-use connections to the database if they are available, but ADO.Net does this for us, it happens at a lower level and so is NOT an argument for maintaining a singleton DbContext)
As a safety precaution, the context will actively block any attempts to re-query while an existing query is still pending.
EF implements the Unit-Of-Work pattern, you are only expected to maintain the same context for the current operation and should dispose of it when you are done. It can be perfectly acceptable to instantiate a DbContext scoped for a single method, you could instantiate multiple contexts if you so need them.
There is some anecdotal advice floating around the web based on previous versions of EF that suggest there is a heavy initialization sequence when you create the context and so they encourage the singleton use of the EF context. This advice worked in non-async environments like WinForms apps, but it was never good advice for entity framework.
When using EF in a HTTP based service architecture, the correct pattern is to create a new context for each HTTP request and not try to maintain the context or state between requests. You can manually do this in each method if you want to, however DI can help to minimise the plumbing code, just make sure that the HTTP request gets a new instance, and not a shared or recycled one.
Because most client-side programming can create multiple concurrent HTTP requests (this of a web site, how many concurrent requests might go to the same server for a single page load) it is a frivolous exercise to synchronise the incoming requests, or introduce a blocking pattern to ensure that the requests to the DbContext are synchronous or queued.
The overheads to creating a new context instance are expected to be minimal and the DbContext is expected to be used in this way especially for HTTP service implementations, so don't try to fight the EF runtime, work with it.
Repositories and EF
When you are using a repository pattern over the top of EF... (IMO an antipattern itself) it is important that each new instance of the repository gets its own unique instance of the DbContext. Your repo should function the same if you instead created the DbContext instance from scratch inside the Repo init logic. The only reason to pass in the context is to have DI or another common routine to pre-create the DbContext instance for you.
Once the DbContext instance is passed into the Repo, we lose the ability to maintain synchronicity of the queries against it, this is an easy pain point that should be avoided.
No amount of await or using synchronous methods on the DbContext will help you if multiple repos are trying to service requests at the same time against the same DbContext.

Unit testing with Effort and SQL CE in parallel fails

I'm evaluating unit tests using EF6 in combination with
http://effort.codeplex.com/ and
SQL CE Server (Local DB file)
http://www.codeproject.com/Articles/460175/Two-strategies-for-testing-Entity-Framework-Effort was a quite good reference but now I'm stuck.
I have 2 test projects (one for Effort and the other for SQL CE). If I'm running both separately everthing's fine. Running both in a row with the ReSharper test runner the last test project always fails. Either
System.InvalidOperationException : The Entity Framework was already
using a DbConfiguration instance before an attempt was made to add an
'Loaded' event handler. 'Loaded' event handlers can only be added as
part of application start up before the Entity Framework is used.
or
System.InvalidOperationException: The default DbConfiguration instance
was used by the Entity Framework before an attempt was made to set an
instance of 'SqlCeConfiguration'.The 'SqlCeConfiguration' instance must
be set at application start before using any Entity Framework features
or must be registered in the application's config file.
It's always the same. The successor inherits the DbConfiguration instance from the predecessor. How can I run both test projects / configuration without side effects?
Here's my DbContext class:
public class DataContext : DbContext
{
public DataContext(string connectionString) : base(connectionString)
{ Configuration.LazyLoadingEnabled = false; }
public DataContext(DbConnection connection) : base(connection, true)
{ Configuration.LazyLoadingEnabled = false; }
}
That's the test fixture with Effort:
[TestFixtureSetUp]
public void TestFixtureSetup()
{
EffortProviderConfiguration.RegisterProvider();
var connection = DbConnectionFactory.CreateTransient();
var dbContext = new DataContext(connection);
...
}
That's the test fixture with SQL CE:
[TestFixtureSetUp]
public void TestFixtureSetup()
{
const string filePath = #"LocalDb.sdf";
var connectionString = string.Format("Data Source={0}; Persist Security Info=False;", filePath);
DbConfiguration.SetConfiguration(new SqlCeConfiguration());
var dbContext = new DataContext(connectionString);
dbContext.Database.Create();
...
}
and the my SqlCeConfiguration:
public class SqlCeConfiguration : DbConfiguration
{
public SqlCeConfiguration()
{
SetProviderServices(SqlCeProviderServices.ProviderInvariantName, SqlCeProviderServices.Instance);
SetDefaultConnectionFactory(new SqlCeConnectionFactory("System.Data.SqlServerCe.4.0"));
}
}
Thank you so much!
Marcel

We were seeing the same errors when running all UnitTests on our build server or when running all UnitTests locally.
System.InvalidOperationException : The Entity Framework was already
using a DbConfiguration instance before an attempt was made to add an
'Loaded' event handler. 'Loaded' event handlers can only be added as
part of application start up before the Entity Framework is used.
Once we moved the Effort Provider registration code from the [TestInitialize] method to the AssemblyInitialize method, everything started working. Based on the message in the reported error, it appears that the registration cannot happen more than once.
[AssemblyInitialize()]
public static void AssemblyInit(TestContext context)
{
Effort.Provider.EffortProviderConfiguration.RegisterProvider();
}

I came across this just now and thought I'd share the cause of my issue.
Everything was working dandy, until I implemented and ActionFilter.
It turned out that in my test the Effort code was running after the instantiation of my web site. Instantiating the web site instantiated my filter, my filter requested a DataContext from the container.
Hence someone had already made use of the datacontext before I tried to configure it with Effort.
Hope this might help someone in the future, albeit it being a bit of a different cause, and I hope you solved your issue int the end!

How to force Entity Framework to always get updated data from the database?

I am using EntityFramework.Extended library to perform batch updates. The only problem is EF does not keep track of the batch updates performed by the library. So when I query the DbContext again it does not return the updated entities.
I found that using AsNoTracking() method while querying disables the tracking and gets fresh data from the database. However, since EF does not keep track of the entities queried with AsNoTracking(), I am not able to perform any update on the queried data.
Is there any way to force EF to get the latest data while tracking changes?

Please try this to refresh a single entity:
Context.Entry<T>(entity).Reload()
Edit:
To get fresh data for a collection of entities is worth trying to dispose the DbContext instance after each request.

I stumbled upon this question while searching for a solution to a problem I was having where the navigation properties were not populating after updating the entity. Whenever I attempted to reload the entity from the database, it would grab the entry from the local store instead which would not populate the navigation properties via lazy loading. Instead of destroying the context and recreating one, I found this allowed me to get fresh data with the proxies working:
_db.Entry(entity).State = EntityState.Detached;
The logic behind it was - my update attached the entity so it would track changes to it. This adds it to the local store. Thereafter, any attempts to retrieve the entity with functional proxies would result in it grabbing the local one instead of going out to the db and returning a fresh, proxy-enabled entity. I tried the reload option above, which does refresh the object from the database, but that doesn't give you the proxied object with lazy-loading. I tried doing a Find(id), Where(t => t.Id = id), First(t => t.Id = id). Finally, I checked the available states that were provided and saw there was a "Detached" state. Eureka! Hope this helps someone.

I declared the entity variable, without assignment, as part of the class. This allowed me to dispose of an instance without losing the variable for reference by other methods. I just came across this so it doesn't have alot of runtime under it's belt, but so far it seems to be working fine.
public partial class frmMyForm
{
private My_Entities entity;
public frmMyForm()
{
InitializeComponent();
}
private void SomeControl_Click(object sender, EventArgs e)
{
db.SaveChanges();
db.Dispose();
entity = new My_Entities();
//more code using entity ...
}

Stumbled onto this problem. My app wasn't returning fresh data from the database.
These seems to be 3 solutions:
Reload on select: first you select the object, then reload. Loading it twice if it's not cached?
Detach after use: if you forget to detach an object after use, it's going to cause bugs in completely separate parts of the application that are going to be extremely hard to track down.
Disposing the DbContext after use. Definitely seems like the way to go.
I was creating my DbContext instance in the Repository class. If the DbContext is declared at the Repository level, then I have no control over how it gets disposed. That's a no-no. If I create a new DbContext on every call, then I cannot call Select, modify data, and then call Update.
Seems like something is fundamentally missing in my Repository pattern.
After some research on fundamental Repository pattern, I found the solution: Unit of Work pattern alongside the Repository pattern.
This is an excellent article on the Unit of Work pattern
Or this article from Microsoft. What I currently have is the Repository further up in the page, and what's missing is the section "Implement a Generic Repository and a Unit of Work Class"
Basically, instead of injecting repositories into your services, you access all repositories via a UnitOfWork that you inject into your service. It will solve many problems.
public class UnitOfWork : IUnitOfWork
{
private readonly ApplicationContext _context;
public UnitOfWork(ApplicationContext context)
{
_context = context;
Developers = new DeveloperRepository(_context);
Projects = new ProjectRepository(_context);
}
public IDeveloperRepository Developers { get; private set; }
public IProjectRepository Projects { get; private set; }
public int Complete()
{
return _context.SaveChanges();
}
public void Dispose()
{
_context.Dispose();
}
}
Remains the question: how to create the IUnitOfWork instance?
If I create it in the class constructor to be injected just like the repository, then it gets created and destroyed exactly the same way and we're back to the same problem. In ASP.NET and MVC, class instances are short-lived so injecting in the constructor may be fine, but in Blazor and desktop apps, class instances are much more long-lived and it's more of a problem.
This article from Microsoft clearly states that Dependency Injection isn't suitable to manage the lifetime of DbContext in Blazor:
In Blazor Server apps, scoped service registrations can be problematic
because the instance is shared across components within the user's
circuit. DbContext isn't thread safe and isn't designed for concurrent
use. The existing lifetimes are inappropriate for these reasons:
Singleton shares state across all users of the app and leads to
inappropriate concurrent use.
Scoped (the default) poses a similar
issue between components for the same user.
Transient results in a new
instance per request; but as components can be long-lived, this
results in a longer-lived context than may be intended.
They suggest using the Factory pattern, which can be implemented like this
/// <summary>
/// Creates instances of UnitOfWork. Repositories and UnitOfWork are not automatically injected through dependency injection,
/// and this class is the only one injected into classes to give access to the rest.
/// </summary>
public class UnitOfWorkFactory : IUnitOfWorkFactory
{
private readonly IDateTimeService _dateService;
private readonly DbContextOptions<PaymentsContext> _options;
public UnitOfWorkFactory(IDateTimeService dateService, DbContextOptions<PaymentsContext> options)
{
_dateService = dateService;
_options = options;
}
/// <summary>
/// Creates a new Unit of Work, which can be viewed as a transaction. It provides access to all data repositories.
/// </summary>
/// <returns>The new Unit of Work.</returns>
public IUnitOfWork Create() => new UnitOfWork(CreateContext(), _dateService);
/// <summary>
/// Creates a new DbContext.
/// </summary>
/// <returns>The new DbContext.</returns>
public PaymentsContext CreateContext() => new(_options);
}
Neither IWorkOfUnit nor any repository will be registered into the IoC container. Only IWorkOfUnitFactory.
And finally... how to share a transaction between various services?
I have a SetStatus method that updates the status field in the database. How is this method supposed to know whether it's a stand-alone operation or part of a larger transaction?
Since class-level dependency injection isn't suitable to manage and share the Work of Unit, then the only option is to pass it as parameters to the methods that need it.
I add an optional IUnitOfWork? workScope = null parameter to every method that needs it, and call Save only if this parameter is null. Here's an implementation.
public virtual async Task<TempOrder?> SetStatusAsync(int orderId, PaymentStatus status, IUnitOfWork? workScope = null)
{
using var unitOfWork = _workFactory.Create();
var work = workScope ?? unitOfWork;
var order = await work.Orders.GetByIdAsync(orderId);
if (order != null)
{
order.Status = status;
work.Orders.Update(order); // DateModified gets set here
if (workScope == null)
{
await work.SaveAsync();
}
}
return order;
}
Another option is to have IUnitOfWorkFactory.Create take the workScope parameter, and when set:
Re-use the existing DbContext
Do not dispose
IUnitOfWork.Save won't submit
My final implementation can be used like this
public virtual async Task<TempOrder?> SetStatusAsync(int orderId, PaymentStatus status, IUnitOfWork? workScope = null)
{
using var unitOfWork = _workFactory.Create(workScope);
var order = await unitOfWork.Orders.GetByIdAsync(orderId);
if (order != null)
{
order.Status = status;
work.Orders.Update(order); // DateModified gets set here
await unitOfWork.SaveAsync(); // Ignored if workScope != null
}
return order;
}
Pheww! That bug was a rabbit hole. It's a pretty long solution but should solve it for good with a solid architecture.

Making the code run on the same context will not yield you updated entities. It will only append new entities created in the database between runs. EF force reload can be done like this:
ObjectQuery _query = Entity.MyEntity;
_query.MergeOption = MergeOption.OverwriteChanges;
var myEntity = _query.Where(x => x.Id > 0).ToList();

For me ...
I access my DbContext like this:
_viewModel.Repo.Context
To force EF to hit the database I do this:
_viewModel.Repo.Context = new NewDispatchContext();
Overwriting the current DbContext with a new instance. Then the next time I use my data services they get the data from the database.

Reloading specific entities was not an option for me because I didn't know the exact entity. I also did not want to create a new DbContext as it is injected by DI. So I resorted to the following trick to "reset" the whole context.
foreach (var entry in db.ChangeTracker.Entries())
{
entry.State = EntityState.Detached;
}

Managing DBContext state behind a data provider interface

I have an interface IDataProvider which exposes (for sake of discussion) just 3 operations:
public interface IDataProvider
{
// get a list of projects (just metadata)
List<Project> ListProjects();
// load the Project by its GUID which we got from the metadata.
Project LoadProject(Guid id);
// save the project. underlying provider should determine to insert or update accordingly.
void SaveProject(Project data);
}
I am using DBContext accessing an SQL CE as the under lying data access layer data provider and I could implement:
public DataProvider : SqlCeDbContext, IDataProvider { ... }
or
public DataProvider : IDataProvider
{
List<Project> ListProjects()
{
using(var ctx = new SqlCeContext())
{
//... manage the life of the context for the API user.
}
}
// ...
}
or
public DataProvider : IDataProvider
{
SqlCeContext _mSqlCeContext = new SqlCeContext();
List<Project> ListProjects() { .. }
// ...
}
The three implementations will of course behave very differently with respect to connection and entity states. Since the interface "rules" does not enforce rules on this, which implementation is better? Or in case we should enforce one or the other, can it be done?

Assuming that this is on a mobile device (which I believe is the case due to the SqlCE references), I think that you may be making your life slightly more complicated than needed.
There is no reason that you can't open a connection at application startup and leave it open through the life of the application since there shouldn't be any other applications using the database.
We've had production WinCE apps for years that use this approach and have never had an issue with it.

How can I use Sql CE 4 databases for functional tests

Due to the potential differences between Linq-to-Entities (EF4) and Linq-to-Objects, I need to use an actual database to make sure my query classes retrieve data from EF correctly. Sql CE 4 seems to be the perfect tool for this however I have run into a few hiccups. These tests are using MsTest.
The problem I have is if the database doesn't get recreated (due to model changes), data keeps getting added to the database after each test with nothing getting rid of the data. This can potentially cause conflicts in tests, with more data being returned by queries than intended.
My first idea was to initialize a TransactionScope in the TestInitialize method, and dispose the transaction in TestCleanup. Unfortunately, Sql CE4 does not support transactions.
My next idea was to delete the database in TestCleanup via a File.Delete() call. Unfortunately, this seems to not work after the first test is run, as the first test's TestCleanup seems to delete the database, but every test after the first does not seem to re-create the database, and thus it gives an error that the database file is not found.
I attempted to change TestInitialize and TestCleanup tags to ClassInitialize and ClassCleanup for my testing class, but that errored with a NullReferenceException due to the test running prior to ClassInitialize (or so it appears. ClassInitialize is in the base class so maybe that's causing it).
I have run out of ways to effectively use Sql CE4 for testing. Does anyone have any better ideas?
Edit: I ended up figuring out a solution. In my EF unit test base class I initiate a new instance of my data context and then call context.Database.Delete() and context.Database.Create(). The unit tests run a tad slower, but now I can unit test effectively using a real database
Final Edit: After some emails back and forth with Microsoft, it turns out that TransactionScopes are now allowed in SqlCE with the latest release of SqlCE. However, if you are using EF4 there are some limitations in that you must explicitly open the database connection prior to starting the transaction. The following code shows a sample on how to successfully use Sql CE for unit/functional testing:
[TestMethod]
public void My_SqlCeScenario ()
{
using (var context = new MySQLCeModelContext()) //ß derived from DbContext
{
ObjectContext objctx = ((IObjectContextAdapter)context).ObjectContext;
objctx.Connection.Open(); //ß Open your connection explicitly
using (TransactionScope tx = new TransactionScope())
{
var product = new Product() { Name = "Vegemite" };
context.Products.Add(product);
context.SaveChanges();
}
objctx.Connection.Close(); //ß close it when done!
}
}

In your TestInitialize you should do the following:
System.Data.Entity.Database.DbDatabase.SetInitializer<YourEntityFrameworkClass>(
new System.Data.Entity.Database.DropCreateDatabaseAlways<YourEntityFrameworkClass>());
This will cause entity framework to always recreate the database whenever the test is run.
Incidentally you can create an alternative class that inherits from DropCreateDatabaseAlways. This will allow you to seed your database with set data each time.
public class DataContextInitializer : DropCreateDatabaseAlways<YourEntityFrameworkClass> {
protected override void Seed(DataContext context) {
context.Users.Add(new User() { Name = "Test User 1", Email = "test#test.com" });
context.SaveChanges();
}
}
Then in your Initialize you would change the call to:
System.Data.Entity.Database.DbDatabase.SetInitializer<YourEntityFrameworkClass>(
new DataContextInitializer());

I found the approach in the "final edit" works for me as well. However, it's REALLY annoying. It's not just for testing, but any time you want to use TransactionScope with Entity Framework and SQL CE. I want to code once and have my app support both SQL Server and SQL CE, but anywhere I use transactions I have to do this. Surely the Entity Framework team should have handled this for us!
In the meantime, I took it one step farther to make it a little cleaner in my code. Add this block to your data context (whatever class you derive from DbContext):
public MyDataContext()
{
this.Connection.Open();
}
protected override void Dispose(bool disposing)
{
if (this.Connection.State == ConnectionState.Open)
this.Connection.Close();
base.Dispose(disposing);
}
private DbConnection Connection
{
get
{
var objectContextAdapter = (IObjectContextAdapter) this;
return objectContextAdapter.ObjectContext.Connection;
}
}
This makes it a lot cleaner when you actually use it:
using (var db = new MyDataContext())
{
using (var ts = new TransactionScope())
{
// whatever you need to do
db.SaveChanges();
ts.Complete();
}
}
Although I suppose that if you design your app such that all changes are committed in a single call to SaveChanges(), then the implicit transaction would be good enough. For the testing scenario, we want to roll everything back instead of calling ts.Complete(), so it's certainly required there. I'm sure there are other scenarios where we need the transaction scope available. It's a shame it isn't supported directly by EF/SQLCE.