Develop Reference

Getting DbUpdateConcurrencyException, but only in production. Cannot reproduce in dev

I'm working on a Quartz.NET hosted job as part of a Blazor Server web application. This job downloads a list of products from a warehouse, and caches them in a local DbContext. In order to optimize performance due to the large number of entries, I'm inserting the new items in batches of 1000 (with DbSet.AddRange()), and after each batch I submit the changes with database.SaveChangesAsync().
The following code is a simplified version of this process:
public async Task Execute(IJobExecutionContext context)
{
// databaseFactory is injected as a dependency (IDbContextFactory<AppDatabase>)
await using var database = await databaseFactory.CreateDbContextAsync();
int page = 0;
while(true)
{
List<WarehouseEntry> dbItems = warehouse.QueryPage(page, 1000); // Retrieves 1000 entries
if(dbItems.Count == 0)
break;
database.ProductWarehouseCache.AddRange(dbItems.Select(p => new CachedWarehouseProduct {
Sku = p.Name,
Barcode = p.Barcode,
Stock = p.Stock,
Price = p.Price
}));
await database.SaveChangesAsync(); // <---- DbUpdateConcurrencyException here
page++;
}
}
Note that there is absolutely no concurrency on the code above. The IJob class has the [DisallowConcurrentExecution] attribute meaning that even if I accidentally trigger this procedure multiple times simultaneously, only one instance will be executing at any given time, so despite the exception message, this is not a concurrency issue. It's also important to note that nothing else is updating/querying the database while this code is running.
This works as intended on my local development machine. However when I tried to deploy the application to a production server for the first time, I've found that this specific part of the code fails with a DbUpdateConcurrencyException. Normally with an exception like this, I would look for concurrency issues, or DbContexts that are used by multiple threads at the same time, or aren't disposed properly. However, as I have explained above, this is not the case here.
The following is the full exception message:
Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException:
The database operation was expected to affect 1 row(s), but actually affected 0 row(s);
data may have been modified or deleted since entities were loaded.
See http://go.microsoft.com/fwlink/?LinkId=527962 for information on
understanding and handling optimistic concurrency exceptions.
What could be causing an exception like this, when there is no concurrency whatsoever? And what could cause this to only happen on the production server, but never in the development workspace.
Additional information:
dotnet 6
EF Core 6.0.6
Local/Dev Database: MySQL 8.0.31
Local/Dev OS: Windows 11
Remote/Prod Database+OS: MySQL 8.0.30-0ubuntu0.20.04.2

I have fixed the issue. I was using DataGrip's built-in import/export tools to clone my database's DDL from local dev DB to remote prod DB. Apparently these tools don't replicate the DDL exactly as they should, which leads to EF core throwing random unexpected errors such as this.
To fix it, I rewrote my deployment pipeline to use the dotnet ef migrations script --idempotent script to generate an .sql file that automatically applies any missing migrations to the production database. By using this dotnet tool, I am no longer getting the exception.

Long running Web API task

I am trying to find the appropriate way to setup a long running api task using Asp.Net Web API. The tasks I am running could take up to 3 minutes to complete. I have found several links such as QueueBackgroundWorkItem or this post here but am unsure on a few things.
Question:
So I guess I am most confused about the overall structure for this and the appropriate way to even go about it. My task seems to long to use a QueueBackgroundWorkItem but I don't want to implement a fire and forget API call.
What I would like:
I would like to be able to fire a web API task, return a OK response, but continue to poll for the returned data after firing the task. I just have no idea how to even begin implementing a queue system or even how to poll for that data from the Web API.
Other Information:
Using .Net Framework 4.5.2
Front end is asp.net MVC web app
Web API method is called through an AJAX call on my front
Currently send about 30 requests at a time with a long time out. Then as they return I update my UI with returned data from the API.
My current code:
This is what I am doing now which works fine. The only issue is I have to set the timeout to some ridiculous amount on my UI application. Which is why I am trying to find the correct way to implement a long running task/polling/queueing mechanism for long running tasks. Overall just unsure what is out there for solving the problem below.
public NewProjectLogDTO CreateProject(string modelNumber, string orderName = "")
{
try
{
string EplanPort = ConnectToEPLAN();
SetContext(modelNumber, orderName);
AddSchematicToDB(modelNumber, orderName); //accesses DB
ExecuteCreateProjectAction(); //Really long running task about 3 minutes long
DecreasePortCounter(EplanPort);
AddMacrosToDb(modelNumber, orderName); //accesses DB
return GetNewProjectLogFromContext(modelNumber,orderName); //return data from long running task
}
catch (Exception ex)
{
}
}

Creating new AppDomain calling method in same class as the AppDomain in made

I want to start below potentially long running thread in it's own AppDomain to prevent the webserver from aborting it during recycling. It compiles fine, however during runtime I get this cryptic error
Type is not resolved for member 'MyCore.MyWebService,MyCore,
Version=5.0.0.0, Culture=neutral, PublicKeyToken=null'.
How do I find out what member is not resolved?
Are there any better ways running a long standing thread in a MVC business service layer, that does not get aborted by the server recycling mechanism?
Here is the code:
namespace MyCore
{
[Serializable]
public class MyWebService : IMyWebService
{
AppDomain domain = AppDomain.CreateDomain("Domain");
Thread.CurrentThread.Name = "MVCThread";
domain.SetData("lDatabaseID", lDatabaseID);
domain.DoCallBack(() =>
{
long lID = Convert.ToInt64(AppDomain.CurrentDomain.GetData("lDatabaseID"));
Thread thread = new Thread(
(() =>
{
PopulateTables(lID );
}));
thread.Name = "DomThread";
thread.Start();
});
}
}

IIS is heavily optimised to respond very quickly to hundreds of small simultaneous requests and just isn't the right tool for what you're attempting. You can try to work around that but in the long term you'll be better off building a tool that is designed for long-running tasks. You've then got a pre-packaged solution the next time this problem arises.
The basic idea is to create an external application that does your background processing with some way to pass tasks to it and get results back. I like using the database to communicate as most web applications that need baground processing already use a database. Add a 'tasks' table with {status, startedDateTime, finishedDateTime, parameters, etc}, then write an external application that will periodically look for a new task, complete it and update the database. Your web site can poll the database for status or your application could make an AJAX call to notify the web site when a job has completed (a small iframe in the web site header that shows waiting / completed tasks can be useful if someone will be waiting for the job to complete and is easy to do).
EDIT: Before you do the above review HangFire (which works inside IIS, as a Windows Service or as a console app). Same principles, but a pre-packaged solution. Note that I haven't implemented this yet but it looks good.
Although it's a bit of work to set up, handing this task off to a Windows Service is a good approach if you might have multiple tasks and need them responded to quickly. There are a lot of tutorials on the web that will help you create a Windows Service, such as http://www.codeproject.com/Articles/106742/Creating-a-simple-Windows-Service but you'll have to build a simple task executor on top of that so if that's the way you want to go I'd look for a pre-built task engine (I couldn't find one quickly but I'm probably using the wrong search phrase).
But that's overkill if turn-around time isn't important and a better approach for you might be to create a small console application that will be started every five minutes by task scheduler. It would connect to the database, execute any waiting tasks then shut down again. This is easier to debug and install than a Windows service and achieves the same goal of moving the task execution out of IIS.
Remember that you still have to detect and handle Windows shutdown so that you don't get half-finished orphaned jobs - at the very least just tag that task as aborted and exit cleanly.

Alright after having mucked with Hangfire, I finally got it to work in .Net 4.0 and MVC 3. Had to install Common.Logging.Core 2.2.0, since the NuGet installed the wrong version (3.3.0)
In my Initial controller I added the following
namespace Core.Controllers
{
...
public void Configuration(IAppBuilder app)
{
app.UseHangfire(config =>
{
config.UseSqlServerStorage(ConnectionString.GetTVConnectionString());
config.UseServer();
});
}
...
}
ConnectionString.GetTVConnectionString() gets the connection string from the config file.
Up top I added the following
[assembly: OwinStartup(typeof(Core.Controllers.BaseController))]
In the code that starts the background thread I added the following, passing in a long instead of the class and having the job load the POCO class from the db.
BackgroundJob.Enqueue(() => PopulateTables(lDatabaseID, JobCancellationToken.Null));
The Enqueue() function returns a job id, that later can be used to cancel the job if needed, through the BackgroundJob.Delete(jobid) function.
In the job method I then have this
while (idxMin < max)
{
try
{
cancellationToken.ThrowIfCancellationRequested();
....
}
catch (JobAbortedException jobEx)
{
....
}
}
It's important to use dependency injection, so my class had a parameter less constructor added that re-reads the connection string rather than have it passed in.
public MyWebService ()
: this(ConnectionString.GetTVConnectionString())
{
}
public MyWebService (string sConnStr)
{
msConnStr = sConnStr;
}
After that it seems to run pretty well. A number of tables are added to the database specified in the connection string. So far it seems like the jobs survive recycling on the webserver.

ASP MVC app resets on long running thread

In my ASP MVC 5 app I have this database related operation that I've to perform once in month maybe and it takes about 30 - 60 minutes.
I start this action like this:
Repository dbForCategories = new Repository();
dbForCategories.Database.CommandTimeout = 60000;
var t = Task.Factory.StartNew(async delegate
{
var crs = new ProductCategoryRelationsSetter(dbForCategories, categoryService);
crs.AddProductsCategoriesRelations(oneLineTree);
}, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
After about 5 minutes of working in background Im getting logged out of application. I think that the app resets because some static variables Im using are reset.
In elmah i don't have any errors. I put my code in Try Catch block.
Only hope in you guys:)

As #David mentioned, it's probably best to go the Windows Service route:
Write the item to a database table
Let a Windows Service poll the table every month (or however often you need it to).
If it finds a new item in the queue, let the Windows Service perform the operation.
Why not do Background threads in ASP.NET?
The server may recycle Application pool at some point and will not know to wait for your Task on a separate thread. (Discussed in this SO question)
.NET 4.5.2 onward. You can fire and forget short tasks
For interest sake you can use HostingEnvironment.QueueBackgroundWorkItem (see here) and the server will respect the background item and not recyle the App pool while it's busy, BUT ONLY for up to 90 seconds. anything longer and Windows Service is your best bet.
HostingEnvironment.QueueBackgroundWorkItem(ct => yourAsyncWork(ct));

Hangfire is wonderful for stuff like this.
https://www.nuget.org/packages/Hangfire/

OptimisticConcurrencyException: Multiple EF based applications using shared AppFabric cache and same database

I am using a web application and a windows service on the same machine as Appfabric.
Both applications reuse same DAL code (dll) which is EF (Entity Framework) Code-First based and accessing the same cache in Appfabric. The code in the windows service is implemented as a Job as part of Quartz.Net
The web application has to support multiple requests off course, and the windows service multiple threads( scheduler and events).
For both, the shared DAL dll creates a DbContext object per http session and thread ContextID or just Thread ContextID for the later. The DAL uses the EFCachingProviders from here. Also, my EF solution uses Optimistic concurrency with a timestamp columns and IsRowVersion in the mapping.
As stated here, the benefit of having a 2nd level cache is to have access to a representation of the original state across processes! But that does not seem to work for me, I get 'OptimisticConcurrencyException' in my use case as following:
restart cache cluster, restart windows service, restart iis -> clean slate :)
Using web app (firefox), I insert a new object A with reference to existing object B. I can see the new row in the database. All ok.
Using webapp in another browser (chrome) = new session, i can see the new object.
Next, the windows service tries to do some background processing and tries to update object B. This results in an 'OptimisticConcurrencyException'. Apparently the process in the windows service is holding a version of Object B with a dated rowversion.
If i restart the windows service, it tries the same logic again and works with no exception....
So both applications are multithreaded, use same DAL code, connect to same database, and same cache cluster and same cache. I would expect the update and insert to be in the appfabric cache. I would expect the EF context of the windows service to use the newest information. Somehow, it seems, that it's 1st level cache in holding on old information...
or something else is going wrong.
Please advice...
Update
Ok, after digging around, i fixed the Update problem of my windows service. Each Manager object with queries the DAL uses a DbContext bound to its Process ID + Thread ID. So in the Execute function of my Quartz Job, all Managers (of different object types) should share the same DbContext which is created by the first Manager.
The problem was, that after the function finished, the DbContext was not Disposed (which happens automatically in the HTTP Session based DbContext manager). So the next time the Job was executed, the same DbContext was found and used, which by that time was dated already (old first level cache???). The 2nd level cache should not be a problem, because that is shared and SHOULD contain newest objects... if any.
So this part is fixed.
New problem
So the web-app creates a new object A, updates an existing object B, the windows-service now works and is able to update the existing (changed) object B with no problem.
Problem:
When i do a refresh of the webapp, it does not see the changes (by the windows service) of object B....
So if the webapp changed a count to 5, 10 minutes later the windows service change that count to 6 and I open the web-app in same or new window/browser, i still see 5, not 6!
A restart of the webapp (iis) does not help, also an iisreset doesn't.
When i do Restart-CacheCluster.... it works and shows 6....
So it looks like the item is in the cache. The windows service updates it, but does not invalidate the item, which is old and used by the webapp....
Or... although the same object, the webapp has its own entry in the cache and the win-app has its own entry (which does get invalidated)....
Which one?
Solution
I solved this myself. The EF wrapper uses the query string as a key to store items in the cache, it seems. So 2 different queries (does not matter if they originate from 2 different application sharing same distributed cache or same application) referencing the same data in the database will have different keys (different query string) and so different places in the cache. Perhaps its not this black-and-white but something like this...
I don't think internally some way of algorithm is used to check if a query touches existing cached objects.
This causes my problem where my windows service does an update and the webapp still sees the old one from the cache which could only be solved by doing a Restart-CacheCluster command.
So how i fixed this:
My windows Service is a batch job triggered by the Quartz Scheduler. After it is done
I clear the whole cache:
private void InvalidateCache()
{
try
{
DataCache myCache = ...
foreach (String region in myCache.GetSystemRegions())
{
myCache.ClearRegion(region);
}
}
catch (Exception ex)
{
eventLog.WriteEntry("InvalidateCache exception : " + ex.Message);
}
}

I don't have an answer, but I hope the thoughts below might point you into the right direction.
If this is only an issue on updates, I would go for reading a fresh instance of the record on every update from the database, and update that. This would avoid optimistic concurrency errors. Note that the DbContext is not thread safe - I don't know if this would cause the issue, but reading every time new would address it.
If you are having this issue on reads, then you would have to track down where the various caches are and which one is not getting updated and why. I am guessing there are various configuration options for caching at each point of usage. Good luck with that.... :)