I'm testing out a bulk import using EF Core and trying to save asynchronously.
I'm trying to add 100 entities at a time and then asynchronously save and repeat until they are all saved. I occasionally get a PK error because it tries to add two entities with the same id to the database. None of the entities being added have an id set, the ids are sequences that are auto generated.
The code:
public async Task<bool> BulkAddAsync(IEnumerable<VehicleCatalogModel> models)
{
_dbContext.ChangeTracker.AutoDetectChangesEnabled = false;
try
{
var entities = models.Select(ToEntity);
_dbContext.Set<VehicleCatalog>().AddRange(entities);
await _dbContext.SaveChangesAsync();
}
catch (Exception ex)
{
_logger.LogError(0, ex, "An Error occurred during the import");
return false;
}
return true;
}
I am calling the method in an xunit test that generates a list of test data and calls the import
var result = manager.BulkAddAsync(modelsToAdd.AsEnumerable());
var counter = 0;
while (!result.IsCompleted && counter < 10)
{
Thread.Sleep(6000);
counter++;
}
Assert.True(result.IsCompleted && result.Result);
It should hold up the execution after 100 entities have been added until they are saved, and then add more but I am still occasionally getting this error. Is there something else I need to add to get this to work correctly? Or a better method of bulk insert?
Related
I have an issue, I want continue to insert data after the exception was raised by SQL Server.
I got an Unique Index on 3 different columns in table to detect duplicates.
For example I am trying to insert 2 rows, the first one is an duplicate, the second one is not.
When the duplicate is detected it goes in the catch, then I'm doing nothing, but when it comes on the second row which is not an duplicate, an exception is raised again for the previous row.
This is my code:
public async Task<IEnumerable<Result>> Handle(NewResultCommandDTO requests, CancellationToken cancellationToken)
{
var results = new List<Result>();
...
for (var j = 0; j < resultDetails.Count(); j++)
{
var rd = resultDetails.ElementAt(j);
var newResult1 = new Result
{
AthleteFEIID = rd.AthleteFEIID,
CompetitionCode = competition.CompetitionCode,
HorseId = horse.Id,
};
results.Add(newResult1);
try
{
await _resultsService.AddResultAsync(newResult1);
await _resultsService.CompleteAsync();
}
catch (Exception ex)
{
var x = ex;
}
}
}
public async Task AddResultAsync(Result result)
{
Context.Results.AddAsync(result);
}
public async Task CompleteAsync()
{
await Context.SaveChangesAsync().ConfigureAwait(false);
}
Thank you for your help !
await _resultsService.CompleteAsync(); is throwing the sql exception statement.
await _resultsService.AddResultAsync(newResult1); statement is already adding the entity in the db context. Even if the next statement throws the exception and it goes to catch block, the duplicated entity is still added in the context. So, when you are adding the next entity in the context and trying to save it, it is throwing exception because of the previous duplicated entity which is not removed from the context.
One solution is to remove the duplicated entity from the context when it goes to catch block.
try
{
await _resultsService.AddResultAsync(newResult1);
await _resultsService.CompleteAsync();
}
catch (Exception ex) {
var x = ex;
_resultsService.RemoveResult(newResult1);
}
public void RemoveResult(Result result)
{
Context.Results.Remove(result);
}
Another solution is to check if the duplication already exists in the table before adding it. For that you will have to write a get method using the unique indexed columns.
What I know is EF creates transaction for DbContext.SaveChanges.
But I need a block of operations including inserts and I need identity results from them to complete my sequence.
So what I do looks like this:
using var dbTransaction = context.DataBase.BeginTransaction();
try {
context.Add(myNewEntity);
context.SaveChanges();
otherEntity.RefId = myNewEntity.Id;
context.Update(otherEntity);
// some other inserts and updates
context.SaveChanges();
dbTransaction.Commit();
}
catch {
dbTransaction.Rollback();
throw;
}
So I call SaveChanges on inserts to get identities and not to break relations.
It looks like transactions in transactions. Is it correct? Is it how it should be done? I mean - Commit doesn't require SaveChanges? I assume it just saves the changes, but I want to be sure.
Your code will be working properly, but I prefer to do it this way:
try {
context.Add(myNewEntity);
var result= context.SaveChanges();
if(result==0){
dbTransaction.Rollback();
... return error
}
otherEntity.RefId = myNewEntity.Id;
context.Update(otherEntity);
// some other inserts and updates
result=context.SaveChanges();
if(result==0){
dbTransaction.Rollback();
... return error
}
dbTransaction.Commit();
}
catch {
dbTransaction.Rollback();
throw;
}
It is very usefull if for example you update or add or delete several records.
In this case the result will return the number of effected records and instead of result==0 I usually use if result < ...effected records I expect.
I want to use System.Transactions and update multiple rows. My database is connected using Entity Framework.
Below is the code I tried but it throws an error :
public void Update(List<PortfolioCompanyLinkModel> record)
{
var transaction = _context.Database.BeginTransaction();
try
{
foreach (var item in record)
{
var portfolioCompanyLink = _context.PortfolioCompanyLink.FirstOrDefault(p => p.Id == item.Id);
portfolioCompanyLink.ModifiedBy = _loggedInUser;
portfolioCompanyLink.ModifiedOn = DateTime.UtcNow;
portfolioCompanyLink.URL = item.URL;
_context.SaveChanges();
//_context.PortfolioCompanyLink.Update(portfolioCompanyLink);
}
transaction.Commit();
}
catch(Exception ex)
{
transaction.Rollback();
}
}
Error:
The configured execution strategy 'SqlServerRetryingExecutionStrategy' does not support user initiated transactions. Use the execution strategy returned by 'DbContext.Database.CreateExecutionStrategy()' to execute all the operations in the transaction as a retriable unit.
Can someone help me on how to proceed with this?
You problem is the SqlServerRetryingExecutionStrategy as described in Microsoft documentation
When not using a retrying execution strategy you can wrap multiple operations in a single transaction. For example, the following code wraps two SaveChanges calls in a single transaction. If any part of either operation fails then none of the changes are applied.
MS docs on resiliency
System.InvalidOperationException: The configured execution strategy 'SqlServerRetryingExecutionStrategy' does not support user initiated transactions. Use the execution strategy returned by 'DbContext.Database.CreateExecutionStrategy()' to execute all the operations in the transaction as a retriable unit.
Solution: Manually Call Execution Strategy
var executionStrategy = _context.db.CreateExecutionStrategy();
executionStrategy.Execute(
() =>
{
// execute your logic here
using(var transaction = _context.Database.BeginTransaction())
{
try
{
foreach (var item in record)
{
var portfolioCompanyLink = _context.PortfolioCompanyLink.FirstOrDefault(p => p.Id == item.Id);
portfolioCompanyLink.ModifiedBy = _loggedInUser;
portfolioCompanyLink.ModifiedOn = DateTime.UtcNow;
portfolioCompanyLink.URL = item.URL;
_context.SaveChanges();
//_context.PortfolioCompanyLink.Update(portfolioCompanyLink);
}
transaction.Commit();
}
catch(Exception ex) {
transaction.Rollback();
}
}
});
You can set the strategy globally too, but that depends on what you are trying to achieve.
I'm trying to insert some potentially poor quality data in to the system and I need a "per row" report of what happened so i've ben trying to do this ...
public async Task<IEnumerable<Result<Invoice>>> AddAllAsync(IEnumerable<Invoice> invoices, Guid bucketId)
{
var results = new List<Result<Invoice>>();
log.Debug(invoices.ToJson());
foreach (var invoice in invoices)
{
try
{
results.Add(new Result<Invoice> { Success = true, Item = await AddAsync(invoice, bucketId), Message = "Imported Successfullly" });
await SaveChangesAsync();
}
catch (Exception ex)
{
results.Add(new Result<Invoice> { Message = ex.Message, Item = invoice });
}
}
return results;
}
My problem is that after a single add call fails that attempted add is left in the change tracker so calling add again with a different item raises the exception again for the first item.
Is there a way to (without rebuilding the context) do "batch inserts" and getting details on a per row / entity level of all the issues not just the first?
I Trying to insert the data into Database By using entity framework but it throwing error following
An error occurred while updating the entries. See the inner exception for details.
Code is Here
public int InsertUserData(UserDetail userDetail, BusinessObjects objects)
{
try
{
UserCredential objCredentials = newPersonEntity.UserCredentials
.First(cd => cd.UserName == objects.UserName);
objCredentials.Status = objects.Status;
newPersonEntity.UserDetails.Add(userDetail);
int result=newPersonEntity.SaveChanges();
return result;
}
catch (Exception ex)
{
CatchError(ex);
return 3;
}
}
Can Any One Tell what mistake i did ?
As i undestand your code snipped correctly, you don't need to perform operations with objCredentials. Or if you want to change its status, you must call
newPersonEntity.UpdateObject(objCredentials);
And the only reason, why you cannot save changes is incorrect userDetail object fields values. Check that all required fields filled correclty and don't conflict with existing key values, stored in dataBase