Develop Reference

What is a good way to archive data with the identity column using EF Core?

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 hour ago.
Improve this question
I'm using .NET 6 and EF Core 6.0.13. I have two databases Foo and FooArchive with identical schemas. I need to archive (migrate) data that are older than a year from Foo to FooArchive for 7 tables. What's the best way to do this with EF Core? I will describe below what I tried and the issues I'm running into.
NOTE: There are no foreign keys or any relationships defined for any table in both DBs and so no navigation properties, etc.
There are FooContext and FooArchiveContext classes both using the same entity models but different connections and injected into the repository class
Query for the customer ids whose account is older than 365 days on a different transaction
var customerIds = GetCustomerIds();
Loop through CustomerIds collection and archive one customer at a time
foreach(var customerId in customerIds)
{
using(var fooTx = _fooContext.Database.BeginTransaction())
using(var fooArchiveTx = _fooArchiveContext.Database.BeginTransaction())
{
//Series of left joins to get the data from 7 tables
var recordsToArchive = (from cust in _fooContext.Customers
join ord in _fooContext.Orders on ord.CustId equals cust.Id into co
from ord in co.DefaultIfEmpty()
join odt in _fooContext.OrderDetails on odt.OrderId equals ord.id into ordt
.
where cust.id = customerId
select new {
Customer = cust,
Order = ord,
OrderDetail = odt,
.
.
}).ToList();
var customer = recordsToArchive.Select(x => x.Customer).Distinct().First();
var orders = recordsToArchive.Select(x => x.Order).Where(x != null).Distinct();
var orderDetails = recordsToArchive.Select(x => x.OrderDetail).Where(x != null).Distinct();
.
.
// Check if the record to be migrated is already in FooArchiveContext
var existingRecord = _fooArchiveContext.Customers.FirstOrDefault(x => x.Id == customerId);
if(existingRecord == null)
{
_fooArchiveContext.Customers.Add(customer);
_fooArchiveContext.Orders.AddRange(orders);
_fooArchiveContext.OrderDetails.AddRange(orderDetails);
.
}
else
{
_fooArchiveContext.Customers.Update(customer);
_fooArchiveContext.Orders.UpdateRange(orders);
_fooArchiveContext.OrderDetails.UpdateRange(orderDetails);
.
}
_fooArchiveContext.SaveChanges();
//Remove the record from fooContext
_fooContext.OrderDetails.RemoveRange(orderDetails);
_fooContext.Orders.RemoveRange(orders);
_fooContext.Customers.Remove(customer);
.
_fooContext.SaveChanges();
fooArchiveTx.Commit();
fooTx.Commit();
}
}
Is what I'm doing the right approach? I think I may have to use the AutoMapper to copy entities in between two contexts. It works in the InMemory database but fails when I try it against the actual SQL Server instances. I get an error
Cannot insert explicit value for identity column in table 'Orders' when IDENTITY_INSERT is set to OFF
I would like to keep the same Ids as in the original db instance (fooContext).
I guess I can remove the Id in the entity object and save. Then query for the new Id and update the related entities but sounds tackier than the code I already have. I've seen SO answers where EF core is turning identity insert option on and off before and after calling SaveChanges() like below but haven't tried.
db.Users.Add(user);
db.Database.ExecuteSqlRaw("SET IDENTITY_INSERT MyDB.Users ON");
db.SaveChanges();
db.Database.ExecuteSqlRaw("SET IDENTITY_INSERT MyDB.Users OFF");
transaction.Commit();
Thanks for your help.

If I understand your problem correctly
Your approach of looping through each customer and archiving their records one at a time seems reasonable. However, there are a few areas where you can improve your implementation.
Firstly, you should avoid querying the database multiple times for the same data. In your code, you are querying the same data multiple times to get the customers, orders, and order details. This can be improved by using the Include method to eagerly load the related entities along with the primary entity.
Secondly, you should avoid duplicating code. In your code, you have duplicate code for adding and updating the entities in the archive database. You can reduce the duplication by using the Attach method to attach the entities to the context and then calling Update or Add depending on whether the entity is already in the context or not.
Thirdly, you should use a bulk insert/update operation instead of adding/updating the entities one at a time. EF Core does not have built-in support for bulk operations, but you can use third-party libraries like Entity Framework Extensions or Z.EntityFramework.Plus to perform bulk operations.
Finally, you should avoid setting the identity column values explicitly. Instead, let the database generate the identity values for you. To do this, you can remove the identity column from your entity models or use the ValueGeneratedOnAdd() method in your entity configuration.
With these improvements in mind, here's an example implementation of your code:
using (var fooTx = _fooContext.Database.BeginTransaction())
using (var fooArchiveTx = _fooArchiveContext.Database.BeginTransaction())
{
var cutoffDate = DateTime.UtcNow.AddYears(-1);
var customerIds = _fooContext.Customers
.Where(c => c.CreatedAt < cutoffDate)
.Select(c => c.Id)
.ToList();
foreach (var customerId in customerIds)
{
var customer = _fooContext.Customers
.Include(c => c.Orders)
.ThenInclude(o => o.OrderDetails)
.FirstOrDefault(c => c.Id == customerId);
if (customer != null)
{
if (_fooArchiveContext.Customers.Any(c => c.Id == customerId))
{
_fooArchiveContext.Attach(customer);
_fooArchiveContext.Update(customer);
}
else
{
_fooArchiveContext.Add(customer);
}
_fooArchiveContext.SaveChanges();
_fooArchiveContext.Orders.BulkInsert(customer.Orders);
_fooArchiveContext.OrderDetails.BulkInsert(customer.Orders.SelectMany(o => o.OrderDetails));
_fooContext.OrderDetails.RemoveRange(customer.Orders.SelectMany(o => o.OrderDetails));
_fooContext.Orders.RemoveRange(customer.Orders);
_fooContext.Customers.Remove(customer);
_fooContext.SaveChanges();
}
}
fooArchiveTx.Commit();
fooTx.Commit();
}
In this code, we first get the list of customer IDs whose accounts are older than a year. We then loop through each customer and retrieve their orders and order details using the Include method. We then check if the customer already exists in the archive database and use the Attach and Update methods to update the existing customer, or the Add method to add a new customer.
We then use the BulkInsert method from the Entity Framework Extensions library to insert the orders and order details in bulk. We also remove the orders, order details, and customer from the source database using the RemoveRange method.
Finally, we call SaveChanges on the archive and source contexts and commit the transactions.

Remove a record in entity framework

I'm working with entity framework for sometimes now and one thing that really irritates me is the fact that to delete an item we have to find the object and then remove it. so if we have the PK of the record we have to get the object and then remove the object.
ex:
Category category = db.Categories.Find(categoryId);
db.Categories.Remove(category);
db.SaveChages();
In this method we are hitting database twice..!!!
is there a way to remove the record with just hitting the database once?
For those none-believers this is the glimpse out come: :)

// no trip to database
var rec = db.TableName.Local.SingleOrDefault(r => r.Id == primaryKey);
if (rec == null) throw NotFoundOrWhateverException();
// still no trip to database, if I remember right
db.TableName.Remove(rec);
// trip to database
db.SaveChanges();

IF you don't want to get the complete object you can try this way using the primary key property of it:
Category category = new Category () { Id = categoryId } ;
db.Categories.Attach(category);
db.DeleteObject(category);
db.Savechanges();
If you are using EF 5 then you can use EntityFramework.Extended Library using NUGETand can do like this:
db.Categories.Delete(c => c.Id == categoryId);

[Answer turned out to be incorrect. Removed content to keep from confusing others. Kept the post for comment thread.]

Request only first item Entity Framework

I'm looking for the most efficiant method to call the first and only the first item from a SQL Server database using Entityframework and linq.
I'm currently using
public static UserProfile GetUserProfile(Guid userID)
{
UserProfile oProfile = null;
using (var context = new MyEntities())
{
var profiles = from c in context.UserProfiles where c.UserID == userID select c;
if(profiles.Any())
{
oProfile = profiles.First();
}
}
return oProfile;
}
This however from what I can tell takes two DB ops to complete, one to check if the record exists and a second to return it. I'm sure there has to be a better pattern / method and I'm just not seeing it.

It's very simple: Use FirstOrDefault().
When there is no entry it will return null (like you have already), else it'll return the first entry. If you don't want your variable to take the null value, use a substitute in between.
edit:
Your code could be equally replaced with the following code, just that this one will just query once against the database.
public static UserProfile GetUserProfile(Guid userID)
{
UserProfile oProfile = null;
using (var context = new MyEntities())
{
oProfile = (from c in context.UserProfiles where c.UserID == userID select c).FirstOrDefault();
}
return oProfile;
}

If only one item should exist (as it looks in this case) then you should consider using Single(). Single will return the first item but throw an exception if more than one item exists - this can help maintain data quality if only a single item should exist.
If only one item should exist but it is optional then use SingleOrDefault which will act as Single but return null if no item exists.

You need to use First() or FirstOrDefault() without calling .Any()
var profile = (from c in context.UserProfiles where c.UserID == userID select c).First();
First() throws an exception if no records returned but FirstOrDefault() returns null in that case
Both will generate SQL like this:
SELECT TOP (1) ...
FROM ...

You should use the Take method:
Entity Framework/Linq to SQL: Skip & Take
It will only require one hit to the database.

What is the recommended practice to update or delete multiple entities in EntityFramework?

In SQL one might sometimes write something like
DELETE FROM table WHERE column IS NULL
or
UPDATE table SET column1=value WHERE column2 IS NULL
or any other criterion that might apply to multiple rows.
As far as I can tell, the best EntityFramework can do is something like
foreach (var entity in db.Table.Where(row => row.Column == null))
db.Table.Remove(entity); // or entity.Column2 = value;
db.SaveChanges();
But of course that will retrieve all the entities, and then run a separate DELETE query for each. Surely that must be much slower if there are many entities that satisfy the criterion.
So, cut a long story short, is there any support in EntityFramework for updating or deleting multiple entities in a single query?

EF doesn't have support for batch updates or deletes but you can simply do:
db.Database.ExecuteSqlCommand("DELETE FROM ...", someParameter);
Edit:
People who really want to stick with LINQ queries sometimes use workaround where they first create select SQL query from LINQ query:
string query = db.Table.Where(row => row.Column == null).ToString();
and after that find the first occurrence of FROM and replace the beginning of the query with DELETE and execute result with ExecuteSqlCommand. The problem with this approach is that it works only in basic scenarios. It will not work with entity splitting or some inheritance mapping where you need to delete two or more records per entity.

Take a look to Entity Framework Extensions (Multiple entity updates). This project allow set operations using lambda expressions. Samples from doc:
this.Container.Devices.Delete(o => o.Id == 1);
this.Container.Devices.Update(
o => new Device() {
LastOrderRequest = DateTime.Now,
Description = o.Description + "teste"
},
o => o.Id == 1);
Digging EFE project source code you can see how automatize #Ladislav Mrnka second approach also adding setting operations:
public override string GetDmlCommand()
{
//Recover Table Name
StringBuilder updateCommand = new StringBuilder();
updateCommand.Append("UPDATE ");
updateCommand.Append(MetadataAccessor.GetTableNameByEdmType(
typeof(T).Name));
updateCommand.Append(" ");
updateCommand.Append(setParser.ParseExpression());
updateCommand.Append(whereParser.ParseExpression());
return updateCommand.ToString();
}
Edited 3 years latter
Take a look to this great answer: https://stackoverflow.com/a/12751429

Entity Framework Extended Library helps to do this.
Delete
//delete all users where FirstName matches
context.Users.Delete(u => u.FirstName == "firstname");
Update
//update all tasks with status of 1 to status of 2
context.Tasks.Update(
t => t.StatusId == 1,
t2 => new Task {StatusId = 2});
//example of using an IQueryable as the filter for the update
var users = context.Users.Where(u => u.FirstName == "firstname");
context.Users.Update(users, u => new User {FirstName = "newfirstname"});
https://github.com/loresoft/EntityFramework.Extended

How to delete elements by ID?

I have a list of ids
IEnumerable<long> ids
How can I delete from a table where the ID matches?

The "standard" way according to the MSDN docs is to use those IDs to pull up each of the objects, then delete them with the DeleteObject command:
context.Where(x=>ids.Contains(x.Id)).ToList().ForEach(context.DeleteObject);
This will produce N+1 roundtrips to the DB. This is a common downside of ORMs; they're excellent at 99% of everyday use of SQL by an app (querying for results, creating/updating/deleting single objects) but are not really designed for these bulk operations.
You could also construct a query using ExecuteStoreCommand, where you "DELETE FROM table WHERE id IN #p1" and specify the list of IDs as a parameter.

Assuming you have Id as primary key in your entities and you have an entity called SampleEntity you could do something like this:
foreach(long id in ids)
{
var sampleEntity = context.SampleEntities.SingleOrDefault( e => e.Id == id);
if(sampleEntity!=null)
context.SampleEntities.DeleteObject(sampleEntity);
}
context.SaveChanges();
EF does not support batch operation, so you will have to delete entities one by one, or alternatively do a direct store query (SQL) to make the deletion.

Generically
var db = GetYourDBContext();
var ItemsToDelete = (from id in ids
from item in db.TableWithItems
where item.id == id
select item).ToList();
foreach (Item item in ItemsToDelete)
db.DeleteObject(item);
db.SaveChanges();

How about something like,
context.Entities.Where(e => ids.Contains(e.id))
.ToList()
.ForEach(e => context.SampleEntities.DeleteObject(e) );
context.saveChanges();