Develop Reference

How to improve the performance of Entity Framework Code?

In the project, I need to call an external API based on time. So, for one day, I may need to call the API 24 times, one call for one hour period. The API result is a XML file which has 6 fields. I will need to insert these data into a table. Averagely, for each hour, it has about 20,000 rows data.
The table has these 6 columns:
col1, col2, col3, col4, col5, col6
When all 6 columns are the same, we consider the rows are the same, and we should not insert duplications.
I'm using C# and Entity Framework for this:
foreach (XmlNode node in nodes)
{
try
{
count++;
CallData data = new CallData();
...
// get all data and set in 'data'
// check whether in database already
var q = ctx.CallDatas.Where(x => x.col1 == data.col1
&& x.col2 == data.col2
&& x.col3 == data.col3
&& x.col4 == data.col4
&& x.col5 == data.col5
&& x.col6 == data.col6
).Any();
if (q)
{
// exists in database, skip
// log info
}
else
{
string key = $"{data.col1}|{data.col2}|{data.col3}|{data.col4}|{data.col5}|{data.col6}";
// check whether in current chunk already
if (dic.ContainsKey(key))
{
// in current chunk, skip
// log info
}
else
{
// insert
ctx.CallDatas.Add(data);
// update dic
dic.Add(key, true);
}
}
}
catch (Exception ex)
{
// log error
}
}
Logger.InfoFormat("Saving changes ...");
if (ctx.ChangeTracker.HasChanges())
{
await ctx.SaveChangesAsync();
}
Logger.InfoFormat("Saving changes ... Done.");
The code works fine. However, we will need to use this code to run for past several months. The issue is: the code runs slow since for each row it will need to check whether it exists already.
Is there any suggestions to improve the performance?
Thanks

You don't show the code on when the context is created or the life-cycle. I'm inclined to point you to your indexes on the table. If these aren't primary keys then you might see the performance issue there. If you are doing full table scans, it will be progressively slower. With that said, there are two separate ways to handle the
The EF Native way: You can explicitly create a new connection on each interaction (avoiding change tracking for all entries reducing progressive slowdown). Also, your save is async but your *Any statement is sync. Using async for that as well might help take some pressure off the current thread if it's waiting.
// Start your context scope closer to the data call, as if the look is long
// running you could be building up tracked changes in the cache, this prevents
// that situation.
using (YourEntity ctx = new YourEntity())
{
CallData data = new CallData();
if (await ctx.CallDatas.Where(x => x.col1 == data.col1
&& x.col2 == data.col2
&& x.col3 == data.col3
&& x.col4 == data.col4
&& x.col5 == data.col5
&& x.col6 == data.col6
).AnyAsync()
)
{
// exists in database, skip
// log info
}
else
{
string key = $"{data.col1}|{data.col2}|{data.col3}|{data.col4}|{data.col5}|{data.col6}";
// check whether in current chunk already
if (dic.ContainsKey(key))
{
// in current chunk, skip
// log info
}
else
{
// insert
ctx.CallDatas.Add(data);
await ctx.SaveChangesAsync();
// update dic
dic.Add(key, true);
}
}
}
Optional Way: Look into inserting the data using a bulk operation via store procedure. 20k rows is trivial, and you can still use entity framework for that as well. See https://stackoverflow.com/a/9837927/1558178
I have created my own version of this (customized for my specific needs) and have found that it works well and give more control for bulk inserts.
I have used this ideology to insert 100k records at a time. I have my logic in the stored procedure for checking for duplicates which gives me better control as well as reducing the over the wire call to 0 reads and 1 write. This should just take a second or two to execute assuming your stored procedure is optimized.

Different approach:
Save all rows with duplicates - should be very efficient
When you use data from the table use DISTINCT for all fields.

For raw, bulk operations like this I would consider avoiding EF entities and context tracking and merely execute SQL through the context:
var sql = $"IF NOT EXISTS(SELECT 1 FROM CallDates WHERE Col1={data.Col1} AND Col2={data.Col2} AND Col3={data.Col3} AND Col4={data.Col4} AND Col5={data.Col5} AND Col6={data.Col6}) INSERT INTO CallDates(Col1,Col2,Col3,Col4,Col5,Col6) VALUES ({data.Col1},{data.Col2},{data.Col3},{data.Col4},{data.Col5},{data.Col6})";
context.Database.ExeculeSqlCommand(sql);
This does without the extra checks and logging, just effectively raw SQL with duplicate detection.

Trying to update a DataRow in C# with LINQ

I'm trying to find, then update, a specific DataRow in a DataTable. I've tried a few things based on my searches, and the code below seems to be the closest I can get. The linq will return one row. With that row, I'd like to update column values (Status, StopTime, Duration). I can't for the life of me find how to do this.. I've tried casting, but I'm new to linq and don't see how to update these values.
private DataTable downloadProcStatusTable;
void UpdateDataDownloadProcedureList(ProcedureStats ProcStats)
{
var currentStatRow = from currentStat in downloadProcStatusTable.AsEnumerable()
where currentStat.Field<String>("ProcedureName") == ProcStats.ProcName
select currentStat;
}

Your query as it stands actually gives you an IEnumerable<DataRow>. You need to do this to get the actual row:
var currentStatRow = (from currentStat in downloadProcStatusTable.AsEnumerable()
where currentStat.Field<String>("ProcedureName") == ProcStats.ProcName
select currentStat).SingleOrDefault();
You should then be able to use the currentStatRow variable to modify the column values.

Outline
Load the existing entity from the database (unless you have one that you can re-attach, in which case you could avoid this additional query)
Update the properties as needed
Submit the changes back to the database using SubmitChanges()
Implementation
I wasn't exactly sure where your variables are and the names, but this should give you a good start...
void UpdateDataDownloadProcedureList(ProcedureStats ProcStats)
{
var currentStatRow = (from currentStat in downloadProcStatusTable.AsEnumerable()
where currentStat.Field<String>("ProcedureName") == ProcStats.ProcName
select currentStat).FirstOrDefault();
currentStatRow.Status = ProcStats.Status;
currentStatRow.StopTime = ProcStats.StopTime;
currentStatRow.Duration = ProcStats.Duration;
downloadProcStatusTable.SubmitChanges();
}

Dynamic LINQ in c#

Hey i am making a simple search machine through alot of different coloumns in 2 tables.
I was trying to get this to abit dynamical.
I read this:
Is there a pattern using Linq to dynamically create a filter?
Which is something that really could do the trick for me.. its just in VB and i need it in c#
here is my code :
private void displayWith1Criteria(string column, string value)
{
Console.WriteLine("entering _1_ display method");
dbcontent = new DBtestEntities();
var studienummerQuery = from members in dbcontent.Medlemmer.Include("Retninger")
where column == value
orderby members.Fornavn
select new { Studienr = members.Studienummer, Fornavn = members.Fornavn, Efternavn = members.Efternavn, Email = members.Email, Studiested = members.Studiested, Betaling = members.BetalingsType, Uddannelses_Retning = members.Retninger.retningNavn };
dataGridView1.DataSource = studienummerQuery;
}
Doesn't return any data at all...
column is being called with members.Fornavn (Fornavn - a column name)
value = Anders (one of the data's in Fornavn column)
What I want to do:
My database is loaded into dbcontent using a .edmx file from ABO entity class.
My database consist of 2 tables, "Retninger" and "Medlemmer".
Medlemmer contains columns things like Fornavn(in english, Firstname), Efternavn(Lastname), Studienummer(study no.)
What i would like is a "dynamic" method that can set both which column to be searched in and the value that needs to be searched for in the set column.

When could your expression column == value possibly return true? Only if string.Equals("Fornavn", "Anders") is true.
Doing dynamic linq is hard. Is usually do it this way:
...
where (!useMycolumn1 || member.mycolumn1 == value1)
&&(!useMycolumn2 || member.mycolumn2 == value2)
&&(!useMycolumn3 || member.mycolumn3 == value3)
...
useMycolumn* is a local boolean variable which is set to true or false, depending on whether the certain condition should be tested or not. This way, unused parts of the query are optimized out at compile time.

I think this answer from Shawn Miller to the question you linked is more what you are looking for:
http://www.albahari.com/nutshell/predicatebuilder.html

Are you remembring to call the DataBind() method on the grid? How do you know nothing is being returned?

I think its because of lazy evaluation of LINQ queries.
You can try using .ToList, as below:
dataGridView1.DataSource = studienummerQuery.ToList();
also .DataBind(), if relevant for your object.
Edit:
Lazy Evaluation: This Link, would serve as a good start

Bulk inserts and duplicate records with LINQ to SQL

Is there a "best practice" way of handling bulk inserts (via LINQ) but discard records that may already be in the table? Or I am going to have to either do a bulk insert into an import table then delete duplicates, or insert one record at a time?
08/26/2010 - EDIT #1:
I am looking at the Intersect and Except methods right now. I am gathering up data from separate sources, converting into a List, want to "compare" to the target DB then INSERT just the NEW records.
List<DTO.GatherACH> allACHes = new List<DTO.GatherACH>();
State.IState myState = null;
State.Factory factory = State.Factory.Instance;
foreach (DTO.Rule rule in Helpers.Config.Rules)
{
myState = factory.CreateState(rule.StateName);
List<DTO.GatherACH> stateACHes = myState.GatherACH();
allACHes.AddRange(stateACHes);
}
List<Model.ACH> newRecords = new List<Model.ACH>(); // Create a disconnected "record set"...
foreach (DTO.GatherACH record in allACHes)
{
var storeInfo = dbZach.StoreInfoes.Where(a => a.StoreCode == record.StoreCode && (a.TypeID == 2 || a.TypeID == 4)).FirstOrDefault();
Model.ACH insertACH = new Model.ACH
{
StoreInfoID = storeInfo.ID,
SourceDatabaseID = (byte)sourceDB.ID,
LoanID = (long)record.LoanID,
PaymentID = (long)record.PaymentID,
LastName = record.LastName,
FirstName = record.FirstName,
MICR = record.MICR,
Amount = (decimal)record.Amount,
CheckDate = record.CheckDate
};
newRecords.Add(insertACH);
}
The above code builds the newRecords list. Now, I am trying to get the records from this List that are not in the DB by comparing on the 3 field Unique Index:
AchExceptComparer myComparer = new AchExceptComparer();
var validRecords = dbZach.ACHes.Intersect(newRecords, myComparer).ToList();
The comparer looks like:
class AchExceptComparer : IEqualityComparer<Model.ACH>
{
public bool Equals(Model.ACH x, Model.ACH y)
{
return (x.LoanID == y.LoanID && x.PaymentID == y.PaymentID && x.SourceDatabaseID == y.SourceDatabaseID);
}
public int GetHashCode(Model.ACH obj)
{
return base.GetHashCode();
}
}
However, I am getting this error:
LINQ to Entities does not recognize the method 'System.Linq.IQueryable1[MisterMoney.LARS.ZACH.Model.ACH] Intersect[ACH](System.Linq.IQueryable1[MisterMoney.LARS.ZACH.Model.ACH], System.Collections.Generic.IEnumerable1[MisterMoney.LARS.ZACH.Model.ACH], System.Collections.Generic.IEqualityComparer1[MisterMoney.LARS.ZACH.Model.ACH])' method, and this method cannot be translated into a store expression.
Any ideas? And yes, this is completely inline with the original question. :)

You can't do bulk inserts with LINQ to SQL (I presume you were referring to LINQ to SQL when you said "LINQ"). However, based on what you're describing, I'd recommend checking out the new MERGE operator of SQL Server 2008.
Inserting, Updating, and Deleting Data by Using MERGE
Another example here.

I recommend you just write the SQL yourself to do the inserting, I find it is a lot faster and you can get it to work exactly how you want it to. When I did something similar to this (just a one-off program) I just used a Dictionary to hold the ID's I had inserted already, to avoid duplicates.
I find LINQ to SQL is good for one record or a small set that does its entire lifespan in the LINQ to SQL.
Or you can try to use SQL Server 2008's Bulk Insert .

One thing to watch out for is if you queue more than 2000 or so records without calling SubmitChanges() - TSQL has a limit on the number of statements per execution, so you cannot simply queue up every record and then call SubmitChanges() as this will throw an SqlException, you need to periodically clear the queue to avoid this.

Check if I have already inserted a particular record

I am going through a massive list of business objects and inserting them into the database using Linq-to-Sql.
Some of the business objects contain a payment method record (cheques, credit card etc..)
When it comes to adding a payment method, I want to check to ensure I havent already added it, cos otherwise it will rant at me when I come to Submit my changes.
if ( !context.PaymentMethods.Any ( paymentMethod => paymentMethod.PaymentMethodID == iPaymentMethod.PaymentMethodID ) )
{
PaymentMethod method = new PaymentMethod ();
method.PaymentMethodID = iPaymentMethod.PaymentMethodID;
// etc...
context.PaymentMethods.InsertOnSubmit ( method );
}
This doesnt work, I presume because Any is checking the database and not the list of objects I am about to Insert on Submit.
I know I can maintain my own list to check if the records have already been added, but to save a lot of hassle, I was just wondering if there was a tidy, Linq way to do this? Any way to check context.PaymentMethods to see if it has been added?

A possible solution would be to check the ChangeSet of the Context:
Func<PaymentMethod,bool> f =
paymentMethod => paymentMethod.PaymentMethodID == iPaymentMethod.PaymentMethodID;
if (!context.PaymentMethods.Any(f) &&
!context.GetChangeSet().Inserts.OfType<PaymentMethod>().Any(f))
{
// Submit
}

Try this:
!context.PaymentMethods.Where(paymentMethod => paymentMethod.PaymentMethodID == iPaymentMethod.PaymentMethodID).Count() = 0