Develop Reference

How to improve the performance of Entity Framework Code?

In the project, I need to call an external API based on time. So, for one day, I may need to call the API 24 times, one call for one hour period. The API result is a XML file which has 6 fields. I will need to insert these data into a table. Averagely, for each hour, it has about 20,000 rows data.
The table has these 6 columns:
col1, col2, col3, col4, col5, col6
When all 6 columns are the same, we consider the rows are the same, and we should not insert duplications.
I'm using C# and Entity Framework for this:
foreach (XmlNode node in nodes)
{
try
{
count++;
CallData data = new CallData();
...
// get all data and set in 'data'
// check whether in database already
var q = ctx.CallDatas.Where(x => x.col1 == data.col1
&& x.col2 == data.col2
&& x.col3 == data.col3
&& x.col4 == data.col4
&& x.col5 == data.col5
&& x.col6 == data.col6
).Any();
if (q)
{
// exists in database, skip
// log info
}
else
{
string key = $"{data.col1}|{data.col2}|{data.col3}|{data.col4}|{data.col5}|{data.col6}";
// check whether in current chunk already
if (dic.ContainsKey(key))
{
// in current chunk, skip
// log info
}
else
{
// insert
ctx.CallDatas.Add(data);
// update dic
dic.Add(key, true);
}
}
}
catch (Exception ex)
{
// log error
}
}
Logger.InfoFormat("Saving changes ...");
if (ctx.ChangeTracker.HasChanges())
{
await ctx.SaveChangesAsync();
}
Logger.InfoFormat("Saving changes ... Done.");
The code works fine. However, we will need to use this code to run for past several months. The issue is: the code runs slow since for each row it will need to check whether it exists already.
Is there any suggestions to improve the performance?
Thanks

You don't show the code on when the context is created or the life-cycle. I'm inclined to point you to your indexes on the table. If these aren't primary keys then you might see the performance issue there. If you are doing full table scans, it will be progressively slower. With that said, there are two separate ways to handle the
The EF Native way: You can explicitly create a new connection on each interaction (avoiding change tracking for all entries reducing progressive slowdown). Also, your save is async but your *Any statement is sync. Using async for that as well might help take some pressure off the current thread if it's waiting.
// Start your context scope closer to the data call, as if the look is long
// running you could be building up tracked changes in the cache, this prevents
// that situation.
using (YourEntity ctx = new YourEntity())
{
CallData data = new CallData();
if (await ctx.CallDatas.Where(x => x.col1 == data.col1
&& x.col2 == data.col2
&& x.col3 == data.col3
&& x.col4 == data.col4
&& x.col5 == data.col5
&& x.col6 == data.col6
).AnyAsync()
)
{
// exists in database, skip
// log info
}
else
{
string key = $"{data.col1}|{data.col2}|{data.col3}|{data.col4}|{data.col5}|{data.col6}";
// check whether in current chunk already
if (dic.ContainsKey(key))
{
// in current chunk, skip
// log info
}
else
{
// insert
ctx.CallDatas.Add(data);
await ctx.SaveChangesAsync();
// update dic
dic.Add(key, true);
}
}
}
Optional Way: Look into inserting the data using a bulk operation via store procedure. 20k rows is trivial, and you can still use entity framework for that as well. See https://stackoverflow.com/a/9837927/1558178
I have created my own version of this (customized for my specific needs) and have found that it works well and give more control for bulk inserts.
I have used this ideology to insert 100k records at a time. I have my logic in the stored procedure for checking for duplicates which gives me better control as well as reducing the over the wire call to 0 reads and 1 write. This should just take a second or two to execute assuming your stored procedure is optimized.

Different approach:
Save all rows with duplicates - should be very efficient
When you use data from the table use DISTINCT for all fields.

For raw, bulk operations like this I would consider avoiding EF entities and context tracking and merely execute SQL through the context:
var sql = $"IF NOT EXISTS(SELECT 1 FROM CallDates WHERE Col1={data.Col1} AND Col2={data.Col2} AND Col3={data.Col3} AND Col4={data.Col4} AND Col5={data.Col5} AND Col6={data.Col6}) INSERT INTO CallDates(Col1,Col2,Col3,Col4,Col5,Col6) VALUES ({data.Col1},{data.Col2},{data.Col3},{data.Col4},{data.Col5},{data.Col6})";
context.Database.ExeculeSqlCommand(sql);
This does without the extra checks and logging, just effectively raw SQL with duplicate detection.

LINQ to Entities- SaveChanges take too much time

Currently, I am struggling with an issue regarding Entity Framework (LINQ to Entities). Most of the time when I try to execute entity.SaveChanges() everything works fine but at some points entity.SaveChanges() takes too much and timesouts. I searched a lot but was unable to find out the answer.
(According to companies policy, I cannot copy code somewhere else. So, I do not have the exact code but I will try to layout the basic structure. I hope it helps you to figure out the problem but if i doesn't then let me know.)
Task:
My task is to scan the whole network for some specific files. Match content of each file with the content of database and based on the matching either insert or update the database with the content of the file. I have around 3000 files on the network.
Problem:
public void PerformAction()
{
DbTransaction tran = null;
entity.Connection.Open(); //entity is a global variable declared like myDatabaseEntity entity = new myDatabaseEntity();
tran = entity.Connection.BeginTransaction();
foreach(string path in listOfPaths)
{
//returns 1 - Multiple matching in database OR
// 2 - One matching file in database OR
// 3 - No Matching found.
int returnValue = SearchDatabase();
if(returnValue == 1)
DoSomething(); //All inserts/updates work perfectly. Save changes also works correctly.
else if(returnValue == 2)
DoSomething(); //Again, everything ok. SaveChanges works perfectly here.
else
{
//This function uses some XML file to generate all the queries dynamically
//Forexample INSERT INTO TABLEA(1,2,3);
GenerateInsertQueriesFromXML();
ExecuteQueries();
SaveChanges(); <---- Problem here. Sometimes take too much time.
}
//Transaction commit/rollback code here
}
}
public bool ExecuteQueries()
{
int result = 0;
foreach(string query in listOfInsertQueries)
{
result = entity.ExecuteStoreCommand(query); //Execute the insert queries
if(result <=0)
return false;
}
entity.TestEntityA a = new entity.TestEntityA();
a.PropertyA = 123;
a.PropertyB = 345;
//I have around 25 properties here
entity.AddToTestEntityA(a);
return true;
}

Found the issue.
The main table where i was inserting all the data had a trigger on INSERT and DELETE.
So, whenever i inserted some new data in the main table, the trigger was firing in the backend and was taking all the time.
Entity framework is FAST and INNOCENT :D

Performance of Related tables in calculated properties

Looking to see if there is a better way to do this.
I am using DB first and have a table called Items. Below is a calculated property that I specify on a partial class to extend it that uses related tables to derive the result. This technically works fine. I like the ease of using it, the fact that all this business logic is defined once in the domain, and that you can use complex code to derive the results.
The only issue I am concerned with is performance, when you pull back multiple records. Using SQL Profiler, I can see that if you pull back 50 rows of Item, it will execute an additional query to retrieve the Work Order Details in this case, 50 times! Not sure why it is not doing a join instead of doing 50 additional reads??? And I have more than one calculated property like this going out to multiple tables and each one is doing an explicit read per row = slow!
The result from pulling back 50 rows from Item table, is 2,735 reads from the database as indicated by SQL Profiler!!! I am not that familiar with SQL Profiler so maybe I am mis-interpreting somthing, but I know it is doing a lot of DB reads.
Why doesn't it do a join instead of doing an explicit read to the related tables for each row in Items?
What is "Best Practice" to accomplish this? Is there a better way?
.
[Display(Name = "Qty Allocated")]
public decimal QtyAllocated
{
get
{
if (this.TrackInventory)
{
var inProcessNonRemnantWorkOrderDetails = this.WorkOrderDetails.Where(wod =>
new[]
{
(int)WorkOrderStatus.Created,
(int)WorkOrderStatus.Released,
(int)WorkOrderStatus.InProcess
}.Contains(wod.WorkOrderHeader.StatusId)
&& wod.EstimatedQuantity >= 1 //Don't count remnants as allocated
);
var inProcessRemnantWorkOrderDetails = this.WorkOrderDetails.Where(wod =>
new[]
{
(int)WorkOrderStatus.Created,
(int)WorkOrderStatus.Released,
(int)WorkOrderStatus.InProcess
}.Contains(wod.WorkOrderHeader.StatusId)
&& wod.EstimatedQuantity > 0 && wod.EstimatedQuantity < 1 //gets just remnants
);
decimal qtyAllocated =
this.WorkOrderDetails == null
? 0
: inProcessNonRemnantWorkOrderDetails.Sum(a => (a.EstimatedQuantity - a.ActualQuantity));
if (qtyAllocated == 0 && inProcessRemnantWorkOrderDetails.Any())
{
qtyAllocated = 0.1M;
}
return qtyAllocated;
}
else
{
return 0;
}
}
}

Aron was correct. When I eager load the related entities by using the Include() method in my query, there is only 1 hit to the database.

Multiple SELECT in NHibernate native SQL

I create a complex search query in native SQL. It's basically something like this:
SELECT ID FROM t_Product WHERE Name LIKE #criteria
SELECT publisher, count(*) as number FROM t_Product GROUP BY publisher
It has 2 SELECT statements and I want it to be sent to DB server in one round trip.
But I can't figure out how to achieve this in Nhibernate.
I considered following options but none seems to work
Use CreateMultiQuery, but this only accept HQL, not native SQL
Use CreateSQLQuery, but call to List() only return result for the first SELECT statement
Moving to a stored procedure is not an option since the whole SQL is very dynamic.
We still use Nhibernate 1.2 thus new features in later version couldn't be used either.
Advice are welcome.

Not possible using NH version 1.2
Futures was released in version 2.1 which allows you to do exactly this.
e.g.
var blogs = s.CreateCriteria<Invoice>()
.SetMaxResults(30)
.Future<Invoice>();
var countOfInvoices = s.CreateCriteria<Invoice>()
.SetProjection(Projections.Count(Projections.Id()))
.FutureValue<int>();
So you are going to either upgrade, fall back to ADO.NET and use multiple recordsets or live with what you have! Sorry!

This is really going to be scenario-specific, but if you're stuck with NH Version 1.2, and eliminating the round-trip is your goal, you could consider rewriting this as a single query using a sub-select.
Something along the lines of:
SELECT publisher, count(*) as number,
(SELECT ID FROM t_Product WHERE Name LIKE #criteria) As theId
FROM t_Product GROUP BY publisher
Would work if your subquery only returned a single value.

I don't think that it is possible, because both queries are SELECTs.
You may try a semicolon after the first query, and two line feeds between them, this is required for some databases. I successfully run query-scripts like this. If it runs, use a debugger to see what you get back ...
If this doesn't work, you need separate round trips or switch to HQL / Criteria.

You can use MultiQuery "Hack" like this:
The procudure:
CREATE PROCEDURE [dbo].[proc_Name]
AS BEGIN
SELECT * FROM t_Question where ...
SELECT * FROM t_Question where ........
END
The NHibernate Query Code:
public void ProcdureMultiTableQuery()
{
var session = Session;
var procSQLQuery = session.CreateSQLQuery("exec [proc_Name] ?,?");// prcodure returns two table
procSQLQuery.SetParameter(0, userId);
procSQLQuery.SetParameter(1, page);
procSQLQuery.AddEntity(typeof(Question));
var multiResults = session.CreateMultiQuery()
.Add(procSQLQuery)
// More table your procedure returns,more empty SQL query you should add
.Add(session.CreateSQLQuery(" ").AddEntity(typeof(Question))) // the second table returns Question Model
.List();
if (multiResults == null || multiResults.Count == 0)
{
return;
}
if (multiResults.Count != 2)
{
return;
}
var questions1 = ConvertObjectsToArray<Question>((System.Collections.IList)multiResults[0]);
var questions2 = ConvertObjectsToArray<Question>((System.Collections.IList)multiResults[1]);
}
static T[] ConvertObjectsToArray<T>(System.Collections.IList objects)
{
if (objects == null || objects.Count == 0)
{
return null;
}
var array = new T[objects.Count];
for (int i = 0; i < array.Length; i++)
{
array[i] = (T)objects[i];
}
return array;
}

Linq update query - Is there no pretty way?

I want to update my database using a LINQ2SQL query.
However this seems for some reason to be a very ugly task compared to the otherwise lovely LINQ code.
The query needs to update two tables.
tbl_subscription
(
id int,
sub_name nvarchar(100),
sub_desc nvarchar(500),
and so on.
)
tbl_subscription2tags
(
sub_id (FK to tbl_subscription)
tag_id (FK to a table called tbl_subscription_tags)
)
Now down to my update function a send a tbl_subscription entity with the tags and everything.
I can't find a pretty way to update my database..
I can only find ugly examples where I suddenly have to map all attributes..
There most be a smart way to perform this. Please help.
C# Example if possible.
I have tried this with no effect:
public void UpdateSubscription(tbl_subscription subscription)
{
db.tbl_subscriptions.Attach(subscription);
db.Refresh(System.Data.Linq.RefreshMode.OverwriteCurrentValues, subscription);
db.SubmitChanges(System.Data.Linq.ConflictMode.FailOnFirstConflict);
}
Source for this code is here:
http://skyeyefive.spaces.live.com/blog/cns!6B6EB6E6694659F2!516.entry

Why don't just make the changes to the objects and perform a SubmitChanges to the DataContext?
using(MyDataContext dc = new MyDataContext("ConnectionString"))
{
foreach(var foo in dc.foo2)
{
foo.prop1 = 1;
}
dc.SubmitChanges();
}
Otherwise you need to tell us more about the lifecycle of the object you want to manipulate
edit: forgot to wrap in brackets for using

Unless I'm misunderstanding your situation, I think that citronas is right.
The best and easiest way that I've found to update database items through LINQ to SQL is the following:
Obtain the item you want to change from the data context
Change whatever values you want to update
Call the SubmitChanges() method of the data context.
Sample Code
The sample code below assumes that I have a data context named DBDataContext that connects to a database that has a Products table with ID and Price parameters. Also, a productID variable contains the ID of the record you want to update.
using (var db = new DBDataContext())
{
// Step 1 - get the item from the data context
var product = db.Products.Where(p => p.ID == productID).SingleOrDefault();
if (product == null) //Error checking
{
throw new ArgumentException();
}
// Step 2 - change whatever values you want to update
product.Price = 100;
// Step 3 - submit the changes
db.SubmitChanges();
}

I found out that you can use "Attach" as seen in my question to update a table, but apparently not the sub tables. So I just used a few Attach and it worked without having to run through parameters!