Develop Reference

Azure SQL DB hanging on to connection until EF times out

I have an intermittent issue that I can't seem to get to the bottom of whereby a .NET Framework 4.6 MVC API (C#) uses an Entity Framework 6 (runtime v4.0.30319) stored procedure call that takes a long time to respond.
The stored procedure is hosted on Azure SQL DB.
It is an unremarkable procedure, it interrogates a couple of tables to give a nutritionist their clients' latest data and website interactions.
ALTER PROCEDURE [PORTAL].[GetNutritionistCustomers]
#id_usernut varchar(20)
AS
WITH Activity AS (
SELECT LastActivity = MAX(ExecutionDate),
ded.UserName
FROM portal.DailyExecutionData ded
WHERE ded.ExecutionDate < GETDATE()
GROUP BY ded.UserName
),
Logging AS (
SELECT LastLogin = MAX(l.LoggedOnDate),
l.UserName
FROM PORTAL.Logins l
WHERE l.LoginType = 'Direct Client'
GROUP BY l.UserName
)
SELECT unc.CompanyCode,
a.ACCOUNT,
ad.ADDRESS1,
un.ID_UserNut,
ueu.ExpirationDate,
Expired = CAST(CASE WHEN ueu.ExpirationDate < GETDATE() THEN 1 ELSE 0 END AS BIT),
LastActive = la.LastActivity,
l.LastLogin
FROM UK_Nutritionist un
JOIN UK_NutCompany unc ON un.ID_UserNut = unc.ID_UserNut
JOIN UK_EnabledUsers ueu ON unc.CompanyCode = ueu.UserName
JOIN Klogix.ACCOUNT a ON ueu.ID_User = a.ACCOUNTID
LEFT JOIN Klogix.ADDRESS ad ON a.ADDRESSID = ad.ADDRESSID AND ad.IsPrimary = 1
LEFT JOIN Activity la ON ueu.UserName = la.UserName
LEFT JOIN Logging l ON ueu.UserName = l.UserName
WHERE un.ID_UserNut = #id_usernut
Run in SSMS or ADS this procedure takes on average about 50-100ms to complete. This is consistent, the tables are well indexed and the query plan is all index seeks. There are no Key or RID lookup red flags.
Having investigated with profiler, I've determined that what happens is EF creates a connection, calls the procedure. I can see the RPC entered, EF reaches it connection timeout period, and then we get RPC completed, and the data returns to EF, and the code spins out a list of pocs to return to the caller in json.
[HttpGet]
[Route("Customers")]
public HttpResponseMessage Get()
{
try
{
if (IsTokenInvalid(Request))
return Request.CreateResponse(HttpStatusCode.Unauthorized);
var nutritionistCustomers = new ExcdbEntities().GetNutritionistCustomers(TokenPayload.Nutritionist).Select(x => new NutritionistCustomers()
{
Account = x.ACCOUNT,
Address = x.ADDRESS1,
CompanyCode = x.CompanyCode,
ExpirationDate = x.ExpirationDate,
expired = x.Expired,
LastActive = x.LastActive,
LastLogin = x.LastLogin
}).ToList();
return Request.CreateResponse(HttpStatusCode.OK, GenerateResponse(nutritionistCustomers))
}
catch (Exception e)
{
return Request.CreateResponse(HttpStatusCode.InternalServerError, GenerateResponse(e));
}
}
If I change the timeout on the connection then the amount of time that SQL waits before releasing the data changes.
I thought it might be something to do with the Azure app service that hosts the API, but it turns out that running it in debug in Visual Studio has the same issue.
The short term fix is to recompile the stored procedure. This will then return the data immediately. But at some undetermined point in the future the same procedure will suddenly manifest this behaviour again.
Only this procedure does this, all other EF interactions, whether linq to sql table or view interrogations, and all other procedures seem to behave well.
This has happened before in a now legacy system that had the same architecture, though it was a different procedure on a different database.
Whilst shortening the timeout is a workaround, it's a system wide value, and even 10 seconds, enough to cope with system hiccouphs, is way too long for this customer selection screen, which should be instantaneous.
Also I would rather fix the underlying problem, if anyone has any clue what might be going on.
I have considered an OPTION_RECOMPILE on the statement, but it just feels wrong to do so.
If anyone else has experienced this and has any insight I'd be most grateful is it's driving me to distraction.

In the end this turned out to be a concatenation issue.
EF sets the CONCAT_NULL_YIELDS_NULL off inits context, and for whatever reason this caused a truly terrible performance issue.
For what I can only call legacy issue of inept previous employees, a date returned by the procedure did some inexplicable chopping up and concatenation of various dateparts of a date.
I replaced this by, you know, returning the date (and amending the EF object obviously), and hey presto no more performance issue.

Performance issues to iterate results with C# SQLite DataReader and attached database

I am using System.Data.SQLite and SQLiteDataReader in my C# project. I am facing performance issues when getting the results of a query with attached databases.
Here is an example of a query to search text into two databases :
ATTACH "db2.db" as db2;
SELECT MainRecord.RecordID,
((LENGTH(MainRecord.Value) - LENGTH(REPLACE(UPPER(MainRecord.Value), UPPER("FirstValueToSearch"), ""))) / 18) AS "FirstResultNumber",
((LENGTH(DB2Record.Value) - LENGTH(REPLACE(UPPER(DB2Record.Value), UPPER("SecondValueToSearch"), ""))) / 19) AS "SecondResultNumber"
FROM main.Record MainRecord
JOIN db2.Record DB2Record ON DB2Record.RecordID BETWEEN (MainRecord.PositionMin) AND (MainRecord.PositionMax)
WHERE FirstResultNumber > 0 AND SecondResultNumber > 0;
DETACH db2;
When executing this query with SQLiteStudio or SQLiteAdmin, this works fine, I am getting the results in a few seconds (the Record table can contain hundreds of thousands of records, the query returns 36000 records).
When executing this query in my C# project, the execution takes a few seconds too, but it takes hours to run through all the results.
Here is my code :
// Attach databases
SQLiteDataReader data = null;
using (SQLiteCommand command = this.m_connection.CreateCommand())
{
command.CommandText = "SELECT...";
data = command.ExecuteReader();
}
if (data.HasRows)
{
while (data.Read())
{
// Do nothing, just iterate all results
}
}
data.Close();
// Detach databases
Calling the Read method of the SQLiteDataReader once can take more than 10 seconds ! I guess this is because the SQLiteDataReader is lazy loaded (and so it doesn't return the whole rowset before reading the results), am I right ?
EDIT 1 :
I don't know if this has something to do with lazy loading, like I said initially, but all I want is being able to get ALL the results as soon as the query is ended. Isn't it possible ? In my opinion, this is really strange that it takes hours to get results of a query executed in few seconds...
EDIT 2 :
I just added a COUNT(*) in my select query in order to see if I could get the total number of results at the first data.Read(), just to be sure that it was only the iteration of the results that was taking so long. And I was wrong : this new request executes in few seconds in SQLiteAdmin / SQLiteStudio, but takes hours to execute in my C# project. Any idea why the same query is so much longer to execute in my C# project?
EDIT 3 :
Thanks to EXPLAIN QUERY PLAN, I noticed that there was a slight difference in the execution plan for the same query between SQLiteAdmin / SQLiteStudio and my C# project. In the second case, it is using an AUTOMATIC PARTIAL COVERING INDEX on DB2Record instead of using the primary key index. Is there a way to ignore / disable the use of automatic partial covering indexes? I know it is used to speed up the queries, but in my case, it's rather the opposite that happens...
Thank you.

Besides finding matching records, it seems that you're also counting the number of times the strings matched. The result of this count is also used in the WHERE clause.
You want the number of matches, but the number of matches does not matter in the WHERE clause - you could try change the WHERE clause to:
WHERE MainRecord.Value LIKE '%FirstValueToSearch%' AND DB2Record.Value LIKE '%SecondValueToSearch%'
It might not result in any difference though - especially if there's no index on the Value columns - but worth a shot. Indexes on text columns require alot of space, so I wouldn't blindly recommend that.
If you haven't done so yet, place an index on the DB2's RecordID column.
You can use EXPLAIN QUERY PLAN SELECT ... to make SQLite spit out what it does to try to make your query perform, the output of that might help diagnose the problem.

Are you sure you use the same version of sqlite in System.Data.SQLite, SQLiteStudio and SQLiteAdmin ?
You can have huge differences.

One more typical reason why SQL query can take different amount of time when executed with ADO.NET and from native utility (like SQLiteAdmin) are command parameters used in CommandText (it is not clear from your code whether parameters are used or not). Depending on ADO.NET provider implementation the following identical CommandText values:
SELECT * FROM sometable WHERE somefield = ? // assume parameter is '2'
and
SELECT * FROM sometable WHERE somefield='2'
may lead to absolutely different execution plan and query performance.
Another suggestion: you may disable journal (specifying "Journal Mode=off;" in the connection string) and synchronous mode ("Synchronous=off;") as these options also may affect query performance in some cases.

NHibernate: Object hierarchy and performance

I've a database with a Customer table. Each of these customers has a foreign key to an Installation table, which further has an foreign key to an Address table (table renamed for simplicity).
In NHibernate I'm trying to query the Customer table like this:
ISession session = tx.Session;
var customers = session.QueryOver<Customer>().Where(x => x.Country == country);
var installations = customers.JoinQueryOver(x => x.Installation, JoinType.LeftOuterJoin);
var addresses = installations.JoinQueryOver(x => x.Address, JoinType.LeftOuterJoin);
if (installationType != null)
{
installations.Where(x => x.Type == installationType);
}
return customers.TransformUsing(new DistinctRootEntityResultTransformer()).List<Customer>();
Which results in a SQL query similar to (catched by NHibernate Profiler):
SELECT *
FROM Customer this_
left outer join Installation installati1_
on this_.InstallationId = installati1_.Id
left outer join Address address2_
on installati1_.AddressId = address2_.Id
WHERE this_.CountryId = 4
and installati1_.TypeId = 1
When I execute the above SQL query in Microsoft SQL Server Management Studio it executes in about 5 seconds but returns ~200.000 records. Nevertheless it takes a lot of time to retrieve the List when running the code. I've been waiting for 10 minutes without any results. The debug-log indicated that a lot of objects are constructed and initiated because of the object hierarchy. Is there a way to fix this performance issue?

I'm not sure what you are trying to do, but loading and saving 200000 records through any OR mapper is not feasable. 200000 objects will take a lot of memory and time to be created. Depending on what you want to do, loading them in pages or make a update query directly on the database (sp or named query) can fix your performance. Batching can be done by:
criteria.SetFirstResult(START).SetMaxResult(PAGESIZE);

NHibernate Profiler shows two times in the duration column x/y, with x being the time to execute the query and y the time to initialize the objects. The first step is to determine where the problem lies. If the query is slow, get the actual query sent to the database using SQL Profiler (assuming SQL Server) and check its performance in SSMS.
However, I suspect your issue may be the logging level. If you have the logging level set to DEBUG, NHibernate will generate very verbose logs and this will significantly impact performance.
Even if you can get it to perform well with 200000 records that's more than you can display to the user in a meaningful way. You should use paging/filtering to reduce the size of the result set.

How to make more efficient query using Nhibernate in asp.net mvc3

I have about 1 million records in my contact table in the DB, now I have to get only 30. How can I make the following query more efficient:
GetContacts Method
private IQueryable<Contact> GetContacts()
{
return this.Query().Where(c => c.ActiveEmployer== true); // here ' this ' is the current service context
}
Gettin Contacts
var contactlist = GetContacts(); // this will get all records from db
var finalcontacts = contactlist.Skip(0).Take(30).Fetch(c.Roles).Fetch(c.SalaryInfo).ThenFetch(c.Employer);
But this query takes almost 10 seconds or more some times, and in the future I can have 30 - 40 million contacts then what would I do?

There may be some people that could come up with a great fix for that query to help you without needing more information, but for that query and all others you can probably gain much more insight by taking a look at the query that is created by NHibernate. If you've not done that before you can enable Log4Net (and possibly NLog) to output the SQL statements NH produces, or you can profile the SQL connection inside SQL Server. You can then run that SQL yourself and select to show the query plan. This will tell you exactly what the query is doing and potentially you can solve the problem yourself. It might be missing index, the original WHERE or one or all of the subsequent FETCH statements. Another question would be, have you hand crafted a SQL statement to do something similar and how long does that take?

Entity model .net querying 1 million records from MySQL performance issues

I am using an ADO .Net Entity Model for querying a MySQL database. I was very happy about its implementation and usage. I decided to see what would happen if I queried 1 million records and it has serious performance issues, and I don't understand why.
The system hangs for sometime and then I get either
A deadlock exception
MySQL Exception
My code is as follows::
try
{
// works very fast
var data = from employees in dataContext.employee_table
.Include("employee_type")
.Include("employee_status")
orderby employees.EMPLOYEE_ID descending
select employees;
// This hangs the system and causes some deadlock exception
IList<employee_table> result = data.ToList<employee_table>();
return result;
}
catch (Exception ex)
{
throw new MyException("Error in fetching all employees", ex);
}
My question is why is ToList() taking such a long time?
Also how can I avoid this exception and what is the ideal way to query a million records?

The ideal way to query a million records would be to use a IQueryable<T> to make sure that you actually aren't executing a query on the database until you need the actual data. I highly doubt that you need a million records at once.
The reason that it is deadlocking is that you are asking the MySQL server to pull those million records from the database then sort then by the EMPLOYEE_ID and then for your program to return that back to you. So I imagine that the deadlocks are from your program waiting for that to finish, and for your program to read that into memory. The MySQL problems are probably related to timeout issues.
The reason that the var data section works quickly is because you actually haven't done anything yet, you've just constructed the query. when you call ToList() then all of the SQL and reading of the SQL is executed. This is what is known as Lazy Loading.
I would suggest try this as follows:
var data = from employees in dataContext.employee_table
.Include("employee_type")
.Include("employee_status")
orderby employees.EMPLOYEE_ID descending
select employees;
Then when you actually need something from the list just call
data.Where(/* your filter expression */).ToList()
So if you needed the employee with ID 10.
var employee = data.Where(e => e.ID == 10).ToList();
Or if you need all the employees that last names start with S (I don't know if your table has a last name column, just an example).
var employees = data.Where(e => e.LastName.StartsWith("s")).ToList();
Or if you want to page through all of the employees in chunks of 100
var employees = data.Skip(page * 100).Take(100).ToList();
If you want to defer your database calls even further, you can not call ToList() and just use the iterator when you need it. So let's say you want to add up all of the salaries of the people that have a name starting with A
var salaries = data.Where(s => s.LastName.StartsWith("A"))
foreach(var employee in salaries)
{
salaryTotal += employee.Salary;
}
This would only do a query that would look something like
Select Salary From EmployeeTable Where ID = #ID
Resulting in a very fast query that is only getting the information when you need it and only just the information that you need.
If for some crazy reason you wanted to actually query all the million records for the database. Ignoring the fact that this would eat up a massive amount of system resources I would suggest doing this in chunks, you would probably need to play around with the chunk size to get the best performance.
The general idea is to do smaller queries to avoid timeout issues from the database.
int ChunkSize = 100; //for example purposes
HashSet<Employee> Employees - new HashSet<Employee>;
//Assuming it's exactly 1 Million records
int RecordsToGet = 1000000;
for(record = 0; record <= RecordsToGet; record += ChunkSize)
{
dataContext.EmployeeTable.Skip(record).Take(ChunkSize).ForEach(e => HashSet.Add(e));
}
I chose to use a HashSet<T> since they are designed for large sets of data, but I don't know what performance would look like a 1,000,000 objects.