Develop Reference

How can I speed up my EF query?

How can I speed up this query? Right now it's taking me around 2 minutes to pull back 210K records.
I turned off LazyLoading as well as set AsNoTracking on my tables.
I know it's a lot of data but surely it shouldn't take 2 minutes to retrieve the data?
context.Configuration.LazyLoadingEnabled = false;
List<MY_DATA> data = context
.MY_DATA.AsNoTracking()
.Include(x => x.MY_DATA_DETAILS)
.Where(x => startDate <= DbFunctions.TruncateTime(x.DB_DATE)
&& endDate >= DbFunctions.TruncateTime(x.DB_DATE)
&& x.MY_DATA_DETAILS.CODE.Trim().ToUpper() == myCode.Trim().ToUpper())
.ToList();

You can do without DbFunctions.TruncateTime() and probably also without these Trim().ToUpper() calls.
If you execute a function on a database column before it is filtered, it's impossible for the query optimizer to use any indexes on this column. This is know as being non-sargable. To execute the query in its present form, the database engine has to transform the data first and then scan all the transformed data to do the filtering. No index involved.
DbFunctions.TruncateTime() is meaningless. You have to choose startDate and endDate wisely and use x.DB_DATE as it is.
Further, if x.MY_DATA_DETAILS.CODE is a varchar column (most text columns are), it will be auto-trimmed in searches. Even if the database value contains trailing spaces, they will be ignored. So Trim isn't necessary. Next, most text columns by default have a case-insensitive database collation. You should check it. If this is SQL Server, look for collations like SQL_Latin1_General_CP1_CI_AS. The CI part means Case-Insensitive. If so, you can also do away with the ToUpper part. If not, you can either change the collation to a case-insensitive one, or you maybe should conclude that the column is case sensitive for a reason, so it does matter whether you look for Abc or abc.
Either way, having these transforming function removed form the database columns, the query should be able to run considerably faster, provided that proper indexes are in place.

Usually defining indexes on the columns that you frequently use in the 'where' clause, can improve your performance in selecting the rows from a large tale.
I recommend that you create a stored procedure and move your query into the SP and apply the performance tuning in Database and in your C# code, call the SP.

In addition to what the other guy said about moving your query to a stored proc and creating the proper indexes, I'd say you'd be better off using SQL reporting rather then trying to import the data into your application and reporting from there. Especially 210K rows. SQL has internal optimizations that you'll never be able to come close to with stored procs and queries.
You can see this very easily:
1) try write a simple console app that tries to pull down that entire table and writes it to a CSV file -- it'll be extremely slow.
2) try use the data export through Sql Mgmt Studio and export to a CSV -- it'll be done in a few seconds.

Do you need all the attributes of the object?
You can do this.
List<MY_DATA> data = context.MY_DATA.Include(x =>
x.MY_DATA_DETAILS).Where(x => startDate <= DbFunctions.TruncateTime(x.DB_DATE) &&
endDate >= DbFunctions.TruncateTime(x.DB_DATE) &&
x.MY_DATA_DETAILS.CODE.Trim().ToUpper() == myCode.Trim().ToUpper()).select(x => new MY_DATA()
{
Value = data
}).ToList();

LINQ query to Azure SQL database timing out

I'm querying my sql database which is in Azure (actually my web app is on Azure as well).
Every time I perform this particular query, there are ever changing errors (e.g. sometimes timeout occurs, sometimes it works perfectly, sometimes it takes extremely long to load).
I have noted that I am using the ToList method here to enumerate the query but I suspect that's why it is degrading.
Is there anyway I can fix this or make it better....or maybe just use native SQL to execute my query?.
I should also note in my webconfig my Database connection timeout is set to 30 seconds. Would this have any performance benefit?
I'm putting the suspect code here:
case null:
lstQueryEvents = db.vwTimelines.Where(s => s.UserID == UserId)
.Where(s => s.blnHide == false)
.Where(s => s.strEmailAddress.Contains(strSearch) || s.strDisplayName.Contains(strSearch) || s.strSubject.Contains(strSearch))
.OrderByDescending(s => s.LatestEventTime)
.Take(intNumRecords)
.ToList();
break;
It's basically querying for the 50 records...I don't understand why it's timing out sometimes.

Here are some tips:
Make sure that your SQL data types matches types in your model
Judging by your code, types should be something like this:
UserID should be int (cannot tell for sure by looking at code);
blnHide should be bit;
strEmailAddress should be nvarchar;
strDisplayName should be nvarchar;
strSubject should be nvarchar;
Make use of indexes
You should create Non-Clustered Indexes on columns that you use to filter and order data.
In order of importance:
LatestEventTime as you order ALL data by this column;
UserID as you filter out most of data by this column;
blnHide as you filter out part of data by this column;
Make use of indexes for text lookup
You could make use of indexes for text lookup if you change your filter behaviour slightly and search text only in the start of column value.
To achieve that:
change .Contains() with .StartsWith() as it would allow index to be used.
create Non-Clustered Indexes on strEmailAddress column:
create Non-Clustered Indexes on strDisplayName column:
create Non-Clustered Indexes on strSubject column:
Try out free text search
Microsoft only recently have introduced full text search in Azure SQL. You can use that to find rows matching by partial string. This is a bit complicated to achieve using EF, but it is certainly doable.
Here are some links to get you started:
Entity Framework, Code First and Full Text Search
https://azure.microsoft.com/en-us/blog/full-text-search-is-now-available-for-preview-in-azure-sql-database/

string.Contains(...) converted to WHERE ... LIKE ... sql-statement. Which is very expensive. Try to reform your query to avoid it.
Plus, Azure SQL has it's own limitations (5 sec as far as I remember, but better check SLA) for query run, so it would generally ignore your web.config settings if they are longer.

What ways exist with linq to compare if a case insensitive and culture using string exists within another

Comparing case insensitive strings with linq is quite easy, even to find out if a specific string is within another. The problems only start when it is also needed (as in my case) to differentate between ss and ß. As far as I have seen linq and string offer only 1 viable option there: Contains but the problem is the only contains overload that takes for example: StringComparer.CurrentCultureIgnoreCase as parameter does not take string as parameter for the part that is searched within the "calling" string but instead takes ONLY a char value.
As it was asked: I'm using linq to gather info from a SQL database thus:
var results = (from c in myEntity.myTablename where
(c.MyStringTextColumn.Contains(myStringTextToCompareWith))
select c.MyStringTextColumn).Distinct().ToList();
is how I'm comparing the strings originally but like I said the problem is this does not differentiate between ss and ß. Thus in the database there are different versions in regards to ss and ß stored, and I need to only find the "correct" one. Even though they are essentially the same one, an example would be strasse and straße (as names are stored in the database I couldn't use them in this example). If I type in strass I only want to find those strings that contain ss and not those with ß (thus straße should not be found)
So my question is: What options exist there?

You should be able to make use of the IndexOf property, which returns a non-negative int if the substring is found (it returns the index of the first character if found, otherwise -1), and also takes a StringComparison argument.
http://msdn.microsoft.com/en-us/library/ms224425(v=vs.110).aspx

Linq to sql doesn't support culture / case sensitive comparison, but it will rely on the database collation and the column type.
Queries do not account for SQL Server collations that might be in
effect on the server, and therefore will provide culture-sensitive,
case-insensitive comparisons by default. This behavior differs from
the default, case-sensitive semantics of the .NET Framework. -
MSDN
One option that you probably have is creating a stored procedure to do the searching like.
create proc sp_TableName
(#name as varchar(max)) -- notice the type is varchar, not nvarchar
as
select * from TableName
where
cast(ColumnName as varchar(max)) COLLATE SQL_Latin1_General_CP1_CI_AS
like '%' + #name + '%'
And using SqlQuery<T> to get the result.
var db = ...; // the context
var param = new SqlParameter("name", "ß");
var results = db.Database.SqlQuery<TableModel>("exec sp_TableName #name", param)
.Distinct()
.ToList();
Or you can just execute all the data first, so that you can do the comparing as linq to objects, not linq to sql.
var results = (from c in myEntity.myTablename.AsEnumerable() where
(c.MyStringTextColumn.Contains(myStringTextToCompareWith))
select c.MyStringTextColumn).Distinct().ToList();

is there a better performance's way to get only few columns in ASP MVC without getting the whole table and filter it with LinQ?

im calling a table with 200.000 rows and 6 columns, but i only want 2 of these columns to be used in one controller, so i want to know if there is a better way to call them from the server without compromising performance, because as i know Linq queries get the whole table and them makes the filtering, i think maybe Views is a good way, but i want to know if there are others and betters, Thanks.
for example:
var items = from i in db.Items select new {i.id,i.name};
in case i have 1.000.000 items, will it be a trouble for the server?

Your initial assumption is incorrect.
In general LINQ queries do not get the whole table. the query is converted into a "server side expression" (i.e. a SQL statement) and the statement is resolved on the server and only the requested data is returned.
Given the statement you provided you will return only two columns but you will get 1,000,000 objects in the result if you do not do any filtering. But that isn't a problem with LINQ, that's a problem with you not filtering. If you included a where clause you would only get the rows you requested.
var items = from i in db.Items
where i.Whatever == SomeValue
select new { i.id, i.name };
Your original query would be translated (roughly) into the following SQL:
SELECT id, name FROM Items
You didn't include a where clause so you're going to get everything.
With the version that included a where clause you'd get the following SQL generated:
SELECT id, name FROM Items WHERE Whatever = SomeValue
Only the rows that match the condition would be returned to your application and converted into objects.

LINQ ; Search with culture invariant

Here is my problem. We've got a list of enterprises, users or anything and we must search into it with a "StartsWith" search type. So on our site, we ain't got a Search field such as a Textbox but we've got a search header including 27 buttons "#", "A", "B", "C", [...] "Z".
The problem we have is that if user click on "E" button, when we querying to get value from database, entreprises name may start with an "É", "È", "Ê", bacause yes, our site is in french. Any idea of how to do it in LINQ.
This is important to know too that we're using LLBLGen Pro. So I guess it need to have something he can convert into a valid SQL Query.
Here is what we've already tryied :
IList<Enterprise> enterprises;
switch (searchChar){
[...]
case "E":
enterprises = from ent in ourContext.Enterprises
where "eèéêë".Any(param => ent.name[0] == param)
select ent;
break;
[...]
}
Which give us this error something relatively to a unconvertable query:
Unable to cast object of type 'System.Linq.Expressions.ConstantExpression' to type 'SD.LLBLGen.Pro.LinqSupportClasses.ExpressionClasses.SetExpression'.
So we've tried to make it basicaly with a simple LINQ query, without querying to DB to know if it's possible.
IList<string> test = new List<string>() { "École", "enlever", "avoir" };
IList<string> result = (from value in test
where "eéèêë".Contains(value[0].ToString())
select value).ToList();
What is weird with this query, is that it ain't crash. But, it ain't work too! When debugging, it go throught it, but when we try to see what is into "result" list, it's like if there's nothing in it. I mean, the list is simply null. But nothing fail into the try catch!
Please help !

The real solution is here is to create an extra column in your database for a searchable name, so whenever you add a new company you also add a value to the searchable name column. You would convert all characters to upper (or lower if you like) invariant and add the "clean" name to the searchable name column. You could also remove punctuation at this point too, as that is often a pain in searches.
This will leave you with a column you will never display but it will be much easier (and also much quicker) to search for matches in this column as you will never need to worry about accented characters at search time.

Just use StartsWith method of the string
IList<string> test = new List<string>() { "École", "enlever", "avoir" };
var result = test
.Where(s => s.StartsWith("e", StringComparison.CurrentCultureIgnoreCase))
.ToList();

If i got you right here is what you want:
var result = test.Where(x => "eéèêë".Contains(Char.ToLowerInvariant(x.FirstOrDefault())));

The Any seem to be not working. Use Contains instead. This is workin.
enterprises = from ent in ourContext.Enterprises
where "eèéêëEÈÉÊËE".Contains(ent.name[0])
select ent;

I'm not sure whether you have any control over the database, and which RDMBS you are using, but an easy way seems to be using a case insensitive, accent insensitive collation in your query - this way SQL does the hard work.
-- Assuming that your existing table is Accent Sensitive
create table Enterprises
(
name nvarchar(200) collate SQL_Latin1_General_CP1_CI_AS
)
GO
-- We can still use an Accent Insensitive View
create view vEnterprises
as
select name collate SQL_Latin1_General_CP1_CI_AI AS name
from Enterprises
GO
insert into Enterprises values ('e')
insert into Enterprises values ('è')
insert into Enterprises values ('é')
insert into Enterprises values ('ê')
insert into Enterprises values ('ë')
insert into Enterprises values ('E')
insert into Enterprises values ('É')
insert into Enterprises values ('È')
insert into Enterprises values ('Ê')
-- returns 'e' and 'E'
select * from Enterprises where name like 'e%'
-- returns everything
select * from vEnterprises where name like 'e%'
i.e. Assuming that you can add the accent insensitive view to LLBLGen you can just pass 'e' to the query.