I am using the following parameterized linq query expression to query from approximately 100,000 records in a SQL server. Is there a better way?
public IList<Article> GetArticles(string language, string category, string subcategory, bool exclusives, int pageIndex, int pageSize = 200)
{
IQueryable<Article> query;
query = db.Articles.Where(t => t.IsActive && t.ArticleStatus);
if (exclusives) { query = query.Where(t => t.IsExclusive); }
if (language.ToUpper() != "ALL")
{
query = query.Where(t => t.Language.ToUpper() == language);
}
if (category.ToUpper() != "ALL")
{
query = query.Where(t => t.Category.ToUpper() == category);
}
if (subcategory.ToUpper() != "ALL")
{
query = query.Where(t => t.SubCategory.ToUpper() == subcategory);
}
query = query.Where(t => t.ArticleDate <= DateTime.Now);
query = query.OrderByDescending(t => t.ArticleNo).Skip((pageIndex - 1) * pageSize).Take(pageSize);
if (query.Any() == false)
{
return null;
}
else
{
return query.ToList();
}
}
Entity Framework isn't known for being the fastest option at runtime--it's supposed to be maintainable, easy, and fast to write. If you want something fast, go with a different ORM that lets you write T-SQL manually (or use EF's Database.SqlQuery<T>(string) methods).
That said, you're actually running your query twice.
if (query.Any() == false)
{
return null;
}
else
{
return query.ToList();
}
Each of these methods will execute a varied form of your query against the database.
Instead, test out performance with only calling with once.
var returnList = query.ToList();
if (returnList.Count == 0)
{
return null;
}
else
{
return returnList;
}
The improvement here should be evident, but it's possible that statistics will be messed up on your database, so definitely test.
Aside from that, I would just check indexes. Entity Framework builds its queries in sometimes-unusual ways, so I like to run the query plan explorer (or SQL Profiler) against them.
Simplest way to do that is to put a breakpoint at your ToList() call for a few inputs, hover or use the Watch windows to get query.ToString(), paste that in something that can get query plans (like VS's database tooling or SSMS), and see if it has anything obvious.
It's sometimes the case that you an rearrange calls to EF to improve the query, but generally the case that you'll want to reassess indexes server-side to meet what EF is giving you.
But, again, remember to do that for varied inputs--you're changing your query, so you want to make sure you've got all your bases covered.
Related
I have a Linq statement which has two optional Where clauses but I cannot work out how to implement this linq (at the moment I have 4 linq statements in this method which off course is not good!):
//Params passed into my method:
int StoreId = 0;
bool ShowEnabledTills = true;
var query = (from Transactions in db.Transactions
join Tills in db.Tills on new { TillId = Convert.ToInt32(Transactions.TillId) } equals new { TillId = Tills.TillId }
join CompanyAddresses in db.CompanyAddresses on new { CompanyId = Convert.ToInt32(Transactions.StoreID) } equals new { CompanyId = CompanyAddresses.CompanyId }
where
CompanyAddresses.CompanyId == StoreId <===== OPTIONAL
&& Tills.Active == ShowEnabledTills <===== OPTIONAL
select new
{
Tills.TillId,
Tills.ComputerName,
Tills.Description,
CompanyAddresses.CompDescription
}).Distinct();
I took a look at PredicateBuilder but I couldn't quite get my head around this but if I can create some re-usable code this would be great!
// get these from params to decide optional or not
var validateStoredId = false;
var validateActiveStatus = false;
And in your where clause do something like:
where
(!validateStoreId || CompanyAddresses.CompanyId == StoreId)
&& (!validateActiveStatus || Tills.Active == ShowEnabledTills)
I have a little to contribute here I think. I came across the same problem, and implemented a similar solution. HOWEVER! This wont short-circuit as you would hope (or at least it didn't in mine!)
This led to me looping through a list of 10k items checking if true == false effectively, which whilst not an expensive check is a check you don't want to do thousands of times nevertheless.
For anyone else coming to this, I'd recommend splitting your queries down into subqueries wrapped in if checks, that is far more performant when dealing with large datasets :)
Edit : I built this little helper for doing exactly this, hope someone else finds it useful, I enjoyed not having to break my method chains :)
public static IEnumerable<T> OptionalWhere<T>(this IEnumerable<T> source, Func<T, bool> predicate, bool excecuteFilter)
{
if (!excecuteFilter)
{
return source;
}
else
{
return source.Where(predicate);
}
}
Try this:
var query = db.Transactions.Join(
db.Tills, //Join to this table
transaction => transaction.TillId //Outer key
, till => till.TillId, //Inner key
(transaction, till) => new {transaction, till}) //Result of join
.Join(db.CompanyAddresses, //Join to this table
r => r.till.StoreId, //Outer key
a => a.CompanyId, //Inner key
(r, address) => new {r.transaction, r.till, address}); //Result of join
if(StoreId != null)
query = query.Where(d => d.address.StoreId == StoreId);
if(ShowEnabledTills)
query = query.Where(d => d.till.Active == true);
var items = query.Select(d => new {
d.Till.TillId,
d.Till.ComputerName,
d.Till.Description,
d.address.CompDescription
}).Distinct();
I did it without knowing much about your schema, but this should give you a good idea.
I disliked the "SQL style" syntax for this very reason (confusing at times). I started using method calls like this and never had a problem since.
I have a question regarding a Linq to SQL query.
I have following situation:
I have a search with lots of options, like location, availability, name, language etc ...
For this options i have to execute a query to retrieve the results according to options selected, how can i best do it, i cannot write a linq query like for each possibility and combination of options, but i cannot write one for all of them as it will not work, for example:
from p in context.people where p.location==model.location && p.availability==model.availability .... select p
In this case imagine availability is not selected and should not be searched for, but in this case it will be passed as false, or if location is not set and is null so it will only search for empty locations, although i just need all.
So my question is how do people handle this kind of behaviour with queries?
As you long as you do not execute the linq query immediately you can just add where clauses to it. You can do this for example:
var query = from p in context.people;
if(searchOnLocation)
{
query = query.where(p => p.location == model.location);
}
if(otherSearch)
{
query = query.where(p => p.someOtherProperty == someotherValue);
}
var result = query.ToList();
As long you don't call ToList() on your IQueryable, the linq will not be translated into SQL. It's only in the last call, that the linq will be translated and executed against the database
IQueryable<Person> query = context.people;
if(model.location != null)
query = query.Where(x => x.location == model.location);
if(model.availability != null)
query = query.Where(x => x.availability == model.availability);
// etc
Basically, you can compose more and more restrictions as you go.
If you want to implement query without if condition than you can use following syntax:
var query = context.people.
where(p => p.location == (model.location ?? p.location)
&& p.availability == (model.availability ?? p.availability))
.ToList();
I am looking for some tips and tricks how to build dynamic queries. I am having an application which lets the user search 10 fields in the database table. Depending on which fields in the UI are filled with a value the query should search in an additional field in the DB.
Currently I am trying to build the query using StringBuilder and adding the where clause but I really don't like this and I am wondering if there is a better way to do this, for example with LINQ if possible.
Maybe someone can bring up ideas or better some example code. Thanks and have a nice day!
With LINQ it is pretty trivial:
IQueryable<User> users = db.Users;
if(name != null) users = users.Where(u => u.Name == name);
if(dept != null) users = users.Where(u => u.Dept == dept);
...
var page = users.OrderBy(u => u.Name).Take(100).ToList();
Each successive Where composes the query with more filters; exactly what you want.
With raw TSQL, StringBuilder isn't unreasonable; just make sure that you fully parameterize it. This might mean adding parameters in each term; for example:
...
if(name != null) {
sql.Append(" and u.Name = #name");
cmd.Parameters.AddWithValue("name", name);
}
if(dept != null) {
sql.Append(" and u.Dept = #dept");
cmd.Parameters.AddWithValue("dept", dept);
}
...
I have a piece of code that's performing badly, and need to rewite it to introduce a proper where clause before starting the .ToList however, that's where I'm getting stuck.
Currently the code looks lke this (roughly, I've taken some of the search criteria out to make it easier to display)
var Widgets = from b in _caspEntities.Widgets.Include("WidgetRegionLogs")
.Include("WidgetStatusLogs").Include("WidgetVoltageTests")
select b;
IEnumerable<Widget> results = Widgets.ToList();
if (comboBoxRegion.SelectedValue.ToString() != "0")
{
results = from b in results
where b.CurrentRegionLog != null && b.CurrentRegionLog.RegionId == int.Parse(comboBoxRegion.SelectedValue.ToString())
select b;
}
if (comboBoxStatus.SelectedValue != null)
{
results = from b in results
where b.CurrentStatusLog != null && b.CurrentStatusLog.StatusId == comboBoxStatus.SelectedValue.ToString()
select b;
}
if (txtCode.Text.Trim().Length > 0)
{
results = from b in results
where b.CodeNumber == txtCode.Text.Trim()
select b;
}
dataGridViewWidget.DataSource = results.ToList();
I can write the SQL easily enough, essentially the model is simple, I have a Widget it has a RegionLog and a StatusLog, both of which store a history. The current region and status are retrieved from this by grouping by WidgetID and selecting the most recent Date Updated (and then going off to Region and Status tables to get the actual value).
So, I need to translate this into LINQ, but to be honest I don't have a clue but am ken and willing to learn. In my head, I think I need to add some better where clauses, and then do the Widget.toList after I have applied the where clauses. I'm struggling with the CurrentRegionLog and CurrentStatusLog concepts as they are not populated until I run the IEnumerable.
If anyone can give some pointers, I'd be grateful,
Thanks
Edit - Added
public BatteryRegionLog CurrentRegionLog
{
get { return _currentRegionLog; }
}
private BatteryRegionLog _currentRegionLog
{
get
{
if (this.BatteryRegionLogs.Count > 0)
{
BatteryRegionLog log = this.BatteryRegionLogs.OrderByDescending(item => item.LastModifiedDate).First();
return log;
}
else
{
return null;
}
}
}
You can compose the query like this:
if (comboBoxRegion.SelectedValue.ToString() != "0")
{
var id = int.Parse(comboBoxRegion.SelectedValue.ToString()
Widgets = from b in Widgets
let currentRegionLog =
b.BatteryRegionLogs
.OrderByDescending(item => item.LastModifiedDate)
.FirstOrDefault()
where currentRegionLog.RegionId == id)
select b;
}
... // Same for the other criteria.
dataGridViewWidget.DataSource = Widgets.ToList();
The whole query is not executed before you do ToList(). As everything is translated to SQL you don't need the null check b.CurrentRegionLog != null. SQL will evaluate b.CurrentRegionLog.RegionId == id just fine when there is no CurrentRegionLog.
Edit
Since CurrentRegionLog is a calculated property of your Widget class it cannot be translated into SQL. I made an effort to incorporate the code of calculated property into the query in a way that only the basic navigation property is used, so EF can translate it to SQL again.
try remove this line:
IEnumerable<Widget> results = Widgets.ToList();
and just use the Widgets variable you get in at the top
The .ToList() goes to the database and materialiaze all the data into entities.
if you don't call the .ToList() the query is still "open" for a where clause
I need to get the list of names that starts with special characters or numbers in the linq to sql query for my asp.net mvc(C#) application.
I have tried like this (Which may not be efficient):
public List<User> GetUsersStartingWithNonCharacter()
{
List<User> _dbUsers = this.GetAllUsers();
return _dbUsers.Where(p => ((p.FirstName != null && p.FirstName != string.Empty && !char.IsLetter(p.FirstName.ToLower()[0])) || (p.FirstName == null || p.FirstName == string.Empty))).ToList();
}
public List<Users> GetAllUsers()
{
return (from u in _context.pu_Users
where !u.Is_Deleted
select new User
{
UserId = u.User_Id,
Email = u.Email_Address,
FirstName = u.First_Name,
LastName = u.Last_Name
}).ToList();
}
Can anyone suggest the most efficient way to do this in linq to sql?
How do you know if it isn't already efficient? Use somekind of profiler tool, like SQL Server Profiler if you're using MSSQL, that way you can trace your call against the database and see the actual SQL. Of course you can only debug the code to see the generated SQL but it's easier with a profiler tool and you'll see how long time the query takes.
EDIT: I see one part in how you can make it more efficient:
public List<User> GetUsersStartingWithNonCharacter()
{
List<User> _dbUsers = this.GetAllUsers();
return _dbUsers.Where(p => ((p.FirstName != null && p.FirstName != string.Empty && !char.IsLetter(p.FirstName.ToLower()[0])) || (p.FirstName == null || p.FirstName == string.Empty))).ToList();
}
public IQueryable<Users> GetAllUsers()
{
return from u in _context.pu_Users
where !u.Is_Deleted
select new User
{
UserId = u.User_Id,
Email = u.Email_Address,
FirstName = u.First_Name,
LastName = u.Last_Name
};
}
Changing your GetAllUsersto return IQueryable will delay the query to execute until you've applied your filters. This might affect some other aspects of your design but you should consider it since that change might make your where clause run in the database instead of in the code which will result in less data traffic between your application and database. Again, use a profiler to see the difference :).
I'll use Regular Expression in this scenerio
Here is my sample code
return _dbUsers.Where(p=>p.FirstName!=String.Empty)
. Where(p => Regex.Match(p.Firstname[0].ToString(), "[a-zA-Z]").Success).ToList();
I suspect all rows will be retrieved and filted in your application due to the condition:
char.IsLetter(p.FirstName.ToLower()[0])
(Using a regular expression like suggested in another answer will also pull in all rows, and filter them on the client.)
It is possible to check characters in a string with the PATINDEX function, but it's seems only to be available for LINQ via the Entity framework.
You could write a stored procedure using PATINDEX directly to check for the first character to optimize your query. Sample queries can be found at http://www.databasejournal.com/features/mssql/article.php/3071531/Using-SQL-Servers-CHARINDEX-and-PATINDEX.htm.
Sometimes LINQ to whatever will not yield the most optimized solution, but that's just life. In most cases it will give clearer code, but special cases might require work arounds in order to use special operators of the underlying system.