Develop Reference

Comparison of Linq to SQL and Stored procedures in terms of performance

I am using SQL Server 2005 database and my application is in ASP.NET MVC4.
The application's business logic is bit complex, containing multiple table joins and search conditions. In extreme conditions I need to join around 10 tables to fetch the data required for a single grid.
I would like to know whether I should use SPs or Linq to SQL application code to maximize my application performance.

SQL Server basically goes through these steps to execute any query (stored procedure call or ad-hoc SQL statement):
syntactically check the query
if it's okay - it checks the plan cache to see if it already has an execution plan for that query
if there is an execution plan - that plan is (re-)used and the query executed
if there is no plan yet, an execution plan is determined
that plan is stored into the plan cache for later reuse
the query is executed
The point is: ad-hoc SQL and stored procedures are treated no differently.
If an ad-hoc SQL query is properly using parameters - as it should anyway, to prevent SQL injection attacks - its performance characteristics are no different and most definitely no worse than executing a stored procedure.
Stored procedure have other benefits (no need to grant users direct table access, for instance), but in terms of performance, using properly parametrized ad-hoc SQL queries is just as efficient as using stored procedures.
Update: using stored procedures over non-parametrized queries is better for two main reasons:
since each non-parametrized query is a new, different query to SQL Server, it has to go through all the steps of determining the execution plan, for each query (thus wasting time - and also wasting plan cache space, since storing the execution plan into plan cache doesn't really help in the end, since that particular query will probably not be executed again)
non-parametrized queries are at risk of SQL injection attack and should be avoided at all costs

You are typically going to find faster performance with stored procedures because the stored procedures will re-use execution plans when possible. Linq queries essentially become ad-hoc queries against the database that without caching are processed as a new request each time.

Using SQL CLR to run a Client Side query in Server Side

I've created a Linq to Entities and I've used
Enumerable.Distinct<T>(this IEnumerable<T> source, IEqualityComparer<T> comparer)
in the query.
Due to my "Comparer" class the query run in the client side. It works as expected and I get the desired result but the query is slow because the involved tables has tons of rows so I was thinking if it's possible to use SQL CLR to implement the whole query including the Comprarer class in that way the whole query run in server side.
Is it possible?
Any idea is welcome.

SQL 2005 CLR only supports .Net 2.0 framwork by default. I have imported the .Net 3.5 framework into SQL server before, but it is somewhat messy and opens up some security holes (don't recall the all the details), but the quick and dirty is in this question -- in particular
CREATE ASSEMBLY [System.Core]
AUTHORIZATION [dbo]
FROM
'C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5\System.Core.dll'
WITH PERMISSION_SET = UNSAFE
The other roadblock that you typically encounter in CLR procs is the security context from the user running the stored proc is not inherited by the CLR proc, so trying to accomplish some things can be harder than you would expect.
Debugging a CLR proc is not as easy. You can do it, and I was never able to get a full stack backtrace (with line number, etc.). So, debug before you move it to CLR as much as possible.
Other than that, I have not had much trouble with CLR procs -- they work pretty well as long as you have a handle on possible performance issues by running complex code inside SQL, don't allocate lots of memory inside your CLR proc -- you will regret it.
ADDED
One other issue I thought of. CLR proc code writing needs a higher level of proficiency than typical client side code or stored procs. This is a maintenance consideration.
Updates to the CLR proc are also more complicated the client side code or sql procs, so this can also be a problem. This also complicates maintenance.
I'm not trying to discourage CLR procs, the upsides are also good -- performance, flexibility, security (have to have DB permissions to update). But, I was trying to documents the issues that they don't tell you about when you read about "how great and simple CLR procs are".
ADDED
You don't give detail, but if you are data bound (lots of rows) and you can write the logic as set-based TSQL performance is almost certain to be much better as set-based. TSQL is slow if you try to do lots of computation via TSQL -- scripting runs slow, database I/O runs fast. TSQL is not very flexible as a programming language, so CLR adds flexibility and faster code execution.
If you are not familiar with using APPLY in a select statement (Sql 2005+), you should take the time to understand as it can be very useful in keeping complex query "set based" -- you don't ever want to use a cursor if you can avoid it -- slow and chews up database resources.

You might save yourself some drag on sending the results across the wire, though if you are on a 1 GB network then it might not matter, especially since it is fairly vague as to what "tons of rows" means. Are we talking in the hundreds of thousands, or millions?
In either case, I am not sure that there is a clear understanding here as to what SQL CLR does based on the statement "use SQL CLR to implement the whole query including the Comprarer class in that way the whole query run in server side". You cannot write a query in SQL CLR. Creating .Net / CLR Stored Procedures and Functions does not replace T-SQL for interaction with the database. This means that you are still going to need to execute a SELECT statement and get the results back. Using LINQ to SQL within a SQL CLR object will still execute the same SQL as it does right now from the client.
If more details are provided as to the end goal of doing the comparison then it might be possible to plan a more appropriate solution. But given the question as asked, it seems doubtful that moving this code into SQL Server, assuming the comparison is still done in the .Net code, will provide much, if any, benefit.
EDIT:
To be clearer: transferring the business logic to run server-side in such a way as to avoid pulling in all rows into memory such that they can be compared via your custom comparer would require that you create a SQL CLR function and use it in a WHERE clause. So the model would be changed to essentially send one row at a time to the function for comparison rather than have all available in a collection.

insert directly or via a stored procedure

I am using sql server and winforms for my application. data would be inserted every minute into the database tables by pressing a button on a Form.
for this, I am using the INSERT query.
But if I create a procedure and include the same insert query in it, then would it be more efficient, what would be the difference then?

Using stored procedures is more secure

A stored procedure would generally be quicker as the query plan is stored and does not need to be created for each call. If this is a simple insert the difference would be minimal.
A stored procedure can be run with execute permissions which is more secure than giving insert permissions to the user.

It depends on what you mean by 'efficient'.
Execution time - if you're only saving to the database only every couple of seconds then any speed difference between SPs and INSERT is most likely insignificant. If the volume is especially high you would probably set up something like a command queue on the server before fine-tuning at this level.
Development time
using INSERT means you can write your SQL directly in your codebase (in a repository or similar). I've seen that described as poor design, but I think that as long as you have integration tests around the query there's no real problem
Stored Procedures can be more difficult to maintain - you need to have a plan to deploy the new SP to the database. Benefits are that you can implement finer-grained security on the database itself (as #b-rain and #mark_s have said) and it is easy to decide between INSERT and UPDATE within the SP, whereas to do the same in code means making certain assumptions.
Personally (at the moment) I use inline SQL for querying and deleting, and stored procedures for inserting. I have a script and a set of migration files that I can run against the production database to deploy table and SP changes, which seems to work pretty well. I also have integration tests around both the inline SQL and the SP calls. If you go for inline SQL you definitely should use parameterised queries, it helps against SQL injection attacks and it is also easier to read and program.

If your DBA is even allowing you to do this without a stored procedure I'd be very suspicious...

Can we convert all SQL scripts to Linq-to-SQL expressions or there is any limitation?

I want to convert all of my db stored procedures to linq to sql expressions, is there any limitation for this work? you must notice that there is some complicated queries in my db.

Several features of SQL Server are not supported by Linq to SQL:
Batch updates (unless you use non-standard extensions);
Table-Valued Parameters;
CLR types, including spatial types and hierarchyid;
DML statements (I'm thinking specifically of table variables and temporary tables);
The OUTPUT INTO clause;
The MERGE statement;
Recursive Common Table Expressions, i.e. hierarchical queries on a nested set;
Optimized paging queries using SET ROWCOUNT (ROW_NUMBER is not the most efficient);
Certain windowing functions like DENSE_RANK and NTILE;
Cursors - although these should obviously be avoided, sometimes you really do need them;
Analytical queries using ROLLUP, CUBE, COMPUTE, etc.
Statistical aggregates such as STDEV, VAR, etc.
PIVOT and UNPIVOT queries;
XML columns and integrated XPath;
...and so on...
With some of these things you could technically write your own extension methods, parse the expression trees and actually generate the correct SQL, but that won't work for all of the above, and even when it is a viable option, it will often simply be easier to write the SQL and invoke the command or stored procedure. There's a reason that the DataContext gives you the ExecuteCommand, ExecuteQuery and ExecuteMethodCall methods.
As I've stated in the past, ORMs such as Linq to SQL are great tools, but they are not silver bullets. I've found that for larger, database-heavy projects, L2S can typically handle about 95% of the tasks, but for that other 5% you need to write UDFs or Stored Procedures, and sometimes even bypass the DataContext altogether (object tracking does not play nice with server triggers).
For smaller/simpler projects it is highly probable that you could do everything in Linq to SQL. Whether or not you should is a different question entirely, and one that I'm not going to try to answer here.

I've found that in almost all cases where I've done a new project with L2S, I've completely removed the need for stored procedures. In fact, many of the cases where I would have been forced to use a stored proc, multivariable filters for instance, I've found that by building the query dynamically in LINQ, I've actually gotten better queries in the vast majority of cases since I don't need to include those parts of the query that get translated to "don't care" in the stored proc. So, from my perspective, yes -- you should be able to translate your stored procs to LINQ.
A better question, thought, might be should you translate your stored procs to LINQ? The answer to that, I think, depends on the state of the project, your relative expertise with C#/VB and LINQ vs SQL, the size of the conversion, etc. On an existing project I'd only make the effort if it improves the maintainability or extensibility of the code base, or if I was making significant changes and the new code would benefit. In the latter case you may choose to incrementally move your code to pure LINQ as you touch it to make changes. You can use stored procs with LINQ so you may not need to change it to make use of LINQ.

I'm not a fan of this approach. This is a major architectural change, because you are now removing a major interface layer you previously put in place to gain a decoupling advantage.
With stored procedures, you have already chosen the interface your database exposes. You will now need to grant users SELECT privileges on all the underlying tables/views instead of EXECUTE on just the application stored procedures and potentially you will need to restrict column read rights at the column level in the tables/views. Now you will need to re-implement at a lower level every explicit underlying table/view/column rights which your stored procedure was previously implementing with a single implicit EXECUTE right.
Whereas before the services expected from the database could be enumerated by an appropriate inventory of stored procedures, now the potential database operations are limited to the exposed tables/views/columns, vastly increasing the coupling and potential for difficulty in estimating scope changes for database refactorings and feature implementations.
Unless there are specific cases where the stored procedure interface is difficult to create/maintain, I see little benefit of changing a working SP-based architecture en masse. In cases where LINQ generates a better implementation because of application-level data coupling (for instance joining native collections to database), it can be appropriate. Even then, you might want to LINQ to the stored procedure on the database side.
If you chose LINQ from the start, you would obviously have done a certain amount of work up front in determining column/view/table permissions and limiting the scope of application code affecting database implementation details.

What does this mean? Does this mean you want to use L2S to call your stored procedures, or do you want to convert all the T-SQL statements in your stored procs to L2S? If it's the later, you should not have too many problems doing this. Most T-SQL statements can be represented in Linq without problem.
I might suggest you investigate a tool like Linqer to help you with your T-SQL conversion. It will convert most any T-SQL statement into Linq. It has saved my quite a bit of time in converting some of my queries.

There are many constructs in T-SQL which have no parallel in LINQ to SQL. Starting with flow control, ability to return multiple row sets, recursive queries.
You will need to approach this on a case by case basis. Remembering any times the SP does significant filtering work on the database much of that filtering may end up on the client, so needing to move far more data from server to client.

If you already have tested and working stored procedures, why convert them at all? That's just making work for no reason.
If you were starting a new product from scratch and were wondering whether to use stored procedures or not, that would be an entirely different question.

Which is faster for multiple SQL calls - parameterized queries or stored procedures?

I'm using SQL Server 2005. I'm looking at opening a SQL connection, looping though a collection and running an update query (stored procedure or parameterized query) with the data from the collection item, then closing the connection.
Which is going to give me better performance and why?

In recent versions of SQL server, execution plans are cached for stored procedures and parametrized queries. The query will have a one time cost to create the execution plan, but this will be very minor and lost in the overhead of even a few calls.
From a performance perspective they will be almost identical.
The exception to this, is if you are doing recursive or nested queries (for each row in query 1, execute query 2), where the round trips between the client and server will add up. In this case, a proc will be much better.
Many companies still have "Everything as a proc" rules, but this is usually due to control of the database and not performance related. As LINQ grows in popularity, this may lessen.
If your application has a single (or few) points of deployment, then use whichever you prefer. On the other hand, if you are deploying to many many installations, then stored procs can in some (but not all) cases allow you to update database functionality without redeploying the client code.

It is difficult to say with certainty as there are a number of factors that can effect performance. In theory the Stored Procedure method should be faster.
There is another solution where you can pass XML to the stored procedure so you don’t have to call it multiple times. Assuming you can and know how to serialize your object into XML. Then you can use Xquery, Open XML or sp_XML_preparedocument to retrieve your data in a set based manner from the XML and perform the update statement.

This can often turn into a religious debate between programmers and DBAs. Many programmers tend to like the prepared statement approach as it allows them complete control over the query that is being executed while DBAs like the stored procedure approach for the same reason. If you don't have that defined line between developer and DBA in your company and you dabble in both development and DBA roles then I would probably lean more toward the stored procedure route because if you need to make slight changes to the query in the future to fine tune performance or fix bugs then you don't have to recompile and redeploy your application.

Stored Procedures are usually the way to go. That said it also depends on how well your proc is written. Try running an Execution Plan on your Stored Procedure to make sure you're getting the best bang for your buck. Additionally using a Stored Procedure is usually the more secure way to go as well as the best performance on your server, provided the SQL instance is not on the same box as your code. When you use the stored procedure you put the load of the work on the SQL box, which will have optimized the query for you in the stored procedure.

Using a direct query or a stored procedure doesn't differ much in performance (if any), but if you are running the same query over and over with different data you should definitely use parameters.
Use the Prepare method to ensure that the same execution plan is reused:
Create the SqlCommand object with the query/procedure.
Create SqlParameter objects with specified data types but without values, and add them to the Parameters collection of the command.
Call the Prepare method on the command object.
Loop through the data just setting the parameter values and execute the command.

Usually stored procedures, because the server can pre-optimize the execution plan. Though well-written parameterized query is better than an over-general stored procedure.