This is going to top some of the weirdest things I've seen. I've tried looking up "simple t-sql delete causing timeout" but all titles are misleading, they say simple but are not. They deal with deleting millions of records or have complex relationships setup. I do not.
I have four tables:
tblInterchangeControl,
tblFunctionalGroup,
tblTransactionSet,
tblSegment
The latter 3 all associate to tblInterchangeConrol via InterchangeControlID. There are no relationships setup. like I said as simple as one could get.
The procedure runs a delete statement on all 4 tables like so...
DELETE FROM tblSegment
WHERE (ID_InterchangeControlID = #InterchangeControlID)
DELETE FROM tblTransactionSet
WHERE (ID_InterchangeControlID = #InterchangeControlID)
DELETE FROM tblFunctionalGroup
WHERE (ID_InterchangeControlID = #InterchangeControlID)
DELETE FROM tblInterchangeControl
WHERE (InterchangeControlID = #InterchangeControlID)
The weird part is if I leave these in the procedure it times out, if I remove them, it does not. I've pinned it to these delete statements that are the cause. But Why?!
I included c# because I'm calling this procedure from a c# application. I don't think this is the issue but maybe. I only say I don't think so because my code work just fine when I remove the delete statements inside the stored procedure. Then if I put them back, an exception is thrown that it's timed out.
In case my comment is the answer.
Most likely you have some locks holding those deletes up.
If you run a query from a command line SQL tool or from SQL Management Studio it will take whatever it needs to complete the query. So yes, most likely it's client part issue. And, because you mentioned c# it's probably ADO.NET command timeout.
Also, I'd suggest to profile the queries by inspecting their execution plans. In case you don't have indexes (primary/unique key constraints) this will result to full-scan, i.e. O(n) operation you don't want.
Update:
OK, looks like it's ADO.NET error. In your code, just prior executing the command increase the timeout:
var myCommand = new SqlCommand("EXEC ..."); // you create it something like this
....
myCommand.CommandTimeout = 300; // 5 minutes
myCommand.ExecuteNonReader(); // assuming your SP doesn't return anything
Related
In our .net application, we have a tool that allows you to type SQL in a browser and submit it, for testing. In this context, though, I need to be able to prevent testers from writing to specific tables. So, based on the parameter passed from the controller (InSupportTool = true, or something), I need to know if SQL Server is allowed to make updates or inserts to, say, an accounts table.
Things I've tried so far:
I have tried looking into triggers, but there is no before trigger available, and I've heard people don't recommend using them if you can help it.
Parsing the passed SQL string to look for references to inserting or updating on that table. This is even more fragile and has countless ways, I'm sure, of getting around it if someone wanted to.
Check constraint, which is the closest I feel I've gotten but I can't quite put it together.
For check constraints, I have this:
ALTER TABLE Accounts WITH NOCHECK
ADD CONSTRAINT chk_read_only_accounts CHECK(*somehow this needs to be dynamic based on parameters passed from C# controller*)
The above works to prevent updates to that table, but only if I put a check like 1 = 0. I've seen a post where people said you could use a function as the check, and pass parameters that way, but I'm at the limit of my familiarity with SQL/.net.
Given what I'm looking to do, does anyone have experience with something like this? Thanks!
Since the application is running under a different account than the end user, you could specify your application name in the connection string (e.g. Application Name=SupportTool) and check that in an after trigger, rolling back the transaction as needed:
CREATE TABLE dbo.example(
col1 int
);
GO
CREATE TRIGGER tr_example
ON dbo.example
AFTER INSERT, UPDATE, DELETE
AS
IF APP_NAME() = N'SupportTool'
BEGIN
ROLLBACK;
THROW 50000, 'This update is not allowed using the support tool', 1;
END;
GO
INSERT INTO dbo.example VALUES(1);
GO
In my program, I want to select some bookIDs into a tempDB for later queries like this (using Dapper extension):
using (var conn = new SqlConnection(connStr)) {
conn.Execute("SELECT bookID INTO #tempdb WHERE ... FROM Books");
int count = conn.ExecuteScalar<int>("SELECT COUNT(*) FROM #tempdb");
var authors = conn.Query("SELECT * FROM #tempdb LEFT JOIN BookAuthors ON ...");
}
However when I execute the page, I get following exception:
Invalid object name '#tempdb'.
It seems that life-cycle of #tempdb is only valid in first query ?
It looks like you're using the implicit connection opening/closing. This will indeed cause problems with transient objects. If you need temp tables between queries, you will need to manually open the connection before you execute any such queries. This should then work fine, and many examples in the test suite make use of temp tables in the way.
However, from a practical standpoint, making use of temporary tables to transfer state between queries is ... awkward. In addition to being brittle, it isn't good for the plan cache, as #foo has a different meaning between all uses on different connection (including reset but reused connections).
I found a previous poster who met the same problem and his solution.
Using dapper, why is a temp table created in one use of a connection not available in a second use of the same connection
The post indicates that you have to "CREATE TABLE #tempdb" explicitly in your SQL first and everything goes fine. Even the poster himself don't know why such style of coding works.
MiniProfiler detects "duplicate" SQL queries and gives you a red exclamation mark in the corner when it detects them (I'm using MiniProfilerEF to link in to my code-first EF database access). This is often very useful, but sometimes these duplicate queries are fully justified, like say when I'm updating 50 users in a parametrized way; 49 are going to be detected as duplicate queries.
Is there a way, then, for me to disable this duplicate SQL query detection for a block of code? Something like:
using (MiniProfiler.Current.Step("Do bulk updates"), Options.NoDuplicateSqlChecking) {
// ...
}
... would be good.
Looking at the MiniProfiler code, this line seems to be checking for duplicates:
https://github.com/SamSaffron/MiniProfiler/blob/master/StackExchange.Profiling/MiniProfiler.IDbProfiler.cs#L91
Doesn't look like there's any way to do this at the moment. I guess stats.IsDuplicate would never have to be set to true if this option were specified, but it's tricky only doing it for a particular block of code. Alternatively maybe the parameters as well as the SQL command string could be taken into account somehow, as they are likely to change, even with bulk operations.
Just found the answer to my own question; you can use MiniProfiler.Current.Ignore() to stop profiling of certain sections of code:
using (MiniProfiler.Current.Ignore()) {
// ...
}
This pretty much does what I want; the only slight annoyance is that it stops profiling altogether. I'd rather it kept listing the SQL queries but didn't label them as duplicates, but oh well.
I am building an application and I want to batch multiple queries into a single round-trip to the database. For example, lets say a single page needs to display a list of users, a list of groups and a list of permissions.
So I have stored procs (or just simple sql commands like "select * from Users"), and I want to execute three of them. However, to populate this one page I have to make 3 round trips.
Now I could write a single stored proc ("getUsersTeamsAndPermissions") or execute a single SQL command "select * from Users;exec getTeams;select * from Permissions".
But I was wondering if there was a better way to specify to do 3 operations in a single round trip. Benefits include being easier to unit test, and allowing the database engine to parrallelize the queries.
I'm using C# 3.5 and SQL Server 2008.
Something like this. The example is probably not very good as it doesn't properly dispose objects but you get the idea. Here's a cleaned up version:
using (var connection = new SqlConnection(ConnectionString))
using (var command = connection.CreateCommand())
{
connection.Open();
command.CommandText = "select id from test1; select id from test2";
using (var reader = command.ExecuteReader())
{
do
{
while (reader.Read())
{
Console.WriteLine(reader.GetInt32(0));
}
Console.WriteLine("--next command--");
} while (reader.NextResult());
}
}
The single multi-part command and the stored procedure options that you mention are the two options. You can't do them in such a way that they are "parallelized" on the db. However, both of those options does result in a single round trip, so you're good there. There's no way to send them more efficiently. In sql server 2005 onwards, a multi-part command that is fully parameterized is very efficient.
Edit: adding information on why cram into a single call.
Although you don't want to care too much about reducing calls, there can be legitimate reasons for this.
I once was limited to a crummy ODBC driver against a mainframe, and there was a 1.2 second overhead on each call! I'm serious. There were times when I crammed a little extra into my db calls. Not pretty.
You also might find yourself in a situation where you have to configure your sql queries somewhere, and you can't just make 3 calls: it has to be one. It shouldn't be that way, bad design, but it is. You do what you gotta do!
Sometimes of course it can be very good to encapsulate multiple steps in a stored procedure. Usually not for saving round trips though, but for tighter transactions, getting ID for new records, constraining for permissions, providing encapsulation, blah blah blah.
Making one round-trip vs three will be more eficient indeed. The question is wether it is worth the trouble. The entire ADO.Net and C# 3.5 toolset and framework opposes what you try to do. TableAdapters, Linq2SQL, EF, all these like to deal with simple one-call==one-resultset semantics. So you may loose some serious productivity by trying to beat the Framework into submission.
I would say that unless you have some serious measurements showing that you need to reduce the number of roundtrips, abstain. If you do end up requiring this, then use a stored procedure to at least give an API kind of semantics.
But if your query really is what you posted (ie. select all users, all teams and all permissions) then you obviosuly have much bigger fish to fry before reducing the round-trips... reduce the resultsets first.
I this this link might be helpful.
Consider using at least the same connection-openning; according to what it says here, openning a connection is almost the top-leader of performance cost in Entity-Framework.
Firstly, 3 round trips isn't really a big deal. If you were talking about 300 round trips then that would be another matter, but for just 3 round trips I would conderer this to definitley be a case of premature optimisation.
That said, the way I'd do this would probably be to executed the 3 stored procuedres using SQL:
exec dbo.p_myproc_1 #param_1 = #in_param_1, #param_2 = #in_param_2
exec dbo.p_myproc_2
exec dbo.p_myproc_3
You can then iterate through the returned results sets as you would if you directly executed multiple rowsets.
Build a temp-table? Insert all results into the temp table and then select * from #temp-table
as in,
#temptable=....
select #temptable.field=mytable.field from mytable
select #temptable.field2=mytable2.field2 from mytable2
etc... Only one trip to the database, though I'm not sure it is actually more efficient.
I have a CSV file and I have to insert it into a SQL Server database. Is there a way to speed up the LINQ inserts?
I've created a simple Repository method to save a record:
public void SaveOffer(Offer offer)
{
Offer dbOffer = this.db.Offers.SingleOrDefault (
o => o.offer_id == offer.offer_id);
// add new offer
if (dbOffer == null)
{
this.db.Offers.InsertOnSubmit(offer);
}
//update existing offer
else
{
dbOffer = offer;
}
this.db.SubmitChanges();
}
But using this method, the program is way much slower then inserting the data using ADO.net SQL inserts (new SqlConnection, new SqlCommand for select if exists, new SqlCommand for update/insert).
On 100k csv rows it takes about an hour vs 1 minute or so for the ADO.net way. For 2M csv rows it took ADO.net about 20 minutes. LINQ added about 30k of those 2M rows in 25 minutes. My database has 3 tables, linked in the dbml, but the other two tables are empty. The tests were made with all the tables empty.
P.S. I've tried to use SqlBulkCopy, but I need to do some transformations on Offer before inserting it into the db, and I think that defeats the purpose of SqlBulkCopy.
Updates/Edits:
After 18hours, the LINQ version added just ~200K rows.
I've tested the import just with LINQ inserts too, and also is really slow compared with ADO.net. I haven't seen a big difference between just inserts/submitchanges and selects/updates/inserts/submitchanges.
I still have to try batch commit, manually connecting to the db and compiled queries.
SubmitChanges does not batch changes, it does a single insert statement per object. If you want to do fast inserts, I think you need to stop using LINQ.
While SubmitChanges is executing, fire up SQL Profiler and watch the SQL being executed.
See question "Can LINQ to SQL perform batch updates and deletes? Or does it always do one row update at a time?" here: http://www.hookedonlinq.com/LINQToSQLFAQ.ashx
It links to this article: http://www.aneyfamily.com/terryandann/post/2008/04/Batch-Updates-and-Deletes-with-LINQ-to-SQL.aspx that uses extension methods to fix linq's inability to batch inserts and updates etc.
Have you tried wrapping the inserts within a transaction and/or delaying db.SubmitChanges so that you can batch several inserts?
Transactions help throughput by reducing the needs for fsync()'s, and delaying db.SubmitChanges will reduce the number of .NET<->db roundtrips.
Edit: see http://www.sidarok.com/web/blog/content/2008/05/02/10-tips-to-improve-your-linq-to-sql-application-performance.html for some more optimization principles.
Have a look at the following page for a simple walk-through of how to change your code to use a Bulk Insert instead of using LINQ's InsertOnSubmit() function.
You just need to add the (provided) BulkInsert class to your code, make a few subtle changes to your code, and you'll see a huge improvement in performance.
Mikes Knowledge Base - BulkInserts with LINQ
Good luck !
I wonder if you're suffering from an overly large set of data accumulating in the data-context, making it slow to resolve rows against the internal identity cache (which is checked once during the SingleOrDefault, and for "misses" I would expect to see a second hit when the entity is materialized).
I can't recall 100% whether the short-circuit works for SingleOrDefault (although it will in .NET 4.0).
I would try ditching the data-context (submit-changes and replace with an empty one) every n operations for some n - maybe 250 or something.
Given that you're calling SubmitChanges per isntance at the moment, you may also be wasting a lot of time checking the delta - pointless if you've only changed one row. Only call SubmitChanges in batches; not per record.
Alex gave the best answer, but I think a few things are being over looked.
One of the major bottlenecks you have here is calling SubmitChanges for each item individually. A problem I don't think most people know about is that if you haven't manually opened your DataContext's connection yourself, then the DataContext will repeatedly open and close it itself. However, if you open it yourself, and then close it yourself when you're absolutely finished, things will run a lot faster since it won't have to reconnect to the database every time. I found this out when trying to find out why DataContext.ExecuteCommand() was so unbelievably slow when executing multiple commands at once.
A few other areas where you could speed things up:
While Linq To SQL doesn't support your straight up batch processing, you should wait to call SubmitChanges() until you've analyzed everything first. You don't need to call SubmitChanges() after each InsertOnSubmit call.
If live data integrity isn't super crucial, you could retrieve a list of offer_id back from the server before you start checking to see if an offer already exists. This could significantly reduce the amount of times you're calling the server to get an existing item when it's not even there.
Why not pass an offer[] into that method, and doing all the changes in cache before submitting them to the database. Or you could use groups for submission, so you don't run out of cache. The main thing would be how long till you send over the data, the biggest time wasting is in the closing and opening of the connection.
Converting this to a compiled query is the easiest way I can think of to boost your performance here:
Change the following:
Offer dbOffer = this.db.Offers.SingleOrDefault (
o => o.offer_id == offer.offer_id);
to:
Offer dbOffer = RetrieveOffer(offer.offer_id);
private static readonly Func<DataContext, int> RetrieveOffer
{
CompiledQuery.Compile((DataContext context, int offerId) => context.Offers.SingleOrDefault(o => o.offer_id == offerid))
}
This change alone will not make it as fast as your ado.net version, but it will be a significant improvement because without the compiled query you are dynamically building the expression tree every time you run this method.
As one poster already mentioned, you must refactor your code so that submit changes is called only once if you want optimal performance.
Do you really need to check if the record exist before inserting it into the DB. I thought it looked strange as the data comes from a csv file.
P.S. I've tried to use SqlBulkCopy,
but I need to do some transformations
on Offer before inserting it into the
db, and I think that defeats the
purpose of SqlBulkCopy.
I don't think it defeat the purpose at all, why would it? Just fill a simple dataset with all the data from the csv and do a SqlBulkCopy. I did a similar thing with a collection of 30000+ rows and the import time went from minutes to seconds
I suspect it isn't the inserting or updating operations that are taking a long time, rather the code that determines if your offer already exists:
Offer dbOffer = this.db.Offers.SingleOrDefault (
o => o.offer_id == offer.offer_id);
If you look to optimise this, I think you'll be on the right track. Perhaps use the Stopwatch class to do some timing that will help to prove me right or wrong.
Usually, when not using Linq-to-Sql, you would have an insert/update procedure or sql script that would determine whether the record you pass already exists. You're doing this expensive operation in Linq, which certainly will never hope to match the speed of native sql (which is what's happening when you use a SqlCommand and select if the record exists) looking-up on a primary key.
Well you must understand linq creates code dynamically for all ADO operations that you do instead handwritten, so it will always take up more time then your manual code. Its simply an easy way to write code but if you want to talk about performance, ADO.NET code will always be faster depending upon how you write it.
I dont know if linq will try to reuse its last statement or not, if it does then seperating insert batch with update batch may improve performance little bit.
This code runs ok, and prevents large amounts of data:
if (repository2.GeoItems.GetChangeSet().Inserts.Count > 1000)
{
repository2.GeoItems.SubmitChanges();
}
Then, at the end of the bulk insertion, use this:
repository2.GeoItems.SubmitChanges();