I got this exception msg:
Transaction (Process ID 55) was deadlocked on lock resources with another process and has been chosen as the deadlock victim.
The only line of my code that was implicated in the Stack Trace was the last one here:
public static DataTable ExecuteSQLReturnDataTable(string sql, CommandType cmdType, params SqlParameter[] parameters)
{
using (DataSet ds = new DataSet())
using (SqlConnection connStr = new SqlConnection(CPSConnStr))
using (SqlCommand cmd = new SqlCommand(sql, connStr))
{
cmd.CommandType = cmdType;
cmd.CommandTimeout = EXTENDED_TIMEOUT;
foreach (var item in parameters)
{
cmd.Parameters.Add(item);
}
try
{
cmd.Connection.Open();
new SqlDataAdapter(cmd).Fill(ds);
This is a general-purpose method that I use for all sorts of queries; I haven't changed it recently, nor have I ever seen this particular exception before.
What can I do to guard against this exception being thrown again?
You can catch the deadlock exception and retry X number of times before giving up.
There's no magic solution to avoid deadlocks. If SQL Server detects a deadlock it's going to pick one of the processes to kill. In some cases you may have had deadlocks where your process was the one that was lucky enough to continue.
You can use SQL Profiler to capture the deadlocks. I had to do this in the past to try and figure out what was actually causing the deadlocks. The less often this happens the harder it is to track down. In our test environment we just created some testing code to hammer the database from a few different machines to try and cause a deadlock.
In our case we made some changes to our indexes and modified database triggers to reduce the deadlocks as best we could. In the end we still had to implement the retries as a "just in case".
It might have helped if you had shown the SQL that was passed to ExecuteSQLReturnDataTable. Meanwhile read Minimizing Deadlocks.
Of course, you may have to also look at whatever else is contributing to the deadlock.
Related
I'm working on an ASP.NET application where, as part of some logic, I want to lock some tables and do work on them. The method runs in a separate thread running as a kind of background task, spawned via a Task. The problem comes in with the error handling...
The code looks more or less like this:
MySqlConnection connection = new MySqlConnection(ConfigurationManager.AppSettings["prDatabase"]);
try
{
connection.Open();
MySqlCommand lock_tables = new MySqlCommand(Queries.lockTables(), connection);
lock_tables.ExecuteNonQuery();
// do a bunch of work here
MySqlCommand unlock_tables = new MySqlCommand(Queries.unlockTables(), connection);
unlock_tables.ExecuteNonQuery();
}
catch (MySqlException mex)
{
// Mostly error logging here
}
finally
{
connection.Close();
}
Pretty simple stuff. Everything works fine and dandy assuming nothing goes wrong. That's a terrible assumption to make, though, so I deliberately set up a situation where things would foul up in the middle and move to the finally block.
The result was that my table locks remained until I closed the app, which I learned by trying to access the tables with a different client once the method completed. Needless to say this isn't my intention, especially since there's another app that's supposed to access those tables once I'm done with them.
I could quickly fix the problem by explicitly releasing the locks before closing the connection, but I'm still left curious about some things. Everything I've read before has sworn that closing a connection should implicitly release the table locks. Obviously in this case it isn't. Why is that? Does connection.Close() not actually completely close the connection? Is there a better way I should be closing my connections?
Try wrapping your Connection and MySqlCommand instance in a using statement. That will release the objects as soon as it leaves the brackets.
using(MySqlConnection conn = new MySqlConnection(connStr))
{
conn.Open();
using(MySqlCommand command = new MySqlCommand("command to execute",conn))
{
//Code here..
}
}
I have five threads. They are doing OracleBulkCopy(1 million records each) into the same table (EXCEL_DATA) at same time. But at some point in time I am getting below error:
ORA-00604: error occurred at recursive SQL level 1
ORA-00054: resource busy and acquire with NOWAIT specified
I am using below code for OracleBulkCopy:
using (OracleConnection con = new OracleConnection(ConnectionString))
{
con.Open();
using (var bulkcopy = new OracleBulkCopy(con, options))
{
OracleTransaction tran =
con.BeginTransaction(IsolationLevel.ReadCommitted);
bulkcopy.DestinationTableName = DestinationTable;
foreach (var mapping in columnMappings)
bulkcopy.ColumnMappings.Add(mapping);
bulkcopy.BulkCopyTimeout = TimeOut.Value;
try
{
bulkcopy.WriteToServer(dataTable);
tran.Commit();
}
catch (Exception ex)
{
tran.Rollback();
}
}
}
It sounds like the table or a section is locked (quite reasonable during a bulk-copy, especially since you have an explicit transaction), and that is blocking other inserts from competing bulk-copies. That doesn't sound very surprising. The best thing I can say is... "don't do that". In particular, this is an IO-bound operation, with your main blockage very likely to be the network, with the secondary limit being the back-end server - which is also required to observe the ACID rules you have specified. For these reasons, doing these operations in parallel is not likely to give any significant performance benefits, but is very likely to cause timeouts due to blocking.
So: instead of doing these in parallel... do them in series.
I have these two exceptions generated when I try to get data from SQL database in C#:
System.Data.SqlClient.SqlException: Transaction (Process ID 97) was deadlocked on lock resources with another process and has been chosen as the deadlock victim.
OR
System.Data.SqlClient.SqlException: Transaction (Process ID 62) was deadlocked on lock resources with another process and has been chosen as the deadlock victim.
OR
System.Data.SqlClient.SqlException: Transaction (Process ID 54) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
this is the code:
using (SqlConnection con = new SqlConnection(datasource))
{
SqlCommand cmd = new SqlCommand("Select * from MyTable Where ID='1' ", con);
cmd.CommandTimeout = 300;
con.Open();
SqlDataAdapter adapter = new SqlDataAdapter(cmd);
DataSet ds = new DataSet();
adapter.Fill(ds);
con.Close();
return ds.Tables[0];
}
These happened every time.
Any ideas on how these can be resolved?
There are a couple of things you can do to lessen the number of deadlocks you receive, and some things you can do to completely eliminate them.
First off, launch SQL Server Profiler and tell it to give you a deadlock graph. Running this trace will tell you the other query which is conflicting with yours. Your query is quite simple, though I seriously doubt you have a SELECT * query off a table called MyTable in your system...
Anyway, armed with the deadlock graph and the other query, you should be able to tell what resources are deadlocking. The classic solution is to change the order of both queries such that the resources are accessed in the same order -- this avoids cycles.
Other things you can do:
Speed up your queries by, among other things, applying the correct indexes to them.
Enable snapshot isolation on the database and use SET TRANSACTION ISOLATION LEVEL SNAPSHOT in your transactions where appropriate. Also enable read committed with row-versioning. In many cases, this is enough to eliminate most deadlocks completely. Read about transaction isolation levels. Understand what you're doing.
Not that this is going to help the deadlock issue, but you should be disposing your other IDisposable objects much like you're disposing your SqlConnection as such:
using (SqlConnection con = new SqlConnection(datasource))
using (SqlCommand cmd = new SqlCommand("Select * from MyTable Where ID='1' ", con))
{
cmd.CommandTimeout = 300;
con.Open();
using (SqlDataAdapter adapter = new SqlDataAdapter(cmd))
using (DataSet ds = new DataSet())
{
adapter.Fill(ds);
return ds.Tables[0];
}
}
You might be able to avoid the lock with a locking hint in your query thusly:
Select * from MyTable with (nolock) Where ID='1'
I want to be clear though, you're allowing for reads of uncommitted data with this solution. It's a risk in a transactional system. Read this answer. Hope this helps.
Basically, the SQL server concurrency model makes it so you can never avoid this exception (eg. completely unrelated transaction might block eachother if they happen to lock the same index page or something). The best you can do is keep your transactions short to reduce the likelyhood, and if you get the exception, do what it says and retry the transaction.
I have the following code:
using (SqlConnection sqlConnection = new SqlConnection("blahblah;Asynchronous Processing=true;")
{
using (SqlCommand command = new SqlCommand("someProcedureName", sqlConnection))
{
sqlConnection.Open();
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("#param1", param1);
command.BeginExecuteNonQuery();
}
}
I never call EndExecuteNonQuery.
Two questions, first will this block because of the using statements or any other reason? Second, will it break anything? Like leaks or connection problems? I just want to tell sql server to run a stored procedure, but I don't want to wait for it and I don't even care if it works. Is that possible? Thanks for reading.
This won't work because you're closing the connection while the query is still running. The best way to do this would be to use the threadpool, like this:
ThreadPool.QueueUserWorkItem(delegate {
using (SqlConnection sqlConnection = new SqlConnection("blahblah;Asynchronous Processing=true;") {
using (SqlCommand command = new SqlCommand("someProcedureName", sqlConnection)) {
sqlConnection.Open();
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("#param1", param1);
command.ExecuteNonQuery();
}
}
});
In general, when you call Begin_Whatever_, you usually must call End_Whatever_ or you'll leak memory. The big exception to this rule is Control.BeginInvoke.
You can't close the connection after you submit the BeginExceuteNotQuery. It will abort the execution. Remove the using block.
In order to close the connection, you must know when the call has completed. For that you must call EndExecuteNonQuery, usually from a callback:
.
command.BeginExecuteNonQuery(delegate (IAsyncResult ar) {
try { command.EndExecuteNonQuery(ar); }
catch(Exception e) { /* log exception e */ }
finally { sqlConnection.Dispose(); }
}, null);
If you want to submit a query and don't care about the results, see Asynchronous T-SQL execution for a reliable pattern that ensures execution even if client diconnects or crashes.
You should always call the EndExecuteNonQuery() method to prevent leaks. It may work now but who knows what will happen in future versions of .NET. The general rule is always follow a BeginExecute... with an EndExecute...
I know this is an old post; just adding my 2c based on our recent (very conclusive) implementation and testing :D
To answer the OP's questions:
If you don't call EndExecuteNonQuery, BeginExecuteNonQuery will execute the procedure, but the operation will be cancelled as soon as the using clause disposes of your sql connection. Hence this is not plausible.
If you call BeginExecuteNonQuery by using a delegate, creating a new thread etc and you do not call EndExecuteNonQuery, chances are good you might get memory leaks depending on what takes place in you stored procedure. (More on this later).
Calling an stored procedure and not waiting for the call to complete, as far I our testing went, is not possible. Irrespective of multitasking, something somewhere will have to wait.
On to our solution:
Refs: BeginExecuteNonQuery -> BENQ, EndExecuteNonQuery -> EENQ
Use Case:
We have a windows service (C#) that makes use of the .Net TPL library. We needed to load data with a stored procedure from one database to another at run time, based on a add hoc request that the service picks up. Our stored procedure had an internal transaction and exception handling with try catch blocks.
First Try:
For our first try we implemented a solution found here MS Solution in this example you will see that MS opts to call BENQ then implements a while loop to block execution and then calls EENQ. This solution was mainly implemented if you don't need a callback method. The problem with this solution is that only BENQ is ignorant to sql connection timeouts. EENQ will timeout. So for a long running query (which is hopefully the reason why you are using BENQ) you will get stuck in the while and once the operation has completed and you call EENQ, you will get an sql timeout connection.
Second Try:
For our second try we thought ok so lets call BENQ, then add a while so that we don't close our sql connection and never call EENQ. This worked, until an exception was thrown in our stored procedure. Because we never called EENQ, the operation was never completed and the exception never bubbled up to our code. Hence we were stuck in a loop/thread/memory leak forever.
Third Try: (The Solution)
For our third try we thought to call BENQ, then directly after call EENQ. What happened was that EENQ effectively blocked execution in the thread until the operation completed. When an exception occurred in the stored procedure it was caught. When the query ran long EENQ did not throw a timeout exception and in all cases our sql connection object was disposed as well as our thread.
Here are some extracts of our code:
Here we open up a new thread for the method that calls the stored procedure.
//Call the load data stored procedure. As this stored procedure can run longer we start it in its own thread.
Task.Factory.StartNew(() => ClassName.MethodName(Parameters));
This is the code inside the method we use to call the stored procedure.
//Because this is a long running stored procedure, we start is up in a new thread.
using (SqlConnection conn = new SqlConnection(ConfigurationManager.ConnectionStrings[ConfigurationManager.AppSettings["ConnectionStringName"]].ConnectionString))
{
try
{
//Create a new instance SqlCommand.
SqlCommand command = new SqlCommand(ConfigurationManager.AppSettings["StoredProcedureName"], conn);
//Set the command type as stored procedure.
command.CommandType = CommandType.StoredProcedure;
//Create input parameters.
command.Parameters.Add(CreateInputParam("#Param1", SqlDbType.BigInt, Param1));
command.Parameters.Add(CreateInputParam("#Param2", SqlDbType.BigInt, Param3));
command.Parameters.Add(CreateInputParam("#Param3", SqlDbType.BigInt, Param3));
//Open up the sql connection.
conn.Open();
//Create a new instance of type IAsyncResult and call the sp asynchronously.
IAsyncResult result = command.BeginExecuteNonQuery();
//When the process has completed, we end the execution of the sp.
command.EndExecuteNonQuery(result);
}
catch (Exception err)
{
//Write to the log.
}
}
I hope this answer save's someone some headache :D We have tested this thoroughly and have not experienced any issues.
Happy coding!
In this case the using statements won't be necessary because you should manually close it yourself rather than allowing the syntactic sugar dispose it for you (i.e. at the }).
It should be as simple as this to ensure you don't have leaks.
using (SqlConnection sqlConnection = new SqlConnection("blahblah;Asynchronous Processing=true;")
{
using (SqlCommand command = new SqlCommand("someProcedureName", sqlConnection))
{
sqlConnection.Open();
command.CommandType = CommandType.StoredProcedure;
command.Parameters.AddWithValue("#param1", param1);
command.BeginExecuteNonQuery((ar) =>
{
var cmd = (SqlCommand)ar.AsyncState;
cmd.EndExecuteNonQuery(ar);
cmd.Connection.Close();
}, command);
}
}
As you can see the lambda expression that is fired once the command is finished (no matter how long it takes) will do all the closing for you.
I'm performing a large number of INSERTS to a SQLite database. I'm using just one thread. I batch the writes to improve performance and have a bit of security in case of a crash. Basically I cache up a bunch of data in memory and then when I deem appropriate, I loop over all of that data and perform the INSERTS. The code for this is shown below:
public void Commit()
{
using (SQLiteConnection conn = new SQLiteConnection(this.connString))
{
conn.Open();
using (SQLiteTransaction trans = conn.BeginTransaction())
{
using (SQLiteCommand command = conn.CreateCommand())
{
command.CommandText = "INSERT OR IGNORE INTO [MY_TABLE] (col1, col2) VALUES (?,?)";
command.Parameters.Add(this.col1Param);
command.Parameters.Add(this.col2Param);
foreach (Data o in this.dataTemp)
{
this.col1Param.Value = o.Col1Prop;
this. col2Param.Value = o.Col2Prop;
command.ExecuteNonQuery();
}
}
this.TryHandleCommit(trans);
}
conn.Close();
}
}
I now employ the following gimmick to get the thing to eventually work:
private void TryHandleCommit(SQLiteTransaction trans)
{
try
{
trans.Commit();
}
catch (Exception e)
{
Console.WriteLine("Trying again...");
this.TryHandleCommit(trans);
}
}
I create my DB like so:
public DataBase(String path)
{
//build connection string
SQLiteConnectionStringBuilder connString = new SQLiteConnectionStringBuilder();
connString.DataSource = path;
connString.Version = 3;
connString.DefaultTimeout = 5;
connString.JournalMode = SQLiteJournalModeEnum.Persist;
connString.UseUTF16Encoding = true;
using (connection = new SQLiteConnection(connString.ToString()))
{
//check for existence of db
FileInfo f = new FileInfo(path);
if (!f.Exists) //build new blank db
{
SQLiteConnection.CreateFile(path);
connection.Open();
using (SQLiteTransaction trans = connection.BeginTransaction())
{
using (SQLiteCommand command = connection.CreateCommand())
{
command.CommandText = DataBase.CREATE_MATCHES;
command.ExecuteNonQuery();
command.CommandText = DataBase.CREATE_STRING_DATA;
command.ExecuteNonQuery();
//TODO add logging
}
trans.Commit();
}
connection.Close();
}
}
}
I then export the connection string and use it to obtain new connections in different parts of the program.
At seemingly random intervals, though at far too great a rate to ignore or otherwise workaround this problem, I get unhandled SQLiteException: Database file is locked. This occurs when I attempt to commit the transaction. No errors seem to occur prior to then. This does not always happen. Sometimes the whole thing runs without a hitch.
No reads are being performed on these files before the commits finish.
I have the very latest SQLite binary.
I'm compiling for .NET 2.0.
I'm using VS 2008.
The db is a local file.
All of this activity is encapsulated within one thread / process.
Virus protection is off (though I think that was only relevant if you were connecting over a network?).
As per Scotsman's post I have implemented the following changes:
Journal Mode set to Persist
DB files stored in C:\Docs + Settings\ApplicationData via System.Windows.Forms.Application.AppData windows call
No inner exception
Witnessed on two distinct machines (albeit very similar hardware and software)
Have been running Process Monitor - no extraneous processes are attaching themselves to the DB files - the problem is definitely in my code...
Does anyone have any idea whats going on here?
I know I just dropped a whole mess of code, but I've been trying to figure this out for way too long. My thanks to anyone who makes it to the end of this question!
brian
UPDATES:
Thanks for the suggestions so far! I've implemented many of the suggested changes. I feel that we are getting closer to the answer...however...
The code above technically works however it is non-deterministic! It is not guaranteed to do anything aside from spin in neutral forever. In practice it seems to work somewhere between the 1st and 10th iteration. If i batch my commits at a reasonable interval damage will be mitigated but I really do not want to leave things in this state...
More suggestions welcome!
It looks like you failed to link the command with the transaction you've created.
Instead of:
using (SQLiteCommand command = conn.CreateCommand())
You should use:
using (SQLiteCommand command = new SQLiteCommand("<INSERT statement here>", conn, trans))
Or you can set its Transaction property after its construction.
While we are at it - your handling of failures is incorrect:
The command's ExecuteNonQuery method can also fail and you are not really protected. You should change the code to something like:
public void Commit()
{
using (SQLiteConnection conn = new SQLiteConnection(this.connString))
{
conn.Open();
SQLiteTransaction trans = conn.BeginTransaction();
try
{
using (SQLiteCommand command = conn.CreateCommand())
{
command.Transaction = trans; // Now the command is linked to the transaction and don't try to create a new one (which is probably why your database gets locked)
command.CommandText = "INSERT OR IGNORE INTO [MY_TABLE] (col1, col2) VALUES (?,?)";
command.Parameters.Add(this.col1Param);
command.Parameters.Add(this.col2Param);
foreach (Data o in this.dataTemp)
{
this.col1Param.Value = o.Col1Prop;
this. col2Param.Value = o.Col2Prop;
command.ExecuteNonQuery();
}
}
trans.Commit();
}
catch (SQLiteException ex)
{
// You need to rollback in case something wrong happened in command.ExecuteNonQuery() ...
trans.Rollback();
throw;
}
}
}
Another thing is that you don't need to cache anything in memory. You can depend on SQLite journaling mechanism for storing incomplete transaction state.
Run Sysinternals Process Monitor and filter on filename while running your program to rule out if any other process does anything to it and to see what exacly your program is doing to the file. Long shot, but might give a clue.
We had a very similar problem using nested Transactions with the TransactionScope class. We thought all database actions occurred on the same thread...however we were caught out by the Transaction mechanism...more specifically the Ambient transaction.
Basically there was a transaction higher up the chain which, by the magic of ado, the connection automatically enlisted in. The result was that, even though we thought we were writing to the database on a single thread, the write didn't really happen until the topmost transaction was committed. At this 'indeterminate' point the database was written to causing it to be locked outside of our control.
The solution was to ensure that the sqlite database did not directly take part in the ambient transaction by ensuring we used something like:
using(TransactionScope scope = new TransactionScope(TransactionScopeOptions.RequiresNew))
{
...
scope.Complete()
}
Things to watch for:
don't use connections across multiple threads/processes.
I've seen it happen when a virus scanner would detect changes to the file and try to scan it. It would lock the file for a short interval and cause havoc.
I started facing this same problem today: I'm studying asp.net mvc, building my first application completely from scratch. Sometimes, when I'd write to the database, I'd get the same exception, saying the database file was locked.
I found it really strange, since I was completely sure that there was just one connection open at that time (based on process explorer's listing of active file handles).
I've also built the whole data access layer from scratch, using System.Data.SQLite .Net provider, and, when I planned it, I took special care with connections and transactions, in order to ensure no connection or transaction was left hanging around.
The tricky part was that setting a breakpoint on ExecuteNonQuery() command and running the application in debug mode would make the error disappear!
Googling, I found something interesting on this site: http://www.softperfect.com/board/read.php?8,5775. There, someone replied the thread suggesting the author to put the database path on the anti-virus ignore list.
I added the database file to the ignore list of my anti-virus (Microsoft Security Essentials) and it solved my problem. No more database locked errors!
Is your database file on the same machine as the app or is it stored on a server?
You should create a new connection in every thread. I would simplefy the creation of a connection, use everywhere: connection = new SQLiteConnection(connString.ToString());
and use a database file on the same machine as the app and test again.
Why the two different ways of creating a connection?
These guys were having similiar problems (mostly, it appears, with the journaling file being locked, maybe TortoiseSVN interactions ... check the referenced articles).
They came up with a set of recommendations (correct directories, changing journaling types from delete to persist, etc). http://sqlite.phxsoftware.com/forums/p/689/5445.aspx#5445
The journal mode options are discussed here: http://www.sqlite.org/pragma.html . You could try TRUNCATE.
Is there a stack trace during the exception into SQL Lite?
You indicate you "batch my commits at a reasonable interval". What is the interval?
I would always use a Connection, Transaction and Command in a using clause. In your first code listing you did, but your third (creating the tables) you didn't. I suggest you do that too, because (who knows?) maybe the commands that create the table somehow continue to lock the file. Long shot... but worth a shot?
Do you have Google Desktop Search (or another file indexer) running? As previously mentioned, Sysinternals Process Monitor can help you track it down.
Also, what is the filename of the database? From PerformanceTuningWindows:
Be VERY, VERY careful what you name your database, especially the extension
For example, if you give all your databases the extension .sdb (SQLite Database, nice name hey? I thought so when I choose it anyway...) you discover that the SDB extension is already associated with APPFIX PACKAGES.
Now, here is the cute part, APPFIX is an executable/package that Windows XP recognizes, and it will, (emphasis mine) ADD THE DATABASE TO THE SYSTEM RESTORE FUNCTIONALITY
This means, stay with me here, every time you write ANYTHING to the database, the Windows XP system thinks a bloody executable has changed and copies your ENTIRE 800 meg database to the system restore directory....
I recommend something like DB or DAT.
While the lock is reported on the COMMIT, the lock is on the INSERT/UPDATE command. Check for record locks not being released earlier in your code.