I'm trying to figure out how to setup my database access correctly while using the SqlClient hitting a Microsoft SQL Server. For the most part it is working, but there's a particular scenario that is giving me trouble. Namely: attempting to simultaneously use two connections in the same thread; one with an open data reader and the other performing a delete operation.
The following code demonstrates my conundrum:
public class Database {
...
internal SqlConnection CreateConnection() => new SqlConnection(connectionString);
...
}
public IEnumerable<Model> GetModel() {
var cmd = new SqlCommand() { ... };
using(var conn = db.CreateConnection()) {
conn.Open();
cmd.Connection = conn;
using(var reader = cmd.ExecuteReader()) {
while(reader.Read()) {
var m = new Model();
// deserialization logic
yield return m;
}
}
}
}
public void Delete(int id) {
var cmd = new SqlCommand() { ... }
using(var conn = db.CreateConnection()) {
conn.Open(); // throwing the error here
cmd.Connection = conn;
cmd.ExecuteNonQuery();
}
}
Application Code:
using(var scope = new TransactionScope()) {
var models = GetModels();
foreach(var m in models) {
Delete(m.Id); // throws an exception
}
scope.Complete();
}
For whatever reason, an exception is thrown by the above code while trying to execute the Delete operation:
quote
System.Transactions.TransactionAbortedException: The transaction has aborted. ---> System.Transactions.TransactionPromotionException: Failure while attempting to promote transaction. ---> System.Data.SqlClient.SqlException: There is already an open DataReader associated with this Command which must be closed first. ---> System.ComponentModel.Win32Exception: The wait operation timed out
quote
Now, I have confirmed that if I either set MultipleActiveResultSets=true or Pooling=false on the ConnectionString, that then the above application code will work without error. However, it doesn't seem like I should need to set either of those. If I open two connections simultaneously, should they not be separate connections? Why then am I getting an error from the Delete connection saying that there's an open DataReader?
Please help <3
By far the easiest fix here is to simply load all the models outside the transaction before you go deleting any. Eg
var models = GetModels().ToList();
using(var scope = new TransactionScope()) {
foreach(var m in models) {
Delete(m.Id); // throws an exception
}
scope.Complete();
}
Even fetching the models inside the transaction shold work
using(var scope = new TransactionScope()) {
var models = GetModels().ToList();
foreach(var m in models) {
Delete(m.Id); // throws an exception
}
scope.Complete();
}
so long as you don't leave the connection open during the iteration. If you allow the connection in GetModels() to close, it will be returned to the connection pool, and be available for use for subsequent methods that are enlisted in the same transaction.
In the current code the connection in GetModels() is kept open during the foreach loop and Delete(id) has to open a second connection and try to create a distributed transaction, which is failing.
Without MultipleActiveResultsets, the GetModels connection can't be promoted to a distributed transaction in the middle of returning query results. Setting pooling=false will not make this error go away.
Here's a simplified repro to play with:
using Microsoft.Data.SqlClient;
using System.Collections.Generic;
using System.Transactions;
namespace SqlClientTest
{
class Program
{
static void Main(string[] args)
{
Setup();
var topt = new TransactionOptions();
topt.IsolationLevel = System.Transactions.IsolationLevel.ReadCommitted;
using (new TransactionScope(TransactionScopeOption.Required, topt ))
{
foreach (var id in GetIds())
{
Delete(id);
}
}
}
static string constr = #"server=.;database=tempdb;Integrated Security=true;TrustServerCertificate=true;";
public static void Setup()
{
using (var con = new SqlConnection(constr))
{
con.Open();
var cmd = con.CreateCommand();
cmd.CommandText = "drop table if exists ids; select object_id id into ids from sys.objects";
cmd.ExecuteNonQuery();
}
}
public static IEnumerable<int> GetIds()
{
using (var con = new SqlConnection(constr))
{
con.Open();
var cmd = con.CreateCommand();
cmd.CommandText = "select object_id id from sys.objects";
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
yield return reader.GetInt32(0);
}
}
}
}
public static void Delete(int id)
{
using (var con = new SqlConnection(constr))
{
con.Open();
var cmd = con.CreateCommand();
cmd.CommandText = "insert into ids(id) values (#id)";
cmd.Parameters.Add(new SqlParameter("#id", id));
cmd.ExecuteNonQuery();
}
}
}
}
And here's what Profiler shows when run:
The main reason here as far as I understand is your yielding iteration.
So the DB connection has not yet called disposed as it's still being used in your iteration (foreach). If for example, you called .ToList() at that point it should return all the entries and then dispose of the connection.
See here for a better explanation on how yield works in an iteration: https://stackoverflow.com/a/58449809/3329836
Related
I have a class and my MysqlConnection is in there:
public class DB
{
private static MySqlConnection _Connection;
public static MySqlConnection Connection
{
get
{
if(_Connection == null)
{
string cs = string.Format("SERVER={0}; DATABASE={1}; UID={2}; PWD={3};", SERVER_ADRESS, DATABASE, UID, PWD);
_Connection = new MySqlConnection(cs);
}
if(_Connection.State == System.Data.ConnectionState.Closed)
try
{
MessageBox.Show("MySQL Connection ist geschlossen. Öffne Sie");
_Connection.Open();
}
catch(MySqlException ex)
{
switch (ex.Number)
{
case 0:
MessageBox.Show("Verbindung zum Server konnte nicht hergestellt werden.");
break;
case 1045:
MessageBox.Show("Ungültiger Benutzername/Passwort.");
break;
default:
MessageBox.Show(ex.Message);
break;
}
}
return _Connection;
}
}
}
So i can use this connection in all the other classes with DB.Connection.
But now I get "DataReader is already open".
But all my DataReader's are in usings.
We start at my login page:
using (loginreader = cmd.ExecuteReader())
{
if (loginreader.Read())
{
DB.Connection.Close();
return true;
}
else
{
DB.Connection.Close();
return false;
}
loginreader.Close();
}
I guess this doesn't work. But the first Error Message after log in i get on another class on line 83:
DataTable schema = null;
using (var schemaCommand = new MySqlCommand("SELECT * FROM " + firmCustomerTablename, connection))
{
using (var reader = schemaCommand.ExecuteReader(CommandBehavior.SchemaOnly))
{
schema = reader.GetSchemaTable();
}
}
which is in a using too. So I don't understand why I get this Error. I guess closing the connections / the DataReaders dont work.
Before this change, i had a connection for every site. But my program had no good performance. So I decided to make 1 connection which is always Open and just call querys to this open connection. And now I get DataReader Errors.
Can someone explain me, why using is not closing the DataReader? And line 83 isn't a DataReader it's a var so i don't know why I get this Error at this line.
It sounds like your issues are regarding connection state management? I may not be fully understanding what you're asking but by design using statements within the context of connections will close the connection. They are syntactic sugar for try {} catch {} finally. Far too often I see examples of Connection objects, Command Objects, etc. not utilizing IDisposable and not being properly disposed/closed.
In this code, I don't see a connection being opened again for the command to execute.
DataTable schema = null;
using (var schemaCommand = new MySqlCommand("SELECT * FROM " + firmCustomerTablename, connection))
{
using (var reader = schemaCommand.ExecuteReader(CommandBehavior.SchemaOnly))
{
schema = reader.GetSchemaTable();
}
}
Here is a basic idea:
using (var conn = new SqlConnection(connectionString: ""))
{
conn.Open();
using (var cmd = new SqlCommand(cmdText: "cmdText", connection: conn))
{
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
//
}
}
}
}
Documentation: MSDN SqlConnection Class
background
I have some code which opens a sql connection, begins a transaction and performs some operations on the DB. This code creates an object from the DB (dequeue), gets some values and saves it back. The whole operation needs to take place in a transaction. All the code works perfectly without the transaction.
using (var connection = new SqlConnection(connectionString))
{
connection.Open();
var transaction = connection.BeginTransaction();
try
{
var myObject = foo.Dequeue(connection, transaction);
var url = myObj.GetFilePathUri(connection, transaction);
//some other code that sets object values
myObj.SaveMessage(connection, transaction);
transaction.Commit(); //error here
}
catch(Exception ex)
{
transaction.Rollback();
//logging
}
finally
{
//cleanup code
}
}
dequeue method code
public foo Dequeue(SqlConnection connection, SqlTransaction transaction)
{
using (var command = new SqlCommand(DEQUEUE_SPROC, connection) {CommandType = CommandType.StoredProcedure, Transaction = transaction})
{
var reader = command.ExecuteReader();
if (reader.HasRows)
{
reader.Read();
ID = (Guid) reader["ID"];
Name = reader["Name"].ToString();
return this;
}
return null;
}
}
Get Path Code
public string GetFilePathUri(SqlConnection connection, SqlTransaction transaction)
{
using (var command = new SqlCommand(FILEPATH_SPROC, connection) {CommandType = CommandType.StoredProcedure, Transaction = transaction})
{
var reader = command.ExecuteReader();
if (reader.HasRows)
{
reader.Read();
return reader["Path"].ToString();
}
return "";
}
}
Save Code
public void SaveMessage(SqlConnection connection, SqlTransaction transaction)
{
using (var command = new SqlCommand(SAVE_SPROC, connection) {CommandType = CommandType.StoredProcedure, Transaction = transaction})
{
command.Parameters.Add("#ID", SqlDbType.UniqueIdentifier).Value = ID;
command.Parameters.Add("#Name", SqlDbType.VarChar).Value = Name;
//other object params here
command.ExecuteNonQuery();
}
}
The problem
When transaction.Commit() is called, I get the following error:
The transaction operation cannot be performed because there are pending requests working on this transaction.
What am I doing wrong?
EDIT: Quick edit to say I have read the other questions about this problem on SO, but couldn't find any related to ADO.net
I have had this issue before and the problem was the reader needed to be closed.
Try this:
public foo Dequeue(SqlConnection connection, SqlTransaction transaction)
{
using (var command = new SqlCommand(DEQUEUE_SPROC, connection) {CommandType = CommandType.StoredProcedure, Transaction = transaction})
{
var reader = command.ExecuteReader();
if (reader.HasRows)
{
reader.Read();
ID = (Guid) reader["ID"];
Name = reader["Name"].ToString();
reader.Close();//Closing the reader
return this;
}
return null;
}
}
public string GetFilePathUri(SqlConnection connection, SqlTransaction transaction)
{
string filePathUri = "";
using (var command = new SqlCommand(FILEPATH_SPROC, connection) {CommandType = CommandType.StoredProcedure, Transaction = transaction})
{
var reader = command.ExecuteReader();
if (reader.HasRows)
{
reader.Read();
filePathUri = reader["Path"].ToString();
}
reader.Close();//Closing the reader
}
return filePathUri;
}
I had this problem when I forgot to use await on an async method that was doing the DB call - the connection was being disposed while the transaction was running because the program was not waiting for the query to complete before trying to dispose everything.
We ran into this problem and while the accepted answer probably would've worked, it wouldn't have been the most correct thing to do -- it did, however, give us a clue as to what we were doing wrong.
In our case, there was more data to read. Here's a simplified version of the problem (using Dapper):
public Foo GetFooById(int fooId, IDbTransaction transaction = null) {
var sql = #"
SELECT * FROM dbo.Foo WHERE FooID = #FooID;
SELECT * FROM dbo.FooBar WHERE FooID = #FooID;
SELECT * FROM dbo.FooBaz WHERE FooID = #FooID;
";
using var data = await _connection.QueryMultipleAsync(sql, new { fooId }, transaction);
var foo = data.ReadSingle<Foo>();
foo.Bars = data.Read<Bar>();
// Oops, didn't read FooBaz.
return foo;
}
Even though data gets disposed, it leaves a "pending request" on the transaction, which then blows up later when transaction.Commit() is called.
The solution is to either read FooBaz records out of data or get rid of the unread SELECT.
For who will come here in the future, I had this issue with a combination of IQueryable and AutoMapper.
I passed an IQueryable for mapping to Mapper.Map<>(myData) by thinking that since it maps data from a class to another, it also materializes it.
Apparently, I was wrong and the issue mentioned in the question was thrown when trying to transaction.Commit().
Solution
As easy as forcing the materialization with a .ToList() in Mapper.Map<>(myData.ToList()).
Just wasted an hour finding that, I hope this will save some time for someone else.
I am attempting to use TransactionScope across two queries to different SQL Server instances (the transaction is promoted to MSDTC). I have it working synchronously without using Tasks but am unable to get the transaction to rollback when using Tasks asynchronously.
I have hidden both the connection strings to the servers and the update statements, but reast assured that they both connect and execute the SQL with no problems. I have intentionally set the SQL in the method runTwo() to fail and raise an exception.
I am using .Net 4.5.2 which includes the enumeration TransactionScopeAsyncFlowOption.Enabled (in the TransactionScope constructor) that I believe should handle a transaction across threads but am unable to get this to work.
static void Main()
{
List<Task> tasks = new List<Task>();
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
try
{
tasks.Add(Task.Run(() => runOne()));
tasks.Add(Task.Run(() => runTwo()));
Task.WhenAll(tasks);
//Complete the scope
scope.Complete();
}
catch (Exception)
{
Transaction.Current.Rollback();
}
}
Console.ReadLine();
}
private static void runOne()
{
//Get the base SQL connection
using (SqlConnection conn = new SqlConnection("Data Source=SERVER1....."))
{
conn.Open();
SqlCommand command1 = new SqlCommand
{
CommandText = "Update .. Complete Successfully",
CommandType = CommandType.Text,
Connection = conn
};
command1.ExecuteNonQuery();
}
}
private static void runTwo()
{
using (
SqlConnection conn =
new SqlConnection("Data Source=SERVER2...")
)
{
conn.Open();
SqlCommand command2 = new SqlCommand
{
CommandText = "Update .... Raises Exception",
CommandType = CommandType.Text,
Connection = conn
};
//Execute the command
command2.ExecuteNonQuery();
}
}
This:
Task.WhenAll(tasks);
Is your problem. Task.WhenAll returns an awaitable, it doesn't block on the method call. Since you're using a console application which cannot be awaited, you'll have to defer to use Task.WaitAll instead which will explicitly block until both requests finish and will propagate any exception via AggregateException
try
{
Task.WaitAll(new[] { Task.Run(() => runOne()),
Task.Run(() => runTwo()) });
scope.Complete();
}
I am getting Database is locked exception from SQLite for some queries only.
Below is my code:
When I execute any select statement it works fine.
When I am executing any write statement on Jobs Table it also works fine.
This works fine:
ExecuteNonQuery("DELETE FROM Jobs WHERE id=1");
But the same way if I am executing queries for Employees table it is throwing an exception that database is locked.
This throws Exception:
ExecuteNonQuery("DELETE FROM Employees WHERE id=1");
Below are my functions:
public bool OpenConnection()
{
if (Con == null)
{
Con = new SQLiteConnection(ConnectionString);
}
if (Con.State == ConnectionState.Closed)
{
Con.Open();
//Cmd = new SQLiteCommand("PRAGMA FOREIGN_KEYS=ON", Con);
//Cmd.ExecuteNonQuery();
//Cmd.Dispose();
//Cmd=null;
return true;
}
if (IsConnectionBusy())
{
Msg.Log(new Exception("Connection busy"));
}
return false;
}
public Boolean CloseConnection()
{
if (Con != null && Con.State == ConnectionState.Open)
{
if (Cmd != null) Cmd.Dispose();
Cmd = null;
Con.Close();
return true;
}
return false;
}
public Boolean ExecuteNonQuery(string sql)
{
if (sql == null) return false;
try
{
if (!OpenConnection())
return false;
else
{
//Tx = Con.BeginTransaction(IsolationLevel.ReadCommitted);
Cmd = new SQLiteCommand(sql, Con);
Cmd.ExecuteNonQuery();
//Tx.Commit();
return true;
}
}
catch (Exception exception)
{
//Tx.Rollback();
Msg.Log(exception);
return false;
}
finally
{
CloseConnection();
}
}
This is the Exception:
At line 103 : Cmd.ExecuteNonQuery();
Exception Found:
Type: System.Data.SQLite.SQLiteException
Message: database is locked
database is locked
Source: System.Data.SQLite
Stacktrace: at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
at System.Data.SQLite.SQLiteDataReader.NextResult()
at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery()
at TimeSheet6.DbOp.ExecuteNonQuery(String sql) in d:\Projects\C# Applications\Completed Projects\TimeSheet6\TimeSheet6\DbOp.cs:line 103
Somewhere along the way a connection is getting left open. Get rid of OpenConnection and CloseConnection and change ExecuteNonQuery to this:
using (SQLiteConnection c = new SQLiteConnection(ConnectionString))
{
c.Open();
using (SQLiteCommand cmd = new SQLiteCommand(sql, c))
{
cmd.ExecuteNonQuery();
}
}
Further, change the way you read data to this:
using (SQLiteConnection c = new SQLiteConnection(ConnectionString))
{
c.Open();
using (SQLiteCommand cmd = new SQLiteCommand(sql, c))
{
using (SQLiteDataReader rdr = cmd.ExecuteReader())
{
...
}
}
}
Do not attempt, to manage connection pooling on your own like you are here. First, it's much more complex than what you have coded, but second, it's handled already inside the SQLiteConnection object. Finally, if you're not leveraging using, you're not disposing these objects properly and you end up with issues like what you're seeing now.
You can use 'using' statement as below, that will make sure connection & command disposed correctly even in exception
private static void ExecuteNonQuery(string queryString)
{
using (var connection = new SQLiteConnection(
ConnectionString))
{
using (var command = new SQLiteCommand(queryString, connection))
{
command.Connection.Open();
command.ExecuteNonQuery();
}
}
}
You should close your DataReader before attempting to write any data to the database. Use:
dr.Close();
after you finish using the DataReader.
In my case it was very stupid of me, I was making changes in SQLite browser and did not click on write changes, which locked the DB to be modified by the services. After I clicked the Write changes button, all the post request worked as expected.
A lot of helpful posts here for folks that may have forgotten to clean up a dangling connection, but there is another way this can happen: SQLite does not support concurrent INSERTs; if you issue two INSERTs at the same time the will be processed in serial. When the INSERTs are quick this is fine, but if an INSERT takes longer than the timeout the second INSERT can fail with this message.
I had this happen when I used a long running transaction to accumulate a bunch of INSERTs into one big commit. Basically I locked the database from any other activity during the transaction. Switching to journal_mode=WAL will allow concurrent writes and reads, but not concurrent writes.
I got rid of the long running transaction and let each INSERT autocommit, and that solved my problem.
Mine was caused by not closing a SqliteDataReader when calling HasRows().
I had this:
using (SQLiteConnection connection = new SQLiteConnection(DbPath))
{
connection.Open();
string sql = $"SELECT * FROM ...";
using (SQLiteCommand command = new SQLiteCommand(sql, connection))
{
return command.ExecuteReader().HasRows;
}
connection.Close();
}
But needed to put a using around the ExecuteReader like so:
using (SQLiteDataReader reader = command.ExecuteReader())
{
return command.ExecuteReader().HasRows;
}
Even though the DbConnection was being disposed and re-created each time the db was still being kept locked by the reader.
I was also getting the same error here:
if (new basics.HindiMessageBox(HMsg, HTitle).ShowDialog()==true)
{
SQLiteConnection m_dbConnection = new SQLiteConnection(MainWindow.con);
m_dbConnection.Open();
sql = "DELETE FROM `users` WHERE `id`=" + SelectedUser.Id;
command = new SQLiteCommand(sql, m_dbConnection);
command.ExecuteNonQuery();
m_dbConnection.Close();
LoadUserDG();
}
but when I just changed SQLiteConnection declaration location
public partial class User : Window
{
SQLiteCommand command;
string sql;
AddUser AddUserObj;
List<basics.users> usersList;
basics.users SelectedUser;
SQLiteConnection m_dbConnection;
// ...
private void DeleteBtn_Click(object sender, RoutedEventArgs e)
{
// ...
if (new basics.HindiMessageBox(HMsg, HTitle).ShowDialog()==true)
{
m_dbConnection = new SQLiteConnection(MainWindow.con);
m_dbConnection.Open();
sql = "DELETE FROM `users` WHERE `id`=" + SelectedUser.Id;
command = new SQLiteCommand(sql, m_dbConnection);
command.ExecuteNonQuery();
m_dbConnection.Close();
LoadUserDG();
}
}
Everything is fine now.
I hope this may work for you, too.
If someone can say how this happened, I would like to know the details to improve my knowledge, please.
I had the same issue when loading a lot of data to different tables from multiple threads.
When trying to do the inserts I was getting database locked because the program was doing too many insert too fast and SQLite didn't have time to complete each transaction before another one came.
The insert are done through threading because I didn't want the interface to be locked and wait for the insert to be done.
My solution is to use BlockingCollection with ThreadPool.QueueUserWorkItem.
This allows me to free the interface while doing the inserts.
All the insert are queued and executed in FIFO (First In First Out) order.
Now the database is never locked while doing any SQL transaction from any thread.
public class DatabaseQueueBus
{
private BlockingCollection<TransportBean> _dbQueueBus = new BlockingCollection<TransportBean>(new ConcurrentQueue<TransportBean>());
private CancellationTokenSource __dbQueueBusCancelToken;
public CancellationTokenSource _dbQueueBusCancelToken { get => __dbQueueBusCancelToken; set => __dbQueueBusCancelToken = value; }
public DatabaseQueueBus()
{
_dbQueueBusCancelToken = new CancellationTokenSource();
DatabaseQueue();
}
public void AddJob(TransportBean dto)
{
_dbQueueBus.Add(dto);
}
private void DatabaseQueue()
{
ThreadPool.QueueUserWorkItem((param) =>
{
try
{
do
{
string job = "";
TransportBean dto = _dbQueueBus.Take(_dbQueueBusCancelToken.Token);
try
{
job = (string)dto.DictionaryTransBean["job"];
switch (job)
{
case "SaveClasse":
//Save to table here
break;
case "SaveRegistrant":
//Save Registrant here
break;
}
}
catch (Exception ex)
{//TODO: Handle this exception or not
}
} while (_dbQueueBusCancelToken.Token.IsCancellationRequested != true);
}
catch (OperationCanceledException)
{
}
catch (Exception ex)
{
}
});
}
}
The inserts are done this way, but without the queuing I was still getting the lock issue.
using (SQLiteConnection c = new SQLiteConnection(BaseDal.SQLiteCon))
{
c.Open();
using (SQLiteCommand cmd = new SQLiteCommand(sql, c))
{
cmd.ExecuteNonQuery();
}
c.Close();
}
Is this code solid? I've tried to use "using" etc. Basically a method to pass as sequenced list of SQL commands to be run against a Sqlite database.
I assume it is true that in sqlite by default all commands run in a single connection are handled transactionally? Is this true? i.e. I should not have to (and haven't got in the code at the moment) a BeginTransaction, or CommitTransaction.
It's using http://sqlite.phxsoftware.com/ as the sqlite ADO.net database provider.
1st TRY
private int ExecuteNonQueryTransactionally(List<string> sqlList)
{
int totalRowsUpdated = 0;
using (var conn = new SQLiteConnection(_connectionString))
{
// Open connection (one connection so should be transactional - confirm)
conn.Open();
// Apply each SQL statement passed in to sqlList
foreach (string s in sqlList)
{
using (var cmd = new SQLiteCommand(conn))
{
cmd.CommandText = s;
totalRowsUpdated = totalRowsUpdated + cmd.ExecuteNonQuery();
}
}
}
return totalRowsUpdated;
}
3rd TRY
How is this?
private int ExecuteNonQueryTransactionally(List<string> sqlList)
{
int totalRowsUpdated = 0;
using (var conn = new SQLiteConnection(_connectionString))
{
conn.Open();
using (var trans = conn.BeginTransaction())
{
try
{
// Apply each SQL statement passed in to sqlList
foreach (string s in sqlList)
{
using (var cmd = new SQLiteCommand(conn))
{
cmd.CommandText = s;
totalRowsUpdated = totalRowsUpdated + cmd.ExecuteNonQuery();
}
}
trans.Commit();
}
catch (SQLiteException ex)
{
trans.Rollback();
throw;
}
}
}
return totalRowsUpdated;
}
thanks
Yes, it's true, each SQLite unnested command is nested in a transaction. So that if you need to run several queries, without fetching the result, there is much gain is explicitly starting a transaction, doing your queries, and committing.