I have an odd situation with TransactionScope and async/synchronous SQL calls that I'm having difficulty understanding. I hope that someone with a deeper understanding of the ins and outs of these kinds of operations can shed some light on the issue.
The situation:
I have a NUnit testfixture which creates a TransactionScope during [SetUp] and Disposes it at [TearDown] to let each test run on the same data. I have a series of tests which kick off an asynchronous operation on the database and then execute a synchronous operation on the database. The first such test completes successfully. The second such test fails with "There is already an open DataReader associated with this Command which must be closed first.".
If I comment out the TransactionScope entirely, all the tests pass.
I tried various different TransactionScope options, and Complete / Dispose, but the same issue occurs.
I am using the Resharper test runner on an NUnit test, .NET 4.5.1.
I realize the "correct" answer may be "make everything async await". That's not an option for me, unfortunately.
I don't want to enable MARS, as this issue only occurs in tests.
I don't want to use GetAwaiter().GetResult() due to the potential deadlocks.
What it looks like to me is that once a TransactionScope.Dispose/Complete is called, the automatic SQLConnection pooling loses track of which connections have open DataReaders. It hands out the same SqlConnection to two simultaneously running operations, and the second dies.
My primary question is "what is causing this behavior (specifically)?"
My secondary question is "is there anything that can be done to safely resolve the issue?"
The replicating code below prints out the client connection Ids. On my machine, the ClientConnectionId for the ASYNC and SYNC calls in the Second test case are always the same.
Replicating Code:
[TestFixture]
public class DataReaderTests
{
private TransactionScope _scope;
private string _connString = #"my connection string";
[SetUp]
public void Setup()
{
var options = new TransactionOptions()
{
IsolationLevel = IsolationLevel.ReadCommitted,
Timeout = TimeSpan.FromMinutes(1)
};
_scope = new TransactionScope(TransactionScopeOption.RequiresNew, options, TransactionScopeAsyncFlowOption.Enabled);
}
[Test]
[TestCase("First")]
[TestCase("Second")]
public void Test(string name)
{
DoAsyncThing().ConfigureAwait(false);
using (var conn = new SqlConnection(_connString))
{
try
{
conn.Open();
Console.WriteLine("SYNC: " + conn.ClientConnectionId);
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = "SELECT 1";
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
int id = reader.GetInt32(0);
}
}
}
}
catch (TransactionAbortedException tax)
{
Console.WriteLine("ERROR: " + ((SqlException)tax.InnerException.InnerException).ClientConnectionId);
throw;
}
}
}
private async Task DoAsyncThing()
{
using (var connection = new SqlConnection(_connString))
{
await connection.OpenAsync();
Console.WriteLine("ASYNC: " + connection.ClientConnectionId);
using (var cmd = connection.CreateCommand())
{
cmd.CommandText = "WAITFOR DELAY '00:02';";
await cmd.ExecuteNonQueryAsync();
Console.WriteLine("ASYNC COMPLETE");
}
}
}
[TearDown]
public void Teardown()
{
_scope.Dispose();
}
}`
Check out this answer
I think the gist is that you cannot have two active sql commands executing over the same connection at the same time without a special connection string property. When you are operating under the transaction scope, you should find that both SqlConnection objects have the same client ID. However, if you remove the transaction scope they are different, which I believe implies that they are operating on separate connections.
Adding "MultipleActiveResultSets=true" to the connection string fixed the issue for me. Another alternative is to replace
DoAsyncThing().ConfigureAwait(false);
with
DoAsyncThing().ConfigureAwait(false).GetAwaiter().GetResult();
which will terminate the first command before starting the second command.
Related
I'm just starting out with async and Task's and my code has stopped processing. It happens when I have an incoming network packet and I try and communicate with the database inside the packet handler.
public class ClientConnectedPacket : IClientPacket
{
private readonly EntityFactory _entityFactory;
public ClientConnectedPacket(EntityFactory entityFactory)
{
_entityFactory= entityFactory;
}
public async Task Handle(NetworkClient client, ClientPacketReader reader)
{
client.Entity = await _entityFactory.CreateInstanceAsync( reader.GetValueByKey("unique_device_id"));
// this Console.WriteLine never gets reached
Console.WriteLine($"Client [{reader.GetValueByKey("unique_device_id")}] has connected");
}
}
The Handle method gets called from an async task
if (_packetRepository.TryGetPacketByName(packetName, out var packet))
{
await packet.Handle(this, new ClientPacketReader(packetName, packetData));
}
else
{
Console.WriteLine("Unknown packet: " + packetName);
}
Here is the method which I think is causing the issue
public async Task<Entity> CreateInstanceAsync(string uniqueId)
{
await using (var dbConnection = _databaseProvider.GetConnection())
{
dbConnection.SetQuery("SELECT COUNT(NULL) FROM `entities` WHERE `unique_id` = #uniqueId");
dbConnection.AddParameter("uniqueId", uniqueId);
var row = await dbConnection.ExecuteRowAsync();
if (row != null)
{
return new Entity(uniqueId, false);
}
}
return new Entity(uniqueId,true);
}
DatabaseProvider's GetConnection method:
public DatabaseConnection GetConnection()
{
var connection = new MySqlConnection(_connectionString);
var command = connection.CreateCommand();
return new DatabaseConnection(_logFactory.GetLogger(), connection, command);
}
DatabaseConnection's constructor:
public DatabaseConnection(ILogger logger, MySqlConnection connection, MySqlCommand command)
{
_logger = logger;
_connection = connection;
_command = command;
_connection.Open();
}
When I comment out this line, it reaches the Console.WriteLine
_connection.Open();
I ran a POC project spinning 100 parallel tasks both with MySql.Data 8.0.19 and MySqlConnector 0.63.2 on .NET Core 3.1 console application. I create, open and dispose the connection into the context of every single task. Both providers runs to completion without errors.
The specifics are that MySql.Data queries run synchronously although the library provide async methods signature e.g. ExecuteReaderAsync() or ExecuteScalarAsync(), while MySqlConnector run truly asynchronously.
You may be running into:
a deadlock situation not specifically related to the mysql provider
not properly handling exceptions inside your tasks (you may inspect the task associated aggregate exception and also monitor mysql db logs)
you execution be still blocked (not returning result) when you assume it’s not working, if you running a high number of parallel tasks with MySql.Data as it executes synchronously
Multi-threading with MySQL must use independent connections. Given that, multithreading is not a MySQL question but an issue for the client language, C# in your question.
That is, build your threads without regard to MySQL, then create a connection in each thread that needs to do queries. It will be on your shoulders if you need to pass data between the threads.
I usually find that optimizing queries eliminates the temptation to multi-thread my applications.
I have a SQL Server 2008 database with two tables, a .NET 4.5 application using a TransactionScope with two queries to the database, each using their own separate connection which shares a single connection string.
This code that works fine:
internal static void Main(string[] args)
{
string connStr = string.Format(
"Data Source={0};Initial Catalog={1};" +
"Integrated Security=True", "My DB Server", "My Database");
using (var scope = new TransactionScope())
{
string query1 = "(actual query not relevant)";
var ds = GetSqlServerDataSet(query1, connStr);
string query2 = "(actual query not relevant)";
var ds1 = GetSqlServerDataSet(query2, connStr);
scope.Complete();
}
}
private static System.Data.DataSet GetSqlServerDataSet(string usingQuery,
string usingConnectionString,
params System.Data.SqlClient.SqlParameter[] withParameters)
{
using (var ds = new System.Data.DataSet())
using (var conn = new System.Data.SqlClient.SqlConnection(usingConnectionString))
using (var command = new System.Data.SqlClient.SqlCommand(usingQuery, conn))
{
command.Parameters.AddRange(withParameters);
using (var adapter = new System.Data.SqlClient.SqlDataAdapter(command))
{
adapter.Fill(ds);
}
return ds;
}
}
Now, if I throw an exception inside the scope to test roll-back, that's when stuff gets strange. I'm using a referenced class library I've written to persist info about exceptions to a database. It's the same type of setup -- same SQL Server, .NET version, identical ADO.NET code, etc. It's just writing to a different database.
Here's how it works: I've added this method to my app:
private static void HandleUnhandledException(Object sender, System.UnhandledExceptionEventArgs e)
{
ExceptionHandling.Core.Main.ProcessException((Exception) e.ExceptionObject);
Environment.Exit(0);
}
and I've added this line to the top of my Main method:
AppDomain.CurrentDomain.UnhandledException += HandleUnhandledException;
Now when I throw an exception, e.g. throw new Exception("blah"); at the bottom of Main right before scope.Complete(), it automatically jumps to HandleUnhandledException, and the rest happens as I've described above. This results in System.Transactions.TransactionManagerCommunicationException with message:
Network access for Distributed Transaction Manager (MSDTC) has been
disabled. Please enable DTC for network access in the security
configuration for MSDTC using the Component Services Administrative
tool.
This happens on the call to Connection.Open() in my class library.
So what's happening here? I know what the error is telling me, and I know how to fix it. My question is why is this happening in the first place? I thought the using statement would take care of rolling back the transaction before it hits my HandleUnhandledException method, so there shouldn't be two transactions involved here. Or am I wrong?
It happens because when your HandleUnhandledException method runs - you are still inside using block and it's finally part has not been run yet. It's easy to verify with this code:
class Program {
static void Main(string[] args) {
AppDomain.CurrentDomain.UnhandledException += OnUnhandledException;
try {
throw new Exception("test");
}
finally {
Console.WriteLine("finally");
}
}
private static void OnUnhandledException(object sender, UnhandledExceptionEventArgs e) {
Console.WriteLine("handle exception");
}
}
Which outputs:
handle exception
Unhandled exception: System.Exception: test ...
finally
You may want to read this question (and comments, from Eric Lipper for example) for discussion of related behavior. But I think reasons are not very relevant to your question - we can say it just works like this.
Given that information it's clear why you observe your exception - you are still inside uncommited transaction scope and trying to open connection to different database.
One solution that comes to mind in your case is check if we are inside transaction and suppress it if yes (because you anyway don't want your logging code to affect ambient transaction in any way):
private static void HandleException(Exception ex) {
TransactionScope scope = null;
if (Transaction.Current != null)
scope = new TransactionScope(TransactionScopeOption.Suppress);
try {
// do stuff
}
finally {
scope?.Dispose();
}
}
Or maybe just always run inside supressed TransactionScope.
Hi all I just had a quick question for you all. For whatever reason, a piece of code periodically does not return and I am not 100% sure yet. To combat this for now, I want to know, using the Close() method below, is there a way to put a timeout on it? So, if it does not finish within 1 minute or so, it just moves on?
Any advice would be appreciated. Thank you,
If it makes any difference, the original writer who wrote this noted that he believed it hangs on the close() and noted "Maybe Too fast?" (The connection is an oledb connection to Netezza, the whole applications is heavily multi-threaded).
Anyways, for now, I just want to be able to get the application to at least finish instead of hanging on that exception catch.
below is the Close(); which I believe is not returning.
catch(Exception){
Close(); //-- if we have an error, close everything down and then return the error
throw;}
public void Close() {
if (null != Command) {
Command.Cancel();
Command.Dispose();
Command = null;
}
if (null != Connection) {
if (Connection.State != System.Data.ConnectionState.Closed)
Connection.Close();
Connection.Dispose();
Connection = null;
}
}
Rather than timeout on a Method do you really mean timeout on a Command?
Based on that Close() you are sharing Command and Connection.
That is not a good design for a heavily multi-threaded application.
That is not a good design from even a lightly multi-threaded application.
DbCommand has a timeout property
Using statement will perform cleanup (including close)
string connectionString = "";
// Wait for 5 second delay in the command
string queryString = "waitfor delay '00:00:05'";
using (OleDbConnection connection = new OleDbConnection(connectionString )) {
connection.Open();
SqlCommand command = new connection.CreateCommand();
// Setting command timeout to 1 second
command.CommandText = queryString;
command.CommandTimeout = 1;
try {
command.ExecuteNonQuery();
}
catch (DbException e) {
Console.WriteLine("Got expected DbException due to command timeout ");
Console.WriteLine(e);
}
}
Assuming you're using .NET 4.0 and above, you can use the TPL to do so using the System.Threading.Tasks.Task object. You create a Task to run a method asynchronously, then Wait on that task for your timeout duration, and if it expires - let the main thread continue.
Task timeoutTask = new Task(Close); // create a Task around the Close method.
timeoutTask.Start(); // run asynchronously.
bool completedSuccessfully = timeoutTask.Wait(TimeSpan.FromMinutes(1));
if (completedSuccessfully)
{
// Yay!
}
else
{
logger.Write("Close command did not return in time. Continuing");
}
In this example, the Close method will keep on running in the background, but your main thread can continue.
Background: I'm writing a function putting long lasting operations in a queue, using C#,
and each operation is kind of divided into 3 steps:
1. database operation (update/delete/add data)
2. long time calculation using web service
3. database operation (save the calculation result of step 2) on the same db table in step 1, and check the consistency of the db table, e.g., the items are the same in step 1 (Pls see below for a more detailed example)
In order to avoid dirty data or corruptions, I use a lock object (a static singleton object) to ensure the 3 steps to be done as a whole transaction. Because when multiple users are calling the function to do operations, they may modify the same db table at different steps during their own operations without this lock, e.g., user2 is deleting item A in his step1, while user1 is checking if A still exists in his step 3. (additional info: Meanwhile I'm using TransactionScope from Entity framework to ensure each database operation as a transaction, but as repeat readable.)
However, I need to put this to a cloud computing platform which uses load balancing mechanism, so actually my lock object won't take effect, because the function will be deployed on different servers.
Question: what can I do to make my lock object working under above circumstance?
This is a tricky problem - you need a distributed lock, or some sort of shared state.
Since you already have the database, you could change your implementation from a "static C# lock" and instead the database to manage your lock for you over the whole "transaction".
You don't say what database you are using, but if it's SQL Server, then you can use an application lock to achieve this. This lets you explicitly "lock" an object, and all other clients will wait until that object is unlocked. Check out:
http://technet.microsoft.com/en-us/library/ms189823.aspx
I've coded up an example implementation below. Start two instances to test it out.
using System;
using System.Data;
using System.Data.SqlClient;
using System.Transactions;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
var locker = new SqlApplicationLock("MyAceApplication",
"Server=xxx;Database=scratch;User Id=xx;Password=xxx;");
Console.WriteLine("Aquiring the lock");
using (locker.TakeLock(TimeSpan.FromMinutes(2)))
{
Console.WriteLine("Lock Aquired, doing work which no one else can do. Press any key to release the lock.");
Console.ReadKey();
}
Console.WriteLine("Lock Released");
}
class SqlApplicationLock : IDisposable
{
private readonly String _uniqueId;
private readonly SqlConnection _sqlConnection;
private Boolean _isLockTaken = false;
public SqlApplicationLock(
String uniqueId,
String connectionString)
{
_uniqueId = uniqueId;
_sqlConnection = new SqlConnection(connectionString);
_sqlConnection.Open();
}
public IDisposable TakeLock(TimeSpan takeLockTimeout)
{
using (TransactionScope transactionScope = new TransactionScope(TransactionScopeOption.Suppress))
{
SqlCommand sqlCommand = new SqlCommand("sp_getapplock", _sqlConnection);
sqlCommand.CommandType = CommandType.StoredProcedure;
sqlCommand.CommandTimeout = (int)takeLockTimeout.TotalSeconds;
sqlCommand.Parameters.AddWithValue("Resource", _uniqueId);
sqlCommand.Parameters.AddWithValue("LockOwner", "Session");
sqlCommand.Parameters.AddWithValue("LockMode", "Exclusive");
sqlCommand.Parameters.AddWithValue("LockTimeout", (Int32)takeLockTimeout.TotalMilliseconds);
SqlParameter returnValue = sqlCommand.Parameters.Add("ReturnValue", SqlDbType.Int);
returnValue.Direction = ParameterDirection.ReturnValue;
sqlCommand.ExecuteNonQuery();
if ((int)returnValue.Value < 0)
{
throw new Exception(String.Format("sp_getapplock failed with errorCode '{0}'",
returnValue.Value));
}
_isLockTaken = true;
transactionScope.Complete();
}
return this;
}
public void ReleaseLock()
{
using (TransactionScope transactionScope = new TransactionScope(TransactionScopeOption.Suppress))
{
SqlCommand sqlCommand = new SqlCommand("sp_releaseapplock", _sqlConnection);
sqlCommand.CommandType = CommandType.StoredProcedure;
sqlCommand.Parameters.AddWithValue("Resource", _uniqueId);
sqlCommand.Parameters.AddWithValue("LockOwner", "Session");
sqlCommand.ExecuteNonQuery();
_isLockTaken = false;
transactionScope.Complete();
}
}
public void Dispose()
{
if (_isLockTaken)
{
ReleaseLock();
}
_sqlConnection.Close();
}
}
}
}
I feel like this behavior should not be happening. Here's the scenario:
Start a long-running sql transaction.
The thread that ran the sql command
gets aborted (not by our code!)
When the thread returns to managed
code, the SqlConnection's state is
"Closed" - but the transaction is
still open on the sql server.
The SQLConnection can be re-opened,
and you can try to call rollback on
the transaction, but it has no
effect (not that I would expect this behavior. The point is there is no way to access the transaction on the db and roll it back.)
The issue is simply that the transaction is not cleaned up properly when the thread aborts. This was a problem with .Net 1.1, 2.0 and 2.0 SP1. We are running .Net 3.5 SP1.
Here is a sample program that illustrates the issue.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data.SqlClient;
using System.Threading;
namespace ConsoleApplication1
{
class Run
{
static Thread transactionThread;
public class ConnectionHolder : IDisposable
{
public void Dispose()
{
}
public void executeLongTransaction()
{
Console.WriteLine("Starting a long running transaction.");
using (SqlConnection _con = new SqlConnection("Data Source=<YourServer>;Initial Catalog=<YourDB>;Integrated Security=True;Persist Security Info=False;Max Pool Size=200;MultipleActiveResultSets=True;Connect Timeout=30;Application Name=ConsoleApplication1.vshost"))
{
try
{
SqlTransaction trans = null;
trans = _con.BeginTransaction();
SqlCommand cmd = new SqlCommand("update <YourTable> set Name = 'XXX' where ID = #0; waitfor delay '00:00:05'", _con, trans);
cmd.Parameters.Add(new SqlParameter("0", 340));
cmd.ExecuteNonQuery();
cmd.Transaction.Commit();
Console.WriteLine("Finished the long running transaction.");
}
catch (ThreadAbortException tae)
{
Console.WriteLine("Thread - caught ThreadAbortException in executeLongTransaction - resetting.");
Console.WriteLine("Exception message: {0}", tae.Message);
}
}
}
}
static void killTransactionThread()
{
Thread.Sleep(2 * 1000);
// We're not doing this anywhere in our real code. This is for simulation
// purposes only!
transactionThread.Abort();
Console.WriteLine("Killing the transaction thread...");
}
/// <summary>
/// The main entry point for the application.
/// </summary>
[STAThread]
static void Main(string[] args)
{
using (var connectionHolder = new ConnectionHolder())
{
transactionThread = new Thread(connectionHolder.executeLongTransaction);
transactionThread.Start();
new Thread(killTransactionThread).Start();
transactionThread.Join();
Console.WriteLine("The transaction thread has died. Please run 'select * from sysprocesses where open_tran > 0' now while this window remains open. \n\n");
Console.Read();
}
}
}
}
There is a Microsoft Hotfix targeted at .Net2.0 SP1 that was supposed to address this, but we obviously have newer DLL's (.Net 3.5 SP1) that don't match the version numbers listed in this hotfix.
Can anyone explain this behavior, and why the ThreadAbort is still not cleaning up the sql transaction properly? Does .Net 3.5 SP1 not include this hotfix, or is this behavior that is technically correct?
Since you're using SqlConnection with pooling, your code is never in control of closing the connections. The pool is. On the server side, a pending transaction will be rolled back when the connection is truly closed (socket closed), but with pooling the server side never sees a connection close. W/o the connection closing (either by physical disconnect at the socket/pipe/LPC layer or by sp_reset_connection call), the server cannot abort the pending transaction. So it really boils down to the fact that the connection does not get properly release/reset. I don't understand why you're trying to complicate the code with explicit thread abort dismissal and attempt to reopen a closed transaction (that will never work). You should simply wrap the SqlConnection in an using(...) block, the implied finally and connection Dispose will be run even on thread abort.
My recommendation would be to keep things simple, ditch the fancy thread abort handling and replace it with a plain 'using' block (using(connection) {using(transaction) {code; commit () }}.
Of course I assume you do not propagate the transaction context into a different scope in the server (you do not use sp_getbindtoken and friends, and you do not enroll in distributed transactions).
This little program shows that the Thread.Abort properly closes a connection and the transaction is rolled back:
using System;
using System.Data.SqlClient;
using testThreadAbort.Properties;
using System.Threading;
using System.Diagnostics;
namespace testThreadAbort
{
class Program
{
static AutoResetEvent evReady = new AutoResetEvent(false);
static long xactId = 0;
static void ThreadFunc()
{
using (SqlConnection conn = new SqlConnection(Settings.Default.conn))
{
conn.Open();
using (SqlTransaction trn = conn.BeginTransaction())
{
// Retrieve our XACTID
//
SqlCommand cmd = new SqlCommand("select transaction_id from sys.dm_tran_current_transaction", conn, trn);
xactId = (long) cmd.ExecuteScalar();
Console.Out.WriteLine("XactID: {0}", xactId);
cmd = new SqlCommand(#"
insert into test (a) values (1);
waitfor delay '00:01:00'", conn, trn);
// Signal readyness and wait...
//
evReady.Set();
cmd.ExecuteNonQuery();
trn.Commit();
}
}
}
static void Main(string[] args)
{
try
{
using (SqlConnection conn = new SqlConnection(Settings.Default.conn))
{
conn.Open();
SqlCommand cmd = new SqlCommand(#"
if object_id('test') is not null
begin
drop table test;
end
create table test (a int);", conn);
cmd.ExecuteNonQuery();
}
Thread thread = new Thread(new ThreadStart(ThreadFunc));
thread.Start();
evReady.WaitOne();
Thread.Sleep(TimeSpan.FromSeconds(5));
Console.Out.WriteLine("Aborting...");
thread.Abort();
thread.Join();
Console.Out.WriteLine("Aborted");
Debug.Assert(0 != xactId);
using (SqlConnection conn = new SqlConnection(Settings.Default.conn))
{
conn.Open();
// checked if xactId is still active
//
SqlCommand cmd = new SqlCommand("select count(*) from sys.dm_tran_active_transactions where transaction_id = #xactId", conn);
cmd.Parameters.AddWithValue("#xactId", xactId);
object count = cmd.ExecuteScalar();
Console.WriteLine("Active transactions with xactId {0}: {1}", xactId, count);
// Check count of rows in test (would block on row lock)
//
cmd = new SqlCommand("select count(*) from test", conn);
count = cmd.ExecuteScalar();
Console.WriteLine("Count of rows in text: {0}", count);
}
}
catch (Exception e)
{
Console.Error.Write(e);
}
}
}
}
This is a bug in Microsoft's MARS implementation. Disabling MARS in your connection string will make the problem go away.
If you require MARS, and are comfortable making your application dependent on another company's internal implementation, familiarize yourself with http://dotnet.sys-con.com/node/39040, break out .NET Reflector, and look at the connection and pool classes. You have to store a copy of the DbConnectionInternal property before the failure occurs. Later, use reflection to pass the reference to a deallocation method in the internal pooling class. This will stop your connection from lingering for 4:00 - 7:40 minutes.
There are surely other ways to force the connection out of the pool and to be disposed. Short of a hotfix from Microsoft, though, reflection seems to be necessary. The public methods in the ADO.NET API don't seem to help.