I am trying to use query cancellation (via cancellation tokens) to cancel a long-running complex query. I have found that in some cases not only does cancellation fail to halt the query but also the call to CancellationToken.Cancel() hangs indefinitely. Here is a simple repro that replicates this behavior (can be run in LinqPad):
void Main()
{
var cancellationTokenSource = new CancellationTokenSource();
var blocked = RunSqlAsync(cancellationTokenSource.Token);
blocked.Wait(TimeSpan.FromSeconds(1)).Dump(); // false (blocked in SQL as expected)
cancellationTokenSource.Cancel(); // hangs forever?!
Console.WriteLine("Finished calling Cancel()");
blocked.Wait();
}
public async Task RunSqlAsync(CancellationToken cancellationToken)
{
var connectionString = new SqlConnectionStringBuilder { DataSource = #".\sqlexpress", IntegratedSecurity = true, Pooling = false }.ConnectionString;
using (var connection = new SqlConnection(connectionString))
{
await connection.OpenAsync().ConfigureAwait(false);
using (var command = connection.CreateCommand())
{
command.CommandText = #"
WHILE 1 = 1
BEGIN
DECLARE #x INT = 1
END
";
command.CommandTimeout = 0;
Console.WriteLine("Running query");
await command.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
}
}
}
Interestingly, the same query run in SqlServer Management Studio cancels instantly via the "Cancel Executing Query" button.
Is there some caveat to query cancellation where it cannot cancel tight WHILE loops?
My version of SqlServer:
Microsoft SQL Server 2012 - 11.0.2100.60 (X64)
Feb 10 2012 19:39:15
Copyright (c) Microsoft Corporation
Express Edition (64-bit) on Windows NT 6.2 (Build 9200: )
I am running on Windows 10, and .NET's Environment.Version is 4.0.30319.42000.
EDIT
Some additional information:
Here is the stack trace pulled from Visual Studio when cancellationToken.Cancel() hangs:
Another thread is stuck here:
Additionally, I tried updating to SqlServer Express 2017 and I am seeing the same behavior.
EDIT
I've filed this as a bug with corefx: https://github.com/dotnet/corefx/issues/26623
I can reproduce the issue in a console application. (The code in the question was code from LINQPad.)
I'm going to make this an answer and say that this is a bug in ADO.NET. ADO.NET should send a query cancellation signal to SQL Server. I can see from the CPU usage, that SQL Server continues executing the loop. Therefore, it did not receive cancellation from the client. We also know that SSMS is able to cancel this loop.
While the loop is running I can see that the console app is using 50% of one CPU core and receiving data from SQL Server at 70MB/sec. I do not know what data this is. It might be ROWCOUNT information or something related.
I think the bug is related to the fact that the loop is continuously sending data so that ADO.NET never has an opportunity to send the cancellation. It's still a bug and it would be a community service if you reported it. You can link to this question.
If the loop is throttled using ...
WHILE 1 = 1
BEGIN
DECLARE #x INT = 1
WAITFOR DELAY '00:00:01' --new
END
... then cancellation is quick.
Also, you can generally not rely on cancellation being quick. If the network dropped it might take 30sec for the client to notice this and throw.
Therefore you need to code your program so that it continues executing and not wait for the query to finish. It could look like this:
var queryTask = ...;
var cancellationToken = ...;
await Task.WhenAll(queryTask, cancellationToken);
That way cancellation always looks instantaneous. Make sure that resources are still disposed. All SQL interaction should be encapsulated in queryTask so that it simply continues in the background and eventually cleans up.
Related
I'm using Serenity C# to develop a website.
When I click on a button it should run a SQL command, which starts a stored procedure.
My code
public ListResponse<MyRow> RunSQL(IDbConnection connection, ListRequest request)
{
string sql = "EXEC SP_A #Username='" + Authorization.UserDefinition.Username + "'";
SqlHelper.ExecuteNonQuery(connection, sql);
return new MyRepository().List(connection, request);
}
This code works fine, but it makes my web slow because my web needs to wait for the query to finish.
I want to kick off the SQL command and not wait for the result. Can I use the Task Parallel Library (TPL) for this?
Can I use TPL (Task Parallel Library)??
No, you can not. You execute ONE statement, I am not even sure where you get the idea that paralellism of one item will do anything. if that query takes a long time, analyze whether it is defective. if it is not defective...
...change the API to be async an return a come back later with a token. It is waiting for the return value because you degiend the API to be synchroneous . you this is not acceptable, then the API is a design error and the design at least of this method should change.
Nothing in async/await/paralellism will change the API design and it will not magically make the request finish faster.
I know this question has been asked many times however none of the answers fit my issue.
I have a thread timer firing every 30 seconds that queries a MSSQL db that is under heavy load. If i need to update the data in the console app that i'm using i use Linq To Sql to update the data stored in memory.
My problem is sometimes I get the error ExecuteReader requires an open and available Connection.
The code from the thread timer fires a Thread.Run(reload());
The connection string is
//example code
void reload(...
string connstring = string.Format("Data Source={0},{1};Initial Catalog={2};User ID={3};Password={4};Application Name={5};Connect Timeout=120;MultipleActiveResultSets=True;Max Pool Size=1524;Pooling=true;"
settings = new ConnectionStringSettings("sqlServer", connstring, "System.Data.SqlClient");
using (var tx = new TransactionScope(TransactionScopeOption.Required,
new TransactionOptions() { IsolationLevel = IsolationLevel.ReadUncommitted }))
{
using (SwitchDataDataContext data = new SwitchDataDataContext(settings.ConnectionString))
{
data.CommandTimeout = 560;
then i do many linqtosql searches. The exceptions happen from time to time but not always on the same query's. it's like the connections is opened and is forced closed.
Sometimes the exceptions says the current status is Open, Closed, Connecting. I add a larger ThreadPool to the SQL db but nothing seems to help.
i also have ADO in other parts of the program without any issues.
I believe that your problem is that the transaction scope also has a timeout. The default timeout is 1 minute according to this answer. So the transaction times out long before your command does (560 seconds = 9.3 minutes or so) . You will need to set the timeout property in the instance of the TransactionOptions object you are creating
new TransactionOptions()
{
IsolationLevel = IsolationLevel.ReadUncommitted,
Timeout = new TimeSpan(0,10,0) /* 10 Minutes */
}
You can verify that is indeed the issue by setting the TransactionScope timeout to a small value to force it to timeout.
I changed Linq To Sql to Entity Framework and received the same type of message. I believe the issues is lazy Loading. I was using the collections before it was ready on a different thread. I just added .Include("Lab") to my collection to load the entire collection and it seems to of fixed the issue.
So I've run into a little issue that puzzles me and I've not been able to find a good explanation for this - I imagine I'm probably mis-using the async/await feature somehow but I really don't know what I'm doing wrong.
So I have some sql code that queries my database and returns a single value. I was therefore using ExecuteScalarAsync to get that value out into c#.
The code is as follows:
public void CheckOldTransactionsSync()
{
CheckOldTransactions().Wait();
}
public async Task CheckOldTransactions()
{
DateTimeOffset beforeThis = DateTime.UtcNow.Subtract(TimeSpan.FromHours(6));
using (SqlConnection connection = new SqlConnection(SqlConnectionString))
{
await connection.OpenAsync(cts.Token);
using (SqlCommand command = new SqlCommand(#"SELECT TOP 1 1 AS value FROM SyncLog WHERE [TimeStamp] < #BeforeThis;", connection))
{
command.Parameters.Add("#BeforeThis", System.Data.SqlDbType.DateTimeOffset, 7);
command.Prepare();
command.Parameters["#BeforeThis"].Value = beforeThis;
Int32 oldTransactions = (Int32)await command.ExecuteScalarAsync(cts.Token);
// do stuff with oldTransactions
}
}
}
So elsewhere in my code the CancellationTokenSource called cts is created and set to expire after 2 minutes using the CancelAfter method.
Now I've stepped through this code with the debugger and I reach the line where I await the call to ExecuteScalarAsync without a problem. However I seem to have two issues with the execution of that line which are that it doesn't seem to return and 2 it ignores my cancellation token and is still running some time after my two minute cancellation token has expired.
Now I've run the sql query in Sql Studio and it returns very quickly - the table has only around 4000 rows at this time.
I've resolved the problem for now by changing that line to:
Int32 oldTransactions = (Int32) command.ExecuteScalar();
Which returns almost instantaneously.
That is the only line of code I've changed and I changed it back just to make sure and the same issue occurred. So my question is, what did I do wrong with the asynchronous call?
You are calling Wait.
That's a classic ASP.NET deadlock. Don't block, or use synchronous IO.
I would like to give a user the ability to cancel a running query. The query is really slow. (Query optimization is besides the point.) This is mainly out of my curiosity.
MSDN says:
If there is nothing to cancel, nothing occurs. However, if there is a
command in process, and the attempt to cancel fails, no exception is
generated.
Cmd - SqlCommand
DA - DataAdapter
Conn - SqlConnection
CurrentSearch - Thread
LongQuery - Singleton
Here's what I have:
var t = new Thread(AbortThread);
t.Start();
void AbortThread()
{
LongQuery.Current.Cmd.Cancel();
LongQuery.Current.Cmd.Dispose();
LongQuery.Current.DA.Dispose();
LongQuery.Current.Conn.Close();
LongQuery.Current.Conn.Dispose();
LongQuery.Current.Cmd = null;
LongQuery.Current.DA = null;
LongQuery.Current.Conn = null;
CurrentSearch.Abort();
CurrentSearch.Join();
CurrentSearch = null;
}
I noticed that CurrentSearch.Abort() was blocking, that's why I wrapped it in a thread, which probably means that the thread is still working.
Finally, is there anything else than this that I can do to cancel a query? Is it actually possible to cancel such a long query from .NET?
IF you really absolutely want to kill it for good use this approach:
store away the session ID right before starting the long-running query by calling SELECT ##SPID AS 'SESSIONID' on the same connection
When you want to kill it:
Open a new DB connection
issue a KILL command for that session ID
BEWARE as the MSDN documentation states you need the permission ALTER ANY CONNECTION to do this
Yes, you can kill a process from .NET. Here is an example. Please note you will need proper permissions and you have to figure out the process in question. I don't have a quick sample of determining which process your query is running under.
You example aborts the thread, but that does not mean the work on SQL Server was terminated. If you think about it this way: when you go through a bad cell zone and the call drops, if you mom/wife/friend was droning on, do they instantly stop talking? That is an analogy of aborting the thread, at least in the case of working with a database server.
Given:
A BenchMark class that lets me know when something has completed.
A very large XML file (~120MB) that has been parsed into multiple Lists
Some code:
SqlConnection con = null;
SqlTransaction transaction = null;
try
{
con = getCon(); // gets a new connection object
con.Open();
transaction = con.BeginTransaction();
var bulkCopy = new SqlBulkCopy(con, SqlBulkCopyOptions.Default, transaction)
{
BatchSize = 1000,
DestinationTableName = "Table1"
};
// assume that the BenchMark class is working
b = new BenchMark("Table1");
bulkCopy.WriteToServer(_insertTable1s.AsDataReader()); // _insertTables1s is a List<Table1>
b.Complete();
LogHelper.WriteLogItem(b);
b = new BenchMark("Table2");
bulkCopy.DestinationTableName = "Table2";
bulkCopy.WriteToServer(_insertTable2s.AsDataReader()); // _insertTables2s is a List<Table2>
b.Complete();
LogHelper.WriteLogItem(b);
// etc... this code does a batch insert into about 7 tables all having about 40,000 records being inserted.
b = new BenchMark("Transaction Commit");
transaction.Commit();
b.Complete();
}
catch (Exception e)
{
transaction.Rollback();
LogHelper.WriteLogItem(
LogLevel.Critical,
LogType.DataProcessing,
e.ToString());
}
finally
{
con.Close();
}
The Problem:
On my local development environment, everything is fine. Its when I run this operation in the cloud that causes it to hang. Using the LogHelper.WriteLogItem method, I can watch the progress of this process. I observe it hang randomly on a particular table. No exception is thrown so the transaction isn't rolled back. Say it hangs on Table2 bulk insert. Using MS SQL Management Studio, I run queries on Table3, Table2 and Table1 with no issue (this means that the transaction was aborted?)
Since it hangs, I'll go rerun the process. This time it hangs sooner so I might get logs like this:
7755 Benchmark LoadXML took 00:00:04.2432816
7756 Benchmark Table1 took 00:00:06.3961230
7757 Benchmark Table2 took 00:00:05.2566890
7758 Benchmark Table3 took 00:00:08.4900921
7759 Benchmark Table4 took 00:00:02.0000123
... it hangs on Table5 (because the BenchMark never completed). I go to run it again and the rest of the log looks like:
7780 Benchmark LoadXML took 00:00:04.1203923
... and it hangs here now.
I'm using rackspace cloud hosting if that helps. I have been able to fix this in the past by deleting all the tables from my dbml file and readding them but this time its not working. I'm wondering if the amount of data being processed is causing the problem?
EDIT: The code in this example is run in an Asynchronous thread. I've found out that the Thread is Aborting for an unknown reason and I need to find out why to solve this problem.
If you have admin to your server or database, you can run
SELECT * FROM sys.dm_tran_session_transactions
to see what transactions are currently active - From Pinal
Additionally, you can run sp_lock to make sure there isn't something blocking your transaction.
Because this process is done asynchronously (i.e. a thread is kicked off to handle this) the thread has a problem which aborts it and that is why I get strange behavior where the code stalls at different places. I've solved this by completing this task synchronously (it works but its not ideal).
I guess the real issue is why my thread is aborting since I'm not aborting it in any of my code. I believe that its due to amount of data that is being processed, but I could be wrong.
Either way, I've solved my problem.