I'm designing a small desktop app that fetches data from SQL server. I used BackgroundWorker to make the query execute in background. The code that fetches data generally comes down to this:
public static DataTable GetData(string sqlQuery)
{
DataTable t = new DataTable();
using (SqlConnection c = new SqlConnection(GetConnectionString()))
{
c.Open();
using (SqlCommand cmd = new SqlCommand(sqlQuery))
{
cmd.Connection = c;
using (SqlDataReader r = cmd.ExecuteReader())
{
t.Load(r);
}
}
}
return t;
}
Since query can take up 10-15 minutes I want to implement cancellation request and pass it from GUI layer to DAL. Cancellation procedure of BackroundWorker won't let me cancel SqlCommand.ExecuteReader() beacuse it only stops when data is fetched from server or an exception is thrown by Data Provider.
I tried to use Task and async/await with SqlCommand.ExecuteReaderAsync(CancellationToken) but I am confused where to use it in multi-layer app (GUI -> BLL -> DAL).
Have you tried using the SqlCommand.Cancel() method ?
Aproach: encapsulate that GetData method in a Thread/Worker and then when you cancel/stop that thread call the Cancel() method on the SqlCommand that is being executed.
Here is an example on how to use it on a thread
using System;
using System.Data;
using System.Data.SqlClient;
using System.Threading;
class Program
{
private static SqlCommand m_rCommand;
public static SqlCommand Command
{
get { return m_rCommand; }
set { m_rCommand = value; }
}
public static void Thread_Cancel()
{
Command.Cancel();
}
static void Main()
{
string connectionString = GetConnectionString();
try
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
Command = connection.CreateCommand();
Command.CommandText = "DROP TABLE TestCancel";
try
{
Command.ExecuteNonQuery();
}
catch { }
Command.CommandText = "CREATE TABLE TestCancel(co1 int, co2 char(10))";
Command.ExecuteNonQuery();
Command.CommandText = "INSERT INTO TestCancel VALUES (1, '1')";
Command.ExecuteNonQuery();
Command.CommandText = "SELECT * FROM TestCancel";
SqlDataReader reader = Command.ExecuteReader();
Thread rThread2 = new Thread(new ThreadStart(Thread_Cancel));
rThread2.Start();
rThread2.Join();
reader.Read();
System.Console.WriteLine(reader.FieldCount);
reader.Close();
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
static private string GetConnectionString()
{
// To avoid storing the connection string in your code,
// you can retrieve it from a configuration file.
return "Data Source=(local);Initial Catalog=AdventureWorks;"
+ "Integrated Security=SSPI";
}
}
You can only do Cancelation checking and Progress Reporting between Distinct lines of code. Usually both require that you disect the code down to the lowest loop level, so you can do both these things between/in the loop itterations. When I wrote my first step into BGW, I had the advantage that I needed to do the loop anyway so it was no extra work. You have one of the worse cases - pre-existing code that you can only replicate or use as is.
Ideal case:
This operation should not take nearly as long is it does. 5-10 minutes indicates that there is something rather wrong with your design.
If the bulk of the time is transmission of data, then you are propably retreiving way to much data. Retrieving everything to do filtering in the GUI is a very common mistake. Do as much filtering in the query as possible. Usign a Distributed Database might also help with transmission performance.
If the bulk of the time is processing as part of the query operation (complex Conditions), something in your general approach might have to change. There are various ways to trade off complex calculation with a bit of memory on the DBMS side. Views afaik can cache the results of operations, while still maintaining transactional consistency.
But it really depends what your backend DB/DBMS and use case are. A lot of the use SQL as Query Language. So it does not allow us to predict wich options you have.
Second best case:
The second best thing if you can not cut it down, would be if you had the actually DB access code down to the lowest loop and would do progress reporting/cancelation checking on it. That way you could actually use the existing Cancelation Token System inherent in BGW.
Everything else
Using any other approach to Cancelation is really a fallback. I wrote a lot on why it is bad, but felt that this might work better if I focus on the core issue - likely something wrong in design of he DB and/or Query. Because those might well eliminate the issue altogether.
Related
I have the following code:
public void Execute(string Query, params SqlParameter[] Parameters)
{
using (var Connection = new SqlConnection(Configuration.ConnectionString))
{
Connection.Open();
using (var Command = new SqlCommand(Query, Connection))
{
if (Parameters.Length > 0)
{
Command.Parameters.Clear();
Command.Parameters.AddRange(Parameters);
}
Command.ExecuteNonQuery();
}
}
}
The method may be called 2 or 3 times for different queries but in same manner.
For example:
Insert an Employee
Insert Employee Certificates
Update Degree of Employee on another table [ Fail can cause here. for example ]
If Point [3] fails, all already committed commands shouldn't execute and must be rolled back.
I know I can put SqlTransaction above and use Commit() method. But what about 3rd point if failed? I think point 3 only will rollback and other point 1,2 will not? How to solve this and what approach should I do??
Should I use SqlCommand[] arrays? What I should I do?
I only find similar question but in CodeProject:
See Here
Without changing your Execute method you can do this
var tranOpts = new TransactionOptions()
{
IsolationLevel = IsolationLevel.ReadCommitted,
Timeout = TransactionManager.MaximumTimeout
};
using (var tran = new TransactionScope(TransactionScopeOption.Required, tranOpts)
{
Execute("INSERT ...");
Execute("INSERT ...");
Execute("UPDATE ...");
tran.Complete();
}
SqlClient will cache the internal SqlConnection that is enlisted in the Transaction and reuse it for each call to Execute. So you even end up with a local (not distributed) transaction.
This is all explained in the docs here: System.Transactions Integration with SQL Server
There are a few ways to do it.
The way that probably involves changing the least code and involves the least complexity is to chain multiple SQL statements into a single query. It's perfectly fine to build a string for the Query argument that runs more than one statement, including BEGIN TRANSACTION, COMMIT, and (if needed) ROLLBACK. Basically, keep a whole stored procedure in your C# code. This also has the nice benefit of making it easier to use version control with your procedures.
But it still feels kind of hackish.
One way to reduce that effect is marking the Execute() method private. Then, have an additional method in the class for each query. In this way, the long SQL strings are isolated, and when you're using the database it feels more like using a local API. For more complicated applications, this might instead be a whole separate assembly with a few types managing logical functional areas, where the core methods like Exectue() are internal. This is a good idea anyway, regardless of how you end up supporting transactions.
And speaking of procedures, stored procedures are also a perfectly fine way to handle this. Have one stored procedure to do all the work, and call it when ready.
Another option is overloading the method to accept multiple queries and parameter collections:
public void Execute(string TransactionName, string[] Queries, params SqlParameter[][] Parameters)
{
using (var Connection = new SqlConnection(Configuration.ConnectionString))
using (var Transaction = new SqlTransaction(TransactionName))
{
connection.Transaction = Transaction;
Connection.Open();
try
{
for (int i = 0; i < Queries.Length; i++)
{
using (var Command = new SqlCommand(Queries[i], Connection))
{
command.Transaction = Transaction;
if (Parameters[i].Length > 0)
{
Command.Parameters.Clear();
Command.Parameters.AddRange(Parameters);
}
Command.ExecuteNonQuery();
}
}
Transaction.Commit();
}
catch(Exception ex)
{
Transaction.Rollback();
throw; //I'm assuming you're handling exceptions at a higher level in the code
}
}
}
Though I'm not sure how the params keyword works with an array of arrays... I've just not tried that option, but something along these lines would work. The weakness here is also that it's not trivial to have a later query depend on a result from an earlier query, and even queries with no parameter would still need a Parameters array as a placeholder.
A final option is extending the type holding your Execute() method to support transactions. The trick here is it's common (and desirable) to have this type be static, but supporting transactions requires re-using common connection and transaction objects. Given the implied long-running nature of a transaction, you have to support more than one at a time, which means both instances and implementing IDisposable.
using (var connection = new SqlConnection(Configuration.ConnectionString))
{
SqlCommand command = connection.CreateCommand();
SqlTransaction transaction;
connection.Open();
transaction = connection.BeginTransaction("Transaction");
command.Connection = connection;
command.Transaction = transaction;
try
{
if (Parameters.Length > 0)
{
command.Parameters.Clear();
command.Parameters.AddRange(Parameters);
}
command.ExecuteNonQuery();
transaction.Commit();
}
catch (Exception e)
{
try
{
transaction.Rollback();
}
catch (Exception ex2)
{
//trace
}
}
}
So the title is a bit vague, but the question is really this: in practice is it best to make a data access class, in this case access to a SQL Server.
Where all the static methods of the class need a connection string and a SQL statement?
Something along these lines:
public static void ExecuteSql(string connStr, string strSqlStatement)
{
SqlConnection conn = new SqlConnection(connStr);
conn.Open();
SqlCommand cmd = new SqlCommand(strSqlStatement, conn);
cmd.ExecuteNonQuery();
conn.Close();
conn.Dispose();
}
And then a method that does something similar except it returns data you queried for.
Or in practice, is it better to create an object for the specific application you are building, and code these items as the objects extensions. So the SQL to run or stored procedure to execute would be wrapped up in that class.
obj.GetSomethingViaQueryThatIsWrittenInTheClassLibrary()
So I guess the first one is more like a service library for accessing and writing data in a SQL Server database. What does one typically do in this situation?
Or could you even use the service library in tandem with the objects library?
There's a ton of, not conflicting, but different opinions on data access and I am really trying to see what is more common a practice.
I disagree with Terry, because:
The connections are pooled, so opening/closing connections is not a problem and you don't want to use a single connection to execute multiple concurrent queries. This is very usefull when having many concurrent threads (for example webservers) but this also applies on a normal application (which might uses Tasks to retrieve data on a separate thread to keep the UI responsive)
So I would create a ConnectionManager class that uses a connection string as constructor parameter, this way the connectionstring wouldn't 'travel' thru your program and is encapsulated in a 'manager' object.
This is a poor example, but I think it work just fine when using ADO.NET
public class ConnectionManager
{
private string _connectionString;
public ConnectionManager(string connectionString)
{
_connectionString = connectionString;
}
public SqlConnection GetConnection()
{
return new SqlConnection(_connectionString);
}
}
Then I would use it something like:
var connectionManager = new ConnectionManager(connectionString);
using(var con = connectionManager.GetConnection())
{
// not all operations require .Open()/.Close()
// multiple queries.
}
using(var con = connectionManager.GetConnection())
{
// not all operations require .Open()/.Close()
// multiple other queries.
}
By using using the connection will be disposed (put back in the pool)
You might even use something like:
public class ConnectionManager
{
private string _connectionString;
public ConnectionManager(string connectionString)
{
_connectionString = connectionString;
}
public void ExecuteNonQuery(string strSqlStatement)
{
using(var connection = new SqlConnection(_connectionString))
using(var command = new SqlCommand(strSqlStatement, connection))
{
connection.Open();
command.ExecuteNonQuery();
}
}
}
Which makes:
var connectionManager = new ConnectionManager(connectionString);
connectionManager.ExecuteNonQuery("SELECT * FROM Whatever");
Tip: You sure need to checkout the SqlParameter to prevent SQL Injection
I wouldn't create, open, and close the connection in an sql execute method like that. Instead, I recommend creating the connection and pass it in to the query method each time a query is needed and when all queries are finished, then close the connection.
Is there a better way than the below to detect if the value retrieved from a database is different to the last retrieved value?
I have a feeling that something better than in infinite poll is available out there?
public void CheckForNewMofificationDate(string username)
{
while(true)
{
OdbcConnection sql = null;
if (!DBClass.Instance.OpenConn(ref sql))
throw new DatabaseConnectionException();
try
{
string query = "SELECT MODIFIED_ON FROM USER_DTLS WHERE USERNAME=?";
using (var cmd = new OdbcCommand(query, sql))
{
cmd.Parameters.Add("USERNAME", OdbcType.VarChar, 50).Value = username;
using (var reader = cmd.ExecuteReader())
{
if (reader.Read())
{
if( OldValue != reader.GetString(0))
{
//use INotifyPropertyChange
}
}
}
}
}
finally
{
DBClass.Instance.CloseConn(ref sql);
}
}
}
Short answer: you would have to employ a polling (looping) mechanism like you suggested.
Or, you could do something crazy with triggers on the database and have the trigger execute a custom function or web service that uses an event bus or WCF to notify your application of a change in data, but I would highly recommend not pursuing this approach.
As recommended by #TimSchmelter, A SqlDependancy is the best approach I found so far, it causes Sql Server to detect changes made to tables assoiciated with a query and fire events based on that:
A SqlDependency object can be associated with a SqlCommand in order to
detect when query results differ from those originally retrieved. You
can also assign a delegate to the OnChange event, which will fire when
the results change for an associated command. You must associate the
SqlDependency with the command before you execute the command. The
HasChanges property of the SqlDependency can also be used to determine
if the query results have changed since the data was first retrieved.
This eliminates the need to have a serprate thread with an infinite loop continuasslt polling to detect changes.
Given the following SQLCLR function:
[Microsoft.SqlServer.Server.SqlProcedure]
public static void ExecSQL(string sql, string connectionString)
{
WindowsIdentity clientId = null;
WindowsImpersonationContext impersonatedUser = null;
clientId = SqlContext.WindowsIdentity;
try
{
try
{
impersonatedUser = clientId.Impersonate();
if (impersonatedUser != null)
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
//** HERE I WILL HAVE MULTIPLE VARIATIONS OF FETCHING THE SQLDATAREADER
connection.Open();
SqlCommand command = new SqlCommand(sql, connection);
SqlDataReader r = command.ExecuteReader();
impersonatedUser.Undo();
SqlContext.Pipe.Send(r);
}
}
else
{ throw new Exception("Impersonation failed."); }
}
finally
{
if (impersonatedUser != null) { impersonatedUser.Undo(); }
}
}
catch
{
throw;
}
}
Are there any performance or other ramifications of moving this section:
connection.Open();
SqlCommand command = new SqlCommand(sql, connection);
SqlDataReader r = command.ExecuteReader();
out into a separate GetDataReader() method?
I ask because I know I am going to want to load data readers for many different scenarios (Sql text, Stored Procedure, Table Valued Function, Scalar Function, etc), so I would like to encapsulate each of those different implementations into their own function rather than having a big switch statement in the middle of this function.
Are there any performance or other ramifications
Not that I can think of or have run into. Of course, when it comes to performance-related questions of this nature, I always recommend testing it out to see which is better, because anyone answering questions can always be wrong, and there are definitely cases of "accepted" answers being incorrect. Your software running on your system, however, is the definitive answer :-).
Beyond that, there are a few notes about the code fragment shown in the question.
You could move the impersonatedUser.Undo(); to just after the connection.Open();. It was only needed to establish the connection. But, then again, perhaps it would be cleaner to keep the Impersonation handling in the main method and not move just the Undo() to the new method.
OR, you could also keep the connection.Open(); in the main method here instead. You are going to have to pass along the connection object in either case.
You could wrap the impersonatedUser = clientId.Impersonate(); in an if condition, testing the connectionString to see if it is "Context Connection = true;". This would allow you to use the Context Connection, which otherwise won't work since it can't be used with Impersonation. I suppose you can just re-work the current if (impersonatedUser != null) since that test would no longer be valid (the new test would only care if impersonatedUser was null if the Context Connection wasn't being use.
WindowsImpersonationContext is Disposable, so it would be better to move your finally block to the outer try, and add impersonatedUser.Dispose(); after the Undo().
Background: I'm writing a function putting long lasting operations in a queue, using C#,
and each operation is kind of divided into 3 steps:
1. database operation (update/delete/add data)
2. long time calculation using web service
3. database operation (save the calculation result of step 2) on the same db table in step 1, and check the consistency of the db table, e.g., the items are the same in step 1 (Pls see below for a more detailed example)
In order to avoid dirty data or corruptions, I use a lock object (a static singleton object) to ensure the 3 steps to be done as a whole transaction. Because when multiple users are calling the function to do operations, they may modify the same db table at different steps during their own operations without this lock, e.g., user2 is deleting item A in his step1, while user1 is checking if A still exists in his step 3. (additional info: Meanwhile I'm using TransactionScope from Entity framework to ensure each database operation as a transaction, but as repeat readable.)
However, I need to put this to a cloud computing platform which uses load balancing mechanism, so actually my lock object won't take effect, because the function will be deployed on different servers.
Question: what can I do to make my lock object working under above circumstance?
This is a tricky problem - you need a distributed lock, or some sort of shared state.
Since you already have the database, you could change your implementation from a "static C# lock" and instead the database to manage your lock for you over the whole "transaction".
You don't say what database you are using, but if it's SQL Server, then you can use an application lock to achieve this. This lets you explicitly "lock" an object, and all other clients will wait until that object is unlocked. Check out:
http://technet.microsoft.com/en-us/library/ms189823.aspx
I've coded up an example implementation below. Start two instances to test it out.
using System;
using System.Data;
using System.Data.SqlClient;
using System.Transactions;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
var locker = new SqlApplicationLock("MyAceApplication",
"Server=xxx;Database=scratch;User Id=xx;Password=xxx;");
Console.WriteLine("Aquiring the lock");
using (locker.TakeLock(TimeSpan.FromMinutes(2)))
{
Console.WriteLine("Lock Aquired, doing work which no one else can do. Press any key to release the lock.");
Console.ReadKey();
}
Console.WriteLine("Lock Released");
}
class SqlApplicationLock : IDisposable
{
private readonly String _uniqueId;
private readonly SqlConnection _sqlConnection;
private Boolean _isLockTaken = false;
public SqlApplicationLock(
String uniqueId,
String connectionString)
{
_uniqueId = uniqueId;
_sqlConnection = new SqlConnection(connectionString);
_sqlConnection.Open();
}
public IDisposable TakeLock(TimeSpan takeLockTimeout)
{
using (TransactionScope transactionScope = new TransactionScope(TransactionScopeOption.Suppress))
{
SqlCommand sqlCommand = new SqlCommand("sp_getapplock", _sqlConnection);
sqlCommand.CommandType = CommandType.StoredProcedure;
sqlCommand.CommandTimeout = (int)takeLockTimeout.TotalSeconds;
sqlCommand.Parameters.AddWithValue("Resource", _uniqueId);
sqlCommand.Parameters.AddWithValue("LockOwner", "Session");
sqlCommand.Parameters.AddWithValue("LockMode", "Exclusive");
sqlCommand.Parameters.AddWithValue("LockTimeout", (Int32)takeLockTimeout.TotalMilliseconds);
SqlParameter returnValue = sqlCommand.Parameters.Add("ReturnValue", SqlDbType.Int);
returnValue.Direction = ParameterDirection.ReturnValue;
sqlCommand.ExecuteNonQuery();
if ((int)returnValue.Value < 0)
{
throw new Exception(String.Format("sp_getapplock failed with errorCode '{0}'",
returnValue.Value));
}
_isLockTaken = true;
transactionScope.Complete();
}
return this;
}
public void ReleaseLock()
{
using (TransactionScope transactionScope = new TransactionScope(TransactionScopeOption.Suppress))
{
SqlCommand sqlCommand = new SqlCommand("sp_releaseapplock", _sqlConnection);
sqlCommand.CommandType = CommandType.StoredProcedure;
sqlCommand.Parameters.AddWithValue("Resource", _uniqueId);
sqlCommand.Parameters.AddWithValue("LockOwner", "Session");
sqlCommand.ExecuteNonQuery();
_isLockTaken = false;
transactionScope.Complete();
}
}
public void Dispose()
{
if (_isLockTaken)
{
ReleaseLock();
}
_sqlConnection.Close();
}
}
}
}