I am working on a setup where a scalable WCF Service Component is connected to a single MS SQL Server Database. The RESTful service allows users to save data into the DB as well as get data from it.
Whilst implementing a class handling the database connections / methods, I started struggling with correctly reusing prepared SqlCommands and the connection. I read up on the MSDN about connection pooling as well as how to use SqlCommand and SqlParameter.
My initial version of the class looks like this:
public class SqlRepository : IDisposable
{
private object syncRoot = new object();
private SqlConnection connection;
private SqlCommand saveDataCommand;
private SqlCommand getDataCommand;
public SqlRepository(string connectionString)
{
// establish sql connection
connection = new SqlConnection(connectionString);
connection.Open();
// save data
saveDataCommand = new SqlCommand("INSERT INTO Table (Operation, CustomerId, Data, DataId, CreationDate, ExpirationDate) VALUES (#Operation, #CustomerId, #Data, #DataId, #CreationDate, #ExpirationDate)", connection);
saveDataCommand.Parameters.Add(new SqlParameter("Operation", SqlDbType.NVarChar, 20));
saveDataCommand.Parameters.Add(new SqlParameter("CustomerId", SqlDbType.NVarChar, 50));
saveDataCommand.Parameters.Add(new SqlParameter("Data", SqlDbType.NVarChar, 50));
saveDataCommand.Parameters.Add(new SqlParameter("DataId", SqlDbType.NVarChar, 50));
saveDataCommand.Parameters.Add(new SqlParameter("CreationDate", SqlDbType.DateTime));
saveDataCommand.Parameters.Add(new SqlParameter("ExpirationDate", SqlDbType.DateTime));
saveDataCommand.Prepare();
// get data
getTripCommand = new SqlCommand("SELECT TOP 1 Data FROM Table WHERE CustomerId = #CustomerId AND DataId = #DataId AND ExpirationDate > #ExpirationDate ORDER BY CreationDate DESC", connection);
getTripCommand.Parameters.Add(new SqlParameter("CustomerId", SqlDbType.NVarChar, 50));
getTripCommand.Parameters.Add(new SqlParameter("DataId", SqlDbType.NVarChar, 50));
getTripCommand.Parameters.Add(new SqlParameter("ExpirationDate", SqlDbType.DateTime));
getTripCommand.Prepare();
}
public void SaveData(string customerId, string dataId, string operation, string data, DateTime expirationDate)
{
lock (syncRoot)
{
saveDataCommand.Parameters["Operation"].Value = operation;
saveDataCommand.Parameters["CustomerId"].Value = customerId;
saveDataCommand.Parameters["CreationDate"].Value = DateTime.UtcNow;
saveDataCommand.Parameters["ExpirationDate"].Value = expirationDate;
saveDataCommand.Parameters["Data"].Value = data;
saveDataCommand.Parameters["DataId"].Value = dataId;
saveDataCommand.ExecuteNonQuery();
}
}
public string GetData(string customerId, string dataId)
{
lock (syncRoot)
{
getDataCommand.Parameters["CustomerId"].Value = customerId;
getDataCommand.Parameters["DataId"].Value = dataId;
getDataCommand.Parameters["ExpirationDate"].Value = DateTime.UtcNow;
using (var reader = getDataCommand.ExecuteReader())
{
if (reader.Read())
{
string data = reader.GetFieldValue<string>(0);
return data;
}
else
{
return null;
}
}
}
}
public void Dispose()
{
try
{
if (connection != null)
{
connection.Close();
connection.Dispose();
}
DisposeCommand(saveDataCommand);
DisposeCommand(getDataCommand);
}
catch { }
}
private void DisposeCommand(SqlCommand command)
{
try
{
command.Dispose();
}
catch (Exception)
{
}
}
}
There are several aspects important to know:
I am using SqlCommand.Prepare() to speed up the process of executing the command
Reusing the commands avoids creating new objects with every call to the GetData and SaveData methods, thus leading to no problem with the garbage collector
There is only one instance of the SqlRepository class, used by the WCF Service.
There are many many calls per minute to this service, so keeping a connection to the DB open is what I want.
Now I read up a bit more about connection pooling and the fact that it is highly recommended to use the SqlConnection object in a using statement to ensure disposal. To my understanding, the connection pooling technology takes care of leaving the connection open even though the Dispose() method of SqlConnection has been called by the using statement.
The way to use this would be to have a using(SqlConnection connection = new SqlConnection(connectionString)) inside the GetData and SaveData methods. However, then - at least to my intuition - I would need to create the SqlCommands inside the GetData / SaveData methods as well. Or not? I could not find any documentation on how to reuse the commands that way. Also wouldn't the call to SqlCommand.Prepare() be meaningless if I need to prepare a new command every time I get into the GetData / SaveData methods?
How do I properly implement the SqlRepository class? The way it is now I believe that if the connection breaks (maybe because the DB server goes down for a while and reboots), then the SqlRepository class will not automatically recover and be functioning. To my best knowledge this sort of failsave scenarios are handled in the pooling technology.
Thanks for ideas and feedback!
Christian
Do not reuse the SqlCommand instances.
You are synchronizing database access.
With your implementation, you are re-using a small object (which is no problem for the GC even if there are thousands) in exchange of concurrent DB operations.
Remove the synchronization locks.
Create new instances of SqlCommands for each database operation.
Do not call Prepare. Prepare speeds up db operations, but after executing ExecuteReader() on a SqlCommand with CommandType = Text and with non-zero number of parameters, the command is unprepared internally.
Related
So the title is a bit vague, but the question is really this: in practice is it best to make a data access class, in this case access to a SQL Server.
Where all the static methods of the class need a connection string and a SQL statement?
Something along these lines:
public static void ExecuteSql(string connStr, string strSqlStatement)
{
SqlConnection conn = new SqlConnection(connStr);
conn.Open();
SqlCommand cmd = new SqlCommand(strSqlStatement, conn);
cmd.ExecuteNonQuery();
conn.Close();
conn.Dispose();
}
And then a method that does something similar except it returns data you queried for.
Or in practice, is it better to create an object for the specific application you are building, and code these items as the objects extensions. So the SQL to run or stored procedure to execute would be wrapped up in that class.
obj.GetSomethingViaQueryThatIsWrittenInTheClassLibrary()
So I guess the first one is more like a service library for accessing and writing data in a SQL Server database. What does one typically do in this situation?
Or could you even use the service library in tandem with the objects library?
There's a ton of, not conflicting, but different opinions on data access and I am really trying to see what is more common a practice.
I disagree with Terry, because:
The connections are pooled, so opening/closing connections is not a problem and you don't want to use a single connection to execute multiple concurrent queries. This is very usefull when having many concurrent threads (for example webservers) but this also applies on a normal application (which might uses Tasks to retrieve data on a separate thread to keep the UI responsive)
So I would create a ConnectionManager class that uses a connection string as constructor parameter, this way the connectionstring wouldn't 'travel' thru your program and is encapsulated in a 'manager' object.
This is a poor example, but I think it work just fine when using ADO.NET
public class ConnectionManager
{
private string _connectionString;
public ConnectionManager(string connectionString)
{
_connectionString = connectionString;
}
public SqlConnection GetConnection()
{
return new SqlConnection(_connectionString);
}
}
Then I would use it something like:
var connectionManager = new ConnectionManager(connectionString);
using(var con = connectionManager.GetConnection())
{
// not all operations require .Open()/.Close()
// multiple queries.
}
using(var con = connectionManager.GetConnection())
{
// not all operations require .Open()/.Close()
// multiple other queries.
}
By using using the connection will be disposed (put back in the pool)
You might even use something like:
public class ConnectionManager
{
private string _connectionString;
public ConnectionManager(string connectionString)
{
_connectionString = connectionString;
}
public void ExecuteNonQuery(string strSqlStatement)
{
using(var connection = new SqlConnection(_connectionString))
using(var command = new SqlCommand(strSqlStatement, connection))
{
connection.Open();
command.ExecuteNonQuery();
}
}
}
Which makes:
var connectionManager = new ConnectionManager(connectionString);
connectionManager.ExecuteNonQuery("SELECT * FROM Whatever");
Tip: You sure need to checkout the SqlParameter to prevent SQL Injection
I wouldn't create, open, and close the connection in an sql execute method like that. Instead, I recommend creating the connection and pass it in to the query method each time a query is needed and when all queries are finished, then close the connection.
How to correctly use one SqlConnection object for multiple queries?
SqlConnection connection = new SqlConnection(connString);
static void SqlQuery(SqlConnection conn, string cmdString)
{
using (conn)
{
if (conn.State != ConnectionState.Open)
{
conn.Close();
conn.Open();
}
SqlCommand cmd = conn.CreateCommand();
cmd.CommandText = cmdString;
cmd.ExecuteNonQuery();
}
}
Function SqlQuery after 1st invoking throws System.InvalidOperationException "ConnectionString property not initialized"
In short don't do it
Creating a new instance of the class SqlConnection does not create a new network connection to SQL Server, but leases an existing connection (or creates a new one). .NET handles the physical connection pooling for you.
When you have finished with your connection (through which you can send multiple queries) just Close() or Dispose() (or use a using{} block preferably).
There is no need, and not good practise, to cache instances of the SqlConnection class.
Update
This is a better pattern for your method, you dont have to worry about the connections state
static void SqlQuery(string cmdString)
{
using (var connection = new SqlConnection(connString))
using (var cmd = connection.CreateCommand(cmdString, connection))
{
connection.Open();
// query
cmd.ExecuteNonQuery();
}
}
It depends on what you really mean/intend to do. If you mean batching a set of commands? Then yes,
it's arguably better to use one connection. Yes, connection pooling does save (all of) us, but if you really thought about it, what does it do? Yup, it reuses connections...
Performing Batch Operations
tips/pointers on SqlCommand as well
Hth.
Background: I'm writing a function putting long lasting operations in a queue, using C#,
and each operation is kind of divided into 3 steps:
1. database operation (update/delete/add data)
2. long time calculation using web service
3. database operation (save the calculation result of step 2) on the same db table in step 1, and check the consistency of the db table, e.g., the items are the same in step 1 (Pls see below for a more detailed example)
In order to avoid dirty data or corruptions, I use a lock object (a static singleton object) to ensure the 3 steps to be done as a whole transaction. Because when multiple users are calling the function to do operations, they may modify the same db table at different steps during their own operations without this lock, e.g., user2 is deleting item A in his step1, while user1 is checking if A still exists in his step 3. (additional info: Meanwhile I'm using TransactionScope from Entity framework to ensure each database operation as a transaction, but as repeat readable.)
However, I need to put this to a cloud computing platform which uses load balancing mechanism, so actually my lock object won't take effect, because the function will be deployed on different servers.
Question: what can I do to make my lock object working under above circumstance?
This is a tricky problem - you need a distributed lock, or some sort of shared state.
Since you already have the database, you could change your implementation from a "static C# lock" and instead the database to manage your lock for you over the whole "transaction".
You don't say what database you are using, but if it's SQL Server, then you can use an application lock to achieve this. This lets you explicitly "lock" an object, and all other clients will wait until that object is unlocked. Check out:
http://technet.microsoft.com/en-us/library/ms189823.aspx
I've coded up an example implementation below. Start two instances to test it out.
using System;
using System.Data;
using System.Data.SqlClient;
using System.Transactions;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
var locker = new SqlApplicationLock("MyAceApplication",
"Server=xxx;Database=scratch;User Id=xx;Password=xxx;");
Console.WriteLine("Aquiring the lock");
using (locker.TakeLock(TimeSpan.FromMinutes(2)))
{
Console.WriteLine("Lock Aquired, doing work which no one else can do. Press any key to release the lock.");
Console.ReadKey();
}
Console.WriteLine("Lock Released");
}
class SqlApplicationLock : IDisposable
{
private readonly String _uniqueId;
private readonly SqlConnection _sqlConnection;
private Boolean _isLockTaken = false;
public SqlApplicationLock(
String uniqueId,
String connectionString)
{
_uniqueId = uniqueId;
_sqlConnection = new SqlConnection(connectionString);
_sqlConnection.Open();
}
public IDisposable TakeLock(TimeSpan takeLockTimeout)
{
using (TransactionScope transactionScope = new TransactionScope(TransactionScopeOption.Suppress))
{
SqlCommand sqlCommand = new SqlCommand("sp_getapplock", _sqlConnection);
sqlCommand.CommandType = CommandType.StoredProcedure;
sqlCommand.CommandTimeout = (int)takeLockTimeout.TotalSeconds;
sqlCommand.Parameters.AddWithValue("Resource", _uniqueId);
sqlCommand.Parameters.AddWithValue("LockOwner", "Session");
sqlCommand.Parameters.AddWithValue("LockMode", "Exclusive");
sqlCommand.Parameters.AddWithValue("LockTimeout", (Int32)takeLockTimeout.TotalMilliseconds);
SqlParameter returnValue = sqlCommand.Parameters.Add("ReturnValue", SqlDbType.Int);
returnValue.Direction = ParameterDirection.ReturnValue;
sqlCommand.ExecuteNonQuery();
if ((int)returnValue.Value < 0)
{
throw new Exception(String.Format("sp_getapplock failed with errorCode '{0}'",
returnValue.Value));
}
_isLockTaken = true;
transactionScope.Complete();
}
return this;
}
public void ReleaseLock()
{
using (TransactionScope transactionScope = new TransactionScope(TransactionScopeOption.Suppress))
{
SqlCommand sqlCommand = new SqlCommand("sp_releaseapplock", _sqlConnection);
sqlCommand.CommandType = CommandType.StoredProcedure;
sqlCommand.Parameters.AddWithValue("Resource", _uniqueId);
sqlCommand.Parameters.AddWithValue("LockOwner", "Session");
sqlCommand.ExecuteNonQuery();
_isLockTaken = false;
transactionScope.Complete();
}
}
public void Dispose()
{
if (_isLockTaken)
{
ReleaseLock();
}
_sqlConnection.Close();
}
}
}
}
I have two method “ExecuteNoQuery” (performs dbCommand.ExecuteNonQuery()) and “Query” performs (dbCommand.ExecuteReader()).
Both the methods are using same connection object. In ExecuteNoQuery method a lock is implemented(using connection object) and Query method implemented with out lock. In case of multiple thred, different thread accessing both the method simultaneously then what will happen?
Note: In Query method custom connection pooling is implemented with the same object.
public int ExecuteNoQuery(string sqlquery, Hashtable htData) {
try {
lock(Myservice.dbcon)
{
using (OracleCommand dbCommand = new OracleCommand(sqlquery, Myservice.dbcon))
{
int rowCount = dbCommand.ExecuteNonQuery();
return 1;
}
}
}
public OracleDataReader Query(string sqlquery, Hashtable htData)
{
try
{
OracleDataReader dbReader = null;
Random ran = new Random();
int randomnumber = ran.Next(1,5);
Myservice.dbcon = (OracleConnection) Myservice.htdbcon
["Connection_" +randomnumber];
if (Myservice.dbcon.State != System.Data.ConnectionState.Executing
|| Myservice.dbcon != System.Data.ConnectionState.Fetching)
{
using (OracleCommand dbCommand = new OracleCommand(sqlquery,
Myservice.dbcon))
{
dbReader = dbCommand.ExecuteReader();
}
}
return dbReader;
}
Both the methods are using same connection object.
Since one method uses a lock and the other does not: bad things. No guarantees are made by the object for this scenario, so you should expect it to fail in interesting ways. You should use the same lock object from both places, or better: only use a connection in isolated code, not a shared connection. With connection pooling, it is very rarely useful to have a shared connection object somewhere. A far more suitable pattern is usually to obtain a connection when you need it, and then dispose it. If the underlying provider supports pooling, this will perform ideally, without any issues of synchronization, and will allow parallel queries etc. For example:
using (var conn = SomeUtilityClass.GetOpenConnection())
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = sqlquery;
int rowCount = dbCommand.ExecuteNonQuery();
return 1;
}
and, importantly, do the same from the Query method; no locks, no global shared connections.
I'd also be concerned by the lack of parameters, btw. That suggests you are opening yourself up to SQL injection errors.
I have a Database class that abstracts the ExecuteNonQuery() and ExecuteReader() of SqlCommand. Due to wrapping the Sqlconnection and SqlCommand around using blocks, the SqlDataReader gets closed after the CustomExecuteReader() is called, therefore I can't read the SqlReaderResultSet at the business level layer. Code below. Thanks guys for the feedback.
public static SqlDataReader SqlReaderResultSet { get; set; }
public static SqlDataReader CustomExecuteReader(string storedProc)
{
using (var conn = new SqlConnection(ConnectionString))
{
var cmd = new SqlCommand(storedProc, conn) {CommandType = CommandType.StoredProcedure};
try
{
conn.Open();
SqlReaderResultSet = cmd.ExecuteReader();
}
catch (InvalidOperationException)
{
if (conn.State.Equals(ConnectionState.Closed))
conn.Open();
}
finally
{
conn.Close();
}
}
return SqlReaderResultSet;
}
"I can't read the SqlReaderResultSet at the business level layer" - and you shouldn't. Data should be passed using data transfer objects, never through a low level data access structure.
I recommend changing your approach so that the method you describe above iterates the records in the datareader, and creates a list of objects. That list of objects is what should be returned and worked on.
Iterator Blocks can be a way around this. It is legal and generally safe to do the following:
IEnumerable<MyFancyData> ResultSet {
get {
using(DbConnection conn = ...)
using(DbCommand cmd = ...) {
conn.Open();
using(DbDataReader reader = cmd.ExecuteReader()) {
while(reader.Read()) {
yield return new MyFancyData(reader[0], reader[42] ...);
}
}
}
}
}
Each time you enumerate the ResultSet property, the connection will be constructed again - and Disposed of afterwards (foreach and other IEnumerator<> consumers will appropriately call the Dispose() method of the generator, allowing the using block to do its thing).
This approach retains the lazy as-you-need it evaluation of the items from the data reader (which can be relevant when your data set becomes large), which still cleaning abstracting away sql-level details from the public API.