Develop Reference

Avoid "The SqlParameter is already contained by another SqlParameterCollection" exception when use retryPolicy with SqlAzureExecutionStrategy

I've read various questions/suggestions about this exception. However what I am supposed to do in order to avoid it when I use retry policy? Connection might not end up closed and so parameters could not be reused?
public class ReliableSqlCommand
{
public List<ResultType> ExecuteReader<ResultType>() where ResultType : new()
{
var list = new List<ResultType>();
var retryPolicy = new DWSqlAzureExecutionStrategy(SqlMaxRetryCount, SqlMaxDelay);
retryPolicy.Execute(() =>
{
list = new List<ResultType>();
using (var sqlConnection = new SqlConnection(ConnectionString))
{
using (var sqlCommand = new SqlCommand(CommandText, sqlConnection))
{
sqlCommand.CommandTimeout = CommandTimeout;
sqlCommand.CommandType = CommandType;
sqlCommand.Parameters.AddRange(Parameters.ToArray());
sqlCommand.Connection = sqlConnection;
sqlConnection.Open();
using (SqlDataReader dataReader = sqlCommand.ExecuteReader())
{
while (dataReader.Read())
{
if (typeof(ResultType).BaseType == typeof(System.ValueType))
{
var sqlValue = dataReader.GetValue(0);
if (sqlValue == DBNull.Value)
list.Add(default);
else
list.Add((ResultType)ChangeType(sqlValue, typeof(ResultType)));
}
else
{
//handle complex types (objects)
ResultType item = new ResultType();
Type itemType = item.GetType();
for (int columnNr = 0; columnNr < dataReader.FieldCount; columnNr++)
{
PropertyInfo prop = itemType.GetProperty(dataReader.GetName(columnNr));
if (prop == null) continue;
var value = dataReader.GetValue(columnNr);
if (value == null || value == DBNull.Value)
{
prop.SetValue(item, null);
}
else
{
prop.SetValue(item, value);
}
}
list.Add(item);
}
}
sqlConnection.Close();
}
sqlCommand.Parameters.Clear();
}
}
});
return list;
}
}
ReliableSqlCommand contains this as a property:
public List<SqlParameter> Parameters { get; } = new List<SqlParameter>();

After reviewing your code, I could imagine the following. (Note that I haven't tested it.)
You pass a function to retryPolicy.Execute(), which seems to correctly handle your database actions, disposing all connections, commands, datareaders, etc.
However, I assume that the retryPolicy can already start executing a new run of that function while a previous run is still active/running (or at least not yet fully completed). In that case, the parameters in ReliableSqlCommand.Parameters will be added to a new instance of SqlCommand, which is clearly not allowed when those parameters are still "alive" in a previous running function call in the background (which is still waiting for a database timeout exception, perhaps).
I do not see a straightforward stable/reliable fix for this.
Within the function, you could try to make new copies/instances of the Parameter objects and assign those copies to the SqlCommand instance. But in case you have output parameters, you will have to update the ReliableSqlCommand.Parameters collection afterwards. When having multiple running/overlapping function calls, that might be tricky as well.

I think what you need to do is either to ensure the parameters are removed from the old command, or cache the command
If I understand correctly, the Execute function retries the lambda, and swallows any exceptions along the way. It does not execute multiple times concurrently.
Unfortunately, SqlCommand.Dispose does not remove the parameters from the command.
So option 1 is:
using (var sqlCommand = new SqlCommand(CommandText, sqlConnection))
{
try
{
.......
}
finally
{
sqlCommand.Parameters.Clear();
}
}
A better option in my opinion, given that a parameter is supposed to be used with only one command, is to cache the command also.
There is nothing wrong with this, as long as the connection is changed each time.
public ReliableSqlCommand
{
public SqlCommand Command { get; set; }
Then instead of using (var sqlCommand = new SqlCommand..., just use the existing _command:
_command.Connection = sqlConnection;
If you don't want to expose your command object directly, you could make a wrapper that adds and deletes the parameters.
It's not strictly necessary to dispose SqlCommand, because it's Dispose does nothing. But for consistency's sake, you may want to have ReliableSqlCommand be disposable as well.

Refactoring to use a generic function with different input objects

I'd like to refactor some code that runs in a console app. The App updates an external database, and it was recently updated to support either MySQL or SQL Server. So now there are two nearly identical methods with a lot of duplicate code in them, because one has a method signature that uses MySqlConnection and MySqlCommand (etc) and the other uses SqlConnection and SqlCommand (etc).
The code is essentially identical, other than the obvious differences in the ADO objects.
What I'd like to do is something like the following. I've seen several posts here on SO (for e.g. How do I use reflection to call a generic method? ) as well as other sites that show how to set this up with a dynamic type, which is great, except that none of the examples do anything more than write foo.GetType() in the generic method to prove that the dynamic type is correct.
So, how do you call a method on that dynamic type? Of course, when I tried to set this up, trying to call the Open() method on the sqlConnection parameter doesn't compile.
Here's sort of what I'm trying to accomplish:
private static void TransferXmlData(ExportManifest m_settings, XmlNodeList xmlNodeList)
{
if (m_Settings.ServerType.ToLower() == "mysql")
{
using (MySqlConnection mySqlConnection = new MySqlConnection(m_Settings.TargetData.ConnectionString))
{
MySqlCommand mySqlCommand =
new MySqlCommand(Program.GetCommandTextTemplate(m_settings), mySqlConnection);
PrepareSqlCommand(mySqlConnection, mySqlCommand, m_settings)
}
}
else
{
using (SqlConnection sqlConnection =
new SqlConnection(m_Settings.TargetData.ConnectionString))
{
SqlCommand sqlCommand =
new SqlCommand(Program.GetCommandTextTemplate(m_settings), sqlConnection);
PrepareSqlCommand(sqlConnection, sqlCommand, m_settings)
}
}
}
private static void PrepareSqlCommand<T>(T sqlConnection, T sqlCommand, ExportManifest m_settings)
{
// Potentially a lot of code here that looks just like the
// code in the else block, Except that it uses the
// MySqlConnection objects instead of SqlConnection
// Do some stuff
sqlConnection.Open(); // obviously doesn't work
}
Thanks in advance!

May be you can Implement factory design pattern(If you do not want to go with generics, this is my opinion you can think about it.)
This will help you to prevent code duplication.
Implement your Factory class.
`
Public class Factory
{
public static IDbConnection createDbInstance(ExportManifest m_settings)
{
if (m_Settings.ServerType.ToLower() == "mysql")
{
return new MySqlConnection();
}
else
return new SqlConnection();
}
} `
and in your actual method you can use IDbConnection and IDbCommand
private static void TransferXmlData(ExportManifest m_settings, XmlNodeList xmlNodeList)
{
IDbConnection db = Factory.createDbInstance(m_settings);
db.ConnectionString = m_Settings.TargetData.ConnectionString;
IDbCommand comnd = db.CreateCommand();
comnd.CommandText = Program.GetCommandTextTemplate(m_settings);
comnd.CommandType = CommandType.Text;
// db.Open(); if you want to open connection here
PrepareSqlCommand(db, comnd, m_settings);
}
private static void PrepareSqlCommand(IDbConnection sqlConnection, IDbCommand sqlCommand, ExportManifest m_settings)
{
// Potentially a lot of code here that looks just like the
// code in the else block, Except that it uses the
// MySqlConnection objects instead of SqlConnection
// Do some stuff
sqlConnection.Open();
}

In order to write your data access code once, but be able to switch out your implementation based on some logic, you should be coding against IDbConnection.
Something to the effect of:
using (IDbConnection db = new SqlConnection(ConfigurationManager.ConnectionStrings["AdventureWorks"].ConnectionString))
{
//execute database actions against IDBConnection
}
In my opinion, this is very well established using the repository pattern, and it shields you from database implementation details, but proper implementation of the pattern might over-complicate your use case. As far as logic to decide which connections get new(), a factory approach as stated above is sufficient, but you could just as easily pass an enum flag if this is a trivial app. In large scale software, you'd typically want to use an Inversion of Control container to control a specifc instance of IDbConnection to get injected in. In any case, reflection, generics, and dynamics would be the wrong tools here (barring any object mapping).

As #Sehnsucht said, you could do something like that:
private static void TransferXmlData(ExportManifest m_settings, XmlNodeList xmlNodeList)
{
if (m_Settings.ServerType.ToLower() == "mysql")
Connect(connectionString => new MySqlConnection(connectionString),
(text, connection) => new MySqlCommand(text, connection));
else
Connect(connectionString => new SqlConnection(connectionString),
(text, connection) => new SqlCommand(text, connection));
}
private static void Connect(ExportManifest m_settings,
Func<string, IDbConnection> createConnection,
Func<string, IDbConnection, IDbCommand> createCommand)
{
using (IDbConnection mySqlConnection =
createConnection(m_Settings.TargetData.ConnectionString))
{
IDbCommand mySqlCommand =
createCommand(Program.GetCommandTextTemplate(m_settings), mySqlConnection);
PrepareSqlCommand(mySqlConnection, mySqlCommand, m_settings);
}
}
private static void PrepareSqlCommand(IDbConnection sqlConnection,
IDbCommand sqlCommand, ExportManifest m_settings)
{
sqlConnection.Open();
}
Both SqlConnection and MySqlConnection inherit from DbConnection which implements IDbConnection. Same go for SqlCommand and MySqlCommand, they implement IDbCommand.
Then, you can use the interfaces to merge your code.
But if, for some reasons, you will need to work with the real types (as return values). You can change your methods for something like this:
private static void Connect<TConnection, TCommand>(ExportManifest m_settings,
Func<string, TConnection> createConnection,
Func<string, TConnection, TCommand> createCommand)
where TConnection : IDbConnection
where TCommand : IDbCommand
{
using (TConnection mySqlConnection =
createConnection(m_Settings.TargetData.ConnectionString))
{
TCommand mySqlCommand =
createCommand(Program.GetCommandTextTemplate(m_settings), mySqlConnection);
PrepareSqlCommand(mySqlConnection, mySqlCommand, m_settings);
}
}
private static void PrepareSqlCommand<TConnection, TCommand>(TConnection sqlConnection,
TCommand sqlCommand, ExportManifest m_settings)
where TConnection : IDbConnection
where TCommand : IDbCommand
{
sqlConnection.Open();
}

Correctly reusing SqlCommand and SqlParameter whilst pooling the connection

I am working on a setup where a scalable WCF Service Component is connected to a single MS SQL Server Database. The RESTful service allows users to save data into the DB as well as get data from it.
Whilst implementing a class handling the database connections / methods, I started struggling with correctly reusing prepared SqlCommands and the connection. I read up on the MSDN about connection pooling as well as how to use SqlCommand and SqlParameter.
My initial version of the class looks like this:
public class SqlRepository : IDisposable
{
private object syncRoot = new object();
private SqlConnection connection;
private SqlCommand saveDataCommand;
private SqlCommand getDataCommand;
public SqlRepository(string connectionString)
{
// establish sql connection
connection = new SqlConnection(connectionString);
connection.Open();
// save data
saveDataCommand = new SqlCommand("INSERT INTO Table (Operation, CustomerId, Data, DataId, CreationDate, ExpirationDate) VALUES (#Operation, #CustomerId, #Data, #DataId, #CreationDate, #ExpirationDate)", connection);
saveDataCommand.Parameters.Add(new SqlParameter("Operation", SqlDbType.NVarChar, 20));
saveDataCommand.Parameters.Add(new SqlParameter("CustomerId", SqlDbType.NVarChar, 50));
saveDataCommand.Parameters.Add(new SqlParameter("Data", SqlDbType.NVarChar, 50));
saveDataCommand.Parameters.Add(new SqlParameter("DataId", SqlDbType.NVarChar, 50));
saveDataCommand.Parameters.Add(new SqlParameter("CreationDate", SqlDbType.DateTime));
saveDataCommand.Parameters.Add(new SqlParameter("ExpirationDate", SqlDbType.DateTime));
saveDataCommand.Prepare();
// get data
getTripCommand = new SqlCommand("SELECT TOP 1 Data FROM Table WHERE CustomerId = #CustomerId AND DataId = #DataId AND ExpirationDate > #ExpirationDate ORDER BY CreationDate DESC", connection);
getTripCommand.Parameters.Add(new SqlParameter("CustomerId", SqlDbType.NVarChar, 50));
getTripCommand.Parameters.Add(new SqlParameter("DataId", SqlDbType.NVarChar, 50));
getTripCommand.Parameters.Add(new SqlParameter("ExpirationDate", SqlDbType.DateTime));
getTripCommand.Prepare();
}
public void SaveData(string customerId, string dataId, string operation, string data, DateTime expirationDate)
{
lock (syncRoot)
{
saveDataCommand.Parameters["Operation"].Value = operation;
saveDataCommand.Parameters["CustomerId"].Value = customerId;
saveDataCommand.Parameters["CreationDate"].Value = DateTime.UtcNow;
saveDataCommand.Parameters["ExpirationDate"].Value = expirationDate;
saveDataCommand.Parameters["Data"].Value = data;
saveDataCommand.Parameters["DataId"].Value = dataId;
saveDataCommand.ExecuteNonQuery();
}
}
public string GetData(string customerId, string dataId)
{
lock (syncRoot)
{
getDataCommand.Parameters["CustomerId"].Value = customerId;
getDataCommand.Parameters["DataId"].Value = dataId;
getDataCommand.Parameters["ExpirationDate"].Value = DateTime.UtcNow;
using (var reader = getDataCommand.ExecuteReader())
{
if (reader.Read())
{
string data = reader.GetFieldValue<string>(0);
return data;
}
else
{
return null;
}
}
}
}
public void Dispose()
{
try
{
if (connection != null)
{
connection.Close();
connection.Dispose();
}
DisposeCommand(saveDataCommand);
DisposeCommand(getDataCommand);
}
catch { }
}
private void DisposeCommand(SqlCommand command)
{
try
{
command.Dispose();
}
catch (Exception)
{
}
}
}
There are several aspects important to know:
I am using SqlCommand.Prepare() to speed up the process of executing the command
Reusing the commands avoids creating new objects with every call to the GetData and SaveData methods, thus leading to no problem with the garbage collector
There is only one instance of the SqlRepository class, used by the WCF Service.
There are many many calls per minute to this service, so keeping a connection to the DB open is what I want.
Now I read up a bit more about connection pooling and the fact that it is highly recommended to use the SqlConnection object in a using statement to ensure disposal. To my understanding, the connection pooling technology takes care of leaving the connection open even though the Dispose() method of SqlConnection has been called by the using statement.
The way to use this would be to have a using(SqlConnection connection = new SqlConnection(connectionString)) inside the GetData and SaveData methods. However, then - at least to my intuition - I would need to create the SqlCommands inside the GetData / SaveData methods as well. Or not? I could not find any documentation on how to reuse the commands that way. Also wouldn't the call to SqlCommand.Prepare() be meaningless if I need to prepare a new command every time I get into the GetData / SaveData methods?
How do I properly implement the SqlRepository class? The way it is now I believe that if the connection breaks (maybe because the DB server goes down for a while and reboots), then the SqlRepository class will not automatically recover and be functioning. To my best knowledge this sort of failsave scenarios are handled in the pooling technology.
Thanks for ideas and feedback!
Christian

Do not reuse the SqlCommand instances.
You are synchronizing database access.
With your implementation, you are re-using a small object (which is no problem for the GC even if there are thousands) in exchange of concurrent DB operations.
Remove the synchronization locks.
Create new instances of SqlCommands for each database operation.
Do not call Prepare. Prepare speeds up db operations, but after executing ExecuteReader() on a SqlCommand with CommandType = Text and with non-zero number of parameters, the command is unprepared internally.

C# 'using' statement question

If you employ a using clause to dispose of a connection, are other items within the clause that implement IDisposable also automatically disposed? If not, how do you handle making sure all IDisposable items are automatically disposed?
public static DataTable ReturnDataTable(
string ConnectionString, string CommandTextString, CommandType CommandType,
int CommandTimeout, List<System.Data.SqlClient.SqlParameter> ParameterList = null)
{
using (System.Data.SqlClient.SqlConnection Connection =
new System.Data.SqlClient.SqlConnection())
{
Connection.ConnectionString = ConnectionString;
System.Data.SqlClient.SqlCommand Command =
new System.Data.SqlClient.SqlCommand();
Command.Connection = Connection;
Command.CommandText = CommandTextString;
Command.CommandType = CommandType;
Command.CommandTimeout = CommandTimeout;
if (ParameterList != null)
{
if (ParameterList.Count > 0)
{
foreach (SqlParameter parameter in ParameterList)
{
Command.Parameters.AddWithValue(
parameter.ParameterName, parameter.Value);
}
}
}
System.Data.DataTable DataTable = new System.Data.DataTable();
System.Data.SqlClient.SqlDataAdapter DataAdapter =
new System.Data.SqlClient.SqlDataAdapter();
DataAdapter.SelectCommand = Command;
DataAdapter.Fill(DataTable);
return DataTable;
}
}

You can stack the statements like this (to initialize all disposable objects early on)
using (...)
using (...)
{
...
}
or you can use nested using statements for each disposable object you need
using (...)
{
using (...) { ... }
using (...) { ... }
}

Only an object created in the using clause will be disposed. If you want to make sure that call to dispose is automatically generated for every disposable object created inside of the using block you will need to wrap each of them in a using clause (or you can just call dispose or close, whichever they support, of course). So, the answer is not.

No. You will have to explicitly call Dispose on the ones that are not inside the parameters of the using statement.

Keeping the SqlDataReader open across an n-tiered layer

I have a Database class that abstracts the ExecuteNonQuery() and ExecuteReader() of SqlCommand. Due to wrapping the Sqlconnection and SqlCommand around using blocks, the SqlDataReader gets closed after the CustomExecuteReader() is called, therefore I can't read the SqlReaderResultSet at the business level layer. Code below. Thanks guys for the feedback.
public static SqlDataReader SqlReaderResultSet { get; set; }
public static SqlDataReader CustomExecuteReader(string storedProc)
{
using (var conn = new SqlConnection(ConnectionString))
{
var cmd = new SqlCommand(storedProc, conn) {CommandType = CommandType.StoredProcedure};
try
{
conn.Open();
SqlReaderResultSet = cmd.ExecuteReader();
}
catch (InvalidOperationException)
{
if (conn.State.Equals(ConnectionState.Closed))
conn.Open();
}
finally
{
conn.Close();
}
}
return SqlReaderResultSet;
}

"I can't read the SqlReaderResultSet at the business level layer" - and you shouldn't. Data should be passed using data transfer objects, never through a low level data access structure.

I recommend changing your approach so that the method you describe above iterates the records in the datareader, and creates a list of objects. That list of objects is what should be returned and worked on.

Iterator Blocks can be a way around this. It is legal and generally safe to do the following:
IEnumerable<MyFancyData> ResultSet {
get {
using(DbConnection conn = ...)
using(DbCommand cmd = ...) {
conn.Open();
using(DbDataReader reader = cmd.ExecuteReader()) {
while(reader.Read()) {
yield return new MyFancyData(reader[0], reader[42] ...);
}
}
}
}
}
Each time you enumerate the ResultSet property, the connection will be constructed again - and Disposed of afterwards (foreach and other IEnumerator<> consumers will appropriately call the Dispose() method of the generator, allowing the using block to do its thing).
This approach retains the lazy as-you-need it evaluation of the items from the data reader (which can be relevant when your data set becomes large), which still cleaning abstracting away sql-level details from the public API.