Develop Reference

C# Using foreach loop to bulkcopy data to SQL Server

I have a C# console app which reads million source data from a CSV file and inserts them into SQL Server in batches.
I group the data by 1000 count and use foreach to loop the groups. Each loop creates a new SqlConnection and a new SqlBulkCopy objects and disposes them at the end of the loop.
for (int index = 0; index < dts.Count; index++)
{
DataTable _dt = dts[index];
try
{
using (SqlConnection connection = new SqlConnection(conn))
{
await connection.OpenAsync();
//using (SqlTransaction trans = connection.BeginTransaction())
using (CancellationTokenSource cts = new CancellationTokenSource())
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
bulkCopy.DestinationTableName = "[dbo].[XXXX]";
bulkCopy.BatchSize = 1000;
bulkCopy.WriteToServer(_dt);
}
}
}
}
The first 10 groups work fast in 1.x seconds, but after that it takes 4x secs to 60secs every group.

If you use other cores of the processor, you will use a little more speed.
splitting data by number of cores using threads saves time.
Thread t1 = new Thread(new ThreadStart(Thread1));
Thread t2 = new Thread(new ThreadStart(Thread2));
t1.Start();
t2.Start();
int piece = dts.Count / 2;
public static void Thread1()
{
for (int index = 0; index < piece; index++)
{
DataTable _dt = dts[index];
try
{
using (SqlConnection connection = new SqlConnection(conn))
{
await connection.OpenAsync();
//using (SqlTransaction trans = connection.BeginTransaction())
using (CancellationTokenSource cts = new CancellationTokenSource())
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
bulkCopy.DestinationTableName = "[dbo].[XXXX]";
bulkCopy.BatchSize = 1000;
bulkCopy.WriteToServer(_dt);
}
}
}
}
}
public static void Thread2()
{
for (int index = piece; index < dts.Count; index++)
{
DataTable _dt = dts[index];
try
{
using (SqlConnection connection = new SqlConnection(conn))
{
await connection.OpenAsync();
//using (SqlTransaction trans = connection.BeginTransaction())
using (CancellationTokenSource cts = new CancellationTokenSource())
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
bulkCopy.DestinationTableName = "[dbo].[XXXX]";
bulkCopy.BatchSize = 1000;
bulkCopy.WriteToServer(_dt);
}
}
}
}
}
if you just want foreach
foreach(DataTable _dt in dts){
try
{
using (SqlConnection connection = new SqlConnection(conn))
{
await connection.OpenAsync();
//using (SqlTransaction trans = connection.BeginTransaction())
using (CancellationTokenSource cts = new CancellationTokenSource())
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
bulkCopy.DestinationTableName = "[dbo].[XXXX]";
bulkCopy.BatchSize = 1000;
bulkCopy.WriteToServer(_dt);
}
}
}
}

SQL dependency using stored procedure

I am using SqlDependency in C# code to alert my application when there is a change in a particular column value in a row in database. I could achieve it using inline SQL.
I would like to replace it with stored procedure but for some reason the code isn't executing the stored procedure. I am pasting few lines of code from my application. Please let me know how to modify it so that I can have inline SQL replaced with a procedure call to achieve the same.
var con = new SqlConnection(ConnectionString);
SqlDependency.Stop(con.ConnectionString);
SqlDependency.Start(con.ConnectionString);
var connection = new SqlConnection(con.ConnectionString);
connection.Open();
try
{
using (var command = new SqlCommand("inlinesql", connection))
{
var dependency = new SqlDependency(command);
dependency.OnChange += new OnChangeEventHandler(OnDependencyChange);
using (SqlDataReader rdr = command.ExecuteReader())
{
try
{
if (rdr.HasRows)
{
_autoEvent.WaitOne();
}
else
{
continue;
}
}
finally
{
rdr.Close();
}
}
}
}
finally
{
connection.Close();
SqlDependency.Stop(con.ConnectionString);
}
void OnDependencyChange(object sender, SqlNotificationEventArgs e)
{
dosomething();
}

Above var dependency... put:
command.CommandType = CommandType.StoredProcedure;
And replace inlinesql with the name of the stored procedure.

Example code that worked for me:
using (var sqlConnection = new SqlConnection (_connectionString)) {
sqlConnection.Open ();
using (var sqlCommand = sqlConnection.CreateCommand ()) {
sqlCommand.Parameters.Clear ();
sqlCommand.CommandType = System.Data.CommandType.StoredProcedure;
// YOUR STORED PROCEDURE NAME AND PARAMETERS SHOULD COME HERE
sqlCommand.CommandText = "sp_InsertData";
sqlCommand.Parameters.AddWithValue ("#Code", e.Entity.Symbol);
sqlCommand.Parameters.AddWithValue ("#Name", e.Entity.Name);
sqlCommand.Parameters.AddWithValue ("#Price", e.Entity.Price);
try {
var sqlDataInsert = sqlCommand.ExecuteNonQuery ();
} catch (Exception ex) {
MessageBox.Show (ex.Message, this.Title);
}
}
}

Is this correct usage of TransactionSope?

I have decided to try using a TransactionScope, rather than the SqlTransaction class.
The following is my code, wrapped in a TransactionScope:
using (var transaction = new System.Transactions.TransactionScope())
{
using (MySqlCommand cmd = new MySqlCommand(sql, connection))
{
if (listParameters != null && listParameters.Count > 0)
{
foreach (string currentKey in listParameters.Keys)
{
cmd.Parameters.Add(new MySqlParameter(currentKey, GetDictionaryValue(listParameters, currentKey)));
}
}
using (MySqlDataReader reader = cmd.ExecuteReader())
{
dtResults.Load(reader);
}
}
transaction.Complete();
}
The code works, however I am not binding the MySqlCommand cmd object with a transaction at any point. Is this a problem?

No, this is not the correct use.
The correct use is to create a connection after creating TransactionScope. Then the connection will detect the ambient TransactionScope and enlist itself.
using (var transaction = new System.Transactions.TransactionScope())
{
using (var connection = new MySqlConnection())
{
...
}
}
If you create the connection before the scope, that connection will be out of that scope, even if you create the command after creating the scope.
Also note that TransactionScope defaults to Serializable level of isolation. This is the most secure level, but also the least concurrent one. You often want to explicitly set a more common isolation level:
using (var transaction = new TransactionScope(
TransactionScopeOption.Required,
new TransactionOptions() { IsolationLevel = IsolationLevel.ReadCommitted }))
{
}

How to rollback a transaction in Date Entity Framework

I am trying to use the Data Entity Framework to create my Data Access Layer. In the work I have done up to now I have used ADO.Net. I am trying to get my head around how transactions work in EF. I ave read loads but its all confussed me even more than I was before!! I would usually do somethink like (simplified for example):
using (SqlConnection conn = new SqlConnection(_connString))
{
using (SqlTransaction trans = conn.BeginTransaction())
{
try
{
using (SqlCommand cmd = new SqlCommand("usp_CreateNewInvoice", conn))
{
cmd.Transaction = trans;
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add(new SqlParameter("#InvoiceName", Invoice.invoicename));
cmd.Parameters.Add(new SqlParameter("#InvoiceAddess", Invoice.invoiceaddress));
_invoiceid = Convert.ToInt32(cmd.ExecuteScalar());
}
foreach (InvoiceLine inLine in Invoice.Lines)
{
using (SqlCommand cmd = new SqlCommand("usp_InsertInvoiceLines", conn))
{
cmd.Transaction = trans;
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add(new SqlParameter("#InvNo", _invoiceid));
cmd.Parameters.Add(new SqlParameter("#InvLineQty", inLine.lineqty));
cmd.Parameters.Add(new SqlParameter("#InvLineGrossPrice", inLine.linegrossprice));
cmd.ExecuteNonQuery();
}
}
trans.Commit();
}
catch (SqlException sqlError)
{
trans.Rollback();
}
}
}
Now I want to do the same in EF:
using (TransactionScope scope = new TransactionScope())
{
try
{
var DbContext = new CCTStoreEntities();
CCTInvoice invHead = new CCTInvoice();
invHead.Name = Invoice.invoicename;
invHead.Address = Invoice.invoiceaddress;
DbContext.CCTInvoices.Add(invHead);
DbContext.SaveChanges(false);
_iid = invHead.InvoiceId;
foreach (InvoiceLine inLine in Invoice.Lines)
{
CCTInvoiceLine invLine = new CCTInvoiceLine();
invLine.InvoiceNo = _iid;
invLine.Quantity = inLine.lineqty;
invLine.GrossPrice = inLine.linegrossprice;
DbContext.CCTInvoiceLines.Add(invHead);
DbContext.SaveChanges(false);
}
DbContext.SaveChanges();
scope.Complete();
}
catch
{
//Something went wrong
//Rollback!
}
}
From what I read SaveChanges(false) means the changes being made will continue to be tracked. But how do I rollback the transaction if something goes wrong?

You don't need to do anything on your catch block. Just by not calling DbContext.SaveChanges no changes will be sent to the database, and they will be lost once DbContext is disposed.
You do have a problem though. DbContext must be wrapped on a using block as follows to be properly disposed. BTW, I don't think DbContext.SaveChanges(false); is needed, your code should work with just the final DbContext.SaveChanges();. EF will take care of wiring up all your Foreign Keys, so you don't need to do that explicitly.
using (TransactionScope scope = new TransactionScope())
{
try
{
using (var DbContext = new CCTStoreEntities())
{
CCTInvoice invHead = new CCTInvoice();
invHead.Name = Invoice.invoicename;
invHead.Address = Invoice.invoiceaddress;
DbContext.CCTInvoices.Add(invHead);
DbContext.SaveChanges(false); // This is not needed
_iid = invHead.InvoiceId; // This is not needed
foreach (InvoiceLine inLine in Invoice.Lines)
{
CCTInvoiceLine invLine = new CCTInvoiceLine();
invLine.InvoiceNo = _iid; // This is not needed
invLine.Quantity = inLine.lineqty;
invLine.GrossPrice = inLine.linegrossprice;
DbContext.CCTInvoiceLines.Add(invHead);
DbContext.SaveChanges(false); // This is not needed
}
DbContext.SaveChanges();
scope.Complete();
}
}
catch
{
//Something went wrong
//Rollback!
}
}

The rollback mechanism in a TransactionScope is implicit.
Basically, if you don't call Complete before the TransactionScope gets disposed it will automatically rollback. See the Rolling back a transaction section in Implementing an Implicit Transaction using Transaction Scope.
So technically, you don't even need to use a try...catch here (unless you want to perform some other action like logging).

sqlite - any improvements for this attach code (running multiple sql commands transactionally in sqlite)

Is this code solid? I've tried to use "using" etc. Basically a method to pass as sequenced list of SQL commands to be run against a Sqlite database.
I assume it is true that in sqlite by default all commands run in a single connection are handled transactionally? Is this true? i.e. I should not have to (and haven't got in the code at the moment) a BeginTransaction, or CommitTransaction.
It's using http://sqlite.phxsoftware.com/ as the sqlite ADO.net database provider.
1st TRY
private int ExecuteNonQueryTransactionally(List<string> sqlList)
{
int totalRowsUpdated = 0;
using (var conn = new SQLiteConnection(_connectionString))
{
// Open connection (one connection so should be transactional - confirm)
conn.Open();
// Apply each SQL statement passed in to sqlList
foreach (string s in sqlList)
{
using (var cmd = new SQLiteCommand(conn))
{
cmd.CommandText = s;
totalRowsUpdated = totalRowsUpdated + cmd.ExecuteNonQuery();
}
}
}
return totalRowsUpdated;
}
3rd TRY
How is this?
private int ExecuteNonQueryTransactionally(List<string> sqlList)
{
int totalRowsUpdated = 0;
using (var conn = new SQLiteConnection(_connectionString))
{
conn.Open();
using (var trans = conn.BeginTransaction())
{
try
{
// Apply each SQL statement passed in to sqlList
foreach (string s in sqlList)
{
using (var cmd = new SQLiteCommand(conn))
{
cmd.CommandText = s;
totalRowsUpdated = totalRowsUpdated + cmd.ExecuteNonQuery();
}
}
trans.Commit();
}
catch (SQLiteException ex)
{
trans.Rollback();
throw;
}
}
}
return totalRowsUpdated;
}
thanks

Yes, it's true, each SQLite unnested command is nested in a transaction. So that if you need to run several queries, without fetching the result, there is much gain is explicitly starting a transaction, doing your queries, and committing.