C# Using foreach loop to bulkcopy data to SQL Server - c#

I have a C# console app which reads million source data from a CSV file and inserts them into SQL Server in batches.
I group the data by 1000 count and use foreach to loop the groups. Each loop creates a new SqlConnection and a new SqlBulkCopy objects and disposes them at the end of the loop.
for (int index = 0; index < dts.Count; index++)
{
DataTable _dt = dts[index];
try
{
using (SqlConnection connection = new SqlConnection(conn))
{
await connection.OpenAsync();
//using (SqlTransaction trans = connection.BeginTransaction())
using (CancellationTokenSource cts = new CancellationTokenSource())
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
bulkCopy.DestinationTableName = "[dbo].[XXXX]";
bulkCopy.BatchSize = 1000;
bulkCopy.WriteToServer(_dt);
}
}
}
}
The first 10 groups work fast in 1.x seconds, but after that it takes 4x secs to 60secs every group.

If you use other cores of the processor, you will use a little more speed.
splitting data by number of cores using threads saves time.
Thread t1 = new Thread(new ThreadStart(Thread1));
Thread t2 = new Thread(new ThreadStart(Thread2));
t1.Start();
t2.Start();
int piece = dts.Count / 2;
public static void Thread1()
{
for (int index = 0; index < piece; index++)
{
DataTable _dt = dts[index];
try
{
using (SqlConnection connection = new SqlConnection(conn))
{
await connection.OpenAsync();
//using (SqlTransaction trans = connection.BeginTransaction())
using (CancellationTokenSource cts = new CancellationTokenSource())
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
bulkCopy.DestinationTableName = "[dbo].[XXXX]";
bulkCopy.BatchSize = 1000;
bulkCopy.WriteToServer(_dt);
}
}
}
}
}
public static void Thread2()
{
for (int index = piece; index < dts.Count; index++)
{
DataTable _dt = dts[index];
try
{
using (SqlConnection connection = new SqlConnection(conn))
{
await connection.OpenAsync();
//using (SqlTransaction trans = connection.BeginTransaction())
using (CancellationTokenSource cts = new CancellationTokenSource())
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
bulkCopy.DestinationTableName = "[dbo].[XXXX]";
bulkCopy.BatchSize = 1000;
bulkCopy.WriteToServer(_dt);
}
}
}
}
}
if you just want foreach
foreach(DataTable _dt in dts){
try
{
using (SqlConnection connection = new SqlConnection(conn))
{
await connection.OpenAsync();
//using (SqlTransaction trans = connection.BeginTransaction())
using (CancellationTokenSource cts = new CancellationTokenSource())
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
{
bulkCopy.DestinationTableName = "[dbo].[XXXX]";
bulkCopy.BatchSize = 1000;
bulkCopy.WriteToServer(_dt);
}
}
}
}

Related

Select Multiple CSV and bulk to Database

I'm using an MDF database with the OpenFileDialog class to import a single CSV file to a database. That works fine for a single CSV file, but I need to open and process multiple CSV files in bulk.
How can I improve my code? I've tried using a for loop.
Here is my code to process a single CSV file:
ofd.Filter = "CSV files (*.csv) | *.csv; | CSV PRN files (*.prn,) |*.prn;";
ofd.FileName = "";
ofd.ShowDialog();
DataTable dt = new DataTable();
string line = null;
int i = 0;
using (StreamReader sr = File.OpenText(ofd.FileName))
{
while ((line = sr.ReadLine()) != null)
{
string[] data = line.Split(',');
if (data.Length > 0)
{
if (i == 0)
{
foreach (var item in data)
{
dt.Columns.Add(new DataColumn());
}
i++;
}
DataRow row = dt.NewRow();
row.ItemArray = data;
dt.Rows.Add(row);
}
}
}
string symbolName = dt.Rows[1][0].ToString();
string strConnection =
#"Data Source =.\SQLEXPRESS; AttachDbFilename = C:\USERS\JEF\DOCUMENTS\DATABASE1.MDF; Integrated Security = True; Connect Timeout = 30; User Instance = True";
SqlConnection condb2 = new SqlConnection(strConnection);
string createTablerow ="create table ["+symbolName+"] (code1 VARCHAR(100) COLLATE Arabic_CI_AI_KS_WS,date1 varchar(50),open1 varchar(50),high1 varchar(50),low1 varchar(50),close1 varchar(50),vol1 varchar(50))";
using (SqlConnection connection = new SqlConnection(strConnection))
{
SqlCommand command1 = new SqlCommand(createTablerow, connection);
connection.Open();
command1.ExecuteNonQuery();
}
using (SqlConnection cn = new SqlConnection(strConnection))
{
cn.Open();
using (SqlBulkCopy copy = new SqlBulkCopy(cn))
{
copy.ColumnMappings.Add(0, "code1");
copy.ColumnMappings.Add(1, "date1");
copy.ColumnMappings.Add(2, "open1");
copy.ColumnMappings.Add(3, "high1");
copy.ColumnMappings.Add(4, "low1");
copy.ColumnMappings.Add(5, "close1");
copy.ColumnMappings.Add(6, "vol1");
copy.DestinationTableName = "[" + symbolName + "]";
copy.WriteToServer(dt);
}
}
I moved some of you code around to make it more efficient :
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.IO;
using System.Data.SqlClient;
namespace WindowsFormsApplication1
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
ofd.FileName = "";
ofd.ShowDialog();
string line = null;
int i = 0;
string strConnection =
#"Data Source =.\SQLEXPRESS; AttachDbFilename = C:\USERS\JEF\DOCUMENTS\DATABASE1.MDF; Integrated Security = True; Connect Timeout = 30; User Instance = True";
using (SqlConnection connection = new SqlConnection(strConnection))
{
connection.Open();
SqlBulkCopy copy = new SqlBulkCopy(connection);
copy.ColumnMappings.Add(0, "code1");
copy.ColumnMappings.Add(1, "date1");
copy.ColumnMappings.Add(2, "open1");
copy.ColumnMappings.Add(3, "high1");
copy.ColumnMappings.Add(4, "low1");
copy.ColumnMappings.Add(5, "close1");
copy.ColumnMappings.Add(6, "vol1");
foreach (string file in ofd.FileNames)
{
using (StreamReader sr = File.OpenText(file))
{
DataTable dt = new DataTable();
while ((line = sr.ReadLine()) != null)
{
string[] data = line.Split(',');
if (data.Length > 0)
{
if (i == 0)
{
foreach (var item in data)
{
dt.Columns.Add(new DataColumn());
}
i++;
}
DataRow row = dt.NewRow();
row.ItemArray = data;
dt.Rows.Add(row);
}
}
string symbolName = dt.Rows[1][0].ToString();
string createTable = string.Format("create table [{0}] (code1 VARCHAR(100) COLLATE Arabic_CI_AI_KS_WS,date1 varchar(50),open1 varchar(50),high1 varchar(50),low1 varchar(50),close1 varchar(50),vol1 varchar(50))",
symbolName);
using (SqlCommand command1 = new SqlCommand(createTable, connection))
{
command1.ExecuteNonQuery();
copy.DestinationTableName = "[" + symbolName + "]";
copy.WriteToServer(dt);
}
}
}
}
}
}
}
In order to process multiple files at once you will need to use a parallel code. Your use of "for" was a nice try, but "for" loops still run concurrently, which means one at a time.
The easiest way to achieve this parallel processing in this case would be to use Parallel.Foreach, here is a Microsoft guide about it: https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/how-to-write-a-simple-parallel-foreach-loop

Run a parameterized query/command asynchronously?

How could I take the following code and make each iteration of myValues asynchronous (in regards to executing the command, not the loop itself)?
using (var connection = new SqlConnection(CONNECTION_STRING))
using (var command = new SqlCommand(query, connection))
{
connection.Open();
command.Parameters.Add("#myParameter", SqlDbType.VarChar)
foreach(string myValue in myValues)
{
command.Parameters["#myParameter"].Value = myValue;
using (var dr = command.ExecuteReader(CommandBehavior.SingleResult))
{
while (dr.Read())
{
//do something
}
}
}
}
Is this what you need?
private void MyMethod()
{
UpdateValues();
}
async Task UpdateValues()
{
using (var connection = new SqlConnection("your_connection_string_here"))
{
using (var command = new SqlCommand("your_sql_statement_here", connection))
{
await connection.OpenAsync();
command.Parameters.Add("#myParameter", SqlDbType.VarChar);
foreach (string myValue in myValues)
{
command.Parameters["#myParameter"].Value = myValue;
var dr = await command.ExecuteReaderAsync(CommandBehavior.SingleResult);
while (await dr.ReadAsync())
{
// do your thing here //
}
}
}
}
}

Will DataAdapter.Fill() call the underlying stored procedure multiple times in a loop?

I'm writing a basic load test to test a stored procedure. Basically I'm trying to eval the performance when the stored procedure is called several times in a row. The results below seem pretty fast so I just wanted to validate that, in the scenario below, DataAdpater.Fill(ds) will not only fill the ds dataset on each iteration with the results but also make a new/fresh call to the underlying stored procedure each time it's called:
private static void LoadTest()
{
//var concurrentResults = new List<DataSet>();
var execTimes = new Dictionary<int, decimal>();
var connString = #"Data Source=myserver;Initial Catalog=reports;Integrated Security=True";
var sprocname = "spMySproc";
using (var conn = new SqlConnection(connString))
{
using (var sqlCmd = new SqlCommand(sprocname, conn))
{
sqlCmd.CommandType = CommandType.StoredProcedure;
var sqlParams = new SqlParameter[]
{
new SqlParameter("#ReportTypeId", 1),
new SqlParameter("#ReportPeriodId", 2)
};
sqlCmd.Parameters.AddRange(sqlParams);
var da = new SqlDataAdapter(sqlCmd);
for (var i = 0; i < 20; i++)
{
var ds = new DataSet();
var start = DateTime.Now;
var result = da.Fill(ds);
var end = DateTime.Now;
var elapsedTime = (end - start).Milliseconds;
execTimes.Add((i + 1), elapsedTime);
}
}
}
}

Returning Data Rows to List<string> with SqlDataReader

I'm trying to create a generic SqlDataReader which converts a table with 33 columns into a list. I would like each list item to contain all 33 column values for each row.
However, my code is assigning each value to an individual list item.
So instead of 1000 list items = 1000 rows of data, I have 33,000 list items.
I would prefer to use a list over a datatable, because the list comparisons I need to do are much simpler.
How can I have 1000 list items with 33 values each?
public static List<string> loadSQL(String query, String connectString)
{
List<string> dataList = new List<string>();
using (SqlConnection connection = new SqlConnection(connectString))
{
using (SqlCommand command = new SqlCommand(query, connection))
{
connection.Open();
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
for (int i = 0; i < reader.FieldCount; i ++)
{
dataList.Add(Convert.ToString(reader.GetValue(i)));
}
}
}
}
return dataList;
}
}
... update ...
corrected to the following. It returns the list items correctly. However, my list contains 33,000 items containing 33 items each. How can I control the loop so it stops after 1000 rows?
using (SqlConnection connection = new SqlConnection(connectString))
{
using (SqlCommand command = new SqlCommand(query, connection))
{
connection.Open();
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
List<string> tempRow = new List<string>();
for (int i = 0; i < reader.FieldCount; i ++)
{
tempRow.Add(Convert.ToString(reader.GetValue(i)));
}
dataList.Add(tempRow);
}
}
}
}
The best option for you to do this task is DataTable, But you don't want to use it. So, the net option will be, Create a class based on the query-output then use a List<objectOftheClass>. But in your case, the Input query will be changed all times so a common class will not be meaningful Since you are trying to make it generic. So the option you can follow is List<List<string>> or List<List<object>>. As per this the method signature will be like the following:
public static List<object[]> loadSQL(string query, string connectString)
{
List<object[]> dataList = new List<object[]>();
using (SqlConnection connection = new SqlConnection(connectString))
{
using (SqlCommand command = new SqlCommand(query, connection))
{
connection.Open();
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
object[] tempRow = new object[reader.FieldCount];
for (int i = 0; i < reader.FieldCount; i++)
{
tempRow[i] = reader[i];
}
dataList.Add(tempRow);
}
}
}
}
return dataList;
}
Why List<object>? why not `List?:
The reader will give you the column data as the same type of column in the table. If it is object then you need not convert it every time.
** Note:-** Change String to string for the arguments in the method signature. You can find a reason here
You can use a List<List<string>> like this:
public static List<List<string>> loadSQL(String query, String connectString)
{
List<List<string>> dataList = new List<List<string>>();
using (SqlConnection connection = new SqlConnection(connectString))
{
using (SqlCommand command = new SqlCommand(query, connection))
{
connection.Open();
using (SqlDataReader reader = command.ExecuteReader())
{
while (reader.Read())
{
var l = new List<string>();
for (int i = 0; i < reader.FieldCount; i ++)
{
l.Add(Convert.ToString(reader.GetValue(i)));
}
dataList.Add(l);
}
}
}
return dataList;
}
}
You can use an array within the list to achieve what you are trying to do. Here is a quick example using your code:
public static List<string[]> loadSQL(String query, String connectString)
{
List<string[]> dataList = new List<string[]>();
using (SqlConnection connection = new SqlConnection(connectString))
{
using (SqlCommand command = new SqlCommand(query, connection))
{
connection.Open();
using (SqlDataReader reader = command.ExecuteReader())
{
int rowcounter = 0;
while (reader.Read())
{
string[] value = new string[reader.FieldCount];
for (int i = 0; i < reader.FieldCount; i ++)
{
value[i] = Convert.ToString(reader.GetValue(i));
}
dataList.Add(value);
rowcounter++;
}
}
}
return dataList;
}
Alertnately, if you want to use the List, you will need to embed the values a single string, using a comma separator or something similar.

SqlBulkCopy - Unexpected existing transaction

I am using SqlBulkCopy to insert large amount of data:
try
{
using (var bulkCopy = new SqlBulkCopy(connection))
{
connection.Open();
using (var tran = connection.BeginTransaction(IsolationLevel.ReadCommitted))
{
bulkCopy.DestinationTableName = "table";
bulkCopy.ColumnMappings.Add("...", "...");
using (var dataReader = new ObjectDataReader<MyObject>(data))
{
bulkCopy.WriteToServer(dataReader);
}
tran.Commit();
return true;
}
}
}
catch (Exception ex)
{
return false;
}
But I always get exception:
Unexpected existing transaction.
Why this exception happens?
"Unexpected existing transaction" ... Why this exception happens?
This happens because using the SqlBulkCopy constructor without specifying a transaction will create its own transaction internally.
Avoid this by creating your transaction and then use it to create the SqlBulkCopy. SqlBulkCopy can be created with the transaction that you want to use, like this:
connection.Open();
using (var tran = connection.BeginTransaction(IsolationLevel.ReadCommitted))
{
using (var bulkCopy = new SqlBulkCopy(connection, SqlBulkCopyOptions.Default, tran))
{
You need to use the constructor that takes in the transaction so SqlBulkCopy will be aware of the transaction
connection.Open();
using (var tran = connection.BeginTransaction(IsolationLevel.ReadCommitted))
{
using (var bulkCopy = new SqlBulkCopy(connection, SqlBulkCopyOptions.Default, tran))
{
bulkCopy.DestinationTableName = "table";
bulkCopy.ColumnMappings.Add("...", "...");
using (var dataReader = new ObjectDataReader<MyObject>(data))
{
bulkCopy.WriteToServer(dataReader);
}
tran.Commit();
return true;
}
}

Categories

Resources