I am migrating my program from Microsoft SQL Server to MySQL. Everything works well except one issue with bulk copy.
In the solution with MS SQL the code looks like this:
connection.Open();
SqlBulkCopy bulkCopy = new SqlBulkCopy(connection);
bulkCopy.DestinationTableName = "testTable";
bulkCopy.WriteToServer(rawData);
Now I try to do something similar for MySQL. Because I think there would be bad performance I don't want to write the DataTable to a CSV file and do the insert from there with the MySqlBulkLoader class.
Any help would be highly appreciated.
Because I think there would be bad performance I don't want to write the DataTable to a CSV file and do the insert from there with the MySqlBulkLoader class.
Don't rule out a possible solution based on unfounded assumptions. I just tested the insertion of 100,000 rows from a System.Data.DataTable into a MySQL table using a standard MySqlDataAdapter#Update() inside a Transaction. It consistently took about 30 seconds to run:
using (MySqlTransaction tran = conn.BeginTransaction(System.Data.IsolationLevel.Serializable))
{
using (MySqlCommand cmd = new MySqlCommand())
{
cmd.Connection = conn;
cmd.Transaction = tran;
cmd.CommandText = "SELECT * FROM testtable";
using (MySqlDataAdapter da = new MySqlDataAdapter(cmd))
{
da.UpdateBatchSize = 1000;
using (MySqlCommandBuilder cb = new MySqlCommandBuilder(da))
{
da.Update(rawData);
tran.Commit();
}
}
}
}
(I tried a couple of different values for UpdateBatchSize but they didn't seem to have a significant impact on the elapsed time.)
By contrast, the following code using MySqlBulkLoader took only 5 or 6 seconds to run ...
string tempCsvFileSpec = #"C:\Users\Gord\Desktop\dump.csv";
using (StreamWriter writer = new StreamWriter(tempCsvFileSpec))
{
Rfc4180Writer.WriteDataTable(rawData, writer, false);
}
var msbl = new MySqlBulkLoader(conn);
msbl.TableName = "testtable";
msbl.FileName = tempCsvFileSpec;
msbl.FieldTerminator = ",";
msbl.FieldQuotationCharacter = '"';
msbl.Load();
System.IO.File.Delete(tempCsvFileSpec);
... including the time to dump the 100,000 rows from the DataTable to a temporary CSV file (using code similar to this), bulk-loading from that file, and deleting the file afterwards.
Similar to SqlBulkCopy, we have MySqlBulkCopy for Mysql.
here is the example how to use it.
public async Task<bool> MySqlBulCopyAsync(DataTable dataTable)
{
try
{
bool result = true;
using (var connection = new MySqlConnector.MySqlConnection(_connString + ";AllowLoadLocalInfile=True"))
{
await connection.OpenAsync();
var bulkCopy = new MySqlBulkCopy(connection);
bulkCopy.DestinationTableName = "yourtable";
// the column mapping is required if you have a identity column in the table
bulkCopy.ColumnMappings.AddRange(GetMySqlColumnMapping(dataTable));
await bulkCopy.WriteToServerAsync(dataTable);
return result;
}
}
catch (Exception ex)
{
throw;
}
}
private List<MySqlBulkCopyColumnMapping> GetMySqlColumnMapping(DataTable dataTable)
{
List<MySqlBulkCopyColumnMapping> colMappings = new List<MySqlBulkCopyColumnMapping>();
int i = 0;
foreach (DataColumn col in dataTable.Columns)
{
colMappings.Add(new MySqlBulkCopyColumnMapping(i, col.ColumnName));
i++;
}
return colMappings;
}
You can ignore the column mapping if you don't have any identity column in your table.
If you have identity column then you have to use the column mapping otherwise it won't insert any records in the table
It will just give message like "x rows were copied but only 0 rows were inserted".
This class i available in the below library
Assembly MySqlConnector, Version=1.0.0.0
Using any of BulkOperation NuGet-package, you can easily have this done.
Here is an example using the package from https://www.nuget.org/packages/Z.BulkOperations/2.14.3/
MySqlConnection conn = DbConnection.OpenConnection();
DataTable dt = new DataTable("testtable");
MySqlDataAdapter da = new MySqlDataAdapter("SELECT * FROM testtable", conn);
MySqlCommandBuilder cb = new MySqlCommandBuilder(da);
da.Fill(dt);
instead of using
......
da.UpdateBatchSize = 1000;
......
da.Update(dt)
just following two lines
var bulk = new BulkOperation(conn);
bulk.BulkInsert(dt);
will take only 5 seconds to copy the whole DataTable into MySQL without first dumping the 100,000 rows from the DataTable to a temporary CSV file.
Related
I have an CSV file with 7 columns, which a user has to upload so it can be added in the database.
I found some help in reading the CSV and putting all the info in a single table, however, the data has to be spread over three tables.
My code for inserting all the data to 1 table:
protected void Upload(object sender, EventArgs e)
{
//Upload and save the file
string csvPath = Server.MapPath("~/Temp/") + Path.GetFileName(FileUpload1.PostedFile.FileName);
FileUpload1.SaveAs(csvPath);
DataTable dt = new DataTable();
dt.Columns.AddRange(new DataColumn[7] {
new DataColumn("Title", typeof(string)),
new DataColumn("Artist", typeof(string)),
new DataColumn("Years", typeof(string)),
new DataColumn("Position", typeof(string)),
new DataColumn("Senddate", typeof(string)),
new DataColumn("Sendfrom", typeof(string)),
new DataColumn("Sendtill", typeof(string))});
string csvData = File.ReadAllText(csvPath);
foreach (string row in csvData.Split('\n'))
{
if (!string.IsNullOrEmpty(row))
{
dt.Rows.Add();
int i = 0;
foreach (string cell in row.Split(';'))
{
dt.Rows[dt.Rows.Count - 1][i] = cell;
i++;
}
}
}
string consString = ConfigurationManager.ConnectionStrings["connection"].ConnectionString;
using (SqlConnection con = new SqlConnection(consString))
{
using (SqlBulkCopy sqlBulkCopy = new SqlBulkCopy(con))
{
//Set the database table name
sqlBulkCopy.DestinationTableName = "dbo.ingevoerd";
con.Open();
sqlBulkCopy.WriteToServer(dt);
con.Close();
}
}
}
As you can see, it takes 7 columns, and puts them in the table [dbo].[ingevoerd]
How can i split the data to put the column 'Title' and 'Years' in a table called Song, 'Artist' in a table called Artiest, and 'Position', 'Senddate', 'Sendfrom' an 'Sendtill' in a table called Lijst?
For more information, put down a comment.
imho this is not the best way to handle this upload because the content is not flat data you can bulk upload in a breeze; there are many entitiest (at least 3) that should be linked.
i would go with the 'old style' approach of calling a insert for each row with proper parameters.
you are already looping through the Whole recordset when reading the CSV so i would make something like:
protected void Upload(object sender, EventArgs e)
{
//Upload and save the file
string csvPath = Server.MapPath("~/Temp/") + Path.GetFileName(FileUpload1.PostedFile.FileName);
FileUpload1.SaveAs(csvPath);
string consString = ConfigurationManager.ConnectionStrings["connection"].ConnectionString;
using (SqlConnection con = new SqlConnection(consString))
{
con.Open();
using (SqlTransaction tran = con.BeginTransaction())
using (SqlCommand cmd = new SqlCommand())
{
cmd.Connection = con;
cmd.Transaction = tran;
cmd.CommandType = System.Data.CommandType.StoredProcedure;
cmd.CommandText = "your_sp_name_here";
cmd.Parameters.Add(new SqlParameter("#title",System.Data.SqlDbType.NVarChar));
cmd.Parameters.Add(new SqlParameter("#artist", System.Data.SqlDbType.NVarChar));
// other parameters follow
// ...
string csvData = File.ReadAllText(csvPath);
foreach (string row in csvData.Split('\n'))
{
if (!string.IsNullOrEmpty(row))
{
// for every row call the command and fill in the parameters with proper values
cmd.Parameters["#title"].Value = row[0];
cmd.Parameters["#artist"].Value = row[1];
// ...
cmd.ExecuteNonQuery();
}
}
// when done commit the transaction
tran.Commit();
}
}
}
inside your stored procedure handle the 'split' of the data in the relevant tables taking all the steps required to avoid duplicates and maybe linking the data among the tables:
create procedure your_sp_name_here(#title nvarchar(50), #artist nvarchar(50), #year int)
as
begin
-- add logic & checks here if needed
-- ...
-- ...
-- if everything is ok insert the rows
insert into songs (title, year) values (#title, #year)
insert into Artiest (Artist) values (#artist)
end
Have you looked into column mappings?
Check out stackoverflow.com/questions/17469349/mapping-columns-in-a-datatable-to-a-sql-table-with-sqlbulkcopy
What am I missing from the following code? In this code fragment, I am reading in a table from a SQLite database. I am then updating a cell, and then reading back the change.
This code is a simplified version of larger code, but it illustrates the problem.
The code reads the table perfectly, however AcceptChanges() does not write anything back. I verified that with the repeated read and by going to SQLiteAdmin and perusing the table.
I tried adding the "oLocalAdapter.Update(oLocalSet.Tables[0]);" line, however that did not make any difference. I saw that doing a search.
using System.Data.SQLite;
// DATABASE (Local): Formulate the SQL command.
String strSqlCommand = "SELECT * FROM [tblTest] ORDER BY [IdPrimary] ASC;";
SQLiteCommand oLocalCommand = new SQLiteCommand(strSqlCommand, ClassEngine.Connection);
// DATABASE (Local): Get the data records.
SQLiteDataAdapter oLocalAdapter = new SQLiteDataAdapter(oLocalCommand);
DataSet oLocalSet = new DataSet();
oLocalAdapter.Fill(oLocalSet, "tblTest");
// Try to write to some changes.
String strValue = oLocalSet.Tables[0].Rows[0][8].ToString();
oLocalSet.Tables[0].Rows[0][8] = 9;
oLocalSet.Tables[0].AcceptChanges();
oLocalAdapter.Update(oLocalSet.Tables[0]);
// Clean up.
oLocalSet.Dispose();
oLocalAdapter.Dispose();
oLocalCommand.Dispose();
oLocalCommand = null;
Okay, got it.
One, I had to relocate/modify the AcceptChanges() line.
That includes the possible line
oLocalAdapter.AcceptChangesDuringUpdate = true;
I then had to add in
SQLiteCommandBuilder oBuilder = new SQLiteCommandBuilder(oLocalAdapter);
oLocalAdapter.UpdateCommand = oBuilder.GetUpdateCommand();
The last line is then the update and it works. That makes the code:
// DATABASE (Local): Formulate the SQL command.
String strSqlCommand = "SELECT * FROM tblTest ORDER BY IdPrimary ASC;";
SQLiteCommand oLocalCommand = new SQLiteCommand(strSqlCommand, ClassEngine.Connection);
// DATABASE (Local): Get the data records.
SQLiteDataAdapter oLocalAdapter = new SQLiteDataAdapter(oLocalCommand);
DataSet oLocalSet = new DataSet();
oLocalAdapter.Fill(oLocalSet, "tblTest");
//
SQLiteCommandBuilder oBuilder = new SQLiteCommandBuilder(oLocalAdapter);
// Try to write to some changes.
String strValue = oLocalSet.Tables[0].Rows[0][8].ToString();
oLocalSet.Tables[0].Rows[0][8] = 9;
strValue = oLocalSet.Tables[0].Rows[0][8].ToString();
oLocalSet.AcceptChanges();
oLocalAdapter.UpdateCommand = oBuilder.GetUpdateCommand();
oLocalAdapter.Update(oLocalSet.Tables[0]);
// Clean up.
oLocalSet.Dispose();
oLocalAdapter.Dispose();
oLocalCommand.Dispose();
oLocalCommand = null;
public void SaveDataTable(DataTable DT)
{
try
{
con.Open();
cmd = con.CreateCommand();
cmd.CommandText = string.Format("SELECT * FROM {0}", DT.TableName);
adapter = new SQLiteDataAdapter(cmd);
SQLiteCommandBuilder builder = new SQLiteCommandBuilder(adapter);
adapter.Update(DT);
con.Close();
}
catch (Exception Ex)
{
System.Windows.MessageBox.Show(Ex.Message);
}
I have a DataTable with a few records. I want to insert all those records into a remote database. What would be the easiest way to do it? I read that most people iterate over the rows of the DataTable and insert record by record. I would like to make just 1 connection to the remote server and do a bulk insert. Is it possible? I am using C# and MySQL.
Although Kemal Taskin's answer is an elegant solution it is horrible on performance with a large DataTable.
I tried it with a 37500 record insert and it took over 15 minutes.
It seems to be inserting one record at a time.
I found that if I generate a MySQL insert statement string with 1000 records in it and loop over the data until its complete I have reduced my insert time down to 6 seconds. It's not BULK LOADING, its CHUNK LOADING. If anyone can come up with a better solution, please let me know.
public void writeToDBTable(DataTable dt)
{
MySqlConnection conn = new MySqlConnection(globalClass.connString);
conn.Open();
String sql = null;
String sqlStart = "insert into MyTable (run_id, model_id, start_frame,water_year, state_id, obligateCover, DTWoodyCover, perennialGrowth, clonalCover) values ";
Console.WriteLine("Write to DB - Start. Records to insert = {0}", dt.Rows.Count);
int x = 0;
foreach (DataRow row in dt.Rows)
{
x += 1;
if (x == 1)
{
sql = String.Format(#"({0},{1},{2},{3},{4},{5},{6},{7},{8})",
row["runId"],
row["modelId"],
row["startFrame"],
row["waterYear"],
row["currentFrame"],
row["obligateCover"],
row["DTWoodyCover"],
row["perennialGrowth"],
row["clonalCover"]
);
}
else
{
sql = String.Format(sql + #",({0},{1},{2},{3},{4},{5},{6},{7},{8})",
row["runId"],
row["modelId"],
row["startFrame"],
row["waterYear"],
row["currentFrame"],
row["obligateCover"],
row["DTWoodyCover"],
row["perennialGrowth"],
row["clonalCover"]
);
}
if (x == 1000)
{
try
{
sql = sqlStart + sql;
MySqlCommand cmd = new MySqlCommand(sql, conn);
cmd.ExecuteNonQuery();
Console.WriteLine("Write {0}", x);
x = 0;
}
catch (Exception ex)
{
Console.WriteLine(sql);
Console.WriteLine(ex.ToString());
}
}
}
// get any straglers
if (x > 0)
{
try
{
sql = sqlStart + sql;
MySqlCommand cmd = new MySqlCommand(sql, conn);
cmd.ExecuteNonQuery();
Console.WriteLine("Write {0}", x);
x = 0;
}
catch (Exception ex)
{
Console.WriteLine(sql);
Console.WriteLine(ex.ToString());
}
}
conn.Close();
Console.WriteLine("Write to DB - End.");
}
I don't know whether this answer is too late or not :)
You can do something like this:
// assume you have a table with one column;
string commandText = "insert into t_test1 (myid) values (#tempid)";
using (MySqlConnection cn = new MySqlConnection(myConnectionString))
{
cn.Open();
using (MySqlCommand cmd = new MySqlCommand(commandText, cn))
{
cmd.UpdatedRowSource = UpdateRowSource.None;
cmd.Parameters.Add("?tempid", MySqlDbType.UInt32).SourceColumn = "tempid";
MySqlDataAdapter da = new MySqlDataAdapter();
da.InsertCommand = cmd;
// assume DataTable dt contains one column with name "tempid"
int records = da.Update(dt);
}
cn.Close();
}
For Kemal Taşkın solution, RowState set must be equal to DataRowState.Added.
If it is not the case do this :
foreach (DataRow row in dt.Rows)
row.SetAdded();
For Mr.Black, it is recommended to use sql parameter and not use data value directly.
When importing data into InnoDB, turn off autocommit mode, because it performs a log flush to disk for every insert. To disable autocommit during your import operation, surround it with SET autocommit and COMMIT statements:
SET autocommit=0;
... SQL import statements ...
COMMIT;
Performance test : insertion of 5400 rows in 2 tables
insertion from CSV file : 3 seconds
LOAD DATA INFILE 'data.csv' INTO TABLE myTable TERMINATED BY '\t';
insertion by using Kemal Taşkın solution: 32 seconds
MySqlDataAdapter.Update (DataTable)
insertion row by row (): 41 seconds
INSERT INTO table (columns) VALUES (values);
INSERT INTO table (columns) VALUES (values);
...
insertion all rows in one query: 143 seconds
INSERT INTO table (columns) VALUES (values), (values), ...;
=> LOAD DATA is the most performant by far !
You can check also this article :
https://dev.mysql.com/doc/refman/8.0/en/insert-optimization.html
I have filled a DataSet with a Table that was created from another database file. The table is NOT in the database file which I want to be able to copy the Table to.
Now I want to save all those records (DataTable) to a newly created SQLite database file...
How can i do that?
Also I really want to avoid loops if this is possible.
The best answer is by me :) so i'll share it.This is loop but writes 100k entries in 2-3secs.
using (DbTransaction dbTrans = kaupykliuduomConn.BeginTransaction())
{
downloadas.Visible = true; //my progressbar
downloadas.Maximum = dataSet1.Tables["duomenys"].Rows.Count;
using (DbCommand cmd = kaupykliuduomConn.CreateCommand())
{
cmd.CommandText = "INSERT INTO duomenys(Barkodas, Preke, kiekis) VALUES(?,?,?)";
DbParameter Field1 = cmd.CreateParameter();
DbParameter Field2 = cmd.CreateParameter();
DbParameter Field3 = cmd.CreateParameter();
cmd.Parameters.Add(Field1);
cmd.Parameters.Add(Field2);
cmd.Parameters.Add(Field3);
while (n != dataSet1.Tables["duomenys"].Rows.Count)
{
Field1.Value = dataSet1.Tables["duomenys"].Rows[n]["Barkodas"].ToString();
Field2.Value = dataSet1.Tables["duomenys"].Rows[n]["Preke"].ToString();
Field3.Value = dataSet1.Tables["duomenys"].Rows[n]["kiekis"].ToString();
downloadas.Value = n;
n++;
cmd.ExecuteNonQuery();
}
}
dbTrans.Commit();
}
In this case dataSet1.Tables["duomenys"] is already filled with all the data i need to transfer to another database. I used loop to fill dataset too.
When you load the DataTable from the source database, set the AcceptChangesDuringFill property of the data adapter to false, so that loaded records are kept in the Added state (assuming that the source database is SQL Server)
var sqlAdapter = new SqlDataAdapter("SELECT * FROM the_table", sqlConnection);
DataTable table = new DataTable();
sqlAdapter.AcceptChangesDuringFill = false;
sqlAdapter.Fill(table);
Create the table in the SQLite database, by executing the CREATE TABLE statement directly with SQLiteCommand.ExecuteNonQuery
Create a new DataAdapter for the SQLite database connection, and use it to Update the db:
var sqliteAdapter = new SQLiteDataAdapter("SELECT * FROM the_table", sqliteConnection);
var cmdBuilder = new SQLiteCommandBuilder(sqliteAdapter);
sqliteAdapter.Update(table);
If the source and target tables have the same column names and compatible types, it should work fine...
The way to import SQL data to SQLite will take long time. When you want to import data in millions, It will take lot of time. So the shortest and easiest way to do that is just fill fetch the data from SQL database in a DataTable and insert all its rows to SQLite database.
public bool ImportDataToSQLiteDatabase(string Proc, string SQLiteDatabase, params object[] obj)
{
DataTable result = null;
SqlConnection conn = null;
SqlCommand cmd = null;
try
{
result = new DataTable();
using (conn = new SqlConnection(ConStr))
{
using (cmd = CreateCommand(Proc, CommandType.StoredProcedure, obj))
{
cmd.Connection = conn;
conn.Open();
result.Load(cmd.ExecuteReader());
}
}
using (SQLiteConnection con = new SQLiteConnection(string.Format("Data Source={0};Version=3;New=False;Compress=True;Max Pool Size=100;", SQLiteDatabase)))
{
con.Open();
using (SQLiteTransaction transaction = con.BeginTransaction())
{
foreach (DataRow row in result.Rows)
{
using (SQLiteCommand sqlitecommand = new SQLiteCommand("insert into table(fh,ch,mt,pn) values ('" + Convert.ToString(row[0]) + "','" + Convert.ToString(row[1]) + "','"
+ Convert.ToString(row[2]) + "','" + Convert.ToString(row[3]) + "')", con))
{
sqlitecommand.ExecuteNonQuery();
}
}
transaction.Commit();
new General().WriteApplicationLog("Data successfully imported.");
return true;
}
}
}
catch (Exception ex)
{
result = null;
return false;
}
finally
{
if (conn.State == ConnectionState.Open)
conn.Close();
}
}
It will take a very few time as compare to upper given answers.
I'm using SqlBulkCopy against two SQL Server 2008 with different sets of columns (going to move some data from prod server to dev). So want to skip some columns not yet existed / not yet removed.
How can I do that? Some trick with ColumnMappings?
Edit:
I do next:
DataTable table = new DataTable();
using (var adapter = new SqlDataAdapter(sourceCommand))
{
adapter.Fill(table);
}
table.Columns
.OfType<DataColumn>()
.ForEach(c => bulk.ColumnMappings.Add(
new SqlBulkCopyColumnMapping(c.ColumnName, c.ColumnName)));
bulk.WriteToServer(table)
and get:
The given ColumnMapping does not match up with any column in the source or destination.
DataTable table = new DataTable();
using (var adapter = new SqlDataAdapter(sourceCommand))
{
adapter.Fill(table);
}
using (SqlBulkCopy bulk = new SqlBulkCopy(targetConnection, SqlBulkCopyOptions.KeepIdentity, null) { DestinationTableName = tableName })
{
foreach (string columnName in GetMapping(stringSource, stringTarget, tableName))
{
bulk.ColumnMappings.Add(new SqlBulkCopyColumnMapping(columnName, columnName));
}
targetConnection.Open();
bulk.WriteToServer(table);
}
private static IEnumerable<string> GetMapping(string stringSource, string stringTarget, string tableName)
{
return Enumerable.Intersect(
GetSchema(stringSource, tableName),
GetSchema(stringTarget, tableName),
StringComparer.Ordinal); // or StringComparer.OrdinalIgnoreCase
}
private static IEnumerable<string> GetSchema(string connectionString, string tableName)
{
using (SqlConnection connection = new SqlConnection(connectionString))
using (SqlCommand command = connection.CreateCommand())
{
command.CommandText = "sp_Columns";
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add("#table_name", SqlDbType.NVarChar, 384).Value = tableName;
connection.Open();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
yield return (string)reader["column_name"];
}
}
}
}
When SqlBulkCopyColumnMapping is used, only columns for which mappings are created will be copied.
If you do not create a mapping for a column, it will be ignored by the copy process.
You can see this in the demo code here - the sample source table in the AdventureWorks demo database contains more columns than are mapped or copied.
EDIT
It's difficult to be certain without more information about the database schema, but at a guess the issue is with this statement:
new SqlBulkCopyColumnMapping(c.ColumnName, c.ColumnName)
From your description, it sounds like not all the columns in the source table exist in the destination table. You need a filter in your SqlBulkCopyColumnMapping construction loop to skip any columns which do not exist in the destination.
My C# is not good enough to give a example which I'm confident will work, but in pseudocode it would be
foreach column c in sourcetable
{
if c.ColumnName exists in destination_table.columns
{
new SqlBulkCopyColumnMapping(c.ColumnName, c.ColumnName)
}
}
(I'm sure it's possible to convert this to a lambda expression)
Note that this is not particularly robust in the scenario where the column names match but the datatypes are incompatible.
Ed Harper, this is what it looks like without pseudo code
(in this case from DataTable dt (fully defined) to an existing table in the db:
using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connectionString))
{
bulkCopy.DestinationTableName = "dbo.DepartmentsItems";
// Write from the source to the destination.
foreach (DataColumn c in dt.Columns)
{
bulkCopy.ColumnMappings.Add(c.ColumnName, c.ColumnName);
}
bulkCopy.WriteToServer(dt);
return dt.Rows.Count;
}
try this:SqlBulkCopyColumnMapping Class
Hope you are looking for the same