I have a DataTable with a few records. I want to insert all those records into a remote database. What would be the easiest way to do it? I read that most people iterate over the rows of the DataTable and insert record by record. I would like to make just 1 connection to the remote server and do a bulk insert. Is it possible? I am using C# and MySQL.
Although Kemal Taskin's answer is an elegant solution it is horrible on performance with a large DataTable.
I tried it with a 37500 record insert and it took over 15 minutes.
It seems to be inserting one record at a time.
I found that if I generate a MySQL insert statement string with 1000 records in it and loop over the data until its complete I have reduced my insert time down to 6 seconds. It's not BULK LOADING, its CHUNK LOADING. If anyone can come up with a better solution, please let me know.
public void writeToDBTable(DataTable dt)
{
MySqlConnection conn = new MySqlConnection(globalClass.connString);
conn.Open();
String sql = null;
String sqlStart = "insert into MyTable (run_id, model_id, start_frame,water_year, state_id, obligateCover, DTWoodyCover, perennialGrowth, clonalCover) values ";
Console.WriteLine("Write to DB - Start. Records to insert = {0}", dt.Rows.Count);
int x = 0;
foreach (DataRow row in dt.Rows)
{
x += 1;
if (x == 1)
{
sql = String.Format(#"({0},{1},{2},{3},{4},{5},{6},{7},{8})",
row["runId"],
row["modelId"],
row["startFrame"],
row["waterYear"],
row["currentFrame"],
row["obligateCover"],
row["DTWoodyCover"],
row["perennialGrowth"],
row["clonalCover"]
);
}
else
{
sql = String.Format(sql + #",({0},{1},{2},{3},{4},{5},{6},{7},{8})",
row["runId"],
row["modelId"],
row["startFrame"],
row["waterYear"],
row["currentFrame"],
row["obligateCover"],
row["DTWoodyCover"],
row["perennialGrowth"],
row["clonalCover"]
);
}
if (x == 1000)
{
try
{
sql = sqlStart + sql;
MySqlCommand cmd = new MySqlCommand(sql, conn);
cmd.ExecuteNonQuery();
Console.WriteLine("Write {0}", x);
x = 0;
}
catch (Exception ex)
{
Console.WriteLine(sql);
Console.WriteLine(ex.ToString());
}
}
}
// get any straglers
if (x > 0)
{
try
{
sql = sqlStart + sql;
MySqlCommand cmd = new MySqlCommand(sql, conn);
cmd.ExecuteNonQuery();
Console.WriteLine("Write {0}", x);
x = 0;
}
catch (Exception ex)
{
Console.WriteLine(sql);
Console.WriteLine(ex.ToString());
}
}
conn.Close();
Console.WriteLine("Write to DB - End.");
}
I don't know whether this answer is too late or not :)
You can do something like this:
// assume you have a table with one column;
string commandText = "insert into t_test1 (myid) values (#tempid)";
using (MySqlConnection cn = new MySqlConnection(myConnectionString))
{
cn.Open();
using (MySqlCommand cmd = new MySqlCommand(commandText, cn))
{
cmd.UpdatedRowSource = UpdateRowSource.None;
cmd.Parameters.Add("?tempid", MySqlDbType.UInt32).SourceColumn = "tempid";
MySqlDataAdapter da = new MySqlDataAdapter();
da.InsertCommand = cmd;
// assume DataTable dt contains one column with name "tempid"
int records = da.Update(dt);
}
cn.Close();
}
For Kemal Taşkın solution, RowState set must be equal to DataRowState.Added.
If it is not the case do this :
foreach (DataRow row in dt.Rows)
row.SetAdded();
For Mr.Black, it is recommended to use sql parameter and not use data value directly.
When importing data into InnoDB, turn off autocommit mode, because it performs a log flush to disk for every insert. To disable autocommit during your import operation, surround it with SET autocommit and COMMIT statements:
SET autocommit=0;
... SQL import statements ...
COMMIT;
Performance test : insertion of 5400 rows in 2 tables
insertion from CSV file : 3 seconds
LOAD DATA INFILE 'data.csv' INTO TABLE myTable TERMINATED BY '\t';
insertion by using Kemal Taşkın solution: 32 seconds
MySqlDataAdapter.Update (DataTable)
insertion row by row (): 41 seconds
INSERT INTO table (columns) VALUES (values);
INSERT INTO table (columns) VALUES (values);
...
insertion all rows in one query: 143 seconds
INSERT INTO table (columns) VALUES (values), (values), ...;
=> LOAD DATA is the most performant by far !
You can check also this article :
https://dev.mysql.com/doc/refman/8.0/en/insert-optimization.html
Related
I am trying to save an online MySql database to a local Sqlite database. What I do is, try to get whole data from MySql to a datatable then from the datatable save it to the Sqlite. So, I tried to make sure that I get all the data from the MySql first counting the datatable through "Console.printLine".
The problem is, when I count the data in the PhpMyAdmin of the MySql, using SELECT COUNT(REFENCE) I get a count of 83,335 rows but in my Console.WriteLine, using the increment +1, I only get up to 3,857 rows which made me to believe that what I am getting is incomplete. Can you show me where I am wrong. If you have better way of copying then Online MySql "table" data to Sqlite table that would be great. Thank you.
private void buttonX1_Click(object sender, EventArgs e)
{
const string sql1 =
#"SELECT reference, DATE(trx_date) as 'date', chassis_nmbr, status, remarks,
approved_by, DATE(date_approved), branch_code FROM jobcomp_hdr;";
var table = new DataTable("onlineData");
using (var conn = new DwoAccess().ConnectToMySql())
{
using (var cmd = new MySqlCommand(sql1, conn))
{
try
{
cmd.CommandTimeout = 60;
var adapt = new MySqlDataAdapter(cmd);
conn.Open();
adapt.Fill(table);
}
catch (Exception ex)
{
//ignore
}
}
}
var x=0;
foreach (DataRow row in table.Rows)
{
Console.WriteLine(row["reference"].ToString());
x += 1;
}
Console.WriteLine(x.ToString());
}
I think the problem is with time of executing query and timeout, number of all rows is to big and executing it is to long. So You can use LIMIT where you set number rows to skip and number rows to get. Using for loop and quantity of all rows in table you will be able take tables divided into several parts 1,000 items in this case.
Here you find more about LIMIT in mysql ,samples
for (int i = 0; i <= rowsNumberInTable; i = i + 1000)
{
...
string sql1 =#"SELECT reference, DATE(trx_date) as 'date', chassis_nmbr, status, remarks, approved_by, DATE(date_approved), branch_code FROM jobcomp_hdr LIMIT " + i + #" ," + i+ 1000;
...
}
I'm having some performance issues when I return a record set with more than 1,000 records.
Sometimes the records are in upwards of 2,100 but can be as low as 10.
I have some bulk actions that I take on all the records by selecting them.
However, when the number is low, the Gridview is fine. When the record count is greater than 500 I see performance issues on the page.
What I want to happen is: if there are more than 500 records, DO NOT DISPLAY THE GRID, instead show a download button that exports to CSV or do other control things on the page.
My issue:
Even if i tell it not to display the grid and instead display a message and a button, the performance is still slow.
Below is my C# code for populating the GridView. Some stuff has been removed that are unimportant and to help with readability.
How can I adjust my C# code for better performance?
SqlConnection conn = new SqlConnection(ConfigurationManager.AppSettings["ConnectString"].ToString());
SqlCommand cmd = conn.CreateCommand();
cmd.CommandType = CommandType.StoredProcedure;
cmd.CommandText = "SomeProcedure";
cmd.Parameters.Add(SearchParam);
try {
DataTable GridData = new DataTable();
conn.Open();
using(SqlDataAdapter Sqlda = new SqlDataAdapter(cmd)) {
Sqlda.Fill(GridData);
}
if (GridData.Rows.Count == 0) {
lblSearchMsg.Text = "No Fee Records are in the Queue at this time.";
} else {
if (GridData.Rows.Count > 500) {
lblSearchMsg.Text = "More than " + gridLimit.ToString() + " records returned.";
//Show the download button
} else {
//Persist the table in the Session object. (for sorting)
Session["GridData"] = GridData;
lblRowCount.Text = "Count: " + GridData.Rows.Count.ToString();
myGridView.DataSource = GridData;
myGridView.DataBind();
myGridView.Visible = true;
}
}
} catch (Exception ex) {
//Do the error stuff
} finally {
if (conn != null) {
conn.Close();
}
}
Create a separate procedure that returns only the row count.
Check that value not the row count of a fully retrieved data set then retrieve the full data set as needed.
Keep in mind you can use the same connection to do both retrievals, no need to close the connection between calls.
if you determine you need to fill a gridview and there is no need to edit the data you can read into the DataTable without the use of an adapter. Here is the basic idea modify with using statements or try/catch as you prefer:
conn = new SqlConnection(connString);
string query = "SELECT * FROM ....";
SqlCommand cmd = new SqlCommand(query, conn);
conn.Open();
SqlDataReader dr = cmd.ExecuteReader();
DataTable dt = new DataTable();
dt.Load(dr);
GridView1.DataSource = dt;
GridView1.DataBind();
I am migrating my program from Microsoft SQL Server to MySQL. Everything works well except one issue with bulk copy.
In the solution with MS SQL the code looks like this:
connection.Open();
SqlBulkCopy bulkCopy = new SqlBulkCopy(connection);
bulkCopy.DestinationTableName = "testTable";
bulkCopy.WriteToServer(rawData);
Now I try to do something similar for MySQL. Because I think there would be bad performance I don't want to write the DataTable to a CSV file and do the insert from there with the MySqlBulkLoader class.
Any help would be highly appreciated.
Because I think there would be bad performance I don't want to write the DataTable to a CSV file and do the insert from there with the MySqlBulkLoader class.
Don't rule out a possible solution based on unfounded assumptions. I just tested the insertion of 100,000 rows from a System.Data.DataTable into a MySQL table using a standard MySqlDataAdapter#Update() inside a Transaction. It consistently took about 30 seconds to run:
using (MySqlTransaction tran = conn.BeginTransaction(System.Data.IsolationLevel.Serializable))
{
using (MySqlCommand cmd = new MySqlCommand())
{
cmd.Connection = conn;
cmd.Transaction = tran;
cmd.CommandText = "SELECT * FROM testtable";
using (MySqlDataAdapter da = new MySqlDataAdapter(cmd))
{
da.UpdateBatchSize = 1000;
using (MySqlCommandBuilder cb = new MySqlCommandBuilder(da))
{
da.Update(rawData);
tran.Commit();
}
}
}
}
(I tried a couple of different values for UpdateBatchSize but they didn't seem to have a significant impact on the elapsed time.)
By contrast, the following code using MySqlBulkLoader took only 5 or 6 seconds to run ...
string tempCsvFileSpec = #"C:\Users\Gord\Desktop\dump.csv";
using (StreamWriter writer = new StreamWriter(tempCsvFileSpec))
{
Rfc4180Writer.WriteDataTable(rawData, writer, false);
}
var msbl = new MySqlBulkLoader(conn);
msbl.TableName = "testtable";
msbl.FileName = tempCsvFileSpec;
msbl.FieldTerminator = ",";
msbl.FieldQuotationCharacter = '"';
msbl.Load();
System.IO.File.Delete(tempCsvFileSpec);
... including the time to dump the 100,000 rows from the DataTable to a temporary CSV file (using code similar to this), bulk-loading from that file, and deleting the file afterwards.
Similar to SqlBulkCopy, we have MySqlBulkCopy for Mysql.
here is the example how to use it.
public async Task<bool> MySqlBulCopyAsync(DataTable dataTable)
{
try
{
bool result = true;
using (var connection = new MySqlConnector.MySqlConnection(_connString + ";AllowLoadLocalInfile=True"))
{
await connection.OpenAsync();
var bulkCopy = new MySqlBulkCopy(connection);
bulkCopy.DestinationTableName = "yourtable";
// the column mapping is required if you have a identity column in the table
bulkCopy.ColumnMappings.AddRange(GetMySqlColumnMapping(dataTable));
await bulkCopy.WriteToServerAsync(dataTable);
return result;
}
}
catch (Exception ex)
{
throw;
}
}
private List<MySqlBulkCopyColumnMapping> GetMySqlColumnMapping(DataTable dataTable)
{
List<MySqlBulkCopyColumnMapping> colMappings = new List<MySqlBulkCopyColumnMapping>();
int i = 0;
foreach (DataColumn col in dataTable.Columns)
{
colMappings.Add(new MySqlBulkCopyColumnMapping(i, col.ColumnName));
i++;
}
return colMappings;
}
You can ignore the column mapping if you don't have any identity column in your table.
If you have identity column then you have to use the column mapping otherwise it won't insert any records in the table
It will just give message like "x rows were copied but only 0 rows were inserted".
This class i available in the below library
Assembly MySqlConnector, Version=1.0.0.0
Using any of BulkOperation NuGet-package, you can easily have this done.
Here is an example using the package from https://www.nuget.org/packages/Z.BulkOperations/2.14.3/
MySqlConnection conn = DbConnection.OpenConnection();
DataTable dt = new DataTable("testtable");
MySqlDataAdapter da = new MySqlDataAdapter("SELECT * FROM testtable", conn);
MySqlCommandBuilder cb = new MySqlCommandBuilder(da);
da.Fill(dt);
instead of using
......
da.UpdateBatchSize = 1000;
......
da.Update(dt)
just following two lines
var bulk = new BulkOperation(conn);
bulk.BulkInsert(dt);
will take only 5 seconds to copy the whole DataTable into MySQL without first dumping the 100,000 rows from the DataTable to a temporary CSV file.
Not sure if this is written correctly but it looks correct. I am wanting to update a record if the id already exists and insert if not.
DataSet ds = new DataSet();
ds.ReadXml(XDocument.Load(Application.StartupPath + #"\xml1.xml").CreateReader());
using (var conn = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0; Data Source=" + Application.StartupPath + "\\Database3.mdb"))
{
conn.Open();
// make two commands here
var commInsert = new OleDbCommand("Insert INTO Table1 (description, active) VALUES (#iq_question,#active);", conn);
var commUpdate = new OleDbCommand("UPDATE Table1 SET description=#iq_question,active=#active WHERE ID=#question_id;", conn);
// here add your parameters with no value
//string question_id = row[0].ToString();
//string iq_question = row[1].ToString();
//string active = row[4].ToString();
commInsert.Parameters.Add(new OleDbParameter("#iq_question", OleDbType.VarChar));
commInsert.Parameters.Add(new OleDbParameter("#active", OleDbType.VarChar));
commUpdate.Parameters.Add(new OleDbParameter("#question_id", OleDbType.AutoNumber));
commUpdate.Parameters.Add(new OleDbParameter("#iq_question", OleDbType.Text));
commUpdate.Parameters.Add(new OleDbParameter("#active", OleDbType.Text));
foreach (DataTable table in ds.Tables)
{
foreach (DataRow row in table.Rows)
{
// here only reset the values
commUpdate.Parameters["#question_id"].Value = row[0].ToString();
commUpdate.Parameters["#iq_question"].Value = row[1].ToString();
commUpdate.Parameters["#active"].Value = row[4].ToString();
int recs = commUpdate.ExecuteNonQuery();
if (recs < 1) // when no records updated do insert
{
commInsert.Parameters["#iq_question"].Value = row[1].ToString();
commInsert.Parameters["#active"].Value = row[4].ToString();
commInsert.ExecuteNonQuery();
}
}
}
commInsert.Dispose();
commUpdate.Dispose();
conn.Close();
}
System.Windows.Forms.MessageBox.Show("Updated Latest Data Was Succesfull");
I either get an error on the insert saying it will create duplicate content, or it creates more rows with different data. So say I should be getting 10 rows from the xml file, the first time I run it I get the 10 rows with the correct data. If I run it again, I end up with 10 more so being 20 but the last 10 rows show different data. I don't think I am identifying the rows in the xml file correctly and I need to do some research on that part.
There is no Exists for MS Access. The engine is much more primitive than Sql Server. See here: Microsoft Access SQL. I think, what you can do is:
myCommand.CommandText = "UPDATE Table1 SET description=#iq_question,active=#active WHERE ID=#currentRow";
......
int recs = myCommand.ExecuteNonQuery();
if (recs < 1) // when no records updated do insert
{
myCommand.Parameters.Clear();
myCommand.CommandText = "Insert INTO Table1 VALUES(#iq_question,#active)";
.....
}
This is still 2 statements but you can save some coding by not doing Select first. Because ExecuteNonQuery will tell you if you updated anything
Another thing is that your code is a bit inefficient. You have nested loop where you can reuse same command and connection. Yuu can do this
using (var conn = new OleDbConnection(.......))
{
conn.Open();
// make two commands here
var commInsert = new OleDbCommand(.....);
var commUpdate = new OleDbCommand(.....);
// here add your parameters with no value
commInsert.Parameters.Add(new OleDbParameter(....));
.......
Foreach (....)
{
Foreach (....)
{
// here only reset the values
commUpdate.Parameters[0].Value = ...
...
int recs = commUpdate.ExecuteNonQuery();
if (recs < 1) // when no records updated do insert
{
commInsert.Parameters[0].Value = iq_question;
.....
}
}
}
commInsert.Dispose();
commUpdate.Dispose();
}
You can also use nested using for commands. Only setting values will be more efficient to what you do right now.
I have to insert 90Mb of data into MySql Tables and I'm using INSERT IGNORE command to avoid the exception of duplicate key. The performance are 8 records a second but it seems very slow. Could I fast it up?
p.s. I'm inserting record per record since i read data from an sql compact database
using (SqlCeConnection sqlConnection = new SqlCeConnection(connectionstrCe))
{
sqlConnection.Open();
SqlCeCommand cmdCe = sqlConnection.CreateCommand();
{
{
mySQLConnection.Open();
foreach (KeyValuePair<string, List<string>> t in tablesCeNames) //reading the tables property from the dictionary - column names and datatypes
{
string tableData = t.Key;
List<string> columnData = t.Value;
//get the values from the table I want to transfer the data
cmdText = "SELECT * FROM " + tableData;
//compose the mysql command
cmdCe.CommandText = cmdText;
SqlCeDataReader dataReader = cmdCe.ExecuteReader(); //read
//InsertTable is a method that get the datareader and convert all the data from this table in a list array with the values to insert
inputValues = InsertTables(dataReader);
MySql.Data.MySqlClient.MySqlTransaction transakcija;
transakcija = mySQLConnection.BeginTransaction();
worker.ReportProgress(4, inputValues.Count);
foreach (string val in inputValues)//foreach row of values of the data table
{
cmdSqlText = "INSERT IGNORE INTO " + tableData + "("; //compose the command for sql
foreach (string cName in columnData) //forach column in a table
{
string[] data = cName.Split(' ');
if (!data[0].ToString().Equals("Id"))
{
cmdSqlText += data[0].ToString() + ","; //write the column names of the values that will be inserted
}
}
cmdSqlText = cmdSqlText.TrimEnd(',');
cmdSqlText += ") VALUES (";
//val contains the values of this current record that i want to insert
cmdSqlText += val; //final command with insert ignore and the values of one record
if (!val.Equals(""))
{
try
{
new MySql.Data.MySqlClient.MySqlCommand(cmdSqlText, mySQLConnection, transakcija).ExecuteNonQuery(); //execute insert on sql database
WriteToTxt("uspješno upisano u mysql" + t.Key);
}
catch (MySql.Data.MySqlClient.MySqlException sqlEx)
{
}
}
}
if (TablicaSveOK)
{
transakcija.Commit();
}
else
{
transakcija.Rollback();
}
}
}
if (mySQLConnection.State != System.Data.ConnectionState.Closed)
{
mySQLConnection.Close();
}
}
What about getting the data from Sql to a file and use LOAD DATA?
http://dev.mysql.com/doc/refman/5.0/es/load-data.html
Rather then sending multiple calls you can send one call to insert all records. Insert it like this
insert into your table(col1, col2)
SELECT 'Name1', 'Location1'
UNION ALL
SELECT 'Name2', 'Location2'
UNION ALL
SELECT 'Name3', 'Location3'
A part from this it is possible that your code is the bottle neck rather than the insert statement. So i would recommend you to first check where the problem lies and then go for the solution.
The latest MySql.Connector has a class called MySqlBulkLoader that could be used to call the LOAD DATA syntax of MySql in a more NET oriented way. The class requires a comma separated value file to load from and is extremely fast.
So your job would be to read all of your Sql Compact records in a datatable and then, using a streamwriter or a specialized CSV Writer write down everything in a file.
Then the code to load your data in a MySql table is simple like this
string connStr = "server=localhost;user=root;database=........";
using(MySqlConnection conn = new MySqlConnection(connStr))
{
MySqlBulkLoader bl = new MySqlBulkLoader(conn);
bl.TableName = "yourdestinationtable";
bl.FieldTerminator = "\t";
bl.LineTerminator = "\n";
bl.FileName = "path_to_your_comma_separated_value_file";
try
{
conn.Open();
int count = bl.Load();
}
}