I have to insert 90Mb of data into MySql Tables and I'm using INSERT IGNORE command to avoid the exception of duplicate key. The performance are 8 records a second but it seems very slow. Could I fast it up?
p.s. I'm inserting record per record since i read data from an sql compact database
using (SqlCeConnection sqlConnection = new SqlCeConnection(connectionstrCe))
{
sqlConnection.Open();
SqlCeCommand cmdCe = sqlConnection.CreateCommand();
{
{
mySQLConnection.Open();
foreach (KeyValuePair<string, List<string>> t in tablesCeNames) //reading the tables property from the dictionary - column names and datatypes
{
string tableData = t.Key;
List<string> columnData = t.Value;
//get the values from the table I want to transfer the data
cmdText = "SELECT * FROM " + tableData;
//compose the mysql command
cmdCe.CommandText = cmdText;
SqlCeDataReader dataReader = cmdCe.ExecuteReader(); //read
//InsertTable is a method that get the datareader and convert all the data from this table in a list array with the values to insert
inputValues = InsertTables(dataReader);
MySql.Data.MySqlClient.MySqlTransaction transakcija;
transakcija = mySQLConnection.BeginTransaction();
worker.ReportProgress(4, inputValues.Count);
foreach (string val in inputValues)//foreach row of values of the data table
{
cmdSqlText = "INSERT IGNORE INTO " + tableData + "("; //compose the command for sql
foreach (string cName in columnData) //forach column in a table
{
string[] data = cName.Split(' ');
if (!data[0].ToString().Equals("Id"))
{
cmdSqlText += data[0].ToString() + ","; //write the column names of the values that will be inserted
}
}
cmdSqlText = cmdSqlText.TrimEnd(',');
cmdSqlText += ") VALUES (";
//val contains the values of this current record that i want to insert
cmdSqlText += val; //final command with insert ignore and the values of one record
if (!val.Equals(""))
{
try
{
new MySql.Data.MySqlClient.MySqlCommand(cmdSqlText, mySQLConnection, transakcija).ExecuteNonQuery(); //execute insert on sql database
WriteToTxt("uspješno upisano u mysql" + t.Key);
}
catch (MySql.Data.MySqlClient.MySqlException sqlEx)
{
}
}
}
if (TablicaSveOK)
{
transakcija.Commit();
}
else
{
transakcija.Rollback();
}
}
}
if (mySQLConnection.State != System.Data.ConnectionState.Closed)
{
mySQLConnection.Close();
}
}
What about getting the data from Sql to a file and use LOAD DATA?
http://dev.mysql.com/doc/refman/5.0/es/load-data.html
Rather then sending multiple calls you can send one call to insert all records. Insert it like this
insert into your table(col1, col2)
SELECT 'Name1', 'Location1'
UNION ALL
SELECT 'Name2', 'Location2'
UNION ALL
SELECT 'Name3', 'Location3'
A part from this it is possible that your code is the bottle neck rather than the insert statement. So i would recommend you to first check where the problem lies and then go for the solution.
The latest MySql.Connector has a class called MySqlBulkLoader that could be used to call the LOAD DATA syntax of MySql in a more NET oriented way. The class requires a comma separated value file to load from and is extremely fast.
So your job would be to read all of your Sql Compact records in a datatable and then, using a streamwriter or a specialized CSV Writer write down everything in a file.
Then the code to load your data in a MySql table is simple like this
string connStr = "server=localhost;user=root;database=........";
using(MySqlConnection conn = new MySqlConnection(connStr))
{
MySqlBulkLoader bl = new MySqlBulkLoader(conn);
bl.TableName = "yourdestinationtable";
bl.FieldTerminator = "\t";
bl.LineTerminator = "\n";
bl.FileName = "path_to_your_comma_separated_value_file";
try
{
conn.Open();
int count = bl.Load();
}
}
Related
I have to update multiple records in SQL Server table from C#. Below are the steps I have to follow and below is the code.
The code is working but the process is taking much longer than expected.
I need a quick way to update 10000 records, not sure whether Bulk Copy would work for Update.
I have seen the other answers which has Bulk insert to temp and then update..But that update has a single statement and here I need to update the records in DB based on Excel data and for this I have to loop each excel record.So how can I achieve faster update.
1) Read the Excel Data and copied the data into a data table
string strDirectory = string. Empty;
strDirectory = System.IO.Directory.GetCurrentDirectory() + "\\" + "Filename.xlsx";
string Connection String = "Provider=Microsoft.ACE.OLEDB.12.0; Data Source = " + strDirectory + "; Extended Properties = \"Excel 12.0;HDR=YES;IMEX=1\"";
using (OleDbConnection conn = new OleDbConnection(Connection String))
{
conn.Open();
DataTable schemaTable = conn.GetOleDbSchemaTableOleDbSchemaGuid.Tables, new object[] { null, null, null, "TABLE" });
DataRow schemaRow = schemaTable. Rows[0];
string sheet = schemaRow["TABLE_NAME"].ToString();
string query = "SELECT * FROM [" + sheet + "]";
OleDbDataAdapter daexcel = new OleDbDataAdapter(query, conn);
daexcel.Fill(dt);
conn.Close();
}
2) Doing some manipulations on datatable data before updating into table.
string strsqlst = string. Empty;
using (SqlConnection sqlConn = new SqlConnection(Connectionstring))
{
sqlConn.Open();
SqlCommand cmd;
StringBuilder sb = new StringBuilder();
sb.AppendLine("DataTable content:");
foreach (DataRow row in dt.Rows)
{
if (row.ItemArray[0].ToString() == "")
break;
strsqlst = "Update table Set col1= " + row.ItemArray[4].ToString() + " ,col2= " + row.ItemArray[5].ToString() + " where <Condition>'";
cmd = new SqlCommand(strsqlst, sqlConn);
cmd.CommandType = CommandType.Text;
cmd.ExecuteNonQuery();
}
sqlConn.Close();
}
The SqlCommand can be a whole SQL batch and is not limited to a single statement. So you can create a single large batch with 10,000 UPDATE statements, or divide it into for example 20 batches of 500 each.
In other words, you can create a single command with CommandText like this:
UPDATE [T] SET Col1='Value1', Col2='Value2' WHERE [Id] = 1;
...
UPDATE [T] SET Col1='Value999', Col2='Value1000' WHERE [Id] = 500;
That said, you should use parameters for all data values (to ensure SQL injection is not possible).
If you want to handle any errors (updates failing due to invalid data) you will need something a bit more sophisticated.
In my current application I have a snippet of code that allows me to select a single row in a data grid view and store all the columns information into a variable to do with what I want. I use this primarily to send information from one SQL database table to another. It's great for only sending specified cells within a row.
Here is how I do a single row:
string ID = dataGridView1.SelectedRows[0].Cells[0].Value + string.Empty;
string itemOne= dataGridView1.SelectedRows[0].Cells[1].Value + string.Empty;
string itemTwo= dataGridView1.SelectedRows[0].Cells[2].Value + string.Empty;
string itemThree= dataGridView1.SelectedRows[0].Cells[3].Value + string.Empty;
var vItemOne = itemOne;
var vItemTwo= itemTwo;
var vItemThree= itemThree;
// ETC..
However, I now want to be able to select Multiple Rows and only insert specified columns within those rows to a SQL database.
I've tried modifying the above code to work... obviously it doesn't work.
I believe I need a loop, I haven't really used loops much so I'm not sure how to make it loop, skip certain columns, then insert into database.
This is what I am currently attempting, however I seem to be messing up somewhere.
using (SqlConnection con = new SqlConnection(Connection.MTRDataBaseConn))
{
for (int i = 0; i < dataGridView1.SelectedRows.Count; i++)
{
con.Open();
SqlCommand cmd = new SqlCommand();
cmd.CommandText = "INSERT INTO dbo.[" + txtJobName.Text + "] ([Item One], [Item Two], [Item Three]) VALUES(#ItemOne,#ItemTwo,#ItemThree)";
cmd.Connection = con;
string strItemOne = this.dataGridView1.SelectedRows[i].Cells[1].Value + string.Empty;
string strItemTwo = this.dataGridView1.SelectedRows[i].Cells[2].Value + string.Empty;
string strItemThree = this.dataGridView1.SelectedRows[i].Cells[3].Value + string.Empty;
//Parameters
cmd.Parameters.AddWithValue("#ItemOne", strItemOne);
cmd.Parameters.AddWithValue("#ItemTwo", strItemTwo);
cmd.Parameters.AddWithValue("#ItemThree", strItemThree);
//execute
cmd.ExecuteNonQuery();
//close connection
con.Close();
}
}
...
While Debugging My dataGridView.SelectedRows.Count; i++ doesn't seem to be increasing and is staying at 0... I'm receiving the error when I try to return the selected row to a string. Shouldn't my selected rows still return a value?
I'm under the assumption my loop is wrong.
Can anyone help me with my issue?
Simply have to use a for each statement
string itemOne= dataGridView1.SelectedRows[0].Cells[1].Value + string.Empty;
string itemTwo= dataGridView1.SelectedRows[0].Cells[2].Value + string.Empty;
string itemThree= dataGridView1.SelectedRows[0].Cells[3].Value + string.Empty;
var vItemOne = itemOne;
var vItemTwo= itemTwo;
var vItemThree= itemThree;
foreach (DataGridViewRow row in dataGridView1.SelectedRows)
{
//Insert Query Here
}
I am trying to save an online MySql database to a local Sqlite database. What I do is, try to get whole data from MySql to a datatable then from the datatable save it to the Sqlite. So, I tried to make sure that I get all the data from the MySql first counting the datatable through "Console.printLine".
The problem is, when I count the data in the PhpMyAdmin of the MySql, using SELECT COUNT(REFENCE) I get a count of 83,335 rows but in my Console.WriteLine, using the increment +1, I only get up to 3,857 rows which made me to believe that what I am getting is incomplete. Can you show me where I am wrong. If you have better way of copying then Online MySql "table" data to Sqlite table that would be great. Thank you.
private void buttonX1_Click(object sender, EventArgs e)
{
const string sql1 =
#"SELECT reference, DATE(trx_date) as 'date', chassis_nmbr, status, remarks,
approved_by, DATE(date_approved), branch_code FROM jobcomp_hdr;";
var table = new DataTable("onlineData");
using (var conn = new DwoAccess().ConnectToMySql())
{
using (var cmd = new MySqlCommand(sql1, conn))
{
try
{
cmd.CommandTimeout = 60;
var adapt = new MySqlDataAdapter(cmd);
conn.Open();
adapt.Fill(table);
}
catch (Exception ex)
{
//ignore
}
}
}
var x=0;
foreach (DataRow row in table.Rows)
{
Console.WriteLine(row["reference"].ToString());
x += 1;
}
Console.WriteLine(x.ToString());
}
I think the problem is with time of executing query and timeout, number of all rows is to big and executing it is to long. So You can use LIMIT where you set number rows to skip and number rows to get. Using for loop and quantity of all rows in table you will be able take tables divided into several parts 1,000 items in this case.
Here you find more about LIMIT in mysql ,samples
for (int i = 0; i <= rowsNumberInTable; i = i + 1000)
{
...
string sql1 =#"SELECT reference, DATE(trx_date) as 'date', chassis_nmbr, status, remarks, approved_by, DATE(date_approved), branch_code FROM jobcomp_hdr LIMIT " + i + #" ," + i+ 1000;
...
}
Not sure if this is written correctly but it looks correct. I am wanting to update a record if the id already exists and insert if not.
DataSet ds = new DataSet();
ds.ReadXml(XDocument.Load(Application.StartupPath + #"\xml1.xml").CreateReader());
using (var conn = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0; Data Source=" + Application.StartupPath + "\\Database3.mdb"))
{
conn.Open();
// make two commands here
var commInsert = new OleDbCommand("Insert INTO Table1 (description, active) VALUES (#iq_question,#active);", conn);
var commUpdate = new OleDbCommand("UPDATE Table1 SET description=#iq_question,active=#active WHERE ID=#question_id;", conn);
// here add your parameters with no value
//string question_id = row[0].ToString();
//string iq_question = row[1].ToString();
//string active = row[4].ToString();
commInsert.Parameters.Add(new OleDbParameter("#iq_question", OleDbType.VarChar));
commInsert.Parameters.Add(new OleDbParameter("#active", OleDbType.VarChar));
commUpdate.Parameters.Add(new OleDbParameter("#question_id", OleDbType.AutoNumber));
commUpdate.Parameters.Add(new OleDbParameter("#iq_question", OleDbType.Text));
commUpdate.Parameters.Add(new OleDbParameter("#active", OleDbType.Text));
foreach (DataTable table in ds.Tables)
{
foreach (DataRow row in table.Rows)
{
// here only reset the values
commUpdate.Parameters["#question_id"].Value = row[0].ToString();
commUpdate.Parameters["#iq_question"].Value = row[1].ToString();
commUpdate.Parameters["#active"].Value = row[4].ToString();
int recs = commUpdate.ExecuteNonQuery();
if (recs < 1) // when no records updated do insert
{
commInsert.Parameters["#iq_question"].Value = row[1].ToString();
commInsert.Parameters["#active"].Value = row[4].ToString();
commInsert.ExecuteNonQuery();
}
}
}
commInsert.Dispose();
commUpdate.Dispose();
conn.Close();
}
System.Windows.Forms.MessageBox.Show("Updated Latest Data Was Succesfull");
I either get an error on the insert saying it will create duplicate content, or it creates more rows with different data. So say I should be getting 10 rows from the xml file, the first time I run it I get the 10 rows with the correct data. If I run it again, I end up with 10 more so being 20 but the last 10 rows show different data. I don't think I am identifying the rows in the xml file correctly and I need to do some research on that part.
There is no Exists for MS Access. The engine is much more primitive than Sql Server. See here: Microsoft Access SQL. I think, what you can do is:
myCommand.CommandText = "UPDATE Table1 SET description=#iq_question,active=#active WHERE ID=#currentRow";
......
int recs = myCommand.ExecuteNonQuery();
if (recs < 1) // when no records updated do insert
{
myCommand.Parameters.Clear();
myCommand.CommandText = "Insert INTO Table1 VALUES(#iq_question,#active)";
.....
}
This is still 2 statements but you can save some coding by not doing Select first. Because ExecuteNonQuery will tell you if you updated anything
Another thing is that your code is a bit inefficient. You have nested loop where you can reuse same command and connection. Yuu can do this
using (var conn = new OleDbConnection(.......))
{
conn.Open();
// make two commands here
var commInsert = new OleDbCommand(.....);
var commUpdate = new OleDbCommand(.....);
// here add your parameters with no value
commInsert.Parameters.Add(new OleDbParameter(....));
.......
Foreach (....)
{
Foreach (....)
{
// here only reset the values
commUpdate.Parameters[0].Value = ...
...
int recs = commUpdate.ExecuteNonQuery();
if (recs < 1) // when no records updated do insert
{
commInsert.Parameters[0].Value = iq_question;
.....
}
}
}
commInsert.Dispose();
commUpdate.Dispose();
}
You can also use nested using for commands. Only setting values will be more efficient to what you do right now.
I have a DataTable with a few records. I want to insert all those records into a remote database. What would be the easiest way to do it? I read that most people iterate over the rows of the DataTable and insert record by record. I would like to make just 1 connection to the remote server and do a bulk insert. Is it possible? I am using C# and MySQL.
Although Kemal Taskin's answer is an elegant solution it is horrible on performance with a large DataTable.
I tried it with a 37500 record insert and it took over 15 minutes.
It seems to be inserting one record at a time.
I found that if I generate a MySQL insert statement string with 1000 records in it and loop over the data until its complete I have reduced my insert time down to 6 seconds. It's not BULK LOADING, its CHUNK LOADING. If anyone can come up with a better solution, please let me know.
public void writeToDBTable(DataTable dt)
{
MySqlConnection conn = new MySqlConnection(globalClass.connString);
conn.Open();
String sql = null;
String sqlStart = "insert into MyTable (run_id, model_id, start_frame,water_year, state_id, obligateCover, DTWoodyCover, perennialGrowth, clonalCover) values ";
Console.WriteLine("Write to DB - Start. Records to insert = {0}", dt.Rows.Count);
int x = 0;
foreach (DataRow row in dt.Rows)
{
x += 1;
if (x == 1)
{
sql = String.Format(#"({0},{1},{2},{3},{4},{5},{6},{7},{8})",
row["runId"],
row["modelId"],
row["startFrame"],
row["waterYear"],
row["currentFrame"],
row["obligateCover"],
row["DTWoodyCover"],
row["perennialGrowth"],
row["clonalCover"]
);
}
else
{
sql = String.Format(sql + #",({0},{1},{2},{3},{4},{5},{6},{7},{8})",
row["runId"],
row["modelId"],
row["startFrame"],
row["waterYear"],
row["currentFrame"],
row["obligateCover"],
row["DTWoodyCover"],
row["perennialGrowth"],
row["clonalCover"]
);
}
if (x == 1000)
{
try
{
sql = sqlStart + sql;
MySqlCommand cmd = new MySqlCommand(sql, conn);
cmd.ExecuteNonQuery();
Console.WriteLine("Write {0}", x);
x = 0;
}
catch (Exception ex)
{
Console.WriteLine(sql);
Console.WriteLine(ex.ToString());
}
}
}
// get any straglers
if (x > 0)
{
try
{
sql = sqlStart + sql;
MySqlCommand cmd = new MySqlCommand(sql, conn);
cmd.ExecuteNonQuery();
Console.WriteLine("Write {0}", x);
x = 0;
}
catch (Exception ex)
{
Console.WriteLine(sql);
Console.WriteLine(ex.ToString());
}
}
conn.Close();
Console.WriteLine("Write to DB - End.");
}
I don't know whether this answer is too late or not :)
You can do something like this:
// assume you have a table with one column;
string commandText = "insert into t_test1 (myid) values (#tempid)";
using (MySqlConnection cn = new MySqlConnection(myConnectionString))
{
cn.Open();
using (MySqlCommand cmd = new MySqlCommand(commandText, cn))
{
cmd.UpdatedRowSource = UpdateRowSource.None;
cmd.Parameters.Add("?tempid", MySqlDbType.UInt32).SourceColumn = "tempid";
MySqlDataAdapter da = new MySqlDataAdapter();
da.InsertCommand = cmd;
// assume DataTable dt contains one column with name "tempid"
int records = da.Update(dt);
}
cn.Close();
}
For Kemal Taşkın solution, RowState set must be equal to DataRowState.Added.
If it is not the case do this :
foreach (DataRow row in dt.Rows)
row.SetAdded();
For Mr.Black, it is recommended to use sql parameter and not use data value directly.
When importing data into InnoDB, turn off autocommit mode, because it performs a log flush to disk for every insert. To disable autocommit during your import operation, surround it with SET autocommit and COMMIT statements:
SET autocommit=0;
... SQL import statements ...
COMMIT;
Performance test : insertion of 5400 rows in 2 tables
insertion from CSV file : 3 seconds
LOAD DATA INFILE 'data.csv' INTO TABLE myTable TERMINATED BY '\t';
insertion by using Kemal Taşkın solution: 32 seconds
MySqlDataAdapter.Update (DataTable)
insertion row by row (): 41 seconds
INSERT INTO table (columns) VALUES (values);
INSERT INTO table (columns) VALUES (values);
...
insertion all rows in one query: 143 seconds
INSERT INTO table (columns) VALUES (values), (values), ...;
=> LOAD DATA is the most performant by far !
You can check also this article :
https://dev.mysql.com/doc/refman/8.0/en/insert-optimization.html