I insert data like this but it takes too long
For 34,000 records it took 20 minutes !! (for example if I insert into SQL Server CE it took only 3 minutes)
conn_sql = new SQLiteConnection(conn_str2);
conn_sql.Open();
cmd_sql = conn_sql.CreateCommand();
for (int i = 0; i < iTotalRows; i++)
{
try
{
Makat = dsView.Tables["Items"].Rows[i]["Makat"].ToString().Trim();
}
catch { Makat = ""; }
try
{
Barcode = dsView.Tables["Items"].Rows[i]["Barcode"].ToString().Trim();
}
catch { Barcode = ""; }
try
{
Des = dsView.Tables["Items"].Rows[i]["Des"].ToString().Trim();
}
catch { Des = ""; }
try
{
Price = dsView.Tables["Items"].Rows[i]["Price"].ToString().Trim();
}
catch { Price = ""; }
SQL = "INSERT INTO Catalog(Makat,Barcode,Des,Price)VALUES('" + Makat + "','" + Barcode + "','" + Des + "','" + Price + "')";
cmd_sql.CommandText = SQL;
cmd_sql.CommandType = CommandType.Text;
cmd_sql.ExecuteNonQuery();
//cmd_sql.Dispose();
}
How to insert faster ?
SQLite implicitly wraps queries within a transaction. Beginning and committing transactions in a loop could be slowing things down. I think you should get a significant boost of speed if you start a transaction and commit it once the loop is completed:
conn_sql.Open();
using(var tran = conn_sql.BeginTransaction()) // <--- create a transaction
{
cmd_sql = conn_sql.CreateCommand();
cmd_sql.Transaction = tran; // <--- assign the transaction to the command
for (int i = 0; i < iTotalRows; i++)
{
// ...
cmd_sql.CommandText = SQL;
cmd_sql.CommandType = CommandType.Text;
cmd_sql.ExecuteNonQuery();
//cmd_sql.Dispose();
}
tran.Commit(); // <--- commit the transaction
} // <--- transaction will rollback if not committed already
If you do it in a single transaction it should be faster:
SqlTransaction transaction;
try
{
conn_sql = new SQLiteConnection(conn_str2);
conn_sql.Open();
cmd_sql = conn_sql.CreateCommand();
transaction = conn_sql.BeginTransaction();
for (int i = 0; i < iTotalRows; i++)
{
// create SQL string
cmd_sql.CommandText = SQL;
cmd_sql.CommandType = CommandType.Text;
cmd_sql.ExecuteNonQuery();
}
transaction.Commit();
}
catch
{
transaction.Rollback();
}
At first, try to use StringBuilder for the string concatenation. Secondly, you are making 34k requests and this is slow, try to make less amount of requests. Let say using StringBuilder concatenate 5k insert statements in one and fire it in transaction, do this until all your data will not be saved.
Related
I'm new here but I need some help. I need to update a SQL Server database from C# with Windows Forms, but I'm having problems. I looked it up but still can't find the right answer. I need to do insert and update by pressing a button for changing or filling the database from the datagridview. I've created a separate function for both I am using this code;
private void InsertPositionen()
{
string qry = "";
SqlCommand insert = new SqlCommand(qry, con);
try
{
for (int i = 0; i < dataGridView1.Rows.Count - 1; i++)
{
qry = "INSERT INTO BelegePositionen (BelID, BelPosId, Artikelnummer, Menge, Preis) VALUES( " + dataGridView1.Rows[i].Cells["BelID"] + ", "
+ dataGridView1.Rows[i].Cells["BelPosId"] + ", "
+ dataGridView1.Rows[i].Cells["Artikelnummer"] + ", "
+ dataGridView1.Rows[i].Cells["Menge"] + ", "
+ dataGridView1.Rows[i].Cells["Preis"];
}
insert.ExecuteNonQuery();
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
private void UpdatePositionen()
{
string updt = "";
SqlCommand update = new SqlCommand(updt, con);
try
{
for (int i = 0; i < dataGridView1.Rows.Count -1; i++)
{
updt = "UPDATE BelegePositionen SET BelID = "
+ dataGridView1.Rows[i].Cells["BelID"] +
", BelPosID = "
+ dataGridView1.Rows[i].Cells["BelPosID"] +
", Atrikelnummer = "
+ dataGridView1.Rows[i].Cells["Artikelnummer"] +
", Menge = "
+ dataGridView1.Rows[i].Cells["Menge"] +
", Preis = "
+ dataGridView1.Rows[i].Cells["Preis"];
}
update.ExecuteNonQuery();
con.Close();
MessageBox.Show("Done!");
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
You should really NOT do your SQL stuff like this!! This leaves your code wide open for SQL injection vulnerabilities! Stop that - right now!
Instead - use parametrized queries - like this:
private void InsertPositionen()
{
string qry = "INSERT INTO BelegePositionen (BelID, BelPosId, Artikelnummer, Menge, Preis) " +
"VALUES(#BelId, #BelPosId, #ArtNr, #Menge, #Preis);";
SqlCommand insert = new SqlCommand(qry, con);
// define the parameters
insert.Parameters.Add("#BelId", SqlDbType.Int);
insert.Parameters.Add("#BelPosId", SqlDbType.Int);
insert.Parameters.Add("#ArtNr", SqlDbType.Int); // maybe this is a string?
insert.Parameters.Add("#Menge", SqlDbType.Int);
insert.Parameters.Add("#Preis", SqlDbType.Decimal, 20, 4);
try
{
// in the loop, only *set* the parameter's values
for (int i = 0; i < dataGridView1.Rows.Count - 1; i++)
{
insert.Parameters["#BelId"].Value = 1;
insert.Parameters["#BelPosId"].Value = 2;
insert.Parameters["#ArtNr"].Value = 3;
insert.Parameters["#Menge"].Value = 4;
insert.Parameters["#Preis"].Value = 99.95;
insert.ExecuteNonQuery();
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
Your Question is quite vague as you state you are having problems, but not quite sure what problems you are having. It will help if you can describe what problems you are having.
In addition to what #marc_c said about sql injection, I can't see how you manage your connection to the database.
From the code it looks like you could run into a situation where you are leaving connection strings open, or not opening them at all.
using the using(...) { } will close the connections when you are done with it.
private void InsertPositionen()
{
//using the using statement you will insure that the connection is closed and resources released
using (SqlConnection connection = new SqlConnection(Properties.Settings.Default.db))
{
string cmd = "INSERT INTO BelegePositionen (BelID, BelPosId, Artikelnummer, Menge, Preis) " +
"VALUES(#BelId, #BelPosId, #ArtNr, #Menge, #Preis);";
//using the using statement will ensure any reasources are released when exiting the code block
using (SqlCommand insert = new SqlCommand(cmd, connection))
{
// define the parameters
insert.Parameters.Add("#BelId", SqlDbType.Int);
insert.Parameters.Add("#BelPosId", SqlDbType.Int);
insert.Parameters.Add("#ArtNr", SqlDbType.Int); // maybe this is a string?
insert.Parameters.Add("#Menge", SqlDbType.Int);
insert.Parameters.Add("#Preis", SqlDbType.Decimal, 20, "4");
try
{
//open the connection
insert.Connection.Open();
// in the loop, only *set* the parameter's values
for (int i = 0; i < dataGridView1.Rows.Count - 1; i++)
{
insert.Parameters["#BelId"].Value = dataGridView1.Rows[i].Cells["BelID"];
insert.Parameters["#BelPosId"].Value = dataGridView1.Rows[i].Cells["BelPosId"];
insert.Parameters["#ArtNr"].Value = dataGridView1.Rows[i].Cells["Artikelnummer"];
insert.Parameters["#Menge"].Value = dataGridView1.Rows[i].Cells["Menge"];
insert.Parameters["#Preis"].Value = dataGridView1.Rows[i].Cells["Preis"];
insert.ExecuteNonQuery();
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
finally
{
MessageBox.Show("Done!");
}
}
}
}
How to implement counter, so that it commit multiple queries, for example every 1000 "queries". Problem is that with or without transaction query is executed by "ExecuteNonQuery()" and it execute one by one, not as I whant e.g. every 1000?
odbc.dbsqlite.Open();
odbc.dbkopito.Open();
OdbcCommand comlite = odbc.dbsqlite.CreateCommand();
OdbcCommand comkopit = odbc.dbkopito.CreateCommand();
OdbcTransaction transaction = null;
comkopit.CommandText =
"SELECT DISTINCT ... "
#region TRY
try
{
OdbcDataReader dbKopitReader = comkopit.ExecuteReader();
var ordinal = new
{
cenik = dbKopitReader.GetOrdinal("sifra"),
ident = dbKopitReader.GetOrdinal("ident"),
klasifikacija = dbKopitReader.GetOrdinal("klasifikacija"),
cena = dbKopitReader.GetOrdinal("cena"),
eankoda = dbKopitReader.GetOrdinal("eankoda"),
};
int stevec = 0;
while (dbKopitReader.Read())
{
var cena = Convert.ToDouble(dbKopitReader.GetDouble(ordinal.cena));
var ident = Convert.ToString(dbKopitReader.GetString(ordinal.ident));
var cenik = Convert.ToString(dbKopitReader.GetString(ordinal.cenik));
var klasi = Convert.ToString(dbKopitReader.GetString(ordinal.klasifikacija));
var eanko = Convert.ToString(dbKopitReader.GetString(ordinal.eankoda));
using (var cmd = odbc.dbsqlite.CreateCommand() )
{
try
{
cmd.CommandText = "INSERT OR REPLACE INTO ARTIKLI (KLASIFIKACIJA, CENA, BARKODA, CENIK, IDENT) " +
"VALUES (?,?,?,?,?);";
cmd.Parameters.AddWithValue("#KLASIFIKACIJA", klasi);
cmd.Parameters.AddWithValue("#CENA", cena);
cmd.Parameters.AddWithValue("#BARKODA", eanko);
cmd.Parameters.AddWithValue("#CENIK", cenik);
cmd.Parameters.AddWithValue("#IDENT", ident);
cmd.ExecuteNonQuery();
if (stevec % 1000 == 0)
{
transaction.Commit();
transaction = odbc.dbsqlite.BeginTransaction();
cmd.Transaction = transaction;
}
stevec++;
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
try
{
transaction.Rollback();
}
catch
{
Console.WriteLine("Transakcija ni aktivna");
}
}
}
}
comlite.Dispose();
odbc.dbsqlite.Close();
dbKopitReader.Close();
comkopit.Dispose();
odbc.dbkopito.Close();
Transaction is not initiated at first iteration (should get null exception).
Transaction is not assigned to command.
You creating transaction and assigning it to command every 1000th, but next 999 commands is created without transactions.
You committing transaction on very first iteration.
Here is example how your code should look like:
var transaction = odbc.dbsqlite.BeginTransaction();
int count = 0;
while(read)
{
count++;
if(count % 1000 == 0)
{
transaction.Commit();
transaction = odbc.dbsqlite.BeginTransaction();
}
//I imagine this CreateCommand() is assigning connection...
using(var cmd = odbc.dbsqlite.CreateCommand())
{
cmd.CommandText = "INSERT...";
//Params...
cmd.Transaction = transaction;
cmd.ExecuteNonQuery();
}
}
The function of application is to select some values from one database (PSQL) and insert it into another database (SQLite). But code below does not work, it stops at executing line and shows no error, but last forever (also if I use SELECT TOP 1 ...).
//... odbc conection to DSN, this works fine
odbc.dbsqlite.Open();
odbc.dbpsql.Open();
//sql command
OdbcCommand comsqlite = odbc.dbsqlite.CreateCommand();
OdbcCommand compsql = odbc.dbpsql.CreateCommand();
//SQL for select ... this works
compsql.CommandText = "SELECT DISTINCT ..."
compsql.Parameters.AddWithValue("#sifra", "VP");
...
// from here is problem
try
{
OdbcDataReader dbReader = compsql.ExecuteReader();
OdbcTransaction transaction = odbc.dbsqlite.BeginTransaction();
var ordinal = new
{
cenik = dbReader.GetOrdinal("sifra"),
ident = dbReader.GetOrdinal("ident"),
klasi = dbReader.GetOrdinal("klasi"),
cena = dbReader.GetOrdinal("cena"),
eankoda = dbReader.GetOrdinal("eankoda"),
};
int count = 0;
while (dbReader.Read())
{
//here single variable gets results
var cena = Convert.ToDouble(dbReader.GetDouble(ordinal.cena));
var ident = Convert.ToString(dbReader.GetString(ordinal.ident));
var cenik = Convert.ToString(dbReader.GetString(ordinal.cenik));
var klasi = Convert.ToString(dbReader.GetString(ordinal.klasi));
var eanko = Convert.ToString(dbReader.GetString(ordinal.eankoda));
comsqlite.CommandText = "INSERT INTO ARTIKLI (KLASI, CENA, BARKODA, CENIK, IDENT) VALUES (?,?,?,?,?);";
comsqlite.Parameters.AddWithValue("#KLASI", klasi);
comsqlite.Parameters.AddWithValue("#CENA", cena);
comsqlite.Parameters.AddWithValue("#BARKODA", eanko);
comsqlite.Parameters.AddWithValue("#CENIK", cenik);
comsqlite.Parameters.AddWithValue("#IDENT", ident);
if (count % 1000 == 0)
{
transaction.Commit();
transaction.Dispose();
**comsqlite.ExecuteNonQuery(); //here it stops and give no results**
transaction = odbc.dbsqlite.BeginTransaction();
}
count++;
}
comsqlite.Dispose();
odbc.dbsqlite.Close();
transaction.Commit();
transaction.Dispose();
dbReader.Close();
compsql.Dispose();
odbc.dbpsql.Close();
}
catch (Exception e)
{
Console.WriteLine("Error: "+ e);
throw;
}
I am not sure what your CommandText looks like at this point, but you should try to set some single quotation marks around the values being strings/characters in your database.
comsqlite.CommandText = "INSERT INTO ARTIKLI (KLASI, CENA, BARKODA, CENIK, IDENT) VALUES ('?','?','?','?','?');";
I am making use of SQL Server 2012 and have a huge file of approx 20 GB size. I want to insert every record inside file into database. I am using SqlBulkCopy class for this purpose. But since, the size of data is very huge I will have to insert it part by part. Here is the code:
String line;
SqlConnection conn = new SqlConnection(ConfigurationManager.ConnectionStrings["conStrtingName"].ConnectionString);
conn.Open();
StreamReader readFile = new StreamReader(filePath);
SqlTransaction transaction = conn.BeginTransaction();
try
{
SqlBulkCopy copy = new SqlBulkCopy(conn, SqlBulkCopyOptions.KeepIdentity, transaction);
copy.BulkCopyTimeout = 600;
copy.DestinationTableName = "Txn";
int counter = 0;
while ((line = readFile.ReadLine()) != null)
{
string[] fields = line.Split('\t');
if (fields.Length == 3)
{
DateTime date = Convert.ToDateTime(fields[0]);
decimal txnCount = Convert.ToDecimal(fields[1]);
string merchantName = fields[2];
if (!string.IsNullOrEmpty(merchantName))
{
long MerchantId = Array.IndexOf(Program.merchantArray, merchantName) + 1;
tables[workerId].Rows.Add(MerchantId, date, txnCount);
counter++;
if (counter % 100000 == 0)
Console.WriteLine("Worker: " + workerId + " - Transaction Records Read: " + counter);
if (counter % 1000000 == 0)
{
copy.WriteToServer(tables[workerId]);
transaction.Commit();
tables[workerId].Rows.Clear();
//transaction = conn.BeginTransaction();
Console.WriteLine("Worker: " + workerId + " - Transaction Records Inserted: " + counter);
}
}
}
}
Console.WriteLine("Total Transaction Records Read: " + counter);
if (tables[workerId].Rows.Count > 0)
{
copy.WriteToServer(tables[workerId]);
transaction.Commit();
tables[workerId].Rows.Clear();
Console.WriteLine("Worker: " + workerId + " - Transaction Records Inserted: " + counter);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
transaction.Rollback();
}
finally
{
conn.Close();
}
It works for first 100000 records. However for the next set of records I get an exception The transaction is either not associated with the current connection or has been completed.
This happens when the control reaches to the transaction.Commit(); for the next set of records.
Can I have a workaround?
The problem is the commented line after the transaction is commit. You need to uncomment it, and also reinitialize your SqlBulkCopy copy variable. You'd better refactor your code, the only places where you need transaction and copy object is when you flush the data table that you are filling, like this (you can further factor out the repetitive part into a separate method):
String line;
SqlConnection conn = new SqlConnection(ConfigurationManager.ConnectionStrings["conStrtingName"].ConnectionString);
conn.Open();
StreamReader readFile = new StreamReader(filePath);
SqlTransaction transaction = null;
try
{
int counter = 0;
while ((line = readFile.ReadLine()) != null)
{
string[] fields = line.Split('\t');
if (fields.Length == 3)
{
DateTime date = Convert.ToDateTime(fields[0]);
decimal txnCount = Convert.ToDecimal(fields[1]);
string merchantName = fields[2];
if (!string.IsNullOrEmpty(merchantName))
{
long MerchantId = Array.IndexOf(Program.merchantArray, merchantName) + 1;
tables[workerId].Rows.Add(MerchantId, date, txnCount);
counter++;
if (counter % 100000 == 0)
Console.WriteLine("Worker: " + workerId + " - Transaction Records Read: " + counter);
if (counter % 1000000 == 0)
{
transaction = conn.BeginTransaction()
SqlBulkCopy copy = new SqlBulkCopy(conn, SqlBulkCopyOptions.KeepIdentity, transaction);
copy.BulkCopyTimeout = 600;
copy.DestinationTableName = "Txn";
copy.WriteToServer(tables[workerId]);
transaction.Commit();
transaction = null;
tables[workerId].Rows.Clear();
Console.WriteLine("Worker: " + workerId + " - Transaction Records Inserted: " + counter);
}
}
}
}
Console.WriteLine("Total Transaction Records Read: " + counter);
if (tables[workerId].Rows.Count > 0)
{
transaction = conn.BeginTransaction()
SqlBulkCopy copy = new SqlBulkCopy(conn, SqlBulkCopyOptions.KeepIdentity, transaction);
copy.BulkCopyTimeout = 600;
copy.DestinationTableName = "Txn";
copy.WriteToServer(tables[workerId]);
transaction.Commit();
transaction = null;
tables[workerId].Rows.Clear();
Console.WriteLine("Worker: " + workerId + " - Transaction Records Inserted: " + counter);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
if (transaction != null) transaction.Rollback();
}
finally
{
conn.Close();
}
The problem thought is that now you cannot rollback ALL the changes in case something goes wrong. Probably the better solution would be to not manually splitting your bulk inserts, but use some sort of a IDataReader implementation to avoid populating a huge DataTable in memory (for instance using Marc Gravell's ObjectReader).
Your transaction is committed every 100000 sets. So it is "gone", you have to start another one then with transaction = conn.BeginTransaction.
Maybe good to rework the code to better reflect the lifespan of the transaction then. You also might to make sure that "copy" is recreated with the new transaction.
You can increase the timeout for your transaction like this (use values appropriate for the expected length of your transaction). The code below is for 15 minutes: Source
using (TransactionScope scope =
new TransactionScope(TransactionScopeOption.Required,
new System.TimeSpan(0, 15, 0)))
{
// working code here
}
I have a datagridview which is created by various action and user's manipulation of data. I want to insert all the data of the gridview to the database at once, I know I could try a code similar to this:
private void btnSaveProducts_Click(object sender, EventArgs e)
{
SqlConnection connection = DBConnectivity.getConnection();
if (connection != null)
{
try
{
for (int i = 0; i < dGvProducts.Rows.Count; i++)
{
string query = "INSERT INTO product (productName) " + "VALUES (#productName)";
SqlCommand command = DBConnectivity.getCommandForQuery(query, connection);
int result = command.ExecuteNonQuery();
Console.WriteLine(result + "");
}
// string query = "Insert into units(name,packing)values('" + txtNameUnit.Text + "' , '" + txtPackingUnit.Text + "')";
// SqlCommand command = DBConnectivity.getCommandForQuery(query, connection);
// int result = command.ExecuteNonQuery();
// Console.WriteLine(result + "");
}
catch (Exception ex)
{
}
finally
{
connection.Close();
}
}
}
As is, the code tries to execute a parameterized query but never assigns a value to the parameter. Even if you do, you never extract the cell values.
The code should look like this:
var query = "INSERT INTO product (productName) VALUES (#productName)";
using var(connection = DBConnectivity.getConnection())
using(var command = new SqlCommand(query, connection))
{
var productParam=command.Parameters.Add("#productName",SqlDbType.NVarChar,50);
connection.Open();
for (int i = 0; i < dGvProducts.Rows.Count; i++)
{
var productName=dGvProducts.Rows[i].Cells[somecolumn].Value;
productParam.Value=productName;
int result = command.ExecuteNonQuery();
Console.WriteLine(result);
}
}