Insert data to Mysql getting slower and slower - c#

I need to insert 388 datas per minute to local Database.
At first when the table is Empty, I only need 5 second to Insert to database.
But when the table gets larger, the program efficacy slow down to more than one minute when the amount of rows comes to 1,026,558.
And the useage of CPU is 100%. It's unusual.
here is my code:
public static void dataToDB(String[] routeIDArray,String[] levelArray,String[] valueArray,String[] travelTimeArray, int amountOfData)
{
MySqlConnection con = new MySqlConnection(connStr);
MySqlCommand cmd = null;
MySqlDataReader rdr = null;
String sqlCmd, updateSqlCmd = "UPDATE `datetimetable` SET ";
for(int counter = 0; counter < amountOfData; counter++)
{
sqlCmd = "ALTER TABLE `datetimetable` ADD COLUMN IF NOT EXISTS `" + routeIDArray[counter] + "` INT NULL;"
+ "INSERT INTO `roadvalue`.`data` (`level`,`value`,`traveltime`) VALUES ("
+ levelArray[counter] + ","
+ valueArray[counter] + ","
+ travelTimeArray[counter] + ");"
+ "SELECT LAST_INSERT_ID() FROM `data`;";
cmd = new MySqlCommand(sqlCmd, con);
con.Open();
rdr = cmd.ExecuteReader();
rdr.Read();
updateSqlCmd += "`" + routeIDArray[counter] + "` = " + rdr[0] + ",";
rdr.Close();
}
updateSqlCmd = updateSqlCmd.TrimEnd(',');
updateSqlCmd += " WHERE EXISTS (SELECT * WHERE dateTime = '" + dateTime.ToString("yyyy-MM-dd HH:mm:00") + "');";
cmd = new MySqlCommand(updateSqlCmd, con);//update data key to datetimetable
cmd.ExecuteNonQuery();
Console.WriteLine("Done.");
con.Close();
}
public static void checkDateTimeExisted()
{
MySqlConnection con = new MySqlConnection(connStr);
MySqlCommand cmd;
String sqlCmd;
sqlCmd = "INSERT INTO `datetimetable` (`dateTime`) SELECT * FROM (SELECT '" + dateTime.ToString("yyyy-MM-dd HH:mm:00")
+ "') AS tmp WHERE NOT EXISTS(SELECT `dateTime` FROM `datetimetable` WHERE `dateTime` = '" + dateTime.ToString("yyyy-MM-dd HH:mm:00") + "') LIMIT 1; ";
con.Open();
cmd = new MySqlCommand(sqlCmd, con);
cmd.ExecuteNonQuery();
con.Close();
}
And Mysql Engine is InooDB, table "data" has one Auto_Increment Primary key, table "datetimetable" has an Auto_Increment Primary key and a not duplicate datetime as index.
What have I done wrong?

I find the answer, the command "SELECT LAST_INSERT_ID() FROM data;" should add LIMIT 1 or it will get all the ID kill the performance.

Do not use ALTER TABLE in a loop -- Plan ahead and provide all the columns before starting.
Do not use multiple statements in a single string. This has security implications, etc.
Do not use WHERE EXISTS... when (I think) a simple WHERE would work.
If there is UNIQUE(datetime), then the final INSERT can be simply
INSERT IGNORE INTO datetimetable
(datetime)
VALUE
('...');
Do batch inserts unless you need the LAST_INSERT_ID(). LIMIT 1 should not be necessary.
Do not 'Normalize' datetime values; it only slows things down. Just put the datetime as is in the main table.

Related

Get RecordId from table

I am new to SQL, I have table with RecordId that is incremented automatically and is primary key. I would like to get RecordId of the row that was inserted into table.
Thanks in advance for help.
myCommand.CommandText = "INSERT INTO " + tableName + " (DateRaised,RaisedBy,WeekNo,Platform,Department,Site,Process, Area,NavErrorNo,RootCauseDescription,Status) " +
"VALUES ('" + currentDate.ToString(format) + "','" +
sender + "'," +
weekNumber + ",'" +
comboBoxPlatform.SelectedItem + "','" +
comboBoxDepartment.SelectedItem + "','" +
comboBoxSite.SelectedItem + "','" +
comboBoxProcess.SelectedItem + "','" +
comboBoxArea.SelectedItem + "','" +
textBoxNavError.Text + "','" +
textBoxIssue.Text + "','Open')";
//int lastInsertedId =
myCommand.ExecuteNonQuery();
lastInsertedId should be int from RecordId in my table.
To do this properly (if this is for SQL Server - you weren't very clear on this), I see two options:
Approach #1 - using SCOPE_IDENTITY
This works well if you're only ever inserting a single row at a time - use something like this:
// set up your query using *PARAMETERS** as you **ALWAYS** should!
// Using SELECT SCOPE_IDENTITY() to get back the newly inserted "Id"
myCommand.CommandText = "INSERT INTO dbo.SomeTable (list-of-columns) " +
"VALUES (#param1, #param2, #param3, ...., #paramN); " +
"SELECT SCOPE_IDENTITY();";
// set up the parameters and theirs values
object result = myCommand.ExecuteScalar();
if (result != null)
{
int lastInsertedId = Convert.ToInt32(result);
}
Approach #2 - using the OUTPUT clause
This works well even if you insert multiple rows at once (typically using a SELECT after the INSERT):
// set up your query using *PARAMETERS** as you **ALWAYS** should!
// Using SELECT SCOPE_IDENTITY() to get back the newly inserted "Id"
myCommand.CommandText = "INSERT INTO dbo.SomeTable (list-of-columns) " +
"OUTPUT Inserted.RecordId " +
"VALUES (#param1, #param2, #param3, ...., #paramN); ";
// set up the parameters and theirs values
object result = myCommand.ExecuteScalar();
if (result != null)
{
int lastInsertedId = Convert.ToInt32(result);
}
First thing this is not a good idea to call direct SQL statement from code it can cause an issue for SQL injection as #Zohar suggested.
You can either user parametrized query or sp.
Inside sp, you can use
SELECT ##IDENTITY AS 'Identity';
after Insert statement, it will return the last auto-incremented value for PK,
then return this value as an output parameter and catch it after .ExecuteNonQuery(); in C# code.
This should do the trick for You
private void SelectLast()
{
string sqlLast = "SELECT TOP(1) RecordId FROM [YourtableName] ORDER BY 1 DESC";
Connection.Open();
using (SqlCommand cmd = new SqlCommand(sqlLast, Connection))
{
cmd.CommandType = CommandType.Text;
{
int insertedID = Convert.ToInt32(cmdAdd.ExecuteScalar());
textBoxID.Text = Convert.ToString(insertedID);
}
Connection.Close();
}
}

How to update selected row values in datagridview?

I've been trying to update the values from a sql database that's set as a data source in my windows form datagridview, but end up updating all rows instead.
I've been working on creating a basic task manager app for a development course that I'm enrolled in.
I'm having a hard time figuring out where the problem is located. I think that my code may not be properly set to the selected row?
I've supplied the code below, any and all help would be appreciated. If anyone need further clarification shoot me a message on the chat. throws a thumbs up
My Current Code:
private void UpdateBtn_Click(object sender, EventArgs e)
{
//Update button [almost done - data is not updating correctly]
string connectionString = "Data Source =ULTRA-COMPUTER; Initial Catalog =test; Persist Security Info =True; User ID = sa; Password = 12345";
SqlConnection con = new SqlConnection(connectionString);
string queryStatement = "SELECT * FROM testtask";
if (Task.Text != "" && Date.Text != "")
{
SqlCommand cmd = new SqlCommand(queryStatement, con);
DataTable task = new DataTable("testtask");
SqlDataAdapter ada = new SqlDataAdapter(cmd);
con.Open();
cmd.CommandText = "UPDATE [testtask] SET Task='" + Task.Text + "',Date='" + Date.Text + "' ";
cmd.Connection = con;
cmd.Parameters.AddWithValue("#Task", Task.Text);
cmd.Parameters.AddWithValue("#Date", Date.Text);
cmd.ExecuteNonQuery();
TaskData.DataSource = task;
MessageBox.Show("Update Inserted!");
ClearTxt();
}
else
{
MessageBox.Show("Please Enter A Task/DueDate To Update");
}
con.Close();
}
First add a hidden column for the primary key of your database table in datagridview. And now when you want to update the selected row that you have edited it use that primary key in where condition of your query.
cmd.CommandText = "UPDATE [testtask] SET Task='" + Task.Text + "',Date='" + Date.Text + "WHERE [TaskId]=#TaskId";
cmd.Parameters.AddWithValue("#TaskId", TaskIdFromDatagridview);
I think the problem is in this line
cmd.CommandText = "UPDATE [testtask] SET Task='" + Task.Text + "',Date='" + Date.Text + "' ";
You've not added the where clause that's why it'll update all rows in the table.
You must add a where clause.
For example you've taskid as primary key in this table and want to update a task with taskid 999 then your query must be
cmd.CommandText = "UPDATE [testtask] SET Task='" + Task.Text + "',Date='" + Date.Text + "' where taskid = 999 ";

Using ExecuteReader to return a primary key

How Do I Find the ID from the first query and return this value so it can be inserted into query2? This is the code that needs done when a user completes a form on front end. I need to populate two tables and they will relate through the ID "StoryID" which is a primary key that is automatically created.
protected void Upload2_Click(object sender, EventArgs e)
{
userStoryForm.Visible = false;
info.Text = "You have successfully added a new user story.";
String connectionString = WebConfigurationManager.ConnectionStrings["ConnectionString"].ConnectionString;
String usernameData = username.Text.ToString();
int captureProjectID = Convert.ToInt32(Request.QueryString.Get("ProjectID"));
String storyno = StoryNoTextBox.Text;
String userstory = StoryTextTextBox.Text;
//Create connection
SqlConnection myConnection = new SqlConnection(connectionString);
//open connection
myConnection.Open();
String query = "INSERT INTO UserStories (StoryNo, StoryText, ProductOwner, ProjectID) " +
"VALUES ('" + storyno + "','" + userstory + "','" + usernameData + "','" + captureProjectID + "')" +
"SELECT SCOPE_IDENTITY() AS StoryID;";
SqlCommand myCommand = new SqlCommand(query, myConnection);
// Call GetOrdinal and assign value to variable.
SqlDataReader reader = myCommand.ExecuteReader();
int StoryIDData = reader.GetOrdinal("StoryID");
// Use variable with GetString inside of loop.
while (reader.Read())
{
Console.WriteLine("StoryID={0}", reader.GetString(StoryIDData));
}
// Call Close when done reading.
reader.Close();
//insert productowner, projectID and storyID into ProductBacklog table
String query2 = "INSERT INTO ProductBacklog (ProductOwner, ProjectID, StoryID) VALUES ('" + usernameData + "', #returnProjectID,'" + StoryIDData + "')";
SqlCommand myCommand2 = new SqlCommand(query2, myConnection);
myCommand2.Parameters.AddWithValue("#returnProjectID", captureProjectID);
//close connection
myConnection.Close();
}
}
Most important - use parameters in your SQL command. Never concatenate strings like that. You're asking for an SQL injection attack.
string query = #"
INSERT INTO UserStories (StoryNo, StoryText, ProductOwner, ProjectID)
VALUES (#storyno, #userstory, #usernameData, #captureProjectID)
SELECT CAST(SCOPE_IDENTITY() AS INT)";
SqlCommand myCommand = new SqlCommand(query);
myCommand.Parameters.Add("#storyno", DbType.String).Value = storyno;
...
To get the returned id, use ExecuteScalar():
int StoryIDData = (int)myCommand.ExecuteScalar();
Also, you don't dispose your resources correctly. If an exception is thrown in the method, the SQLConnection will not be closed. You should put it in a using statement.

Parameterize a SELECT ROWCOUNT sql statement

I am using VS2005 C# and SQL Server 2005.
I have a a few SQL queries which I am converting them from using parameters instead concatenations for SQL injection prevention.
Below is a SELECT query which is parameter-ed:
string loggedinuser = (User.Identity.Name);
SqlDataSource1.SelectCommand = "SELECT * FROM [UserTable] where [" + DropDownList1.Text + "] like #searchtb AND [LoggedInUser] LIKE #userlog";
SqlDataSource1.SelectParameters.Add("searchtb", "%" + searchTB.Text + "%");
SqlDataSource1.SelectParameters.Add("userlog", "%" + loggedinuser+ "%");
The above sql query searches for records base on the user's input in a search textbox and return results which matches the search input and username in the database.
I have another SQL query which is also a SELECT statement. However, this time it does not use SqlDataSource, but using cmd instead. Thus I need some help in converting the SQL statement below to parameter form:
string loggedinuser = (User.Identity.Name);
string stmt = "SET ROWCOUNT 1 SELECT COUNT(*) FROM MP.dbo.UserTable where [" + DropDownList1.Text + "] like '%" + searchTB.Text + "%' AND [LoggedInUser] LIKE '%"+loggedinuser +"%'";
int count = 0;
using (SqlCommand cmdCount = new SqlCommand(stmt, thisConnection))
{
thisConnection.Open();
count = (int)cmdCount.ExecuteScalar();
thisConnection.Close();
}
This SQL query searches for number of records that the user is trying to search base on his search input and username. And if countuser returns a 0 value, I will prompt the user after that.
I need help in converting the 2nd SQL statement into parameter form.
Thank you.
Try,
string stmt = "SELECT COUNT(*) FROM MP.dbo.UserTable where [" + DropDownList1.Text + "]
like #searchTb AND [LoggedInUser] LIKE #loggedinuser";
int count = 0;
using (SqlCommand cmdCount = new SqlCommand(stmt, thisConnection))
{
cmdCount.Parameters.Add("#searchTb",SqlDbType.VarChar,40).Value="%" + searchTB.Text + "%";
cmdCount.Parameters.Add("#loggedinuser",SqlDbType.VarChar,40).Value="%" + loggedinuser + "%";
thisConnection.Open();
count = (int)cmdCount.ExecuteScalar();
thisConnection.Close();
}
Using stored procedures is your best bet, but if you cannot use them, this code should work:
SqlDataAdapter myCommand = new SqlDataAdapter(
"SELECT au_lname, au_fname FROM Authors WHERE au_id = #au_id", conn);
SQLParameter parm = myCommand.SelectCommand.Parameters.Add("#au_id",
SqlDbType.VarChar, 11);
Parm.Value = Login.Text;
This is from the MSDN article on SQL injection.
http://msdn.microsoft.com/en-us/library/ms161953.aspx

how to improve this code

// conn is read from handydrive
//conn2 read from C:\
this code is for write new record in to DB in C:\ by check exist first.
my problem is too slow for alot of records. and how to improve it to be faster...
SqlCeCommand cmd1 = new SqlCeCommand("Select * from bill_discount ", conn);
SqlCeDataReader dr1 = cmd1.ExecuteReader();
while (dr1.Read() != false)
{
SqlCeCommand cmd4 = new SqlCeCommand("Select * from bill_discount where bill_no='" + dr1.GetInt32(0) + "' AND bill_shopdc='" + dr1.GetString(2) + "' ", conn2);
SqlCeDataReader dr4 = cmd4.ExecuteReader();
if (dr4.Read() == false)
{
SqlCeCommand cmd2 = new SqlCeCommand("INSERT INTO bill_discount (bill_no,bill_value,bill_shopdc) VALUES ('" + dr1.GetInt32(0) + "','" + dr1.GetDouble(1) + "','" + dr1.GetString(2) + "') ", conn2);
// SqlCeDataReader dr2 = cmd2.ExecuteReader();
cmd2.ExecuteNonQuery();
}
}
//-------------------------------------------------------------------
I would take a look at the SqlBulkCopy Class:
Lets you efficiently bulk load a SQL
Server table with data from another
source.
BTW: In your code above, selecting the entire bill_discount table is not really a good idea, especially if the table is large.
[Also, it appears you could perform a single TSQL statement rather than looping through each row and round-tripping to the database.]
This example should be of help: SqlBulkCopy - Copy Table Data Between SQL Servers at High Speeds - ADO.NET 2.0 New Feature
Let's start by make the code more readable. Here's the result:
SqlCeCommand getAllBills = new SqlCeCommand("select * from bill_discount", primaryConnection);
SqlCeDataReader allBillsReader = getAllBills.ExecuteReader();
while (allBillsReader.Read())
{
SqlCeCommand getBill = new SqlCeCommand("select * from bill_discount where bill_no = '" + allBillsReader.GetInt32(0) + "' and bill_shopdc = '" + allBillsReader.GetString(2) + "' ", secondaryConnection);
SqlCeDataReader billReader = getBill.ExecuteReader();
if (!billReader.Read())
{
SqlCeCommand addMissingBill = new SqlCeCommand("insert into bill_discount (bill_no, bill_value, bill_shopdc) values ('" + allBillsReader.GetInt32(0) + "', '" + allBillsReader.GetDouble(1) + "', '" + allBillsReader.GetString(2) + "')", secondaryConnection);
addMissingBill.ExecuteNonQuery();
}
}
Disposable objects must be disposed. Let's do it.
Let's also remove SQL Injections.
Finally, let's optimize the second query: you don't need to select something and executing the reader if you just want to check if the value exists in the database.
using (SqlCeCommand getAllBills = new SqlCeCommand("select bill_no, bill_value, bill_shopdc from [bill_discount]", primaryConnection))
{
using (SqlCeDataReader allBillsReader = getAllBills.ExecuteReader())
{
while (allBillsReader.Read())
{
using (SqlCeCommand getBill = new SqlCeCommand("if exists(select * from [bill_discount] where [bill_no] = #billNumber and bill_shopdc = #billShop) select 1 else select 0", secondaryConnection))
{
getBill.Parameters.AddWithValue("#billNumber", allBillsReader["bill_no"]);
getBill.Parameters.AddWithValue("#billShop", allBillsReader["bill_shopdc"]);
bool billExists = Convert.ToBoolean(getBill.ExecuteScalar());
if (!billExists)
{
using (SqlCeCommand addMissingBill = new SqlCeCommand("insert into [bill_discount] ([bill_no], [bill_value], [bill_shopdc]) values (#billNumber, #billValue, #billShop)", secondaryConnection))
{
addMissingBill.Parameters.AddWithValue("#billNumber", allBillsReader["bill_no"]);
addMissingBill.Parameters.AddWithValue("#billValue", allBillsReader["bill_value"]);
addMissingBill.Parameters.AddWithValue("#billShop", allBillsReader["bill_shopdc"]);
int countAffectedRows = addMissingBill.ExecuteNonQuery();
Debug.Assert(countAffectedRows == 1, "The data was not inserted.");
}
}
}
}
}
}
So here we are.
Now, it's still a low performance solution. To be more effective, you might want to do the same thing in a single SQL query with joins. Since two tables are probably situated on different servers, you may look at linked servers: a feature that enables to execute a single query over several tables from several servers.
I see you are using SqlCe, which has number of limitations when inserting bulk data. The main limitation is the actual SqlCe Engine. You can however bypass this by using direct table inserts:
using (var command = connection.CreateCommand())
{
command.Transaction = transaction;
command.CommandType = CommandType.TableDirect;
command.CommandText = TABLE_NAME_IN_SQL;
using (var rs = command.ExecuteResultSet(ResultSetOptions.Updatable))
{
var rec = rs.CreateRecord();
rec.SetInt32(0, value0); // the index represents the column numbering
rec.SetString(1, value1);
rec.SetInt32(2, value2);
rs.Insert(rec);
}
}

Categories

Resources