Insert multiple object to MySQL - c#

Is there an other way to insert multiple objects to an MySQL database than the way shown here. This works but takes time to execute.
using (MySql.Data.MySqlClient.MySqlConnection conn = new MySql.Data.MySqlClient.MySqlConnection(connStr))
{
//Goes thrue the List<object>
foreach(List<object> sub in listSubject)
{
MySql.Data.MySqlClient.MySqlCommand cmd = new MySql.Data.MySqlClient.MySqlCommand();
cmd.Connection = conn;
cmd.CommandText = "CALL stp_InsertSubject(#param_SubjectId, #param_ProjectId, #param_Used);";
cmd.Parameters.AddWithValue("#param_SubjectId",Convert.ToInt32(sub[0]) );
cmd.Parameters.AddWithValue("#param_ProjectId", Convert.ToInt32(sub[1]));
cmd.Parameters.AddWithValue("#param_Used", Convert.ToBoolean(sub[2]) );
conn.Open();
cmd.ExecuteNonQuery();
conn.Close();
}
My Stored procedure:
CREATE DEFINER=`mortenstarck`#`%` PROCEDURE `stp_InsertSubject`(param_SubjectId int(45), param_ProjectId int(45), param_Used tinyint(1))
BEGIN
INSERT INTO Subject_has_Projects(Subject_Id, Projects_Id, Used) VALUES (param_SubjectId, param_ProjectId, param_Used);
END

Few things to improve:
Open the connection just one time outside the loop (no need to close
with using)
Create the command, assign connection just one time before the loop
Create the parameters all before the loop with dummy values
Assign only the value inside the loop and call the ExecuteScalar()
using(MySql.Data.MySqlClient.MySqlConnection conn = new MySql.Data.MySqlClient.MySqlConnection(connStr))
{
conn.Open();
MySql.Data.MySqlClient.MySqlCommand cmd = new MySql.Data.MySqlClient.MySqlCommand();
cmd.Connection = conn;
cmd.CommandText = "CALL stp_InsertSubject(#param_SubjectId, #param_ProjectId, #param_Used);";
cmd.Parameters.AddWithValue("#param_SubjectId",0 );
cmd.Parameters.AddWithValue("#param_ProjectId", 0);
cmd.Parameters.AddWithValue("#param_Used", false );
foreach(List<object> sub in listSubject)
{
cmd.Parameters["#param_SubjectId"].Value = Convert.ToInt32(sub[0]) ;
cmd.Parameters["#param_ProjectId"].Value = Convert.ToInt32(sub[1]);
cmd.Parameters["#param_Used"].Value = Convert.ToBoolean(sub[2]);
Id = (Int64)cmd.ExecuteScalar();
}
}

You can try. Open connection outside foreach loop. This will save time in opening and closing connection every time in loop. This will improve performance.
Int64 Id = 0;
using (MySql.Data.MySqlClient.MySqlConnection conn = new MySql.Data.MySqlClient.MySqlConnection(connStr))
{
//Goes through the List<object>
conn.Open();
foreach(List<object> sub in listSubject)
{
MySql.Data.MySqlClient.MySqlCommand cmd = new MySql.Data.MySqlClient.MySqlCommand();
cmd.Connection = conn;
cmd.CommandText = "CALL stp_InsertSubject(#param_SubjectId, #param_ProjectId, #param_Used);";
cmd.Parameters.AddWithValue("#param_SubjectId",Convert.ToInt32(sub[0]) );
cmd.Parameters.AddWithValue("#param_ProjectId", Convert.ToInt32(sub[1]));
cmd.Parameters.AddWithValue("#param_Used", Convert.ToBoolean(sub[2]) );
Id = (Int64)cmd.ExecuteScalar();
}
conn.Close();
}
How bad is opening and closing a SQL connection for several times? What is the exact effect?

Have you thought about surrounding these calls with a single transaction?

Related

C# OleDB Update statement not updating

I'm trying to update an Access database through my C# web application but I'm currently facing a problem in WHERE statement not updating the records and not returning any errors.
I'm trying update a text field and the condition in my WHERE statement is an integer.
OleDbCommand cmd = new OleDbCommand("UPDATE Data SET Title = #Title WHERE ID = #ID");
cmd.CommandType = CommandType.Text;
cmd.Connection = conn;
cmd.Parameters.AddWithValue("#Title", TextBox1.Text);
cmd.Parameters.AddWithValue("#ID", param2);
I even tried doing it this way
OleDbCommand cmd = new OleDbCommand("UPDATE Data SET Title = ? WHERE ID = ?");
cmd.CommandType = CommandType.Text;
cmd.Connection = conn;
cmd.Parameters.AddWithValue("?", TextBox1.Text);
cmd.Parameters.AddWithValue("?", param2);
But it's still not updating!
What I found out trying to fix it is that when i replace the first parameter with a string between single quotes (see below), it actually updates the table.
OleDbCommand cmd = new OleDbCommand("UPDATE Data SET Title = 'test' WHERE ID = #ID");
Does any of you guys have an idea why is this happening?
Edit: This is the full code
using (OleDbConnection conn = new OleDbConnection(connectionString))
{
OleDbCommand cmd = new OleDbCommand("UPDATE Data SET Title = #Title WHERE ID = #ID");
cmd.CommandType = CommandType.Text;
cmd.Connection = conn;
cmd.Parameters.AddWithValue("#Title", TextBox1.Text);
cmd.Parameters.AddWithValue("#ID", param2);
conn.Open();
try
{
cmd.ExecuteNonQuery();
}
catch (Exception ex)
{
Response.Write(ex.Message);
}
Edit 2:
This is my code after trying to set the data types, it's still not working.
To clarify, in my Access database ID is "AutoNumber" and Title is "Long Text"
using (OleDbConnection conn = new OleDbConnection(connectionString))
{
OleDbCommand cmd = new OleDbCommand("UPDATE Data SET Title = ? WHERE ID = ?");
cmd.CommandType = CommandType.Text;
cmd.Connection = conn;
cmd.Parameters.AddWithValue("Title", TextBox1.Text).OleDbType = OleDbType.VarChar;
cmd.Parameters.AddWithValue("ID", param2).OleDbType = OleDbType.Integer;
conn.Open();
try
{
var recordsUpdated = cmd.ExecuteNonQuery();
}
catch (Exception ex)
{
Response.Write(ex.Message);
}
When I check the value of recordsUpdated, it returns "1" but the database isn't being updated.
You need to call ExecuteNonquery which executes the statement.
// your OleDbConnection should also be wrapped in a using statement to ensure it is closed
using (OleDbConnection conn = new OleDbConnection(connectionString))
{
conn.Open(); // open the connection
using(OleDbCommand cmd = new OleDbCommand("UPDATE Data SET Title = ? WHERE ID = ?")) // wrap in using block because OleDbCommand implements IDisposable
{
cmd.CommandType = CommandType.Text;
cmd.Connection = conn;
cmd.Parameters.AddWithValue("Title", TextBox1.Text).OleDbType = OleDbType.VarChar; // Title, also set the parameter type
cmd.Parameters.AddWithValue("ID", param2).OleDbType = OleDbType.Integer; // ID, I am guessing its an integer but you should replace it with the correct OleDbType
var recordsUpdated = cmd.ExecuteNonQuery(); // execute the query
// recordsUpdated contains the number of records that were affected
}
}
The order op the Parameters that you add must match the order of the parameters in the sql statement.
Be sure you specify your OleDbType for each parameter so the command does not have to guess what it is. I guessed Title is a varchar and ID is integer, correct this if it is a wrong assumption.
You can replace "?" with parameter names to make the addition of parameters easier to read and keep the ? placeholders in the update statement. Microsoft does this in many of their examples in their documentation
ExecuteNonQuery returns the number of rows affected, capture the result and use it for debugging (see code update)
When I check the value of recordsUpdated, it returns "1" but the database isn't being updated.
If 1 is being returned then 1 record is being updated. Here are 2 trouble shooting tips.
You should make sure that your query is updating the same record you are checking. During debugging capture the value of param2 and manually use that in a select statement in Access to see the Title column of that record SELECT Title FROM Data WHERE ID = [id here]. Also get the value of TextBox1.Text in debugging and check this value against what is returned from the manually executed select query.
As this is Ms Access and the file resides on disk, make sure that you are manually checking against the same database that you are connecting to in your connection string. If you have 2 copies of the database this could easily lead to a wrong conclusion that an update is not being performed.

How to actually execute a command?

I'm playing around making a POC and I've created the following call.
public string DoStuff()
{
try
{
using (SqlDataAdapter adapter = new SqlDataAdapter())
{
SqlConnection connection = new SqlConnection("Server...");
string command = "insert into Records values (...)";
adapter.InsertCommand = new SqlCommand(command, connection);
}
}
catch (Exception exception)
{
return exception.Message + " " + exception.InnerException;
}
return "WeeHee!";
}
The text I'm seeing returned is the happy one, so I conclude there's no exceptions. Hence, I conclude that the call to the DB is performed as supposed to. However, there's no new lines in the DB being created.
I'm using the same connection string as I have in my config file and the command in pasted in from SQL Manager, where it works.
So my suspicion was that although I create an insert command, I never actually execute it but according to MSDN that's how it's supposed to work.
What stupid thing do I miss here?
You are missing connection.Open(); and adapter.InsertCommand.ExecuteNonQuery();
using (SqlDataAdapter adapter = new SqlDataAdapter())
{
SqlConnection connection = new SqlConnection("Server...");
connection.Open();
string command = "insert into Records values (...)";
adapter.InsertCommand = new SqlCommand(command, connection);
adapter.InsertCommand.ExecuteNonQuery();
}
You should use ExecuteNonQuery instead. Using an SqlDataAdapter for an INSERT query does not make sense.
Also you should Open your connection just before you execute it.
You can:
using(SqlConnection connection = new SqlConnection("Server..."))
{
SqlCommand command = connection.CreateCommand();
command.CommandText = "insert into Records values (...)";
connection.Open();
int craeted = command.ExecuteNonQuery();
}
The example you linked to returned a SQLAdapter for later use.
You don't need one at all:
using (SqlConnection connection = new SqlConnection("Server..."))
{
string command = "insert into Records values (...)";
connection.Open();
var command = new SqlCommand(command, connection);
command.ExecuteNonQuery();
}
Note that there are other execution methods, depending on expected return values and whether you want asynchronous operation: https://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlcommand(v=vs.110).aspx

C# - SQL - Reading every row in a database, datamining the row, then saving results in another database - How to increase speed

My problem: Code works great, but the speed is far too slow for the number of rows it needs to process.
What I am doing: I initiate a COUNT(*) to pull total number of rows (which last night was ~58000) and I use that to create the loop to do the following: pull two columns of data from that row, datamine one column for text patterns.
Once I had completed that, I search a second table to see if that individual by username is already existing - if they exist, I update their row. If not, I add a new one.
There are 44 columns of data, one being the name the other 43 storing values of my datamining results.
In about 8 hours, it has completed 26500 out of the 58000 when it first started (in that same period, the table has grown to ~100000, but I am not worried about that).
Is there a better method to increase read/write rate?
Portion of my code - I have removed many of the int declarations and Regex.Matches as they are copies of the first with altered Match values.
azCheck is to determine if the message even contains anything we are looking for, it is remains '0', then we don't bother with the last portion of the code.
using (new MySqlConnection(ConnectiongString))
{
using (MySqlCommand cmd = new MySqlCommand("select count(*) from messages", connection))
{
using (MySqlDataReader reader = cmd.ExecuteReader())
{
StringBuilder sb = new StringBuilder();
while (reader.Read())
{
sb.Append(reader.GetInt32(0).ToString());
}
total_messages = int.Parse(sb.ToString());
}
}
}
Console.WriteLine(total_messages.ToString());
connection.Close();
for (int i = 1; i <= total_messages; i++)
{
connection.Open();
using (new MySqlConnection(ConnectiongString))
{
using (MySqlCommand cmd = new MySqlCommand("select * from messages WHERE id="+i+"", connection))
{
using (MySqlDataReader reader = cmd.ExecuteReader())
{
StringBuilder sb = new StringBuilder();
while (reader.Read())
{
username = reader["username"].ToString();
message = reader["message"].ToString();
}
}
}
}
connection.Close();
Console.Write("\r{0} ", i);
int aiCount = 0;
aiCount += Regex.Matches(message, "ai", RegexOptions.IgnoreCase).Count;
azCheck += aiCount;
//There are ~42 of the regex.matches after the first one.
MySqlCommand cmd1 = connection.CreateCommand();
connection.Open();
cmd1.CommandText = "SELECT username FROM users";
cmd1.CommandType = CommandType.Text;
cmd1.Connection = connection;
MySqlDataReader dr = cmd1.ExecuteReader();
while (dr.Read())
{
if (dr[0].ToString() == username)
{
check++;
}
}
connection.Close();
if (check == 0)
{
MySqlConnection connection2 = new MySqlConnection(ConnectiongString);
connection2.Open();
try
{
MySqlCommand cmd2 = connection2.CreateCommand();
cmd2.CommandText = "INSERT INTO users (username,aiCount) VALUES (#username,#aiCount)";
cmd2.Parameters.AddWithValue("#username", username);
cmd2.Parameters.AddWithValue("#aiCount", aiCount);
cmd2.ExecuteNonQuery();
connection2.Close();
}
catch (Exception)
{
throw;
}
} else {
int aiCount_old = 0;
if (azCheck > 0)
{
//Here we are taking the existing values from this users row,
//which we then add the new values from above and save.
MySqlConnection connection4 = new MySqlConnection(ConnectiongString);
connection4.Open();
try
{
MySqlCommand cmd2 = connection4.CreateCommand();
cmd2.CommandType = CommandType.Text;
cmd2.CommandText = "SELECT * from users WHERE username = #username";
cmd2.Parameters.AddWithValue("#username", username);
MySqlDataReader reader = cmd2.ExecuteReader();
while (reader.Read())
{
aiCount_old = Convert.ToInt32(reader["aiCount"].ToString());
}
}
catch (Exception)
{
throw;
}
connection4.Close();
aiCount += aiCount_old;
MySqlConnection connection5 = new MySqlConnection(ConnectiongString);
connection5.Open();
try
{
MySqlCommand cmd4 = connection5.CreateCommand();
cmd4.CommandType = CommandType.Text;
cmd4.CommandText = "UPDATE users SET aiCount = #aiCount WHERE LOWER(LTRIM(RTRIM(username))) = #username";
cmd4.Parameters.AddWithValue("#username", username.Trim().ToLower());
cmd4.Parameters.AddWithValue("#aiCount", aiCount.ToString());
cmd4.ExecuteNonQuery();
Console.WriteLine("User updated.");
}
catch (Exception ex)
{
throw;
}
connection5.Close();
You have several inefficiencies that I can spot right off the bat.
You are constantly opening and closing your connection string. This is probably your biggest bottleneck. Open the connection once, then close it once when all your processing is done, and you'll probably see a massive increase in performance.
You also use different connection objects when one will do, which will reduce your need to open and close connections.
You also seem to have a misunderstanding of the use of "using" on connection objects. I see using (new MySqlConnection(ConnectiongString)), but that code is completely useless, as it doesn't do anything but initialize a connection object, which is immediately lost since it is not assigned to an object.
Since you are processing everything sequentially, use connection as your connection object in every case, opening it only at the start of your processing, and closing it when processing is complete, then executing the Dispose method (the point of the using statement).
This change alone might reduce processing time by an order of magnitude.
NOTE: You will need a separate connection for your datareader if you need to do updates or another query while the datareader is open.

using the same instance of SQLCommand more than one time in the same code for more than one query?

I have question about using why i can not use the same instance of SQLCommand more than one time in the same code?
I tried the code down here and it runs good for the gridview but when i changed the query by using cmd.CommandText() method it keeps saying:
There is already an open DataReader associated with this Command which must be closed first.
This is the code:
string cs = ConfigurationManager.ConnectionStrings["MyDB"].ConnectionString;
SqlConnection con = new SqlConnection(cs);
try
{
SqlCommand cmd = new SqlCommand();
cmd.Connection = con;
con.Open();
cmd.CommandText = "Select top 10 FirstName, LastName, Address, City, State from Customers";
GridView1.DataSource = cmd.ExecuteReader();
GridView1.DataBind();
cmd.CommandText = "SELECT TOP 10 COUNT(CreditLimit) FROM Customers";
int total = (int)cmd.ExecuteScalar();
TotalCreditLble.Text = "The total Credit :" + total.ToString();
}
catch(Exception exp)
{
Response.Write(exp.Message);
}
finally
{
con.Close();
}
The problem is that you are using the SqlCommand object to generate a DataReader via the command.ExecuteReader() command. While that is open, you can't re-use the command.
This should work:
using (var reader = cmd.ExecuteReader())
{
GridView1.DataSource = reader;
GridView1.DataBind();
}
//now the DataReader is closed/disposed and can re-use command
cmd.CommandText = "SELECT TOP 10 COUNT(CreditLimit) FROM Customers";
int total = (int)cmd.ExecuteScalar();
TotalCreditLble.Text = "The total Credit :" + total.ToString();
There is already an open DataReader associated with this Command which must be closed first.
This is the very reason you don't share a command. Somewhere in your code you did this:
cmd.ExecuteReader();
but you didn't leverage the using statement around the command because you wanted to share it. You can't do that. See, ExecuteReader leaves a connection to the server open while you read one row at a time; however that command is locked now because it's stateful at this point. The proper approach, always, is this:
using (SqlConnection c = new SqlConnection(cString))
{
using (SqlCommand cmd = new SqlCommand(sql, c))
{
// inside of here you can use ExecuteReader
using (SqlDataReader rdr = cmd.ExecuteReader())
{
// use the reader
}
}
}
These are unmanaged resources and need to be handled with care. That's why wrapping them with the using is imperative.
Do not share these objects. Build them, open them, use them, and dispose them.
By leveraging the using you will never have to worry about getting these objects closed and disposed.
Your code, written a little differently:
var cs = ConfigurationManager.ConnectionStrings["MyDB"].ConnectionString;
var gridSql = "Select top 10 FirstName, LastName, Address, City, State from Customers";
var cntSql = "SELECT TOP 10 COUNT(CreditLimit) FROM Customers";
using (SqlConnection con = new SqlConnection(cs))
{
con.Open();
try
{
using (SqlCommand cmd = new SqlCommand(gridSql, con))
{
GridView1.DataSource = cmd.ExecuteReader();
GridView1.DataBind();
}
using (SqlCommand cmd = new SqlCommand(cntSql, con))
{
int total = (int)cmd.ExecuteScalar();
TotalCreditLble.Text = "The total Credit :" + total.ToString();
}
}
catch(Exception exp)
{
Response.Write(exp.Message);
}
}
Thank u quys but for the guys who where talking about using block !
why this code work fine which i seen it on example on a video ! It's the same thing using the same instance of SqlCommand and passing diffrent queries by using the method CommanText with the same instance of SqlCommand and it's execute just fine , this is the code :
using (SqlConnection con = new SqlConnection(cs))
{
SqlCommand cmd = new SqlCommand();
cmd.Connection = con;
con.Open();
cmd.CommandText = "Delete from tbleProduct where ProductID= 4";
int TotalRowsAffected = cmd.ExecuteNonQuery();
Response.Write("Total rows affected :" + TotalRowsAffected );
cmd.CommandText = "Insert into tbleProduct values (4, 'Calculator', 100, 230)";
TotalRowsAffected = cmd.ExecuteNonQuery();
Response.Write("Total rows affected :" + TotalRowsAffected );
cmd.CommandText = "ypdate tbleProduct set QtyAvailbe = 234 where ProductID = 2";
TotalRowsAffected = cmd.ExecuteNonQuery();
Response.Write("Total rows affected :" + TotalRowsAffected );
}

Trying to make two SQL INSERT statements atomic in C# code

Below is the code I'm using in an SSIS script task. I am trying to make both inserts atomic as they deal with similar customers.
The first .executeNonQuery() works fine, locking the SQL table as it should.
The second .executNonQuery() throws an error:
ExecuteNonQuery requires the command to have a transaction when the
connection assigned to the command is in a pending local transaction.
The Transaction property of the command has not been initialized.
Code:
ConnectionManager cm;
SqlTransaction sqlTrans;
SqlConnection sqlConn;
SqlCommand sqlComm;
cm = Dts.Connections["connectionManager"];
try
{
//Set 'global' variables
SqlParameter agentID = new SqlParameter("#agentID", 1000018); //retrievedMessage.Substring(2, 10));//Primary key
SqlParameter lastChangeOperator = new SqlParameter("#lastChangeOperator", "LVO");
SqlParameter lastChangeDate = new SqlParameter("#lastChangeDate", DateTime.Now);
SqlParameter controlId = new SqlParameter("#controlID", 1); //Hard-coded value for testing - CHANGE LATER
//Set variables for Agent table
SqlParameter entityType = new SqlParameter("#entityType", "P");//retrievedMessage.Substring(162, 1));
SqlParameter fName = new SqlParameter("#fName", "test");//retrievedMessage.Substring(12, 25));
SqlParameter lName = new SqlParameter("#lName", "test");//retrievedMessage.Substring(37, 35));
SqlParameter suffix = new SqlParameter("#suffix", "test");//retrievedMessage.Substring(72, 10));
SqlParameter corporateName = new SqlParameter("#corporateName", "Initech");//retrievedMessage.Substring(82, 80));
//Insert record into Agent table
sqlConn = (SqlConnection)cm.AcquireConnection(Dts.Transaction);
sqlComm = new SqlCommand
(
"SET IDENTITY_INSERT Agent ON " +
"INSERT INTO Agent (UniqueAgentId, EntityType, FirstName, LastName, NameSuffix, CorporateName, LastChangeOperator, LastChangeDate, ControlId) " +
"VALUES (#agentID, #entityType, #fName, #lName, #suffix, #corporateName, #lastChangeOperator, #lastChangeDate, #controlID)" +
"SET IDENTITY_INSERT Agent OFF",
sqlConn//, sqlTrans
);
sqlTrans = sqlConn.BeginTransaction("SqlAgentTableUpdates");
sqlComm.Parameters.Add(agentID);
sqlComm.Parameters.Add(lastChangeOperator);
sqlComm.Parameters.Add(lastChangeDate);
sqlComm.Parameters.Add(controlId);
sqlComm.Parameters.Add(entityType);
sqlComm.Parameters.Add(fName);
sqlComm.Parameters.Add(lName);
sqlComm.Parameters.Add(suffix);
sqlComm.Parameters.Add(corporateName);
sqlComm.Transaction = sqlTrans;
sqlComm.ExecuteNonQuery();
//Set variables for AgentIdentification table
SqlParameter taxIdType = new SqlParameter("taxIdType", "S");//Hard-coded value for testing - CHANGE LATER
SqlParameter agentTaxId = new SqlParameter("#agentTaxId", "999999999");//Hard-coded value for testing - CHANGE LATER
//Insert record into AgentIdentification table
sqlConn = (SqlConnection)cm.AcquireConnection(Dts.Transaction);
sqlComm = new SqlCommand
(
"INSERT INTO AgentIdentification (UniqueAgentId, TaxIdType, AgentTaxId, LastChangeOperator, LastChangeDate, ControlId) " +
"VALUES (#agentID, #taxIdType, #agentTaxId, #lastChangeOperator, #lastChangeDate, #controlId)",
sqlConn//, sqlTrans
);
sqlComm.Parameters.Add(taxIdType);
sqlComm.Parameters.Add(agentTaxId);
sqlComm.Transaction = sqlTrans;
sqlComm.ExecuteNonQuery();
}
catch (Exception)
{
sqlTrans.Rollback();
cm.ReleaseConnection(sqlConn);
}
finally
{
sqlTrans.Commit();
cm.ReleaseConnection(sqlConn);
}
EDIT
I was able to make this transaction work by eliminating the second connection. However, both queries use a couple of the same variables (SqlParameters). I was forced to duplicate these in order for this to run without errors. Is there a way for them to share the variables so I do not have to waste space re-creating them?
I think the problem might be with the connection, or when you set the command to a new command for the second insert, you can use two different commands with the same connection or reuse one command just changing the CommandText property.
Hope this helps... Using SqlTransaction
A transaction cannot span multiple connections... does cm.AcquireConnection return a new connection each time? If so, try using the same connection for both commands.
use transactionscope
using(TransactionScope ts = new TransactionScope())
{
using(SqlConnection conn = new SqlConnection(myconnstring)
{
conn.Open();
//call first executenonquery
//call second executenonquery
ts.Complete();
conn.Close();
}
}

Categories

Resources