I'm creating a program in .NET that uses the IBM DB2 .NET provider(IBM.Data.DB2) to connect to a DB2 database(v9.7) and run select queries.
The program should terminate the SQL execution, if a submitted select query takes more than 5 seconds to return the data.
To implement this as per IBM's documentation I could use the QueryTimeout parameter in the connection.
string connStr = "Server=server:12345;Database=db;UID=user;PWD=pass; QueryTimeout = 5;";
DB2Connection conn = null;
DB2Command cmd = null;
conn = new DB2Connection(connStr);
conn.Open();
Console.WriteLine("IBM DB2: " + conn.IsOpen);
if (conn.IsOpen)
{
Console.WriteLine(conn.ConnectionTimeout);
cmd = conn.CreateCommand();
cmd.CommandText = "select * from user.orders";
DB2DataReader reader = cmd.ExecuteReader();
int counter = 0;
while (reader.Read())
{
counter += 1;
Console.WriteLine(reader.GetDB2Int64(0));
}
reader.Close();
}
conn.Close();
When I ran the query in winSQL, the query took about 20 seconds to execute. But when I execute it in this program, it took me the same 20 seconds. As per the documentation the query should have terminated in 5 seconds.
Why has the execution not stopped?
PS: I've also tried setting the cmd.CommandTimeout to 5 and it still would not stop the execution.
Read DB2Command.CommandTimeout property tutorial on IBM website. I hope this will solve your problem. C# example code is below:
[C#]
public void CreateMyDB2Command()
{
string mySelectQuery = "SELECT * FROM EMPLOYEE ORDER BY EMPNO";
DB2Command myCommand = new DB2Command(mySelectQuery);
myCommand.CommandTimeout = 20;
}
The problem with the above example is that ExecuteReader() creates a cursor to the database that pulls the data as an when its read, so the query never times out.
But using a DataAdapter to a dataset, pulls the data in a single go. So the below seems to kill the execution.
string connStr = "Server=server:12345;Database=db;UID=user;PWD=pass;";
conn = new DB2Connection(connStr);
conn.Open();
if (conn.IsOpen)
{
Console.WriteLine(conn.ConnectionTimeout);
cmd = conn.CreateCommand();
cmd.CommandText = "select * from orders";
cmd.CommandTimeout = 5;
DB2DataAdapter adp = new DB2DataAdapter(cmd);
DataSet ds = new DataSet();
adp.Fill(ds);
foreach (DataRow row in ds.Tables[0].Rows){
Console.WriteLine(row[0]);
}
}
conn.Close();
Related
I'm playing around making a POC and I've created the following call.
public string DoStuff()
{
try
{
using (SqlDataAdapter adapter = new SqlDataAdapter())
{
SqlConnection connection = new SqlConnection("Server...");
string command = "insert into Records values (...)";
adapter.InsertCommand = new SqlCommand(command, connection);
}
}
catch (Exception exception)
{
return exception.Message + " " + exception.InnerException;
}
return "WeeHee!";
}
The text I'm seeing returned is the happy one, so I conclude there's no exceptions. Hence, I conclude that the call to the DB is performed as supposed to. However, there's no new lines in the DB being created.
I'm using the same connection string as I have in my config file and the command in pasted in from SQL Manager, where it works.
So my suspicion was that although I create an insert command, I never actually execute it but according to MSDN that's how it's supposed to work.
What stupid thing do I miss here?
You are missing connection.Open(); and adapter.InsertCommand.ExecuteNonQuery();
using (SqlDataAdapter adapter = new SqlDataAdapter())
{
SqlConnection connection = new SqlConnection("Server...");
connection.Open();
string command = "insert into Records values (...)";
adapter.InsertCommand = new SqlCommand(command, connection);
adapter.InsertCommand.ExecuteNonQuery();
}
You should use ExecuteNonQuery instead. Using an SqlDataAdapter for an INSERT query does not make sense.
Also you should Open your connection just before you execute it.
You can:
using(SqlConnection connection = new SqlConnection("Server..."))
{
SqlCommand command = connection.CreateCommand();
command.CommandText = "insert into Records values (...)";
connection.Open();
int craeted = command.ExecuteNonQuery();
}
The example you linked to returned a SQLAdapter for later use.
You don't need one at all:
using (SqlConnection connection = new SqlConnection("Server..."))
{
string command = "insert into Records values (...)";
connection.Open();
var command = new SqlCommand(command, connection);
command.ExecuteNonQuery();
}
Note that there are other execution methods, depending on expected return values and whether you want asynchronous operation: https://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlcommand(v=vs.110).aspx
We have a performance Issue in our ASP.NET based web App and it's visible only once when we login in the morning. As per Log looks like below query takes more than 1 minute 20 secs. Next time onward when user login to the app and try to access same page doesn't find any issue. Can you please let me know how to optimize this query ? Any thoughts how we can fix this problem ?
Log -
"11/13/15","08:38:27","ExecuteSql - ---3---- ","8",""
"11/13/15","08:38:27","ExecuteSql - ---4---- : SQL : SELECT TOP 1 CONVERT(varchar(15), Period_End_Date, 107) as PDate FROM PBHISTORY..STATEMENT_OF_CHANGE
ORDER BY Period_End_Date DESC","8",""
"11/13/15","08:39:48","ExecuteSql - ---5---- ","8",""
SQL :
SELECT TOP 1
CONVERT(varchar(15), Period_End_Date, 107) as PDate
FROM
PBHISTORY..STATEMENT_OF_CHANGE
ORDER BY
Period_End_Date DESC
C# ASP.NET -
public string GetDateRangeReportingDate(int reportId)
{
var report = GetReportInfoById(reportId);
string sql = string.Format(#"SELECT TOP 1 CONVERT(varchar(15), Period_End_Date, 107) as PDate FROM {0}..{1} ORDER BY Period_End_Date DESC", _historyDatabase, report.SourceTableName);
var data = ExecuteSql(sql);
while (data.Read())
{
return data["PDate"].ToString();
}
return null;
}
private SqlDataReader ExecuteSql(string sql)
{
SqlDataReader reader;
SqlConnection conn;
var commandTimeOut = ConfigurationManager.AppSettings["PBReportCommandTimeout"].ToString();
string connString = ConfigurationManager.ConnectionStrings["PBReportCS"].ConnectionString;
conn = new SqlConnection(connString);
conn.Open();
SqlCommand cmd = new SqlCommand(sql, conn);
cmd.CommandTimeout = int.Parse(commandTimeOut);
reader = cmd.ExecuteReader();
return reader;
}
I found solution of my performance issue I mentioned above. Issue was wrong indexing in table and I fixed this issue by changing the indexing of table where from we are fetching records in production.
Also, I fixed Framework level entity class by using SQL Data Adapter and using statements. App runs super fast in production. Thanks for your help.
private string ExecuteSqlNew(string sql)
{
string connectionString = ConfigurationManager.ConnectionStrings["PBReportCS"].ConnectionString;
string commandTimeOut = ConfigurationManager.AppSettings["PBReportCommandTimeout"].ToString();
DataSet result = new DataSet();
string pDate = "";
try
{
using (SqlConnection conn = new SqlConnection(connectionString))
{
using (SqlCommand cmd = new SqlCommand(sql, conn))
{
conn.Open();
cmd.CommandTimeout = int.Parse(commandTimeOut);
SqlDataAdapter adapter = new SqlDataAdapter();
adapter.SelectCommand = cmd;
adapter.Fill(result);
adapter.Dispose();
conn.Close();
pDate = result.Tables[0].Rows[0]["PDate"].ToString();
}
}
}
catch(Exception ex)
{
throw ex;
}
return pDate;
}
My problem: Code works great, but the speed is far too slow for the number of rows it needs to process.
What I am doing: I initiate a COUNT(*) to pull total number of rows (which last night was ~58000) and I use that to create the loop to do the following: pull two columns of data from that row, datamine one column for text patterns.
Once I had completed that, I search a second table to see if that individual by username is already existing - if they exist, I update their row. If not, I add a new one.
There are 44 columns of data, one being the name the other 43 storing values of my datamining results.
In about 8 hours, it has completed 26500 out of the 58000 when it first started (in that same period, the table has grown to ~100000, but I am not worried about that).
Is there a better method to increase read/write rate?
Portion of my code - I have removed many of the int declarations and Regex.Matches as they are copies of the first with altered Match values.
azCheck is to determine if the message even contains anything we are looking for, it is remains '0', then we don't bother with the last portion of the code.
using (new MySqlConnection(ConnectiongString))
{
using (MySqlCommand cmd = new MySqlCommand("select count(*) from messages", connection))
{
using (MySqlDataReader reader = cmd.ExecuteReader())
{
StringBuilder sb = new StringBuilder();
while (reader.Read())
{
sb.Append(reader.GetInt32(0).ToString());
}
total_messages = int.Parse(sb.ToString());
}
}
}
Console.WriteLine(total_messages.ToString());
connection.Close();
for (int i = 1; i <= total_messages; i++)
{
connection.Open();
using (new MySqlConnection(ConnectiongString))
{
using (MySqlCommand cmd = new MySqlCommand("select * from messages WHERE id="+i+"", connection))
{
using (MySqlDataReader reader = cmd.ExecuteReader())
{
StringBuilder sb = new StringBuilder();
while (reader.Read())
{
username = reader["username"].ToString();
message = reader["message"].ToString();
}
}
}
}
connection.Close();
Console.Write("\r{0} ", i);
int aiCount = 0;
aiCount += Regex.Matches(message, "ai", RegexOptions.IgnoreCase).Count;
azCheck += aiCount;
//There are ~42 of the regex.matches after the first one.
MySqlCommand cmd1 = connection.CreateCommand();
connection.Open();
cmd1.CommandText = "SELECT username FROM users";
cmd1.CommandType = CommandType.Text;
cmd1.Connection = connection;
MySqlDataReader dr = cmd1.ExecuteReader();
while (dr.Read())
{
if (dr[0].ToString() == username)
{
check++;
}
}
connection.Close();
if (check == 0)
{
MySqlConnection connection2 = new MySqlConnection(ConnectiongString);
connection2.Open();
try
{
MySqlCommand cmd2 = connection2.CreateCommand();
cmd2.CommandText = "INSERT INTO users (username,aiCount) VALUES (#username,#aiCount)";
cmd2.Parameters.AddWithValue("#username", username);
cmd2.Parameters.AddWithValue("#aiCount", aiCount);
cmd2.ExecuteNonQuery();
connection2.Close();
}
catch (Exception)
{
throw;
}
} else {
int aiCount_old = 0;
if (azCheck > 0)
{
//Here we are taking the existing values from this users row,
//which we then add the new values from above and save.
MySqlConnection connection4 = new MySqlConnection(ConnectiongString);
connection4.Open();
try
{
MySqlCommand cmd2 = connection4.CreateCommand();
cmd2.CommandType = CommandType.Text;
cmd2.CommandText = "SELECT * from users WHERE username = #username";
cmd2.Parameters.AddWithValue("#username", username);
MySqlDataReader reader = cmd2.ExecuteReader();
while (reader.Read())
{
aiCount_old = Convert.ToInt32(reader["aiCount"].ToString());
}
}
catch (Exception)
{
throw;
}
connection4.Close();
aiCount += aiCount_old;
MySqlConnection connection5 = new MySqlConnection(ConnectiongString);
connection5.Open();
try
{
MySqlCommand cmd4 = connection5.CreateCommand();
cmd4.CommandType = CommandType.Text;
cmd4.CommandText = "UPDATE users SET aiCount = #aiCount WHERE LOWER(LTRIM(RTRIM(username))) = #username";
cmd4.Parameters.AddWithValue("#username", username.Trim().ToLower());
cmd4.Parameters.AddWithValue("#aiCount", aiCount.ToString());
cmd4.ExecuteNonQuery();
Console.WriteLine("User updated.");
}
catch (Exception ex)
{
throw;
}
connection5.Close();
You have several inefficiencies that I can spot right off the bat.
You are constantly opening and closing your connection string. This is probably your biggest bottleneck. Open the connection once, then close it once when all your processing is done, and you'll probably see a massive increase in performance.
You also use different connection objects when one will do, which will reduce your need to open and close connections.
You also seem to have a misunderstanding of the use of "using" on connection objects. I see using (new MySqlConnection(ConnectiongString)), but that code is completely useless, as it doesn't do anything but initialize a connection object, which is immediately lost since it is not assigned to an object.
Since you are processing everything sequentially, use connection as your connection object in every case, opening it only at the start of your processing, and closing it when processing is complete, then executing the Dispose method (the point of the using statement).
This change alone might reduce processing time by an order of magnitude.
NOTE: You will need a separate connection for your datareader if you need to do updates or another query while the datareader is open.
I have question about using why i can not use the same instance of SQLCommand more than one time in the same code?
I tried the code down here and it runs good for the gridview but when i changed the query by using cmd.CommandText() method it keeps saying:
There is already an open DataReader associated with this Command which must be closed first.
This is the code:
string cs = ConfigurationManager.ConnectionStrings["MyDB"].ConnectionString;
SqlConnection con = new SqlConnection(cs);
try
{
SqlCommand cmd = new SqlCommand();
cmd.Connection = con;
con.Open();
cmd.CommandText = "Select top 10 FirstName, LastName, Address, City, State from Customers";
GridView1.DataSource = cmd.ExecuteReader();
GridView1.DataBind();
cmd.CommandText = "SELECT TOP 10 COUNT(CreditLimit) FROM Customers";
int total = (int)cmd.ExecuteScalar();
TotalCreditLble.Text = "The total Credit :" + total.ToString();
}
catch(Exception exp)
{
Response.Write(exp.Message);
}
finally
{
con.Close();
}
The problem is that you are using the SqlCommand object to generate a DataReader via the command.ExecuteReader() command. While that is open, you can't re-use the command.
This should work:
using (var reader = cmd.ExecuteReader())
{
GridView1.DataSource = reader;
GridView1.DataBind();
}
//now the DataReader is closed/disposed and can re-use command
cmd.CommandText = "SELECT TOP 10 COUNT(CreditLimit) FROM Customers";
int total = (int)cmd.ExecuteScalar();
TotalCreditLble.Text = "The total Credit :" + total.ToString();
There is already an open DataReader associated with this Command which must be closed first.
This is the very reason you don't share a command. Somewhere in your code you did this:
cmd.ExecuteReader();
but you didn't leverage the using statement around the command because you wanted to share it. You can't do that. See, ExecuteReader leaves a connection to the server open while you read one row at a time; however that command is locked now because it's stateful at this point. The proper approach, always, is this:
using (SqlConnection c = new SqlConnection(cString))
{
using (SqlCommand cmd = new SqlCommand(sql, c))
{
// inside of here you can use ExecuteReader
using (SqlDataReader rdr = cmd.ExecuteReader())
{
// use the reader
}
}
}
These are unmanaged resources and need to be handled with care. That's why wrapping them with the using is imperative.
Do not share these objects. Build them, open them, use them, and dispose them.
By leveraging the using you will never have to worry about getting these objects closed and disposed.
Your code, written a little differently:
var cs = ConfigurationManager.ConnectionStrings["MyDB"].ConnectionString;
var gridSql = "Select top 10 FirstName, LastName, Address, City, State from Customers";
var cntSql = "SELECT TOP 10 COUNT(CreditLimit) FROM Customers";
using (SqlConnection con = new SqlConnection(cs))
{
con.Open();
try
{
using (SqlCommand cmd = new SqlCommand(gridSql, con))
{
GridView1.DataSource = cmd.ExecuteReader();
GridView1.DataBind();
}
using (SqlCommand cmd = new SqlCommand(cntSql, con))
{
int total = (int)cmd.ExecuteScalar();
TotalCreditLble.Text = "The total Credit :" + total.ToString();
}
}
catch(Exception exp)
{
Response.Write(exp.Message);
}
}
Thank u quys but for the guys who where talking about using block !
why this code work fine which i seen it on example on a video ! It's the same thing using the same instance of SqlCommand and passing diffrent queries by using the method CommanText with the same instance of SqlCommand and it's execute just fine , this is the code :
using (SqlConnection con = new SqlConnection(cs))
{
SqlCommand cmd = new SqlCommand();
cmd.Connection = con;
con.Open();
cmd.CommandText = "Delete from tbleProduct where ProductID= 4";
int TotalRowsAffected = cmd.ExecuteNonQuery();
Response.Write("Total rows affected :" + TotalRowsAffected );
cmd.CommandText = "Insert into tbleProduct values (4, 'Calculator', 100, 230)";
TotalRowsAffected = cmd.ExecuteNonQuery();
Response.Write("Total rows affected :" + TotalRowsAffected );
cmd.CommandText = "ypdate tbleProduct set QtyAvailbe = 234 where ProductID = 2";
TotalRowsAffected = cmd.ExecuteNonQuery();
Response.Write("Total rows affected :" + TotalRowsAffected );
}
Is there an other way to insert multiple objects to an MySQL database than the way shown here. This works but takes time to execute.
using (MySql.Data.MySqlClient.MySqlConnection conn = new MySql.Data.MySqlClient.MySqlConnection(connStr))
{
//Goes thrue the List<object>
foreach(List<object> sub in listSubject)
{
MySql.Data.MySqlClient.MySqlCommand cmd = new MySql.Data.MySqlClient.MySqlCommand();
cmd.Connection = conn;
cmd.CommandText = "CALL stp_InsertSubject(#param_SubjectId, #param_ProjectId, #param_Used);";
cmd.Parameters.AddWithValue("#param_SubjectId",Convert.ToInt32(sub[0]) );
cmd.Parameters.AddWithValue("#param_ProjectId", Convert.ToInt32(sub[1]));
cmd.Parameters.AddWithValue("#param_Used", Convert.ToBoolean(sub[2]) );
conn.Open();
cmd.ExecuteNonQuery();
conn.Close();
}
My Stored procedure:
CREATE DEFINER=`mortenstarck`#`%` PROCEDURE `stp_InsertSubject`(param_SubjectId int(45), param_ProjectId int(45), param_Used tinyint(1))
BEGIN
INSERT INTO Subject_has_Projects(Subject_Id, Projects_Id, Used) VALUES (param_SubjectId, param_ProjectId, param_Used);
END
Few things to improve:
Open the connection just one time outside the loop (no need to close
with using)
Create the command, assign connection just one time before the loop
Create the parameters all before the loop with dummy values
Assign only the value inside the loop and call the ExecuteScalar()
using(MySql.Data.MySqlClient.MySqlConnection conn = new MySql.Data.MySqlClient.MySqlConnection(connStr))
{
conn.Open();
MySql.Data.MySqlClient.MySqlCommand cmd = new MySql.Data.MySqlClient.MySqlCommand();
cmd.Connection = conn;
cmd.CommandText = "CALL stp_InsertSubject(#param_SubjectId, #param_ProjectId, #param_Used);";
cmd.Parameters.AddWithValue("#param_SubjectId",0 );
cmd.Parameters.AddWithValue("#param_ProjectId", 0);
cmd.Parameters.AddWithValue("#param_Used", false );
foreach(List<object> sub in listSubject)
{
cmd.Parameters["#param_SubjectId"].Value = Convert.ToInt32(sub[0]) ;
cmd.Parameters["#param_ProjectId"].Value = Convert.ToInt32(sub[1]);
cmd.Parameters["#param_Used"].Value = Convert.ToBoolean(sub[2]);
Id = (Int64)cmd.ExecuteScalar();
}
}
You can try. Open connection outside foreach loop. This will save time in opening and closing connection every time in loop. This will improve performance.
Int64 Id = 0;
using (MySql.Data.MySqlClient.MySqlConnection conn = new MySql.Data.MySqlClient.MySqlConnection(connStr))
{
//Goes through the List<object>
conn.Open();
foreach(List<object> sub in listSubject)
{
MySql.Data.MySqlClient.MySqlCommand cmd = new MySql.Data.MySqlClient.MySqlCommand();
cmd.Connection = conn;
cmd.CommandText = "CALL stp_InsertSubject(#param_SubjectId, #param_ProjectId, #param_Used);";
cmd.Parameters.AddWithValue("#param_SubjectId",Convert.ToInt32(sub[0]) );
cmd.Parameters.AddWithValue("#param_ProjectId", Convert.ToInt32(sub[1]));
cmd.Parameters.AddWithValue("#param_Used", Convert.ToBoolean(sub[2]) );
Id = (Int64)cmd.ExecuteScalar();
}
conn.Close();
}
How bad is opening and closing a SQL connection for several times? What is the exact effect?
Have you thought about surrounding these calls with a single transaction?