I am writing my C# application that connects to database, does couple selects and then inserts record back to the server in the network.
But I have around 40k records and my program processes one records like for 1 second.
I don't know how to improve the performance. Here are my sql getter and inserter. Any suggestions?
public bool insert_and_ConfirmSQL(String Query, String comments)
{
bool success = false;
NpgsqlCommand cmd = new NpgsqlCommand();
NpgsqlConnection mycon = new NpgsqlConnection();
string connstring = String.Format("Server={0};Port={1}; User Id={2};Password={3};Database={4};", tbHost, tbPort, tbUser, tbPass, tbDataBaseName);
mycon.ConnectionString = connstring;
cmd = mycon.CreateCommand();
cmd.CommandText = Query;
mycon.Open();
int temp = 0;
try
{
temp = cmd.ExecuteNonQuery();
success = true;
}
catch
{
success = false;
}
finally
{
if (mycon.State.ToString() == "Open")
{
mycon.Close();
}
}
return success;
}
public String getString(String sql, NpgsqlConnection conn)
{
using (DataSet ds = new DataSet())
{
using (NpgsqlDataAdapter da = new NpgsqlDataAdapter(sql, conn))
{
da.Fill(ds);
if (ds.Tables.Count > 0)
{
DataTable dt = ds.Tables[0];
// check count of Rows
if (dt.Rows.Count > 0)
{
object o = dt.Rows[0][0];
if (o != DBNull.Value && o != null)
{
return o.ToString();
}
}
}
}
}
// Return default value
return "";
}
There is no chance to help you with your sql statements because we don't know them.
The only (guessing) visible problem is that you are trying to loop for 40K records (you know how) with a new connection each time. Both routines provided as your code do exactly this. So do you need 40K calls to insert_and_ConfirmSQL and/or another 40K to getString routines?
If you really loop then update your code to use only one connection without closing it; the same can be done with a dataset (you have to clear it before use: ds.Clear()).
Needless to say that if your queries are huge (in terms of data), and/or indexes do not cover the queries, then delays are expected.
Try with this approach and let us know.
Sometimes you can improve performance by using a sql Datareader instead of DataSets.
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldatareader(v=vs.100).aspx#Y0
You then can Inspect the data while you are reading it from the database.
So I'd implement the above code with a DataReader and retime it.
Edit: Especially that getString method. if that ds.Fill was taking 40k rows that could be the cause of your performance problem.
Related
I have a method to retrieve data from a database using a stored procedure as a DataTable like:
public DataTable GetTableBySQL(string sql)
{
SqlCommand cmd = new SqlCommand(sql.ToString(), this.dbconn)
{
CommandTimeout = 0,
CommandType = CommandType.Text
};
DataTable tbl = new DataTable("Table1")
{
Locale = System.Globalization.CultureInfo.InvariantCulture
};
SqlDataAdapter da = new SqlDataAdapter(cmd);
try
{
da.SelectCommand.CommandTimeout = 0;
da.Fill(tbl);
}
catch (SqlException e)
{
this.HandleSQLError(e, "GetTableBySQL", sql.ToString());
}
finally
{
cmd.Dispose();
da.Dispose();
}
return tbl;
}
Now I call the stored procedure like this:
var empList = db.GetTableBySQL("$exec getMySP");
But when I execute, it just don't return any columns.
What am I doing wrong? Regards
There are three main problems here (other smaller ones, but three that are important):
The $exec part of the SQL doesn't mean anything. Maybe you just want exec.
When the bad SQL fails, the error is hidden from the program, so you don't really know what happened.
The method signature doesn't support query parameters, and therefore will force you to write horribly insecure code that will result in someone hacking your application. Probably sooner rather than later. This is really bad, and you should not ignore it.
Try something more like this:
public DataTable GetTableBySQL(string sql, params SqlParameter[] parameters)
{
var result = new DataTable();
//ADO.Net really does work better when you create a **NEW** connection
// object for most queries. Just share the connection string.
//Also: "using" blocks are a better way to make sure the connection is closed.
using (var dbconn = new SqlConnection(this.dbConnectionString))
using (var cmd = new SqlCommand(sql, dbconn))
using (var da = new SqlDataAdapter(cmd))
{
cmd.CommandTimeout = 0;
// A number of the properties set on the cmd and tbl variables just set the same value that was already there, didn't accomplish anything
//It's hard to understate how important it is to use parameterized queries.
if (parameters != null && parameters.Length > 0)
{
cmd.Parameters.AddRange(parameters);
}
try
{
da.Fill(result);
}
catch (SqlException e)
{
this.HandleSQLError(e, "GetTableBySQL", sql.ToString());
//you may want to re-throw here,
// or even just remove the try/catch and let the error bubble up to calling code
}
}
return result;
}
Here it is again without all the extra explanatory comments, so you can see that doing it right is less code, rather than more:
public DataTable GetTableBySQL(string sql, params SqlParameter[] parameters)
{
var result = new DataTable();
using (var dbconn = new SqlConnection(this.dbConnectionString))
using (var cmd = new SqlCommand(sql, dbconn))
using (var da = new SqlDataAdapter(cmd))
{
cmd.CommandTimeout = 0;
if (parameters != null && parameters.Length > 0)
{
cmd.Parameters.AddRange(parameters);
}
da.Fill(result);
}
return result;
}
Then call it like this:
var empList = db.GetTableBySQL("exec getMySP");
I'm having some performance issues when I return a record set with more than 1,000 records.
Sometimes the records are in upwards of 2,100 but can be as low as 10.
I have some bulk actions that I take on all the records by selecting them.
However, when the number is low, the Gridview is fine. When the record count is greater than 500 I see performance issues on the page.
What I want to happen is: if there are more than 500 records, DO NOT DISPLAY THE GRID, instead show a download button that exports to CSV or do other control things on the page.
My issue:
Even if i tell it not to display the grid and instead display a message and a button, the performance is still slow.
Below is my C# code for populating the GridView. Some stuff has been removed that are unimportant and to help with readability.
How can I adjust my C# code for better performance?
SqlConnection conn = new SqlConnection(ConfigurationManager.AppSettings["ConnectString"].ToString());
SqlCommand cmd = conn.CreateCommand();
cmd.CommandType = CommandType.StoredProcedure;
cmd.CommandText = "SomeProcedure";
cmd.Parameters.Add(SearchParam);
try {
DataTable GridData = new DataTable();
conn.Open();
using(SqlDataAdapter Sqlda = new SqlDataAdapter(cmd)) {
Sqlda.Fill(GridData);
}
if (GridData.Rows.Count == 0) {
lblSearchMsg.Text = "No Fee Records are in the Queue at this time.";
} else {
if (GridData.Rows.Count > 500) {
lblSearchMsg.Text = "More than " + gridLimit.ToString() + " records returned.";
//Show the download button
} else {
//Persist the table in the Session object. (for sorting)
Session["GridData"] = GridData;
lblRowCount.Text = "Count: " + GridData.Rows.Count.ToString();
myGridView.DataSource = GridData;
myGridView.DataBind();
myGridView.Visible = true;
}
}
} catch (Exception ex) {
//Do the error stuff
} finally {
if (conn != null) {
conn.Close();
}
}
Create a separate procedure that returns only the row count.
Check that value not the row count of a fully retrieved data set then retrieve the full data set as needed.
Keep in mind you can use the same connection to do both retrievals, no need to close the connection between calls.
if you determine you need to fill a gridview and there is no need to edit the data you can read into the DataTable without the use of an adapter. Here is the basic idea modify with using statements or try/catch as you prefer:
conn = new SqlConnection(connString);
string query = "SELECT * FROM ....";
SqlCommand cmd = new SqlCommand(query, conn);
conn.Open();
SqlDataReader dr = cmd.ExecuteReader();
DataTable dt = new DataTable();
dt.Load(dr);
GridView1.DataSource = dt;
GridView1.DataBind();
I ran into a problem where passing a connection from a TableAdapter to some methods throws an exception stating the connectionstring isn't initialized. There are quite a few questions on SO with this exception but none were passing the connection and most were because the ConnectionString was null. Weird thing is I used MessageBox.Show(connection.ConnectionString); through out the chain of methods and I receive a valid connection string at every step. This is a somewhat complicated program that is in production but I will try to simplify the code for this question...
This is the postInventoryData method, which takes a DataGridView with inventory items and iterates through it posting them to the inventory. I use a TransactionScope to ensure the changes are safely rolled back in the event of an error. If an item is a kit(an item comprised of other items) I must iterate through those items and remove them from the inventory. The problem occurs when I check whether or not the item is a kit.
public bool postInventoryData(DataGridView dgv)
{
bool successful = true;
TestDataSetTableAdapters.inentoryTrxTableAdapter inventoryTrxAdapter =
new TestDataSetTableAdapters.inentoryTrxTableAdapter();
try
{
using (TransactionScope trxScope = new TransactionScope
(TransactionScopeOption.Required, new System.TimeSpan(0, 15, 0)))
{
MessageBox.Show(inventoryTrxAdapter.Connection.ConnectionString); // <-- Valid ConnectionString
inventoryTrxAdapter.OpenConnection();
for (int i = 0; i < dgv.Rows.Count; i++)
{
//parameter values
string departmentCode = dgv.Rows[i].Cells["Department_Code"].Value.ToString();
string machineCode = dgv.Rows[i].Cells["Machine_Code"].Value.ToString();
string operatorCode = dgv.Rows[i].Cells["Operator_Code"].Value.ToString();
string itemNumber = dgv.Rows[i].Cells["Item_Number"].Value.ToString();
double? qtyProduced = Convert.ToDouble(dgv.Rows[i].Cells["Quantity"].Value.ToString());
bool isKit =
businessLayer.isItemNumberKit
(inventoryTrxAdapter.Connection, itemNumber); // <-- CULPRIT!
// Inserts the item
dailyProductionInsertQty(
departmentCode,
machineCode,
operatorCode,
itemNumber,
isKit,
qtyProduced,
inventoryTrxAdapter,
trxScope);
}
inventoryTrxAdapter.CloseConnection();
trxScope.Complete();
}
}
catch (System.Exception ex)
{
successful = false;
MessageBox.Show(ex.ToString());
}
return successful;
}
The isItemNumberKit method
public bool isItemNumberKit(SqlConnection connection, string itemNumber)
{
bool contains;
MessageBox.Show(connection.ConnectionString); // <-- Valid ConnectionString
DataTable dt = getKit(connection, itemNumber); // <-- CULPRIT!
if (dt.Rows.Count > 0)
{
contains = true;
}
else
{
contains = false;
}
return contains;
}
The getKit method
public DataTable getKit(SqlConnection connection, string itemNumber)
{
DataTable dt = new DataTable();
SqlConnection myConnection = connection;
MessageBox.Show(myConnection.ConnectionString); // <-- Valid ConnectionString
SqlParameter paramItemNumber = new SqlParameter();
paramItemNumber.ParameterName = "#ItemNumber";
paramItemNumber.Value = itemNumber;
paramItemNumber.SqlDbType = System.Data.SqlDbType.VarChar;
try
{
using (myConnection)
{
string sql =
#"SELECT kits.Row_Id,
kits.Kit_Item_Number,
kits.Location_Code
FROM Inventory.dbo.Z_PV_Kits kits
WHERE kits.Kit_Item_Number=#ItemNumber";
//myConnection.Open();
using (SqlCommand myCommand = new SqlCommand(sql, myConnection))
{
myCommand.Parameters.Add(paramItemNumber);
SqlDataReader reader = myCommand.ExecuteReader();
dt.Load(reader);
}
}
}
catch (Exception ex)
{
dt = null;
MessageBox.Show(ex.ToString());
}
return dt;
}
When I execute postInventoryData the program throws an exception with the message, "The connectionstring property has not been initialized." with the line numbers pointing to isItemNumberKit and getKit. As you can see in the code above, I used a MessageBox.Show(connection.ConnectionString) throughout the process and each time I received a valid Connection string. I have created a workaround which stores a cached DataTable containing all the kit items I can run linq statements on. I am not in emergency mode or anything but I thought this to be weird and an opportunity for me to learn. Thanks in advance for any help!
It might be possible that you have 2 app.config files in your solution with 2 different connection strings.
OK, I figured it out and now when I think about it the answer was somewhat obvious. I always use using(){} blocks to ensure connections and similar objects are properly disposed and taken care of after they are used. The solution was to simply remove the using(myConnection){} block from the getKit method like this:
public DataTable getKit(SqlConnection connection, string itemNumber)
{
DataTable dt = new DataTable();
SqlConnection myConnection = connection;
MessageBox.Show(myConnection.ConnectionString);
SqlParameter paramItemNumber = new SqlParameter();
paramItemNumber.ParameterName = "#ItemNumber";
paramItemNumber.Value = itemNumber;
paramItemNumber.SqlDbType = System.Data.SqlDbType.VarChar;
try
{
string sql =
#"SELECT kits.Row_Id,
kits.Kit_Item_Number,
kits.Location_Code
FROM Inventory.dbo.Z_PV_Kits kits
WHERE kits.Kit_Item_Number=#ItemNumber
";
//myConnection.Open();
using (SqlCommand myCommand = new SqlCommand(sql, myConnection))
{
myCommand.Parameters.Add(paramItemNumber);
SqlDataReader reader = myCommand.ExecuteReader();
dt.Load(reader);
}
}
catch (Exception ex)
{
dt = null;
MessageBox.Show(ex.ToString());
}
return dt;
}
This will leave the connection intact but properly dispose of the command. Sorry for the long winded question with a short simple answer. Hope this might help someone someday.
I am trying to show number of rows from a grid view using a label. I used SQL Count statement but it doesn't work. The problem is I only get only number 1 shown in the label which is not matched with the number of rows I have in my table! I have posted similar question but unfortunately no body gave a clear answer!
My codes as:
Basket.ac
public int td()
{
int customers;
//I tried this select query but still gets number 1
//String sql = String.Format("SELECT COUNT(*) FROM dbo.Baskets");
string sql = string.Format("SELECT COUNT(*) FROM Baskets");
customers = Db.RunQuery(sql).Rows.Count;
//customers = Convert.ToInt32(Db.RunQuery(sql).Rows.Count);
return customers;
}
DataBaseConn.ac
public class DataBaseConn
{
SqlConnection conn;
SqlCommand cmd;
DataTable tbl;
private void Intialise(CommandType commandtype, string DataBase)
{
conn = new SqlConnection();
cmd = new SqlCommand();
//Requirements
conn.ConnectionString = ConfigurationManager.ConnectionStrings[1].ToString();
cmd.Connection = conn;
cmd.CommandType = commandtype;
cmd.CommandText = DataBase;
conn.Open();
}
public int RunProcedure(string Procedure, SortedList ParameterV)
{
Intialise(CommandType.StoredProcedure, Procedure);
for (int i = 0; i < ParameterV.Count; i++)
try
{
if (ParameterV.GetByIndex(i) != null)
cmd.Parameters.AddWithValue(ParameterV.GetKey(i).ToString(),
PrameterV.GetByIndex(i).ToString());
}
catch { ;}
return RunUpdate();
}
public int RunUpdate(string InsDelUpd)
{
Intialise(CommandType.Text, InsDelUpd);
return RunUpdate();
}
private int RunUpdate()
{
try
{
int x = cmd.ExecuteNonQuery();
conn.Close();
return x;
}
catch (SqlException ex)
{
conn.Close();
return ex.Number;
}
}
public DataTable RunQuery(string Select)
{
Intialise(CommandType.Text, Select);
tbl = new DataTable();
tbl.Load(cmd.ExecuteReader());
conn.Close();
return tbl;
}
public bool EData(string selection)
{
if (RunQuery(selection).Rows.Count > 0)
return true;
else
return false;
}
}
Basket.aspx
lblQueue.Text = _b.td().ToString();
You don't want to return the data table's .Rows.Count - this will always be 1 (as in 1 row(s) affected) for your count(*) query.
You should instead look at using ExecuteScalar to return the first column of the first row from your query
I'm not sure how you want to build this into your DataBaseConn data helper class, but the gist of it is you need the following sequence:
using (var conn = new SqlConnection(connectionStringHere))
using (var cmd = new SqlCommand(conn))
{
cmd.CommandType = CommandType.Text;
cmd.CommandText = "SELECT COUNT(*) FROM Baskets";
return (int)cmd.ExecuteScalar();
}
Edit
If you aren't able to extend your DataBaseConn helper with an ExecuteScalar, then I guess you'll be able to use the existing RunQuery method which returns a DataTable. Just scrape the first column of the first row like so:
return Db.RunQuery(sql).Rows[0].Field<int>(0);
As a side note, you might look at replacing your DataHelper entirely with the Microsoft Patterns and Practices Data Application Access Block (DAAB), or, if you prefer, look at using an ORM like Entity Framework. By upgrading to a mainstream Data Access encapsulation, you won't need to spend as much time debugging data access issues like this one :)
I am given connection to a database which is owned by another company. The user that they gave to me has restricted privilidges, meaining that I can only make select queries on certain views.
I got a little problem here since the other company is not being so cooperative. They change my users password without telling me, or they change the names of the views. Since there are more than 40 views I want to make an automatic system that checks if everything is alright.
My question is what kind of checks I can make on the views and database? is just trying the connection to open and making select * queries for each view enough?
BTW the database is SQLServer 2008 R2 and I use C#.
Here is a function for checking that all required views exists:
bool IsAllviewsExists()
{
string DatabaseName= "Your_DB_NAME";
string[] viewsInDB = GetAllViewsNamesInDB();
for (int i = 0; i < viewsInDB.Length; ++i)
{
using (SqlCommand cmd = CreateSqlCommand(String.Format("SELECT id FROM sysobjects WHERE ID = OBJECT_ID('{0}.dbo.{1}') AND (type = 'V')", DatabaseName,viewsInDB [i])))
{
using (DataTable objects = ExecuteDataTableQuery(cmd))
{
if (objects.Rows.Count == 0)
{
return false;
}
}
}
}
return true;
}
The functions that are called from IsAllviewsExists :
(Pleas note that they assume you have a data member of a connection called _conn)
SqlCommand CreateSqlCommand(string sql, SqlParameterCollection parameters)
{
SqlCommand cmd = _conn.CreateCommand();
cmd.Connection = _conn;
cmd.CommandText = sql;
if (parameters != null)
foreach (SqlParameter param in parameters)
cmd.Parameters.Add(param);
return cmd;
}
DataTable ExecuteDataTableQuery(SqlCommand cmd)
{
DataTable table = null;
using (SqlDataAdapter adapter = new SqlDataAdapter(cmd))
{
table = new DataTable();
try
{
adapter.Fill(table);
}
catch (SqlException sqlEx)
{
rethrow;
}
}
return table;
}