Develop Reference

Stored procedure causing connection time execution out error when fetch large data

I am calling a SQL Server stored procedure to get huge amount of records from the database, but it's causing timeout execution errors when run. How can we make it fast in this way?
I am using stored procedure call in Entity Framework. The stored procedure takes input parameters and out parameter is total record in table for pagination in the fronted. I want to get all data with 10 record per page. Database table name is Player_account_flow and store and stored procedure name is also given. I am mapping my db model when the data load.
using (SqlConnection conn = dbContext.Database.GetDbConnection() as SqlConnection)
{
conn.Open();
using (SqlCommand cmd = new SqlCommand())
{
cmd.Connection = conn;
cmd.CommandText = "Pro_GetPageData";
cmd.CommandType = CommandType.StoredProcedure;
// input parameters to procedure
cmd.Parameters.AddWithValue("#TableName", "Player_Account_Flow");
cmd.Parameters.AddWithValue("#OrderString", "Create_Time Desc");
cmd.Parameters.AddWithValue("#ReFieldsStr", "*");
cmd.Parameters.AddWithValue("#PageIndex", 1);
cmd.Parameters.AddWithValue("#PageSize", 10);
cmd.Parameters.AddWithValue("#WhereString", query);
cmd.Parameters.AddWithValue("#TotalRecord", SqlDbType.Int);
cmd.Parameters["#TotalRecord"].Direction = ParameterDirection.Output;
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
var map = new PlayerAccountFlow();
map.Id = (Int32)reader["Id"];
map.AccountId = (Int32)reader["AccountId"];
map.Type = (byte)reader["Type"];
map.BeforeAmout = (decimal)reader["BeforeAmout"];
map.AfterAmout = (decimal)reader["AfterAmout"];
map.Amout = (decimal)reader["Amout"];
map.Source = (Int32)reader["Source"];
map.Memo = reader["Memo"].ToString();
map.CreateTime = (DateTime)reader["Create_Time"];
playerAccountFlowsList.Add(map);
}
}
obj = cmd.Parameters["#TotalRecord"].Value;
}
}
PagingData<PlayerAccountFlow> pagingData = new PagingData<PlayerAccountFlow>();
pagingData.Countnum = (Int32)obj;
pagingData.Data = playerAccountFlowsList;
return pagingData;

There's multiple things to look into here. You could set a longer command timeout than the default. You could look into the stored procedure to see why is it running so slow.
For the CommandTimeout just give it a value in seconds. In the example below that would be 2 minutes. Although it would be a bad user experience to make them wait minutes to display their result. I would advise on optimizing the stored procedure first(look at indexes, multiple joins, sub queries, and so on). Also, are you actually displaying that huge dataset on the user interface, or are you splitting it in pages? You could change the stored procedure to just return the current page, and call it again when the user goes to the next page. A well written stored procedure should be a lot faster when you only return a small subset of the whole data.
using (SqlCommand cmd = new SqlCommand())
{
cmd.Connection = conn;
cmd.CommandTimeout = 120;
cmd.CommandText = "Pro_GetPageData";
cmd.CommandType = CommandType.StoredProcedure;

In this case you should revisit your stored procedure for example multiple joins. You should also ask yourself if all data should be loaded immediately or not. I would suggest to display initial data and load further data on user request. That means that you can use simple count on database and calculate pages based on result, then fetch data only from selected range.

DataTable Load very slow

I'm using a datatable as the datasource of some dropdowns on a page, but have noticed that the page is very slow during the postbacks.
I've tracked it through to here:
DataTable dt = new DataTable();
dt.Load(sqlCmd.ExecuteReader()); // this takes ages
The sql command is a parametrised query, not a stored procedure (the return values and where are quite 'dynamic' so this wouldn't be practicable), but nevertheless a simple select union query.
Usually returns between 5 and 20 options per dropdown, depending on what's been selected on the other dropdowns.
When I run the query in the management studio, it's done in under a second. Here it can take up to 7 seconds per dropdown, with 6 dropdowns on the page it soon adds up.
I have also tried with a SqlDataAdapter:
SqlDataAdapter sqlDa = new SqlDataAdapter(sqlCmd);
sqlDa.Fill(dt); // this takes ages
but this was just as slow.
I have this on 2 different systems and on both have the same performance issues.
If anyone knows a better (faster) methord, or knows why this is so slow that would be great.

Not the best thread I've seen on the issue, but there's good links inside, & it's in my post history:
SQL Query that runs fine in SSMS runs very slow in ASP.NET
The SQL Optimizer sometimes likes to decide what's best & you'll have to break out your query through some tracing and logging of data execution plans. It may very well be something as buried as a bad index, or your query code might need optimization. Seeing as we don't have the query code, and having it may or may not be helpful. I'd recommend you follow the guides linked to in the above post and close your question.

here is an example on how you can load a DataTable very quickly notice how I show specific Columns that I want to return
private DataTable GetTableData()
{
string sql = "SELECT Id, FisrtName, LastName, Desc FROM MySqlTable";
using (SqlConnection myConnection = new SqlConnection(connectionString))
{
using (SqlCommand myCommand = new SqlCommand(sql, myConnection))
{
myConnection.Open();
using (SqlDataReader myReader = myCommand.ExecuteReader())
{
DataTable myTable = new DataTable();
myTable.Load(myReader);
myConnection.Close();
return myTable;
}
}
}
}
If you want to use DataAdapter to Fill the DataTable here is a simple example
private void FillAdapter()
{
using (SqlConnection conn = new SqlConnection(Your ConnectionString))
{
conn.Open();
using (SqlDataAdapter dataAdapt = new SqlDataAdapter("SELECT * FROM EmployeeIDs", conn))
{
DataTable dt = new DataTable();
dataAdapt.Fill(dt);
// dataGridView1.DataSource = dt;//if you want to display data in DataGridView
}
}
}

Multiple Select result and Linq to Entities

I have an SP which returns an unknown amount of data, here is an example for my query:
MySP:
WHILE (#counter <= #SomeParameter)
BEGIN
Select *
From tblFoo
Where tblFoo.Counter=#counter
#counter=#counter+1
END
In order to store the data efficiently I would like to use DataSet , that will store in each of it's DataTable the result for each of the Selects.
Since my application is based on EF 5 , I tried to call my SP with my dbContext object, here is what I tried to do.
var ds=db.Database.SqlQuery<DataSet>("MySP #counter #SomeParameter", value1,value2);
That doesn't seem to work properly.
I thought of using the classic ADO.NET to solve this matter, and use SqlDataAdapter , But I am not sure how to pass the original Connection reference from the dbContext to the SqlDataAdapter.Connection Property since its not the same type.
Note: The reason I am using DataSet and not Entities Collection at this matter is because the results I am getting from the SP could have different columns and therefore I'm not sure if Entities Collection will do.
I would like to know how to call my SP using Entities (or SqlAdapter) to fill each of the tables at my DataSet with the results of each Select from my SP.
I'm fresh at EF so if I am thinking or doing anything wrong any tip would be appriciated.

You can try with this code - based on SqlDataAdapter class
var connectionString = "...";
using (SqlConnection connection =
new SqlConnection(connectionString))
{
SqlDataAdapter adapter = new SqlDataAdapter();
var cmd = new SqlCommand("YourSP", connection);//Adjust your stored procedure name
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add(new SqlParameter("#SomeParameter", YourValue));//Adjust your value
adapter.SelectCommand = cmd;
adapter.Fill(dataset);
return dataset;
}

Making SqlDataAdapter/Datareader "Really Read-Only"

Updated Question: Is there a way to force dataadapter accept only commands which do not include any update/drop/create/delete/insert commands other than verifying the command.text before sending to dataadapter (otherwise throw exception). is there any such built-in functionality provided by dot net in datareader dataadapter or any other?
Note: DataReader returns results it also accepts update query and returns result. (I might be omitting some mistake but I am showing my update command just before executing reader and then show message after its success which is all going fine

Could you search the string for some keywords? Like CREATE,UPDATE, INSERT, DROP or if the query does not start with SELECT? Or is that too flimsy?
You might also want to create a login for this that the application uses that only has read capability. I don't know if the object has that property but you can make the server refuse the transaction.

All you need to do is ensure there are no INSERT, UPDATE, or DELETE statements prepared for the DataAdapter. Your code could look something like this:
var dataAdapter = new SqlDataAdapter("SELECT * FROM table", "connection string");
OR
var dataAdapter = new SqlDataAdapter("SELECT * FROM table", sqlConnectionObject);
And bam, you have a read-only data adapter.

If you just wanted a DataTable then the following method is short and reduces complexity:
public DataTable GetDataForSql(string sql, string connectionString)
{
using(SqlConnection connection = new SqlConnection(connectionString))
{
using(SqlCommand command = new SqlCommand())
{
command.CommandType = CommandType.Text;
command.Connection = connection;
command.CommandText = sql;
connection.Open();
using(SqlDataReader reader = command.ExecuteReader())
{
DataTable data = new DataTable();
data.Load(reader);
return data;
}
}
}
}
usage:
try{
DataTable results = GetDataForSql("SELECT * FROM Table;", ApplicationSettings["ConnectionString"]);
}
catch(Exception e)
{
//Logging
//Alert to user that command failed.
}
There isn't really a need to use the DataAdapter here - it's not really for what you want. Why even go to the bother of catching exceptions etc if the Update, Delete or Insert commands are used? It's not a great fit for what you want to do.
It's important to note that the SelectCommand property doesn't do anything special - when the SelectCommand is executed, it will still run whatever command is passed to it - it just expects a resultset to be returned and if no results are returned then it returns an empty dataset.
This means that (and you should do this anyway) you should explicitly grant only SELECT permissions to the tables you want people to be able to query.
EDIT
To answer your other question, SqlDataReader's are ReadOnly because they work via a Read-Only firehose style cursor. What this effectively means is:
while(reader.Read()) //Reads a row at a time moving forward through the resultset (`cursor`)
{
//Allowed
string name = reader.GetString(reader.GetOrdinal("name"));
//Not Allowed - the read only bit means you can't update the results as you move through them
reader.GetString(reader.GetOrdina("name")) = name;
}
It's read only because it doesn't allow you to update the records as you move through them. There is no reason why the sql they execute to get the resultset can't update data though.

If you have a read-only requirement, have your TextBox use a connection string that uses an account with only db_datareader permissions on the SQL database.
Otherwise, what's stopping the developer who is consuming your control from just connecting to the database and wreaking havoc using SqlCommand all on their own?

How to do a batch update?

I am wondering is there a way to do batch updating? I am using ms sql server 2005.
I saw away with the sqlDataAdaptor but it seems like you have to first the select statement with it, then fill some dataset and make changes to dataset.
Now I am using linq to sql to do the select so I want to try to keep it that way. However it is too slow to do massive updates. So is there away that I can keep my linq to sql(for the select part) but using something different to do the mass update?
Thanks
Edit
I am interested in this staging table way but I am not sure how to do it and still not clear how it will be faster since I don't understand how the update part works.
So can anyone show me how this would work and how to deal with concurrent connections?
Edit2
This was my latest attempt at trying to do a mass update using xml however it uses to much resources and my shared hosting does not allow it to go through. So I need a different way so thats why I am not looking into a staging table.
using (TestDataContext db = new TestDataContext())
{
UserTable[] testRecords = new UserTable[2];
for (int count = 0; count < 2; count++)
{
UserTable testRecord = new UserTable();
if (count == 1)
{
testRecord.CreateDate = new DateTime(2050, 5, 10);
testRecord.AnotherField = true;
}
else
{
testRecord.CreateDate = new DateTime(2015, 5, 10);
testRecord.AnotherField = false;
}
testRecords[count] = testRecord;
}
StringBuilder sBuilder = new StringBuilder();
System.IO.StringWriter sWriter = new System.IO.StringWriter(sBuilder);
XmlSerializer serializer = new XmlSerializer(typeof(UserTable[]));
serializer.Serialize(sWriter, testRecords);
using (SqlConnection con = new SqlConnection(connectionString))
{
string sprocName = "spTEST_UpdateTEST_TEST";
using (SqlCommand cmd = new SqlCommand(sprocName, con))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.CommandType = System.Data.CommandType.StoredProcedure;
SqlParameter param1 = new SqlParameter("#UpdatedProdData", SqlDbType.VarChar, int.MaxValue);
param1.Value = sBuilder.Remove(0, 41).ToString();
cmd.Parameters.Add(param1);
con.Open();
int result = cmd.ExecuteNonQuery();
con.Close();
}
}
}
# Fredrik Johansson I am not sure what your saying will work. Like it seems to me you want me to make a update statement for each record. I can't do that since I will have need update 1 to 50,000+ records and I will not know till that point.
Edit 3
So this is my SP now. I think it should be able to do concurrent connections but I wanted to make sure.
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[sp_MassUpdate]
#BatchNumber uniqueidentifier
AS
BEGIN
update Product
set ProductQty = 50
from Product prod
join StagingTbl stage on prod.ProductId = stage.ProductId
where stage.BatchNumber = #BatchNumber
DELETE FROM StagingTbl
WHERE BatchNumber = #BatchNumber
END

You can use the sqlDataAdapter to do a batch update. It dosen’t matter how you fill your dataset. L2SQL or whatever, you can use different methods to do the update. Just define the query to run using the data in your datatable.
The key here is the UpdateBatchSize. The dataadapter will send the updates in batches of whatever size you define. You need to expirement with this value to see what number works best, but typicaly numbers of 500-1000 do best. SQL can then optimize the update and execute a little faster. Note that when doing batchupdates, you cannot update the row source of the datatable.
I use this method to do updates of 10-100K and it usualy runs in under 2 minutes. It will depend on what you are updating though.
Sorry, this is in VB….
Using da As New SqlDataAdapter
da.UpdateCommand = conn.CreateCommand
da.UpdateCommand.CommandTimeout = 300
da.AcceptChangesDuringUpdate = False
da.ContinueUpdateOnError = False
da.UpdateBatchSize = 1000 ‘Expirement for best preformance
da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None 'Needed if UpdateBatchSize > 1
sql = "UPDATE YourTable"
sql += " SET YourField = #YourField"
sql += " WHERE ID = #ID"
da.UpdateCommand.CommandText = sql
da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None
da.UpdateCommand.Parameters.Clear()
da.UpdateCommand.Parameters.Add("#YourField", SqlDbType.SmallDateTime).SourceColumn = "YourField"
da.UpdateCommand.Parameters.Add("#ID", SqlDbType.SmallDateTime).SourceColumn = "ID"
da.Update(ds.Tables("YourTable”)
End Using
Another option is to bulkcopy to a temp table, and then run a query to update the main table from it. This may be faster.

As allonym said, Use SqlBulkCopy, which is very fast(I found speed improvements of over 200x - from 1500 secs to 6s). However you can use the DataTable and DataRows classes to provide data to SQlBulkCopy (which seems easier). Using SqlBulkCopy this way has the added advantage of bein .NET 3.0 compliant as well (Linq was added only in 3.5).
Checkout http://msdn.microsoft.com/en-us/library/ex21zs8x%28v=VS.100%29.aspx for some sample code.

Use SqlBulkCopy, which is lightning-fast. You'll need a custom IDataReader implementation which enumerates over your linq query results. Look at http://code.msdn.microsoft.com/LinqEntityDataReader for more info and some potentially suitable IDataReader code.

You have to work with the expression trees directly, but it's doable. In fact, it's already been done for you, you just have to download the source:
Batch Updates and Deletes with LINQ to SQL
The alternative is to just use stored procedures or ad-hoc SQL queries using the ExecuteMethodCall and ExecuteCommand methods of the DataContext.

You can use SqlDataAdapter to do a batch-update even if a datatable is filled manually/programmatically (from linq of any other source).
Just remember to manually set the RowState for the rows in the datatable. Use dataRow.SetModified() for this.