I'm currently trying to use C# to read through an SQL DB. To do so, I use OleDB with a select statement. This goes into a dataset, which then populates a data adapter. I then iterate through each row and calculate stuff.
First of all, I feel like there's a better/more efficient way of doing this because I NEVER actually write back to the SQL DB. I just calculate based on what I'm selecting.
Anyways, past a certain point I get out of memory errors and/or an error from Ssms.exe saying "a new guard page for the stack cannot be created."
From the other questions I've seen, I need to use DataReader but I can't seem to get it to work the same way as the data adapter (which I suppose isn't that surprising).
The code I have now:
OleDbConnection myConn = new OleDbConnection(#"SQLDB connection string here");
OleDbCommand cmd = new OleDbCommand();
cmd.CommandText = <selectstatement here>
cmd.Connection = myConn;
cmd.CommandTimeout = 0;
OleDbDataAdapter da = new OleDbDataAdapter(cmd);
DataSet ds = new DataSet();
da.Fill(ds);
myConn.Close();
foreach (DataTable table in ds.Tables)
{
foreach (DataRow dr in table.Rows)
{
//do stuff
I guess my question is twofold, like I said above. One would DataReader solve my problem and allow me to iterate through the data, and two how do I adapt the first code snippet above to support that?
Also, since I've seen it elsewhere, I'm using x64 on the application.
Related
Sorry for the vague title but I'm not sure how best to word it.
Basically, the issue that I'm having is that I initially was using an oledbconnection in C# to initiate a connection to a spreadsheet, query it, and load the results into a .net datatable. Unfortunately, it maxes out at 255 fields which I have around 600.
So, what I did instead was I created a dataset and tried to load 4 separate data tables with separate queries. Now for some reason, what's crazy to me is, if I load lets say, the first data table with 190 fields, and then I go on to query the spreadsheet again, if I go over that 250 mark (which I'd have 60 left), I get the following error:
An exception of type 'System.Data.OleDb.OleDbException' occurred in System.Data.dll but was not handled in user code
Additional information: No value given for one or more required parameters.
If I reduce the amount of fields in the second table to equal less than 250 though, it works. Anyway, is there a way that I can somehow clear the cache of the oledbconnection to somehow drop the first 190 fields or whatever is holding the result of the excel query so I can move on to the next? I tried doing a data adapter dispose but that didn't work, same issue. If I do a connection.dispose, I have to re-initialize the connection anyway. Is there a way I can still do this without having to drop the connection? Code below:
OleDbConnection cnn = new OleDbConnection(Settings.ExcelCN);
OleDbCommand fillT1 = new OleDbCommand(Settings.Excel_Proj_Select_1, cnn);
OleDbCommand fillT2 = new OleDbCommand(Settings.Excel_Proj_Select_2, cnn);
OleDbCommand fillT3 = new OleDbCommand(Settings.Excel_Proj_Select_3, cnn);
OleDbCommand fillT4 = new OleDbCommand(Settings.Excel_Proj_Select_4, cnn);
OleDbCommand updateCnt = new OleDbCommand(Settings.Excel_Update_Count_Select, cnn);
cnn.Open();
OleDbDataAdapter adp1 = new OleDbDataAdapter(fillT1);
OleDbDataAdapter adp2 = new OleDbDataAdapter(fillT2);
OleDbDataAdapter adp3 = new OleDbDataAdapter(fillT3);
OleDbDataAdapter adp4 = new OleDbDataAdapter(fillT4);
DataTable dt1 = new DataTable();
DataTable dt2 = new DataTable();
DataTable dt3 = new DataTable();
DataTable dt4 = new DataTable();
DataSet ds1 = new DataSet();
adp1.Fill(dt1);
ds1.Tables.Add(dt1);
adp2.Fill(dt2);
ds1.Tables.Add(dt2);
adp3.Fill(dt3);
ds1.Tables.Add(dt3);
adp4.Fill(dt4);
ds1.Tables.Add(dt4);
int rowcount = updateCnt.ExecuteNonQuery();
cnn.Close();
I'm using a datatable as the datasource of some dropdowns on a page, but have noticed that the page is very slow during the postbacks.
I've tracked it through to here:
DataTable dt = new DataTable();
dt.Load(sqlCmd.ExecuteReader()); // this takes ages
The sql command is a parametrised query, not a stored procedure (the return values and where are quite 'dynamic' so this wouldn't be practicable), but nevertheless a simple select union query.
Usually returns between 5 and 20 options per dropdown, depending on what's been selected on the other dropdowns.
When I run the query in the management studio, it's done in under a second. Here it can take up to 7 seconds per dropdown, with 6 dropdowns on the page it soon adds up.
I have also tried with a SqlDataAdapter:
SqlDataAdapter sqlDa = new SqlDataAdapter(sqlCmd);
sqlDa.Fill(dt); // this takes ages
but this was just as slow.
I have this on 2 different systems and on both have the same performance issues.
If anyone knows a better (faster) methord, or knows why this is so slow that would be great.
Not the best thread I've seen on the issue, but there's good links inside, & it's in my post history:
SQL Query that runs fine in SSMS runs very slow in ASP.NET
The SQL Optimizer sometimes likes to decide what's best & you'll have to break out your query through some tracing and logging of data execution plans. It may very well be something as buried as a bad index, or your query code might need optimization. Seeing as we don't have the query code, and having it may or may not be helpful. I'd recommend you follow the guides linked to in the above post and close your question.
here is an example on how you can load a DataTable very quickly notice how I show specific Columns that I want to return
private DataTable GetTableData()
{
string sql = "SELECT Id, FisrtName, LastName, Desc FROM MySqlTable";
using (SqlConnection myConnection = new SqlConnection(connectionString))
{
using (SqlCommand myCommand = new SqlCommand(sql, myConnection))
{
myConnection.Open();
using (SqlDataReader myReader = myCommand.ExecuteReader())
{
DataTable myTable = new DataTable();
myTable.Load(myReader);
myConnection.Close();
return myTable;
}
}
}
}
If you want to use DataAdapter to Fill the DataTable here is a simple example
private void FillAdapter()
{
using (SqlConnection conn = new SqlConnection(Your ConnectionString))
{
conn.Open();
using (SqlDataAdapter dataAdapt = new SqlDataAdapter("SELECT * FROM EmployeeIDs", conn))
{
DataTable dt = new DataTable();
dataAdapt.Fill(dt);
// dataGridView1.DataSource = dt;//if you want to display data in DataGridView
}
}
}
I have a datatable fetching values from a stored procedure written as below:
SqlConnection sqlcon = new SqlConnection(ConfigurationManager.ConnectionStrings["db"].ConnectionString);
sqlcon.Open();
DataTable dt = new DataTable("tmp");
DataSet ds = new DataSet();
SqlDataAdapter da = new SqlDataAdapter();
SqlCommand cmd = new SqlCommand("usp_abc", sqlcon);
cmd.CommandType = CommandType.StoredProcedure;
da.SelectCommand = cmd;
da.Fill(dt);
Now, I need to loop through the datatable and get the values of this datatable and pass it is as a parameter to my stored proc.
I believe such operation is better done in the DB if possible instead of going to the DB for each row in the datatable row collection.
You can use foreach to loop through the datatable, each DataRow representing a row in the returned result.
foreach(DataRow row in dt.Rows)
{
var col1 = row[0]; //access using column index/position
var firstNameCol = row["FirstName"].ToString(); //access through column name
}
You can throw more light on what you want to do if it's possible to move it to DB (using stored procedure)
DataTable.Rows
Update: Passing value to stored proc using command.Parameters
cmd.Parameters.AddWithValue("#FirstName", firstNameCol);
You can use:
foreach (DataRow row in dt.Rows)
{
foreach(object o in row.ItemArray)
{
//do something with o
}
}
To set parameters in a sproc:
SqlCommand cmd = new SqlCommand("usp_abc", sqlcon);
cmd.CommandType = CommandType.StoredProcedure;
cmd.parameters.AddWithValue("#foo", "bar");
cmd.parameters.AddWithValue("#plugh", Integer.valueOf(17));
This will set the parameter "#foo" to the value "bar" and the parameter "#plugh" to the value 17. The value can be String, Integer, Boolean, probably others.
Read the documentation on the SqlParameterCollection object for other functions to set parameters. There are others that let you explicitly set the SQL data type, instead of making it guess a SQL datatype from your C# datatype. The only time I've ever needed this was when I wanted to set an Image value.
Side note that tripped me up early on: You cannot pass in NULL. If you want a value to be null, you have to pass in System.DBNull.Value. There is no doubt some good reason for this but I find it an annoying pain.
If you only need the data to pass as a parameter to a stored procedure consider DataReader.
DataTable has overhead.
DataReader is more efficient and simpler syntax.
SqlDataReader Class
If you need the DataTable for other stuff (like bind to a DataGrid) then stay with DataTable and the answer from codingbiz +1 seems correct to me.
Servy feels strongly the answer should include how to reference a values in a DataRow.
You can do it by ordinal position or column name.
row[0];
row["colx"];
But you already have that answer from condigbiz and should give the check to codingbiz if the is what you use.
SqlDataReader reader = command.ExecuteReader();
// Call Read before accessing data.
while (reader.Read())
{
if (!reader.IsDBnull(0))
{
cmd.Parameters.AddWithValue("#param1", Reader.GetString(0));
}
}
// Call Close when done reading.
reader.Close();
OP commented Reader would lock table.
You could use with (no lock) or read into a List<>;
But you could also just stay with DataTable and that seems like the best plan to me.
This question already has answers here:
Is datareader quicker than dataset when populating a datatable?
(9 answers)
Closed 10 years ago.
I am using the following code (Variant DataReader):
public DataTable dtFromDataReader(list<String> lstStrings)
{
OleDBConn_.Open();
using (OleDbCommand cmd = new OleDbCommand())
{
DataTable dt = new DataTable();
OleDbDataReader reader = null;
cmd.Connection = OleDBConn_;
cmd.CommandText = "SELECT * from TableX where SUID=?";
foreach (String aString in lstStrings)
{
cmd.Parameters.AddWithValue("?", aNode.SUID);
reader = cmd.ExecuteReader();
if (reader != null)
dt.Load(reader);
cmd.Parameters.Clear();
}
return dt;
}
}
and compare it to (Variant DataAdapter):
public DataTable dtFromDataAdapter(list<String> lstStrings)
{
dt = new DataTable();
foreach (string aString in lstStrings)
{
sOledb_statement = String.Concat("SELECT * FROM TableX where SUID='", aString, "'");
OleDbDataAdapter oleDbAdapter;
using (oleDbAdapter = new OleDbDataAdapter(sOledb_statement, OleDBConn_))
{
GetOleDbRows = oleDbAdapter.Fill(dt);
}
}
}
When i connect to an offline database (microsoft access) my reading time is (~1.5k retrieved items):
DataReader 420 ms
DataAdapter 5613 ms
When reading from oracle server (~30k retrieved items):
DataReader 323845 ms
DataAdapter 204153 ms
(several tests, times do not change much)
Even changing the order of the commands (dataadapter before datareader) didn't change much (i thought that there may have been some precaching..).
I thought DataTable.Load should be somewhat faster than DataAdapter.Fill?
And i still believe, even though i see the results, that it should be faster. Where am i losing my time? (There are no unhandled exceptions..)
Your comparison isn't really an Adapter vs DataReader with the way you have the code setup. You are really comparing the Adapter.Fill vs DataTable.Load methods.
The DataReader would normally be faster on a per-record basis because you would be traversing the records one at a time and can react accordingly when you read each record.
Since you are returning a DataTable in both instances, the Adapter.Fill method would probably be the optimal choice to use. It was designed to do just that.
I don't use DataSets much. Usually find myself using an ORM or just a basic sqlReader.Read() followed by some GetValues(). I'm working on some legacy code that has DataSets all over the place, and while fixing a bug was trying to DRY some of it up.
However, I can't seem to actually get the data loaded into a non-typed DataSet.
public static DataSet ExecuteStoredProcedure(string storedProcedure, DBEnum db, IEnumerable<SqlParameter> parameters)
{
DataSet result = new DataSet();
using (SqlConnection connection = SqlHelper.GetSqlConnection(db))
{
SqlCommand command = connection.CreateCommand();
command.CommandType = CommandType.StoredProcedure;
command.CommandText = storedProcedure;
if (parameters != null)
foreach (SqlParameter parameter in parameters)
command.Parameters.Add(parameter);
connection.Open();
DataTable table = new DataTable();
using (SqlDataReader reader = command.ExecuteReader())
{
table.Load(reader);
}
result = table.DataSet; // table.DataSet is always empty!
}
return result;
}
I assumed table.Load(reader) does all the necessary reader.Read() calls ... but I went ahead and tried it both with and without reader.Read() before the table.Load(), to no avail.
I know that the stored procedure being called is actually returning data. If I do something like this, I see the data just fine:
using(SqlDataReader reader = command.ExecuteReader())
{
reader.Read();
object test = reader.GetValue(0); // returns the expected value
}
Seems like I'm missing something simple here, but I've been scratching my head over this one for a while now.
This is in .NET 3.5.
If you can, I would suggest using a SqlDataAdapter to populate the DataTable
using(SqlDataAdapter sqlDA = new SqlDataAdapter(command))
{
sqlDA.Fill(table);
}
You logic shows the DataTable being loaded with data from the reader but DataTable is never added to a dataset.
I believe the dataset should be created first. In fact, you could use DataSet.Load instead of the DataTable.Load. The DataSet.Load should create data tables, but it won't work the other way around.
A DataSet contains DataTables not the other way round. So if you want to create a DataSet to return then you'll probably want to create a new dataset and then add your DataTable into it before returning it. If a DataTable is not in a DataSet then the reference to its parent dataset will always be null.
That having been said I do also recommend Jason Evans' suggestion of using teh SqlDataAdapter.
Think of a dataset as a collection of tables and optionally information about the relationships between them.
In your example code you are creating an independent Table that does not belong to a DataSet. To pragmatically create a table that is part of a dataset you could do the following:
DataSet ds = new DataSet();
DataTable dt = ds.Tables.Add();
//or
ds.Tables.Add(MyAlreadyCreatedTable);
Jason Evans above is also correct, populating DataTables and DataSets is much simpler using SqlDataAdaptors as he demonstrated.
Finally, the Method as you have it written is meant to return a DataSet. But it only captures a single result set from the stored procedure it is calling. It's possible that a procedure could return any number of separate results.
All you should need to do is change the following:
DataTable table = new DataTable();
using (SqlDataReader reader = command.ExecuteReader())
{
table.Load(reader);
}
to
//you can skip creating a new DataTable object
using (SqlDataAdapter da = new SqlDataAdapter(command))
{
da.Fill(result); // the result set you created at the top
}