Develop Reference

DataTable Takes Forever to Load Data from SqlDataReader

I am loading data from an MS SQL Server Table using the following code:
using (SqlDataReader rdr = cmd.ExecuteReader())
{
if (rdr.HasRows)
{
dt.Load(rdr); //takes forever to load
}
if (dt.Rows.Count > 0 && !dt.HasErrors)
{
Parallel.For (0, dt.Rows.Count, i =>
{
byte[] docBytes = (byte[])(dt.Rows[i]["DocObject"]); File.WriteAllBytes(Path.Combine(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "Documents\\"), $"{dt.Rows[i]["FileName"].ToString().ToLower()}"), docBytes);
});
}
}
}
The SQL query executes in less than one second. The data contains an SQL image column that holds binary document data. I used Stopwatch from System.Diagnostics to time the execution and found that this single dt.Load(rdr) statement is taking approximately 5 minutes to load about 5,000 records. My application needs to load several millions of rows and at this rate the app would be unusable. This is a Windows Forms application built using standard Windows Forms. Any ideas why dt.Load(rdr) takes forever? Any ideas on either rewriting this code or improving its performance would be greatly appreciated.

Try something like this, instead of loading all the rows into memory on the client:
using (SqlDataReader rdr = cmd.ExecuteReader(CommandBehavior.SequentialAccess))
{
while (rdr.Read())
{
string fn = rdr.GetString(0);
using (var rs = rdr.GetStream(1))
{
var fileName = $"c:\\temp\\{fn}.txt";
using (var fs = File.OpenWrite(fileName))
{
rs.CopyTo(fs);
}
Console.WriteLine(fileName);
}
}
}

Below code is untested.It is just an idea.
Another approach will be to define entity class and populate the list with SqldataReader.And Do not use DataTable at all.
Also one should close Database connection as soon as possible.So while fetching do not do other work.
Hope you are using connection pool in connection string
public class Example
{
public byte DocObject {get;set;}
public string FileName {get;set;}
}
List<Example> objList=new List<Example>();
using (SqlDataReader rdr = cmd.ExecuteReader())
{
while (rdr.Read())
{
Example obj=new Example();
obj.DocObject=(byte[])rdr["DocObject"] //suitable cast
obj.FileName =rdr["FileName "].toSting() //suitable cast
objList.Add(obj);
}
}
}
if (objList.Count > 0)
{
Parallel.For (0, objList.Count, i =>
{
byte[] docBytes = (byte[])(objList[i]["DocObject"]); File.WriteAllBytes(Path.Combine(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "Documents\\"), $"{objList[i]["FileName"].ToString().ToLower()}"), docBytes);
});
}
}

ODBCDataReader has rows but can not access data

So in C#, I have an ODBCDataReader that returns that it has rows, but when I try to access the data it returns an object not set to a reference of an object error. I tested the sql directly on the db and it does return a row without any nulls
OdbcDataReader results;
try
{
// Initialize & open odbc connection
using (OdbcConnection conn = new OdbcConnection(connectionString.ToString()))
{
conn.Open();
// Initialiaze odbc command object
using (OdbcCommand comm = new OdbcCommand(query.ToString(), conn))
{
results = comm.ExecuteReader();
}
}
}
catch
{
//detailed error messaging here (which does not get hit)
}
temp = results;
if (temp.HasRows == false)
{
//error messaging here does not get hit.
}
while (temp.Read())
{
try
{
//I attempted to access the data by creating an object array:
object [] objarray = new object[temp.FieldCount)
temp.GetValues(objarray); //this causes error
}
catch{ // error is caught here "object not set to a reference of an object" }
for (i = 0; i < temp.FieldCount; i++)
{
try
{
//I also attempted other ways to access the data including:
temp[i].ToString(); // this causes error
temp.GetInt32(i).ToString(); // this causes error
temp.GetName(i); //this causes error
}
catch
{
// error is caught here "object not set to a reference of an object"
}
}
}

You are using it outside the using blocks. Move the part where you use [results] inside the using blocks (immediately after the ExecuteReader() call) and you should be in a much better place.

I ran into this same issue. The problem in my case was that I was not binding my parameters correctly. I was binding using #:
SELECT * FROM MyTable WHERE MyField = #MyField
For some reason, this is valid in MySQL and doesn't produce an error, but will not return data. The solution was to bind using ?:
SELECT * FROM MyTable WHERE MyField = ?
Then in C# bind the parameters:
cmd.Parameters.AddWithValue("#MyField", myFieldValue);
Old question, but it's the first result on Google and unanswered. Hope it's helpful.

Class function to return different objects

In short I am writing a class handler to handle to database integration of some software I am writing for myself, however as there is not always a connection to the remote database I thought I would use SQLCE to create a local database buffer so when a connection is made the changes can be synchronized.
So far it is going well except for the parameters. The function I am looking to call is shown below however this function is complaining about invalid arguments.
public Object run(string query, List<Object> dbparams = null)
{
if (MyDB.isConnected())
{
return MyDB.run(query, dbparams);
}
else
{
SqlCeCommand sql = _OfflineConnection.CreateCommand();
sql.CommandText = query;
if (dbparams.Count > 0)
{
sql.Parameters.AddRange(dbparams.ToArray());
}
return sql;
}
}
MyDB.run is the exact same code as in the else statement except for mysql, the line that it is moaning about is the return mydb.run as the mydb class is expecting the dbparams list to be of mysqlparameters.
Does anyone know how I can achieve this? I attempted to use LINQ to do a convert but that failed miserably.
EDIT
At present I have the following working but I am sure there is a cleaner solution
public Object run(string query, List<Object> dbparams = null)
{
if (MyDB.isConnected())
{
List<MySqlParameter> mydbparams = null;
for (int i = 0; i < dbparams.Count; i++)
{
mydbparams.Add((MySqlParameter)dbparams[i]);
}
return MyDB.run(query, mydbparams);
}
else
{
SqlCeCommand sql = _OfflineConnection.CreateCommand();
sql.CommandText = query;
if (dbparams.Count > 0)
{
sql.Parameters.AddRange(dbparams.ToArray());
}
return sql;
}
}

A bit cleaner solution would be
mydbparams = dbparams.Cast<MySqlParameters>().ToList();
Also, you should check for and handle the null condition of dbparams.

Reading and writing data into sql server simultaneously

I have a service which continuously writes data in a separate thread into SQL database.Now from the same service if i am trying to read from the same table, since i already am writing into it,I get this exception : There is already an open DataReader associated with this Command which must be closed first.
So can anyone help me how to do this simultaneously?
Here s my code for reading data:
public Collection ReadData(string query)
{
{
_result = new Collection<string[]>();
string[] tempResult;
SqlDataReader _readerRead;
using (_command = new SqlCommand(query, _readConnection))
{
_readerRead = _command.ExecuteReader();
while (_readerRead.Read())
{
tempResult = new string[4];
tempResult[0] = _reader[0].ToString();
tempResult[1] = _reader[1].ToString();
tempResult[2] = _reader[2].ToString();
tempResult[3] = _reader[3].ToString();
_result.Add(tempResult);
//Console.WriteLine("Name : {0} Type : {1} Value : {2} timestamp : {3}", _reader[0], _reader[1], _reader[2], _reader[3]);
}
if (_readerRead != null)
{
_readerRead.Close();
}
_readConnection.Close();
return _result;
}
}
}
and here it is for writing to it :
public void WriteData(Collection<TagInfo> tagInfoList)
{
int i = 0;
for (i = 0; i < tagInfoList.Count; i++)
{
using( _command = new SqlCommand(insert statement here)
{
_command.Parameters.AddWithValue("Name", tagInfoList[i].Name);
_command.Parameters.AddWithValue("Type", tagInfoList[i].TagType);
_command.Parameters.AddWithValue("Value", tagInfoList[i].Value);
_reader = _command.ExecuteReader();
if (_reader != null)
{
_reader.Close();
}
}
}
}

You need a different SQLConnection to the database for your writer. You cannot use the same db connection for both.

Although its possible to do, using a separate connection I would question why you need to do this.
If you are reading and writing data to one table in the same service you will be placing unnecessary load on one SQL table, and depending on the number of queries you intend to make this could cause you problems. If you already have this data (in a different thread) why not Marshall the data from the background thread to where you need it as you write it into the database, and you don't need to read the data anymore.
However.... it is difficult to give an fair answer without seeing the code/what you are looking to achieve.

Storing reader information in C#

I know what I asking might not make a lot of sense for C# experts but I'll explain what I want to do and then you can suggest me how to do it in a better way if you want ok?
I have a C# class called DatabaseManager that deals with different MySQL queries (ado.net NET connector, not linq or any kind of ActiveRecord-ish library).
I am doing something like
categories = db_manager.getCategories();
The list of categories is quite small (10 items) so I'd like to know what's the best way of accessing the retrieved information without a lot of additional code.
Right now I'm using a Struct to store the information but I'm sure there's a better way of doing this.
Here's my code:
public struct Category
{
public string name;
}
internal ArrayList getCategories()
{
ArrayList categories = new ArrayList();
MySqlDataReader reader;
Category category_info;
try
{
conn.Open();
reader = category_query.ExecuteReader();
while (reader.Read())
{
category_info = new Category();
category_info.name = reader["name"].ToString();
categories.Add(category_info);
}
reader.Close();
conn.Close();
}
catch (MySqlException e)
{
Console.WriteLine("ERROR " + e.ToString());
}
return categories;
}

Example:
public IEnumerable<Category> GetCategories()
{
using (var connection = new MySqlConnection("CONNECTION STRING"))
using (var command = new MySqlCommand("SELECT name FROM categories", connection))
{
connection.Open();
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
yield return new Category { name = reader.GetString(0) };
}
}
}
}
Remarks:
Let ADO.NET connection pooling do the right work for you (avoid storing connections in static fields, etc...)
Always make sure to properly dispose unmanaged resources (using "using" in C#)
Always return the lowest interface in the hierarchy from your public methods (in this case IEnumerable<Category>).
Leave the callers handle exceptions and logging. These are crosscutting concerns and should not be mixed with your DB access code.

The first thing I would do is to replace you use of ArrayList with List that will provide compile-time type checkig for your use of the category list (so you will not have to type cast it when using it in your code).

There's nothing wrong with returning them in an like this. However, a few things stand out:
Your catch block logs the error but
then returns either an empty array or
a partially populated array. This
probably isn't a good idea
If an exception is thrown in the try
block you won't close the connection
or dispose of the reader. Consider
the using() statement.
You should use the generic types
(List<>) instead of ArrayList.

From your code I guess you are using .NET 1.1, becuase you are not using the power of generics.
1) Using a struct that only contains a string is an overkill. Just create an arraylist of strings (or with generics a List )
2) When an exception occurs in your try block, you leave your connection and reader open... Use this instead:
try
{
conn.open();
//more code
}
catch (MySqlException e) { // code
}
finally {
conn.close()
if (reader != null)
reader.close();
}