Develop Reference

C# - SqlCommand.ExecuteReader() gets more entries from database, then the same query directly

I wrote a query in SQL Server Management Studio and saved it. Then I wrote this code:
static public List<DataRow> SendQuery(string query, SqlParameter[] parameters)
{
using (var connection = new SqlConnection(connectionString))
{
connection.Open();
SqlCommand sqlCommand = new SqlCommand(query, connection);
foreach (var parameter in parameters)
sqlCommand.Parameters.Add(parameter);
var reader = sqlCommand.ExecuteReader();
var dataTable = new DataTable();
dataTable.Load(reader);
List<DataRow> rows = dataTable.AsEnumerable().ToList();
return rows;
}
}
and ran it. To my surprise rows had more items then actual number of rows that I got when ran the same query in SSMS.
Comparing data is not an option because there is too much of it. Why is it happening and what can I do about it?

Thanks to Bora Karaca suggestion I figured it out. First of all I can't post query here, because it exposes my DB structure which is IP of my employee. In turned out that passing Date parameter to the query is a bit tricky since it passes it with time 0:00:00. When I uses 2023-01-31 in C# it somehow converted it to the end of the day with 23:59:59 but passed 0:00:00 as parameter.

C# vs Python query against SQLite database

I have a SQLite database that has a single table with 18 million rows and 24 columns. The schema is along the lines of,
Date (VARCHAR)
CompanyName (VARCHAR)
Amount (REAL)
AggCode (VARCHAR)
Level1 ... Level20 (VARCHAR)
I am querying the table two ways - first with a Python script, and then with a C# function that is exposed to Excel with ExcelDNA (ultimately I'd prefer to use Excel to run the queries as some queries will return rows of data that need to be further manipulated).
I find that Python usually outperforms the Excel add-in by a factor of 3-5x, and I was wondering if there was something wrong with my code. Sample query below,
query = "SELECT Sum(Amount) FROM my WHERE Level9='STIN_J' AND (AggCode='R_REAL' AND Date='05DEC2016')"
The queries are usually run combining the fields Level9, Level5, AggCode, Date, CompanyName in the WHERE clause. So apart from the raw table, I have also configured the following four indices,
CREATE INDEX idx1 on my(Level09, AggCode);
CREATE INDEX idx2 on my(Level05, AggCode);
CREATE INDEX idx3 on my(CompanyName, AggCode);
CREATE INDEX idx4 on my(Date, AggCode);
This is the sample Python code to run a query,
import sqlite3 as lite
...
con = lite.connect("C:\temp\my.db")
cur = con.cursor()
cur.execute(query)
data = cur.fetchall
for row in data:
for i in range(len(row)):
print row[i],
print "\t",
On the whole this code works rather well.
This is the sample C# code to run a query,
using System.Data.SQLite;
...
string constr = "Data Source=C:\temp\my.db;Version=3;";
SQLiteConnection conn = new SQLiteConnection(constr);
SQLiteCommand command = new SQLiteCommand(query, conn);
conn.Open();
SQLiteDataAdapter sda = new SQLiteDataAdapter(command);
DataTable dt = new DataTable();
sda.Fill(dt);
sda.Dispose();
command.Dispose();
conn.Dispose();
object[,] ret = new object[dt.Rows.Count, dt.Columns.Count];
int rowCount = 0;
int colCount = 0;
foreach (DataRow row in dt.Rows)
{
foreach (DataColumn col in dt.Columns)
{
ret[rowCount, colCount] = col.ColumnName;
colCount++;
}
rowCount++;
}
...
return ret;
Are either the Python or C# codes sub-optimal? For example, should I use SQLiteDataReader instead of SQLiteDataAdapter? Would appreciate any thoughts.
The result sets themselves are pretty small, in some cases just a single number, so I wouldn't have thought that ExcelDNA was adding headroom to the process. Sample Python query take about 15 seconds, whereas C# up to 1 minute.
Finally how would amending the PRAGMA settings affect performance? Any suggestions on some generic settings given my top priority is the speed of the query?
Also, any suggestions on how to actually implement these settings in Python, C# or persist them would be much appreciated.

I wasn't purging the datatable, SQL command or connection properly. Once I stuck the Dispose methods into a finally block, the response of the C# code increased drastically and outperforming Python in most cases.

Calling a stored procedure results in an incorrectly empty DataTable

I have a functioning stored procedure which returns the correct data when executed manually. There are several rows of data in the output. However, the following code I have is always resulting in no rows of data being added to the DataTable.
var commandString = string.Format(#"EXEC MyStoredProcedure {0}", SomeParameter);
var dataTable = new DataTable();
using (var connection = new SqlConnection(ConnectionString))
{
connection.Open();
using (var adapter = new SqlDataAdapter(commandString, ConnectionString))
{
using (new SqlCommandBuilder(adapter))
{
adapter.Fill(dataTable);
adapter.Update(dataTable);
}
}
}
var result = (from DataRow row in dataTable.Rows
select new MyModelClass
{
SomeString = (string) row["SomeString"],
SomeValue = (string) row["SomeValue"],
}).ToList();
Debug.WriteLine("Results: " + result.Count);
I am not sure why the code is resulting in no rows of data. Where am I going wrong? I suspect it is because I have an incorrect understanding of how DataTable works. How should I fix the code?

Basically, your code should look something like this:
string ConnectionString = "<Your connection string here>";
string procedureName = "<your stored procedure name here>";
string ParamName = "#<Parameter name>"; // NOTE: the '#' is INSIDE the string!
DataSet ds = new DataSet();
using (var connection = new SqlConnection(ConnectionString))
{
var cmd = new SqlCommand(procedureName, connection);
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add(ParamName, SqlDbType.Int).Value = 5;
using (var adapter = new SqlDataAdapter(cmd))
{
adapter.Fill(ds);
}
}

Sorry to be late to the party but I just want to confirm something.
I noticed that in the OP the asker uses the adapter to fill a DataTable
In the accepted answer, however, the adapter is used to fill a DataSet
Well, that's great - I also discovered this myself. The DataTable will contain 0 rows ase well as 0 columns, but use it to fill a DataSet instead, and it works fine.
So the real question here is: Why does it work with a DataSet when it doesn't with a DataTable?
I have a suspicion and I'm really here to try and prove it right or wrong. In my case I found that the SP started returning this weird empty DataTable as soon as my SP contained anything like a DELETE or an INSERT. In my case, where the original SP was just a simple SELECT from some data, I had to add some complexity which meant I first had to populate a Table Variable using INSERTs, and then SELECT from that. And that's when I ran into this problem.
So, #user9993, if you're still around can you just tell me.... does your SP do more than just SELECT? Does it do updates or deletes even if it's just to something like a Temporary Table or a Table Variable?

Boost select statement performance of sqlite database in C#

I have a sqlite database consist of 50 columns and more than 1.2 million rows. I'm using System.Data.Sqlite to work with Visual Studio 2013 C# language.
I used a very simple code to retrieve my data from database but it is taking too much time.
private SQLiteConnection sqlite;
public MySqlite(string path)
{
sqlite = new SQLiteConnection("Data Source="+path+"\\DBName.sqlite");
}
public DataTable selectQuery(string query)
{
SQLiteDataAdapter ad;
DataTable dt = new DataTable();
try
{
SQLiteCommand cmd;
sqlite.Open();
cmd = sqlite.CreateCommand();
cmd.CommandText = query; //set the passed query
ad = new SQLiteDataAdapter(cmd);
ad.Fill(dt); //fill the datasource
}
catch (SQLiteException ex)
{
//exception code here.
}
sqlite.Close();
return dt;
}
And, the select statement is:
select * from table
As I told you, it is a very simple code.
I want to know how to boost select operation performance to get the appropriate result. for this code the process takes up to 1 minute which I want to get to less than 1 second.
and another thing is that there seems to be some tags for configuring sqlite database but I don't know where to apply them. could some one tell me how to configure sqlite database with System.Data.Sqlite;

Consider narrowing your result set by getting necessary columns or paging.

How to know when to stop filling an OracleDataAdapter

I'm using the OPD.NET dll in a project that is accessing oracle.
Users can type in any SQL into a text box, that is then executed against the db. I've been trying to use the OracleDataAdapter to populate a datatable with the resultset, but I want to be able to return the resultset in stages (for large select queries).
An example of my problem is...
If a select query returns 13 rows of data, the code snippet below will execute without issue until the fourth time oda.Fill (start row is 15 which doesn't exist) is called, I presume because it is calling into a reader that has closed or something similar.
It then will throw a System.InvalidOperationException with the message - Operation is not valid due to the current state of the object.
How can I find out how many rows in total the command will eventually contain (so that I don't encounter the exception)?
OracleDataAdapter oda = new OracleDataAdapter(oracleCommand);
oda.Requery = false;
var dts = new DataTable[] { dt };
DataTable dt = new DataTable();
oda.Fill(0, 5, dts);
var a = dts[0].Rows.Count;
oda.Fill(a, 5, dts);
var b = dts[0].Rows.Count;
oda.Fill(b, 5, dts);
var c = dts[0].Rows.Count;
oda.Fill(c, 5, dts);
var d = dts[0].Rows.Count;
Note: I've omitted the connection and oracle command objects for brevity.
EDIT 1:
I've just thought I could just wrap the SQL entered by the user in another query and execute it...
SELECT COUNT(*) FROM (...intial query in here...)
but this isn't exactly a clean solution, and surely there is a method somewhere that I haven't seen?
Thanks in advance.

For paging in Oracle, see: http://www.oracle.com/technology/oramag/oracle/06-sep/o56asktom.html
There is no way to know the record set count without running a separate count(*) query. This is by design. The DataReader and DataAdapter are forward-only, read only.
If efficiency is a concern (i.e., large record sets), one should let the database do the paging and not ask the OracleDataAdapter to run the full query. Imagine if Google filled a DataTable with all 1M+ results for each user search!! The following article addresses this concern, although the examples are in sql:
http://www.asp.net/data-access/tutorials/efficiently-paging-through-large-amounts-of-data-cs
I've revised my example below to allow paging on any sql query. The calling procedure is responsible for keeping track of the user's current page and page size. If the result set is less than the requested page size, there are no more pages.
Of course, running custom sql from user input is a huge security risk. But that wasn't the question at hand.
Good luck! --Brett
DataTable GetReport(string sql, int pageIndex, int pageSize)
{
DataTable table = new DataTable();
int rowStart = pageIndex * pageSize + 1;
int rowEnd = (pageIndex + 1) * pageSize;
string qry = string.Format(
#"select *
from (select rownum ""ROWNUM"", a.*
from ({0}) a
where rownum <= :rowEnd)
where ""ROWNUM"" >= :rowStart
", sql);
try
{
using (OracleConnection conn = new OracleConnection(_connStr))
{
OracleCommand cmd = new OracleCommand(qry, conn);
cmd.Parameters.Add(":rowEnd", OracleDbType.Int32).Value = rowEnd;
cmd.Parameters.Add(":rowStart", OracleDbType.Int32).Value = rowStart;
cmd.CommandType = CommandType.Text;
conn.Open();
OracleDataAdapter oda = new OracleDataAdapter(cmd);
oda.Fill(table);
}
}
catch (Exception)
{
throw;
}
return table;
}

You could add an Analytic COUNT to your query:
SELECT foo, bar, COUNT(*) OVER () TheCount WHERE ...;
That way the count of the entire query is returned with each row in TheCount, and you could set your loop to terminate accordingly.

To gain control over Fill DataTable Loop you need own the loop.
Then build your own Function to Fill DataTable using OracleDataReader.
To get Columns information, you can use dataReader.GetSchemaTable
To Fill the Table:
MyTable.BeginLoadData
Dim Values(mySchema.rows.count-1)
Do while myReader.read
MyReader.GetValues(Values)
MyTable.Rows.add(Values)
'Include here your control over load Count
Loop
MyTable.EndLoadData