Develop Reference

SQLiteDataReader Appears to be Holding DB Connection Open

Synopsis
I have a small local SQLite DB with a set records that are used to populate a ComboBoxList. The list is populated, and the DB connection released. No DB locking issues observed.
Then when a record is selected from the list, that record is retrieved and used to populate a LocalDBInstallRecord object. However, after populating the object, the DB connection seems to remain open and prevents any further access.
The Code
public LocalDBInstallRecord GetRecord(string recordID)
{
LocalDBInstallRecord lir = new LocalDBInstallRecord();
string query = "SELECT * FROM InstallationRecords WHERE RecordID = " + recordID;
using (SQLiteConnection localDBConnection = new SQLiteConnection(ConnectionStr))
{
localDBConnection.Open();
using (SQLiteCommand cmd = new SQLiteCommand(query, localDBConnection))
{
using (SQLiteDataReader rdr = cmd.ExecuteReader(CommandBehavior.KeyInfo))
{
while (rdr.Read())
{
lir.RecordID = (uint)rdr.GetInt32(rdr.GetOrdinal("RecordID"));
lir.DateCreated = rdr.GetDateTime(rdr.GetOrdinal("DateCreated"));
lir.Model = CheckDBNull(rdr, "Model");
lir.SCFirmwareVer = CheckDBNull(rdr, "SCFirmwareVer");
lir.DispFirmwareVer = CheckDBNull(rdr, "DispFirmwareVer");
lir.UCCFirmwareVer = CheckDBNull(rdr, "UCCFirmwareVer");
lir.Title = CheckDBNull(rdr, "Title");
lir.FOC = rdr.GetBoolean(rdr.GetOrdinal("FOC"));
lir.MCManufacturer = CheckDBNull(rdr, "MCManufacturer");
lir.MCType = CheckDBNull(rdr, "MCType");
lir.RailConvertor = CheckDBNull(rdr, "RailConvertor");
lir.PlantID = CheckDBNull(rdr, "PlantID");
lir.PlantOwner = CheckDBNull(rdr, "PlantOwner");
lir.PlantOwnerID = CheckDBNull(rdr, "PlantOwnerID");
lir.BoomLength = rdr.GetFloat(rdr.GetOrdinal("BoomLength"));
lir.ArticLength = rdr.GetFloat(rdr.GetOrdinal("ArticLength"));
lir.DipperLength = rdr.GetFloat(rdr.GetOrdinal("DipperLength"));
lir.MachineRecord = ReadDBBlob(rdr, "MachineRecord");
lir.DutyRecord = ReadDBBlob(rdr, "DutyRecord");
lir.CalibRecord = ReadDBBlob(rdr, "CalibRecord");
}
}
}
}
return lir;
}
Called Functions...
CheckDBNull
private static string CheckDBNull(SQLiteDataReader rdr, string col)
{
string str = null;
if (!rdr.IsDBNull(rdr.GetOrdinal(col)))
str = rdr.GetString(rdr.GetOrdinal(col));
return str;
}
ReadDBBlob
private static byte[] ReadDBBlob(SQLiteDataReader rdr, string col)
{
byte[] data = null;
if (!rdr.IsDBNull(rdr.GetOrdinal(col)))
{
SQLiteBlob blob = rdr.GetBlob(rdr.GetOrdinal(col), true);
data = new byte[blob.GetCount()];
blob.Read(data, blob.GetCount(), 0);
}
return data;
}
My Thoughts
I'm sure there's easier ways to load the object, but that's not the issue...
I think the DB connection is being held open because the SQLiteDataReader can't be disposed of cleanly. I suspect that something in this function (in the LocalDBInstallRecord object) has got hold of an object by reference. I just can't see what.
I've tried not accessing the DateTime object in the record, but that's not it.
Should I be using a different method to access the DB in this case?
Any suggestions greatly appreciated, I think it needs a fresh set of eyes. Thanks.
Further Research...
I've found that if I try and us the SQLite DB with another tool, whilst the GetRecord() function has hold of the reader, I'm able to read the DB, but can't write. I get a database locked (RELEASE 'RESTOREPOINT';) error message from "DB Browser SQLite" application. But I think that's just because it's waiting for the reader to release.

The problem came in the function ReadDBBlob. The SQLiteBlob is IDisposable, and it's use wasn't encapsulated in a using clause. As a result the SQLiteBlob wasn't being diposed of, and this held the reader and hence kept the connection open...
New ReadDBBlob
private static byte[] ReadDBBlob(SQLiteDataReader rdr, string col)
{
byte[] data = null;
if (!rdr.IsDBNull(rdr.GetOrdinal(col)))
{
using (SQLiteBlob blob = rdr.GetBlob(rdr.GetOrdinal(col), true))
{
data = new byte[blob.GetCount()];
blob.Read(data, blob.GetCount(), 0);
}
}
return data;
}
I hope this saves someone some time.

WCF streaming large number objects

I have a WCF service that query a database and returns a large number of records. There is so many records, that the server runs out of memory and fails before it can return.
So I want to send the records back as I fetch them from the database, or a set number back at a time.
For additional clarity, I cannot collect call records fetched into a collection on the server, as the server runs out of memory before I have collected all the records. I want to try and find away to send them back one by one or in chunks, in one call.
For example, in chunks:
Fetch first 1000 records
Add to collection
Send collection to client
Clear collection
Fetch next 1000 records, and repeat from step 2
So the idea I have how the web service code will look something like this:
Public IEnumerable<Customer> GetAllCustomers()
{
// Setup Query
string query = PrepareQuery();
// Create Connection
connection = new SqlConnection(ConnectionString);
connection.Open();
var sqlcommand = connection.CreateCommand();
sqlcommand.CommandText = query.ToString();
// Read Results
var reader = sqlcommand.ExecuteReader();
while (reader.Read())
{
Customer customer = new Customer();
foreach (var column in Columns)
{
int fieldIndex = reader.GetOrdinal(column);
object value = reader.GetValue(fieldIndex);
customer[column.Name] = value;
}
yield return customer;
}
}
I don't want to consider paging as the Order By on the SQL server is slow.
Looking for way to do this in WCF

I think you answer your own question. There are 2 ways to do it, stream or chunk.
You can do streaming in wcf - see https://learn.microsoft.com/en-us/dotnet/framework/wcf/feature-details/large-data-and-streaming
You get a Stream to write to, so you need to handle yourself how you are going to encode your data on that stream, and how you are going decode it at the client.
The alternative is you do chunking/paging. You just modify your service so it accepts e.g. a page number or some other way to indicate which page is needed.
Which one you do depends on the application, eg how much data? what is the nature of the client? is it possible to use some field to page on? etc etc
Here is some psudo code for making a stream that can do this on the server side. It is based on the example here: https://learn.microsoft.com/en-us/dotnet/framework/wcf/feature-details/how-to-enable-streaming
I'm not writing the full compilable code for you, but this is the gist of it.
In the server:
public Stream GetBigData()
{
return new BigDataStream();
}
BigDataStream (the non-implimented methods are not shown):
class BigDataStream : Stream
{
public BigDataStream()
{
// open DB connection
// run your query
// get a DataReader
}
// you need a buffer to encode your data between calls to Read
List<byte> _encodeBuffer = new List<byte>();
public override int Read(byte[] buffer, int offset, int count)
{
// read from the DataReader and populate the _encodeBuffer
// until the _encodeBuffer contains at least count bytes
// (or until there are no more records)
// for example:
while (_encodeBuffer.Count < count && _reader.Read())
{
// (1)
// encode the record into a byte array. How to do this?
// you can read into a class and then use the data
// contract serialization for example. If you do this, you
// will probably find it easier to prepend an integer which
// specifies the length of the following encoded message.
// This will make it easier for the client to deserialize it.
// (2)
// append the encoded record bytes (plus any length prefix
// etc) to _encodeBuffer
}
// remove up to the first count bytes from _encodeBuffer
// and copy them into buffer at the offset requested
// return the number of bytes added
}
public override void Close()
{
// close the reader + db connection
base.Close();
}
}

Thank to mikelegg & Reniuz for helping come to a solution. I wish I could give them the tick for the right answer, but I am a afraid the next developer to read this question would not fully benefit. So where is what I ended up with.
Setup the config files for the Server and Client (Follow link: Large Data and Streaming)
Followed this solution, can download source code from here
I had to change the DBRowStream.DBThreadProc method a bit to work so I post the source code:
DBRowStream Class:
void DBThreadProc(object o)
{
SqlConnection con = null;
SqlCommand com = null;
try
{
con = new System.Data.SqlClient.SqlConnection(/*ConnectionString*/);
com = new SqlCommand();
com.Connection = con;
com.CommandText = PrepareQuery();
con.Open();
SqlDataReader reader = com.ExecuteReader();
int count = 0;
MemoryStream memStream = memStream1;
memStreamWriteStatus = 1;
readyToWriteToMemStream1.WaitOne();
while (reader.Read())
{
// Populate
Customer customer = new Customer();
foreach (var column in Columns)
{
int fieldIndex = reader.GetOrdinal(column);
object value = reader.GetValue(fieldIndex);
customer[column.Name] = value;
}
// Serialize: I used a custom Serializer
// but BinaryFormatter should be fine
DBDataFormatter.Serialize(memStream, customer);
count++;
if (count == PAGESIZE) // const int PAGESIZE = 10000
{
switch (memStreamWriteStatus)
{
case 1: // done writing to stream 1
{
memStream1.Position = 0;
readyToSendFromMemStream1.Set();
// write stream 1 is done...waiting for stream 2
readyToWriteToMemStream2.WaitOne();
memStream = memStream2;
memStream.Position = 0;
memStream.SetLength(0); // Added:To Reset the stream. Else was getting garbage data back
memStreamWriteStatus = 2;
break;
}
case 2: // done writing to stream 2
{
memStream2.Position = 0;
readyToSendFromMemStream2.Set();
// Write on stream 2 is done...waiting for stream 1
readyToWriteToMemStream1.WaitOne();
// done waiting for stream 1
memStream = memStream1;
memStreamWriteStatus = 1;
memStream.Position = 0;
memStream.SetLength(0); // Added: Reset the stream. Else was getting garbage data back
break;
}
}
count = 0;
}
}
if (count > 0)
{
switch (memStreamWriteStatus)
{
case 1: // done writing to stream 1
{
memStream1.Position = 0;
readyToSendFromMemStream1.Set();
// END write stream 1 is done...waiting for stream 2
break;
}
case 2: // done writing to stream 2
{
memStream2.Position = 0;
readyToSendFromMemStream2.Set();
// END write stream 2 is done...waiting for stream 1
break;
}
}
}
bDoneWriting = true;
bCanRead = false;
}
catch
{
throw;
}
finally
{
if (com != null)
{
com.Dispose();
com = null;
}
if (con != null)
{
con.Close();
con.Dispose();
con = null;
}
}
}
And then the Client side:
private static void TestGetRecordsAndDump()
{
const string FILE_NAME = "Records.CSV";
File.Delete(FILE_NAME);
var file = File.AppendText(FILE_NAME);
long count = 0;
try
{
ServiceReference1.ServiceClient service = new ServiceReference1.DataServiceClient();
var stream = service.GetDBRowStream();
Console.WriteLine("Records Retrieved : ");
Console.WriteLine("File Size (MB) : ");
var canDoLastRead = true;
while (stream.CanRead && canDoLastRead)
{
try
{
Customer customer = DBDataFormatter.Deserialize(stream); // Used custom Deserializer, but BinaryFormatter should be fine
file.Write(customer.ToString());
count++;
}
catch
{
canDoLastRead = false; // Bug: stream.CanRead is not set to false at the end of stream, so I do this trick to know if I finished retruning all records.
}
finally
{
Console.SetCursorPosition("Records Retrieved : ".Length, 0);
Console.Write(string.Format("{0} ", count));
Console.SetCursorPosition("File Size (MB) : ".Length, 1);
Console.Write(string.Format("{0:G} ", file.BaseStream.Length / 1024f / 1024f));
}
}
finally
{
file.Close();
}
}
}
There is a bug I cannot seem to solve, that stream.CanRead is not set to false, then all the records have been returned, have not been able to work out why, but at least now, I can query large data sets, and return all records, with out the server or client running out of memory.

Using ExecuteNonQueryAsync and Reporting Progress

I thought I was trying to do something very simple. I just want to report a running number on the screen so the user gets the idea that the SQL Stored Procedure that I'm executing is working and that they don't get impatient and start clicking buttons.
The problem is that I can't figure out how to actually call the progress reporter for the ExecutNonQueryAsync command. It gets stuck in my reporting loop and never executes the command but, if I put it after the async command, it will get executed and result will never not equal zero.
Any thoughts, comments, ideas would be appreciated. Thank you so much!
int i = 0;
lblProcessing.Text = "Transactions " + i.ToString();
int result = 0;
while (result==0)
{
i++;
if (i % 500 == 0)
{
lblProcessing.Text = "Transactions " + i.ToString();
lblProcessing.Refresh();
}
}
// Yes - I know - the code never gets here - that is the problem!
result = await cmd.ExecuteNonQueryAsync();

The simplest way to do this is to use a second connection to monitor the progress, and report on it. Here's a little sample to get you started:
using System;
using System.Collections.Generic;
using System.Data;
using System.Data.SqlClient;
using System.Text;
using System.Threading.Tasks;
namespace Microsoft.Samples.SqlServer
{
public class SessionStats
{
public long Reads { get; set; }
public long Writes { get; set; }
public long CpuTime { get; set; }
public long RowCount { get; set; }
public long WaitTime { get; set; }
public string LastWaitType { get; set; }
public string Status { get; set; }
public override string ToString()
{
return $"Reads {Reads}, Writes {Writes}, CPU {CpuTime}, RowCount {RowCount}, WaitTime {WaitTime}, LastWaitType {LastWaitType}, Status {Status}";
}
}
public class SqlCommandWithProgress
{
public static async Task ExecuteNonQuery(string ConnectionString, string Query, Action<SessionStats> OnProgress)
{
using (var rdr = await ExecuteReader(ConnectionString, Query, OnProgress))
{
rdr.Dispose();
}
}
public static async Task<DataTable> ExecuteDataTable(string ConnectionString, string Query, Action<SessionStats> OnProgress)
{
using (var rdr = await ExecuteReader(ConnectionString, Query, OnProgress))
{
var dt = new DataTable();
dt.Load(rdr);
return dt;
}
}
public static async Task<SqlDataReader> ExecuteReader(string ConnectionString, string Query, Action<SessionStats> OnProgress)
{
var mainCon = new SqlConnection(ConnectionString);
using (var monitorCon = new SqlConnection(ConnectionString))
{
mainCon.Open();
monitorCon.Open();
var cmd = new SqlCommand("select ##spid session_id", mainCon);
var spid = Convert.ToInt32(cmd.ExecuteScalar());
cmd = new SqlCommand(Query, mainCon);
var monitorQuery = #"
select s.reads, s.writes, r.cpu_time, s.row_count, r.wait_time, r.last_wait_type, r.status
from sys.dm_exec_requests r
join sys.dm_exec_sessions s
on r.session_id = s.session_id
where r.session_id = #session_id";
var monitorCmd = new SqlCommand(monitorQuery, monitorCon);
monitorCmd.Parameters.Add(new SqlParameter("#session_id", spid));
var queryTask = cmd.ExecuteReaderAsync( CommandBehavior.CloseConnection );
var cols = new { reads = 0, writes = 1, cpu_time =2,row_count = 3, wait_time = 4, last_wait_type = 5, status = 6 };
while (!queryTask.IsCompleted)
{
var firstTask = await Task.WhenAny(queryTask, Task.Delay(1000));
if (firstTask == queryTask)
{
break;
}
using (var rdr = await monitorCmd.ExecuteReaderAsync())
{
await rdr.ReadAsync();
var result = new SessionStats()
{
Reads = Convert.ToInt64(rdr[cols.reads]),
Writes = Convert.ToInt64(rdr[cols.writes]),
RowCount = Convert.ToInt64(rdr[cols.row_count]),
CpuTime = Convert.ToInt64(rdr[cols.cpu_time]),
WaitTime = Convert.ToInt64(rdr[cols.wait_time]),
LastWaitType = Convert.ToString(rdr[cols.last_wait_type]),
Status = Convert.ToString(rdr[cols.status]),
};
OnProgress(result);
}
}
return queryTask.Result;
}
}
}
}
Which you would call something like this:
class Program
{
static void Main(string[] args)
{
Run().Wait();
}
static async Task Run()
{
var constr = "server=localhost;database=tempdb;integrated security=true";
var sql = #"
set nocount on;
select newid() d
into #foo
from sys.objects, sys.objects o2, sys.columns
order by newid();
select count(*) from #foo;
";
using (var rdr = await SqlCommandWithProgress.ExecuteReader(constr, sql, s => Console.WriteLine(s)))
{
if (!rdr.IsClosed)
{
while (rdr.Read())
{
Console.WriteLine("Row read");
}
}
}
Console.WriteLine("Hit any key to exit.");
Console.ReadKey();
}
}
Which outputs:
Reads 0, Writes 0, CPU 1061, RowCount 0, WaitTime 0, LastWaitType SOS_SCHEDULER_YIELD, Status running
Reads 0, Writes 0, CPU 2096, RowCount 0, WaitTime 0, LastWaitType SOS_SCHEDULER_YIELD, Status running
Reads 0, Writes 0, CPU 4553, RowCount 11043136, WaitTime 198, LastWaitType CXPACKET, Status suspended
Row read
Hit any key to exit.

You're not going to be able to get ExecuteNonQueryAsync to do what you want here. To do what you're looking for, the result of the method would have to be either row by row or in chunks incremented during the SQL call, but that's not how submitting a query batch to SQL Server works or really how you would want it to work from an overhead perspective. You hand a SQL statement to the server and after it is finished processing the statement, it returns the total number of rows affected by the statement.

Do you just want to let the user know that something is happening, and you don't actually need to display current progress?
If so, you could just display a ProgressBar with its Style set to Marquee.
If you want this to be a "self-contained" method, you could display the progress bar on a modal form, and include the form code in the method itself.
E.g.
public void ExecuteNonQueryWithProgress(SqlCommand cmd) {
Form f = new Form() {
Text = "Please wait...",
Size = new Size(400, 100),
StartPosition = FormStartPosition.CenterScreen,
FormBorderStyle = FormBorderStyle.FixedDialog,
MaximizeBox = false,
ControlBox = false
};
f.Controls.Add(new ProgressBar() {
Style = ProgressBarStyle.Marquee,
Dock = DockStyle.Fill
});
f.Shown += async (sender, e) => {
await cmd.ExecuteNonQueryAsync();
f.Close();
};
f.ShowDialog();
}

That is an interesting question. I have had to implement similar things in the past. In our case the priority was to:
Keep client side responsive in case the user doesn't want to stick around and wait.
Update the user of action and progress.
What I would do is use threading to run the process in the background like:
HostingEnvironment.QueueBackgroundWorkItem(ct => FunctionThatCallsSQLandTakesTime(p, q, s));
Then using a way to estimate work time I would increment a progress bar from client side on a clock. For this, query your data for a variable that gives you a linear relationship to the work time needed by FunctionThatCallsSQLandTakesTime.
For example; the number of active users this month drives the time FunctionThatCallsSQLandTakesTime takes. For each 10000 user it takes 5 minutes. So you can update your progress bar accordingly.

I'm wondering if this might be a reasonable approach:
IAsyncResult result = cmd2.BeginExecuteNonQuery();
int count = 0;
while (!result.IsCompleted)
{
count++;
if (count % 500 == 0)
{
lblProcessing.Text = "Transactions " + i.ToString();
lblProcessing.Refresh();
}
// Wait for 1/10 second, so the counter
// does not consume all available resources
// on the main thread.
System.Threading.Thread.Sleep(100);
}

Async I/O intensive code is running slower than non-async, why?

I am refactoring an application and trying to add an asynchronous version of an existing function to improve performance times in an ASP.NET MVC application. I understand that there is an overhead involved with asynchronous functions, but I expected that with enough iterations, the I/O intensive nature of loading the data from the database would more than compensate for the overhead penalty and that I would receive significant performance gains.
The TermusRepository.LoadByTermusId function loads data by retrieving a bunch of datatables from the database (using ADO.NET and the Oracle Managed Client), populates a model, and returns it. TermusRepository.LoadByTermusIdAsync is similar, except it does so asynchronously, with a slightly different method of loading up datatable download tasks when there's multiple datatables to retrieve.
public async Task<ActionResult> AsyncPerformanceTest()
{
var vm = new AsyncPerformanceTestViewModel();
Stopwatch watch = new Stopwatch();
watch.Start();
for (int i = 0; i < 60; i++)
{
TermusRepository.LoadByTermusId<Termus2011_2012EndYear>("1");
TermusRepository.LoadByTermusId<Termus2011_2012EndYear>("5");
TermusRepository.LoadByTermusId<Termus2011_2012EndYear>("6");
TermusRepository.LoadByTermusId<Termus2011_2012EndYear>("7");
}
watch.Stop();
vm.NonAsyncElapsedTime = watch.Elapsed;
watch.Reset();
watch.Start();
var tasks = new List<Task<Termus2011_2012EndYear>>();
for (int i = 0; i < 60; i++)
{
tasks.Add(TermusRepository.LoadByTermusIdAsync<Termus2011_2012EndYear>("1"));
tasks.Add(TermusRepository.LoadByTermusIdAsync<Termus2011_2012EndYear>("5"));
tasks.Add(TermusRepository.LoadByTermusIdAsync<Termus2011_2012EndYear>("6"));
tasks.Add(TermusRepository.LoadByTermusIdAsync<Termus2011_2012EndYear>("7"));
}
await Task.WhenAll(tasks.ToArray());
watch.Stop();
vm.AsyncElapsedTime = watch.Elapsed;
return View(vm);
}
public static async Task<T> LoadByTermusIdAsync<T>(string termusId) where T : Appraisal
{
var AppraisalHeader = new OracleCommand("select tu.termus_id, tu.manager_username, tu.evaluee_name, tu.evaluee_username, tu.termus_complete_date, termus_start_date, tu.termus_status, tu.termus_version, tn.managername from tercons.termus_users tu left outer join tercons.termus_names tn on tu.termus_id=tn.termus_id where tu.termus_id=:termusid");
AppraisalHeader.BindByName = true;
AppraisalHeader.Parameters.Add("termusid", termusId);
var dt = await Database.GetDataTableAsync(AppraisalHeader);
T Termus = Activator.CreateInstance<T>();
var row = dt.AsEnumerable().Single();
Termus.TermusId = row.Field<decimal>("termus_id").ToString();
Termus.ManagerUsername = row.Field<string>("manager_username");
Termus.EvalueeUsername = row.Field<string>("evaluee_username");
Termus.EvalueeName = row.Field<string>("evaluee_name");
Termus.ManagerName = row.Field<string>("managername");
Termus.TERMUSCompleteDate = row.Field<DateTime?>("termus_complete_date");
Termus.TERMUSStartDate = row.Field<DateTime>("termus_start_date");
Termus.Status = row.Field<string>("termus_status");
Termus.TERMUSVersion = row.Field<string>("termus_version");
Termus.QuestionsAndAnswers = new Dictionary<string, string>();
var RetrieveQuestionIdsCommand = new OracleCommand("select termus_question_id from tercons.termus_questions where termus_version=:termus_version");
RetrieveQuestionIdsCommand.BindByName = true;
RetrieveQuestionIdsCommand.Parameters.Add("termus_version", Termus.TERMUSVersion);
var QuestionIdsDt = await Database.GetDataTableAsync(RetrieveQuestionIdsCommand);
var QuestionIds = QuestionIdsDt.AsEnumerable().Select(r => r.Field<string>("termus_question_id"));
//There's about 60 questions/answers, so this should result in 60 calls to the database. It'd be a good spot to combine to a single DB call, but left it this way so I could see if async would speed it up for learning purposes.
var DownloadAnswersTasks = new List<Task<DataTable>>();
foreach (var QuestionId in QuestionIds)
{
var RetrieveAnswerCommand = new OracleCommand("select termus_response, termus_question_id from tercons.termus_responses where termus_id=:termus_id and termus_question_id=:questionid");
RetrieveAnswerCommand.BindByName = true;
RetrieveAnswerCommand.Parameters.Add("termus_id", termusId);
RetrieveAnswerCommand.Parameters.Add("questionid", QuestionId);
DownloadAnswersTasks.Add(Database.GetDataTableAsync(RetrieveAnswerCommand));
}
while (DownloadAnswersTasks.Count > 0)
{
var FinishedDownloadAnswerTask = await Task.WhenAny(DownloadAnswersTasks);
DownloadAnswersTasks.Remove(FinishedDownloadAnswerTask);
var AnswerDt = await FinishedDownloadAnswerTask;
var Answer = AnswerDt.AsEnumerable().Select(r => r.Field<string>("termus_response")).SingleOrDefault();
var QuestionId = AnswerDt.AsEnumerable().Select(r => r.Field<string>("termus_question_id")).SingleOrDefault();
if (!String.IsNullOrEmpty(Answer))
{
Termus.QuestionsAndAnswers.Add(QuestionId, System.Net.WebUtility.HtmlDecode(Answer));
}
}
return Termus;
}
public static async Task<DataTable> GetDataTableAsync(OracleCommand command)
{
DataTable dt = new DataTable();
using (var connection = GetDefaultOracleConnection())
{
command.Connection = connection;
await connection.OpenAsync();
dt.Load(await command.ExecuteReaderAsync());
}
return dt;
}
public static T LoadByTermusId<T>(string TermusId) where T : Appraisal
{
var RetrieveAppraisalHeaderCommand = new OracleCommand("select tu.termus_id, tu.manager_username, tu.evaluee_name, tu.evaluee_username, tu.termus_complete_date, termus_start_date, tu.termus_status, tu.termus_version, tn.managername from tercons.termus_users tu left outer join tercons.termus_names tn on tu.termus_id=tn.termus_id where tu.termus_id=:termusid");
RetrieveAppraisalHeaderCommand.BindByName = true;
RetrieveAppraisalHeaderCommand.Parameters.Add("termusid", TermusId);
var AppraisalHeaderDt = Database.GetDataTable(RetrieveAppraisalHeaderCommand);
T Termus = Activator.CreateInstance<T>();
var AppraisalHeaderRow = AppraisalHeaderDt.AsEnumerable().Single();
Termus.TermusId = AppraisalHeaderRow.Field<decimal>("termus_id").ToString();
Termus.ManagerUsername = AppraisalHeaderRow.Field<string>("manager_username");
Termus.EvalueeUsername = AppraisalHeaderRow.Field<string>("evaluee_username");
Termus.EvalueeName = AppraisalHeaderRow.Field<string>("evaluee_name");
Termus.ManagerName = AppraisalHeaderRow.Field<string>("managername");
Termus.TERMUSCompleteDate = AppraisalHeaderRow.Field<DateTime?>("termus_complete_date");
Termus.TERMUSStartDate = AppraisalHeaderRow.Field<DateTime>("termus_start_date");
Termus.Status = AppraisalHeaderRow.Field<string>("termus_status");
Termus.TERMUSVersion = AppraisalHeaderRow.Field<string>("termus_version");
Termus.QuestionsAndAnswers = new Dictionary<string, string>();
var RetrieveQuestionIdsCommand = new OracleCommand("select termus_question_id from tercons.termus_questions where termus_version=:termus_version");
RetrieveQuestionIdsCommand.BindByName = true;
RetrieveQuestionIdsCommand.Parameters.Add("termus_version", Termus.TERMUSVersion);
var QuestionIdsDt = Database.GetDataTable(RetrieveQuestionIdsCommand);
var QuestionIds = QuestionIdsDt.AsEnumerable().Select(r => r.Field<string>("termus_question_id"));
//There's about 60 questions/answers, so this should result in 60 calls to the database. It'd be a good spot to combine to a single DB call, but left it this way so I could see if async would speed it up for learning purposes.
foreach (var QuestionId in QuestionIds)
{
var RetrieveAnswersCommand = new OracleCommand("select termus_response from tercons.termus_responses where termus_id=:termus_id and termus_question_id=:questionid");
RetrieveAnswersCommand.BindByName = true;
RetrieveAnswersCommand.Parameters.Add("termus_id", TermusId);
RetrieveAnswersCommand.Parameters.Add("questionid", QuestionId);
var AnswersDt = Database.GetDataTable(RetrieveAnswersCommand);
var Answer = AnswersDt.AsEnumerable().Select(r => r.Field<string>("termus_response")).SingleOrDefault();
if (!String.IsNullOrEmpty(Answer))
{
Termus.QuestionsAndAnswers.Add(QuestionId, System.Net.WebUtility.HtmlDecode(Answer));
}
}
return Termus;
}
public static DataTable GetDataTable(OracleCommand command)
{
DataTable dt = new DataTable();
using (var connection = GetDefaultOracleConnection())
{
command.Connection = connection;
connection.Open();
dt.Load(command.ExecuteReader());
}
return dt;
}
public static OracleConnection GetDefaultOracleConnection()
{
return new OracleConnection(ConfigurationManager.ConnectionStrings[connectionstringname].ConnectionString);
}
Results for 60 iterations are:
Non Async 18.4375460 seconds
Async 19.8092854 seconds
The results of this test are consistent. No matter how many iterations I go through of the for loop in AsyncPerformanceTest() action method, the async stuff runs about 1 second slower than the non-async. (I run the test multiple times in a row to account for the JITter warming up.) What am I doing wrong that's causing the async to be slower than the non-async? Am I misunderstanding something fundamental about writing asynchronous code?

The asynchronous version will always be slower than the synchronous version when there is no concurrency. It's doing all of the same work as the non-async version, but with a small amount of overhead added to manage the asynchrony.
Asynchrony is advantageous, with respect to performance, by allowing improved availability. Each individual request will be slower, but if you make 1000 requests at the same time, the asynchronous implementation will be able to handle them all more quickly (at least in certain circumstances).
This happens because the asynchronous solution allows the thread that was allocated to handle the request to go back to the pool and handle other requests, whereas the synchronous solution forces the thread to sit there and do nothing while it waits for the asynchronous operation to complete. There is overhead in structuring the program in a way that allows the thread to be freed up to do other work, but the advantage is the ability of that thread to go do other work. In your program there is no other work for the thread to go do, so it ends up being a net loss.

Turns out the Oracle Managed Driver is "fake async", which would partially explain why my async code is running slower.

Using the async Datareader

We are pretty new to this, but we're trying to make use of the new async datareader
we have been following the sample from http://blogs.msdn.com/b/adonet/archive/2012/07/15/using-sqldatareader-s-new-async-methods-in-net-4-5-beta-part-2-examples.aspx
but experiencing problems.
What we are trying to achieve is :-
The stored proc returns 2 tables, using a datareader, we are trying to populate 2 models asynchronously, ie the 2 models will be created at the same time and not have to wait for the first model to be built before the second one can begin
(If that makes sense?)
Below is what we have at the moment :-
public async Task<UsertResultsModel> GetResults(string user_ID)
{
UsertResultsModel rm = new UsertResultsModel();
List<TableColumn> ltc = new List<TableColumn>();
List<List<TableColumn>> lltc = new List<List<TableColumn>>();
var col = 0;
try
{
using (SqlConnection con = new SqlConnection(Config.DatabaseStringCDA))
{
await con.OpenAsync();
SqlCommand com = new SqlCommand(#"USP_GET_USER_RESULTS", con);
com.CommandType = CommandType.StoredProcedure;
using (SqlDataReader rdr = await com.ExecuteReaderAsync())
{
col = rdr.FieldCount;
while (await rdr.ReadAsync())
{
for (int i = 0; i < rdr.FieldCount; i++)
{
string colName = rdr.GetName(i).ToSafeString();
ltc.Add(new TableColumn(colName, rdr[i].ToSafeString(), 50, EnumColumnType.String, colName));
}
lltc.Add(ltc);
}
rm.Summary = new TableViewModel { Grid = lltc };
await rdr.NextResultAsync();
lltc = new List<List<TableColumn>>();
while (rdr.Read())
{
for (int i = 0; i < rdr.FieldCount; i++)
{
string colName = rdr.GetName(i).ToSafeString();
ltc.Add(new TableColumn(colName, rdr[i].ToSafeString(), 50, EnumColumnType.String, colName));
}
lltc.Add(ltc);
}
rm.Trend = new TableViewModel { Grid = lltc };
}
}
}
catch (Exception ex)
{
log.Error(MethodBase.GetCurrentMethod(), ex);
return null;
}
await Task.WhenAll();
return rm;
// return lltc;
}
however, stepping through the code, as soon as we hit any await, The application simply loads, and no further code is hit..
Any advice or recommendations for this would be greatly appreciated.

The async part is the DB I/O, meaning that while you are awaiting an open, or a read, your app is free to do other things rather than blocking on the DB.
await/async allows you to write code in a sequential manner even though parts may take a while. The code flows exactly as you would expect in sequence, the line after each await doesn't run until the awaited item has finished, but the current thread is unwound back up the stack so it can get on with other things.
If you want to do two things at once using await/async, then you need to wrap each one in a function that returns a Task just like traditional TPL. That way you can keep hold of the tasks WITHOUT awaiting their results until you want to.