The below method doesn't compile. Alternatives?
public static async Task<IEnumerable<object[]>> GetRecordsAsync(
this Transaction transaction,
string commandText,
params SqlParameter[] parameters)
{
// Get a SqlDataReader
var reader = await transaction.GetReaderAsync(commandText, parameters);
var fieldCount = -1;
// Begin iterating through records asynchronously
while (await reader.ReadAsync()) // Note we don't loop until .ReadAsync returns a boolean
{
// Grab all the field values out
if (fieldCount < 0)
fieldCount = reader.FieldCount;
var fields = new object[fieldCount];
reader.GetValues(fields);
// Yield return the field values from this record
yield return fields;
}
}
Error message:
The body of 'TransactionExtensions.GetRecordsAsync(Transaction, string, params SqlParameter[])' cannot be an iterator block because 'Task>' is not an iterator interface type
I don't see a way to adapt this answer to a similar sounding (but different) question, because I don't know a priori how many times the loop will go.
Edit: fixed formatting
Based on #SLaks's comment to the question, here's a general alternative using Reactive Extensions:
/// <summary>
/// Turns the given asynchronous functions into an IObservable
/// </summary>
static IObservable<T> ToObservable<T>(
Func<Task<bool>> shouldLoopAsync,
Func<Task<T>> getAsync)
{
return Observable.Create<T>(
observer => Task.Run(async () =>
{
while (await shouldLoopAsync())
{
var value = await getAsync();
observer.OnNext(value);
}
observer.OnCompleted();
}
)
);
}
Example usage, tailored to solve the question's specific case:
/// <summary>
/// Asynchronously processes each record of the given reader using the given handler
/// </summary>
static async Task ProcessResultsAsync(this SqlDataReader reader, Action<object[]> fieldsHandler)
{
// Set up async functions for the reader
var shouldLoopAsync = (Func<Task<bool>>)reader.ReadAsync;
var getAsync = new Func<SqlDataReader, Func<Task<object[]>>>(_reader =>
{
var fieldCount = -1;
return () => Task.Run(() =>
{
Interlocked.CompareExchange(ref fieldCount, _reader.FieldCount, -1);
var fields = new object[fieldCount];
_reader.GetValues(fields);
return fields;
});
})(reader);
// Turn the async functions into an IObservable
var observable = ToObservable(shouldLoopAsync, getAsync);
// Process the fields as they become available
var finished = new ManualResetEventSlim(); // This will be our signal for when the observable completes
using (observable.Subscribe(
onNext: fieldsHandler, // Invoke the handler for each set of fields
onCompleted: finished.Set // Set the gate when the observable completes
)) // Don't forget best practice of disposing IDisposables
// Asynchronously wait for the gate to be set
await Task.Run((Action)finished.Wait);
}
(Note that getAsync could be simplified in the above code block, but I like how explicit it is about the closure that's being created)
...and finally:
// Get a SqlDataReader
var reader = await transaction.GetReaderAsync(commandText, parameters);
// Do something with the records
await reader.ProcessResultsAsync(fields => { /* Code here to process each record */ });
Don't return a Task<IEnumerable<T>> and don't even use Task at all for this; instead, return an IAsyncEnumerable<T>. No need for third-party libraries or other workarounds, no need to even alter the body of your original method.
public static async IAsyncEnumerable<object[]> GetRecordsAsync(
this Transaction transaction,
string commandText,
params SqlParameter[] parameters)
{
// Get a SqlDataReader
var reader = await transaction.GetReaderAsync(commandText, parameters);
var fieldCount = -1;
// Begin iterating through records asynchronously
while (await reader.ReadAsync()) // Note we don't loop until .ReadAsync returns a boolean
{
// Grab all the field values out
if (fieldCount < 0)
fieldCount = reader.FieldCount;
var fields = new object[fieldCount];
reader.GetValues(fields);
// Yield return the field values from this record
yield return fields;
}
}
I solved it without third-party extensions:
public async Task<IEnumerable<Item>> GetAllFromDb()
{
OracleConnection connection = null;
DbDataReader reader = null;
try
{
connection = new OracleConnection(connectionString);
var command = new OracleCommand(queryString, connection);
connection.Open();
reader = await command.ExecuteReaderAsync();
return this.BuildEnumerable(connection, reader);
}
catch (Exception)
{
reader?.Dispose();
connection?.Dispose();
throw;
}
}
private IEnumerable<Item> BuildEnumerable(OracleConnection connection, DbDataReader reader)
{
using (connection)
using (reader)
{
while (reader.Read())
{
var item = new Item()
{
Prop = reader.GetString(0),
};
yield return item;
}
}
}
This example is for Oracle Data Reader but the same approach is applicable to any asynchronous operation combined with yield return
Related
I have created a supposed to be asynchronous method. And inside of that it has several await statements.
I am calling this method (in another class) from the main method.
But it doesn't return the control back to the caller (main), to execute the next statement (to get more data),
even though the operation takes long time to complete.
What is wrong here?
Could it be something with the return statement? Or do I need to wrap the logic inside a Task?
internal async static Task<List<Cars>> GetCarsAsync()
{
var cars = new List<Cars>();
try
{
using (var connection = new OracleConnection(ConfigurationManager.ConnectionStrings["KWC"].ConnectionString))
{
await connection.OpenAsync(); //first await
logger.Info("Opened database connection.");
StringBuilder selectStatement = new StringBuilder(Properties.Settings.Default.SelectCarsView);
using (var command = connection.CreateCommand())
{
command.CommandText = selectStatement.ToString();
logger.Info("About to execute following query on database: {0}", command.CommandText);
using (var reader = await command.ExecuteReaderAsync()) //second await
{
logger.Info("Executed query: {0}", command.CommandText);
while (await reader.ReadAsync()) //third await
{
var car = new Car { model = reader.GetString(0) };
//more code
cars.Add(car);
}
}
}
}
return cars;
}
catch (OracleException ex)
{
logger.Fatal(ex.Message);
throw;
}
}
static async Task Main(string[] args)
{
var getCarsTask = CarsManager.GetCarsAsync();
var getSomethingElseTask = DummyManager.GetMoreDataAsync();
// more similar tasks
var cars = await getCarsTask;
var moreData = await getSomethingElseTask;
// more code
}
I migrated an Access database to SQL using Microsoft SQL Server Migration Assistant.
Now, I am having trouble reading data.
return reader.GetInt32(0); Throws Invalid attempt to read when no data is present exception after 32 rows retrieved. If I add CommandBehavior.SequentialAccess to my command, I am able to read 265 rows, every time.
The data
The query (in SQL Management Studio):
SELECT *
FROM Products2
WHERE Products2.PgId = 337;
There is nothing special about row 32, and if I reverse the order, it is still row 32 that kills it.
Row 265, just for good measure.
The code
The query:
SELECT *
FROM Products2
WHERE Products2.PgId = #productGroupId;
The parameter:
Name = "productGroupId"
Value = 337
The execution:
public async Task ExecuteAsync(string query, Action<IDataReader> onExecute, params DataParameter[] parameters)
{
using(var command = builder.BuildCommand(query))
{
foreach(var parameter in parameters)
command.AddParameter(parameter);
if(!connection.IsOpen)
connection.Open();
await Task.Run(async () =>
{
using(var reader = await command.ExecuteAsync())
if(await reader.ReadAsync())
onExecute(reader);
});
}
}
And reading:
await executor.ExecuteAsync(query, async reader =>
{
do
{
products.Add(GetProduct(reader));
columnValues.Enqueue(GetColumnValues(reader).ToList());
} while(await reader.ReadAsync());
}, parameters.ToArray());
await reader.ReadAsync() returns true, but when GetProduct(reader) calls reader.GetInt32(0); for the 32nd time, it throws the exception.
It works fine if the data is less than 32 rows, or 265 in case of SequentialAccess.
I tried increasing the CommandTimeout, but it didn't help. When I swap the connection to OleDb again, it works just fine.
Thanks in advance.
EDIT
If I replace * in the query with just a few specific columns, it works. When I read ~12 columns, it fails, but later than row 32.
As per request, GetProduct:
private Product GetProduct(IDataReader productReader)
{
return new Product
{
Id = productReader.ReadLong(0),
Number = productReader.ReadString(2),
EanNo = productReader.ReadString(3),
Frequency = productReader.ReadInt(4),
Status = productReader.ReadInt(5),
NameId = productReader.ReadLong(6),
KMPI = productReader.ReadByte(7),
OEM = productReader.ReadString(8),
CurvesetId = productReader.ReadInt(9),
HasServiceInfo = Convert.ToBoolean(productReader.ReadByte(10)),
ColumnData = new List<ColumnData>()
};
}
GetColumnData:
private IEnumerable<long> GetColumnValues(IDataReader productReader)
{
var columnValues = new List<long>();
int columnIndex = 11;
while(!productReader.IsNull(columnIndex))
columnValues.Add(productReader.ReadLong(columnIndex++));
return columnValues;
}
And the adapter:
public long ReadLong(int columnIndex)
{
return reader.GetInt32(columnIndex);
}
Alright, it is getting long. :)
Thanks to #Igor, I tried creating a small working example. This seem to work fine:
private static async Task Run()
{
var result = new List<Product>();
string conString = #" ... ";
var con = new SqlConnection(conString);
con.Open();
using(var command = new SqlCommand("SELECT * FROM Products2 WHERE Products2.PgId = #p;", con))
{
command.Parameters.Add(new SqlParameter("p", 337));
await Task.Run(async () =>
{
using(var productReader = await command.ExecuteReaderAsync())
while(await productReader.ReadAsync())
{
result.Add(new Product
{
Id = productReader.GetInt32(0),
Number = productReader.GetString(2),
EanNo = productReader.GetString(3),
Frequency = productReader.GetInt16(4),
Status = productReader.GetInt16(5),
NameId = productReader.GetInt32(6),
KMPI = productReader.GetByte(7),
OEM = productReader.GetString(8),
CurvesetId = productReader.GetInt16(9),
HasServiceInfo = Convert.ToBoolean(productReader.GetByte(10))
});
GetColumnValues(productReader);
}
});
}
Console.WriteLine("End");
}
private static IEnumerable<long> GetColumnValues(SqlDataReader productReader)
{
var columnValues = new List<long>();
int columnIndex = 11;
while(!productReader.IsDBNull(columnIndex))
columnValues.Add(productReader.GetInt32(columnIndex++));
return columnValues;
}
}
Here's the data in Access, just in case:
I managed to fix the problem.
I still have questions though, since I don't understand why this didn't affect Access.
I changed:
public async Task ExecuteAsync(string query, Action<IDataReader> onExecute, params DataParameter[] parameters)
{
using(var command = builder.BuildCommand(query))
{
foreach(var parameter in parameters)
command.AddParameter(parameter);
if(!connection.IsOpen)
connection.Open();
await Task.Run(async () =>
{
using(var reader = await command.ExecuteAsync())
if(await reader.ReadAsync())
onExecute(reader);
});
}
}
To:
public async Task ExecuteAsync(string query, Func<IDataReader, Task> onExecute, params DataParameter[] parameters)
{
using(var command = builder.BuildCommand(query))
{
foreach(var parameter in parameters)
command.AddParameter(parameter);
if(!connection.IsOpen)
connection.Open();
using(var reader = await command.ExecuteAsync())
await onExecute(reader);
}
}
If someone can explain why this helped, I would really appreciate it.
I have dilemma on how to actually create proper awaitable function. I don't get the entire concept fully, probably due the language barrier :)
A
public async Task<int> InsertASync(string tableName, Dictionary<string, string> data)
{
int r = 0;
using (SQLiteConnection con = NewConnection())
{
await con.OpenAsync();
using (SQLiteCommand com = new SQLiteCommand(Query_Insert(tableName, data), con))
{
r = await com.ExecuteNonQueryAsync();
}
con.Close();
}
return r;
}
or
B
public Task<int> InsertASync(string tableName, Dictionary<string, string> data)
{
return Task<int>.Run(async () =>
{
int r = 0;
using (SQLiteConnection con = NewConnection())
{
await con.OpenAsync();
using (SQLiteCommand com = new SQLiteCommand(Query_Insert(tableName, data), con))
{
r = await com.ExecuteNonQueryAsync();
}
con.Close();
}
return r;
});
}
Reason i'm wondering about it is because the way i create awaitable methods via cheap way.
Example
public void DoHeavyWork() //this will block UI
{
//Do work
}
public Task DoHeavyWorkASync() //this won't
{
return Task.Run(() =>
{
DoHeavyWork();
});
}
You want to go the A route. The compiler will take care of creating the returned task and returning the actual integer, when available.
You can read Asynchronous Programming - Pause and Play with ... for some good in-depth information on what the compiler does internally (how your method is converted into a state machine)
I have some async method
public static Task<JObject> GetUser(NameValueCollection parameters)
{
return CallMethodApi("users.get", parameters, CallType.HTTPS);
}
And I write method below
public static IEnumerable<JObject> GetUsers(IEnumerable<string> usersUids, Field fields)
{
foreach(string uid in usersUids)
{
var parameters = new NameValueCollection
{
{"uids", uid},
{"fields", FieldsUtils.ConvertFieldsToString(fields)}
};
yield return GetUser(parameters).Result;
}
}
This method is asynchronous? How to write this using Parallel.ForEach?
Something kind of like this.
public static IEnumerable<JObject> GetUsers(IEnumerable<string> usersUids, Field fields)
{
var results = new List<JObject>
Parallel.ForEach(usersUids, uid => {
var parameters = new NameValueCollection
{
{"uids", uid},
{"fields", FieldsUtils.ConvertFieldsToString(fields)}
};
var user = GetUser(parameters).Result;
lock(results)
results.Add(user);
});
return results;
}
NOTE: The results won't be in the same order as you expect.
Your method is not asynchronous. Assuming your GetUser method already starts an asynchronous task, Parallel.ForEach would use additional threads just to start off your tasks, which is probably not what you want.
Instead, what you probably want to do is to start all of the tasks and wait for them to finish:
public static IEnumerable<JObject> GetUsers(IEnumerable<string> usersUids, Field fields)
{
var tasks = usersUids.Select(
uid =>
{
var parameters = new NameValueCollection
{
{"uids", uid},
{"fields", FieldsUtils.ConvertFieldsToString(fields)}
};
return GetUser(parameters);
}
).ToArray();
Task.WaitAll(tasks);
var result = new JObject[tasks.Length];
for (var i = 0; i < tasks.Length; ++i)
result[i] = tasks[i].Result;
return result;
}
If you also want to start them in parallel you can use PLINQ:
var tasks = usersUids.AsParallel().AsOrdered().Select(
uid =>
{
var parameters = new NameValueCollection
{
{"uids", uid},
{"fields", FieldsUtils.ConvertFieldsToString(fields)}
};
return GetUser(parameters);
}
).ToArray();
Both code snippets preserve relative ordering of uids and returned objects - result[0] corresponds to usersUids[0], etc.
I'm facing a pretty weird construct. The Foo type returned in an IEnumerable loses its data as soon as the enumeration ends. This means that I can't do a enumeration.First() because the data would be lost right away.
A loop over it works, but since I know it will contain only a single element that would be weird.
int Test(out int something)
IEnumerable<Foo> enumeration = ...
for (var foo in enumeration) {
something = foo.GetSomething ();
return foo.GetAnInt ();
}
something = 42;
return 0;
}
Another way I though of is abusing a Linq Select, but that's just as horrible.
Is there a way to work around this limitation? Fixing the root cause is obviously superior, but difficult in this case.
Edit: It's an IEnumerable<IDataRecord> that is yield returned from a transactioned SQL data reader.
public IEnumerable<IDataRecord> ExecuteReader (SqlCommand cmd)
{
using (var con = GetConnection()) {
con.Open ();
using (var tr = con.BeginTransaction ()) {
cmd.Connection = con;
var reader = cmd.ExecuteReader ();
while (reader.Read ()) {
yield return reader;
}
tr.Commit ();
}
}
}
The problem is that your ExecuteReader method does simply return the SqlDataReader itself (which implements IDataRecord), instead of returning a block of data. So when you do this:
var list = ExecuteReader(...).ToList();
In that case all elements of the list will be the same SqlDataReader instance, but after the ToList has been executed, the reader has been closed. I'm a bit surprised that you don't get an ObjectDisposedException.
For this to work, you need to return a copy of the data in the IDataRecord. You think you can iterate the elements in the data record. An other option is to change the ExecuteReader to the following:
public IEnumerable<T> ExecuteReader<T>(SqlCommand cmd,
Func<IDataRecord, T> recordCreator)
{
using (var con = GetConnection()) {
con.Open ();
using (var tr = con.BeginTransaction()) {
cmd.Connection = con;
var reader = cmd.ExecuteReader();
while (reader.Read()) {
yield return recordCreator(reader);
}
tr.Commit();
}
}
}
This way you can do the following:
var list = ExecuteReader(command, record => new
{
Item1 = record.GetInt("id"),
Item2 = record.GetString("name")
});
Note: I'm not sure why you need a transaction for this anyway.
How about
int Test(out int something)
{
IEnumerable<Foo> enumeration = ...
var values = enumeration
.Select(foo => new
{
something = foo.GetSomething(),
anInt = foo.GetAnInt()
})
.FirstOrDefault();
if (values != null)
{
something = values.something;
return values.anInt;
}
else
{
something = 42;
return 0;
}
}
GetSomething and GetAnInt are called while inside the enumeration.
Another idea could be to convert the result type of the method from IEnumerable to IEnumerator. That way, scope control is much easier, and returning single results does not require any (fake) loop, too.
Edit: I think I found a way to refactor the whole issue. This circumvents the initial problem by using new disposable class that contains the logic formerly found in the method. It's very readable and less code even.
public TransactedConnection GetConnection (string text)
{
return new TransactedConnection (_ConnectionString, text);
}
public class TransactedConnection : IDisposable
{
private readonly SQLiteCommand _Cmd;
private readonly SQLiteConnection _Con;
private readonly SQLiteTransaction _Tr;
public TransactedConnection (string connection, string text)
{
_Cmd = new SQLiteCommand (text);
_Con = new SQLiteConnection (connection);
_Con.Open ();
_Cmd.Connection = _Con;
_Tr = _Con.BeginTransaction ();
}
public void Dispose ()
{
_Tr.Commit ();
_Tr.Dispose ();
_Con.Dispose ();
_Cmd.Dispose ();
}
public SQLiteParameterCollection Parameters
{
get
{
return _Cmd.Parameters;
}
}
public int ExecuteNonQuery ()
{
return _Cmd.ExecuteNonQuery ();
}
public object ExecuteScalar ()
{
return _Cmd.ExecuteScalar ();
}
public SQLiteDataReader ExecuteReader ()
{
return _Cmd.ExecuteReader ();
}
}
public void Test (string match)
{
var text = "SELECT * FROM Data WHERE foo=#bar;";
using (var cmd = GetConnection (text)) {
cmd.Parameters.Add ("#bar", DbType.String).Value = match;
using (var reader = cmd.ExecuteReader ()) {
while (reader.Read ()) {
Console.WriteLine (reader["foo"]);
}
}
}
}