Develop Reference

How to make parallel database calls and bind to model ASP.NET Core MVC

I'm using ASP.NET Core MVC 2.0 and I've got an API that calls a method which opens a DB connection and inserts values into a model.
I'm hitting the same method 7 times and I'm just passing different information into it the parameters to build out my model. I HAVE to hit it seven different times, no rewrite of code will prevent this.
Instead of creating a loop and calling that method I wish to make parallel calls to the db so the response is faster. I've found multiple articles explaining how to do this e.g. How to properly make asynchronous / parallel database calls but there's enough difference that I can't seem to get it to work.
Following is the my method hitting the DB:
public async Task<RS_Model> Get_Results(RequestS_Model values)
{
//Deleted a bunch of code to keep it simple
string strQS = #"EXEC Get param1,param2,etc";
using (SqlConnection conSQL = new SqlConnection(connectionString))
{
using (SqlCommand cmdSQL = new SqlCommand(strQS, conSQL))
{
conSQL.Open();
using (SqlDataReader dtrSQL = cmdSQL.ExecuteReader())
{
while (dtrSQL.Read())
{
Double.TryParse(dtrSQL["Value1"].ToString(), out dblVal1);
} //Ends While
} //end SQLDataReader
} //Ends cmdSQL
} //ends using
results.Price = dblVal1;
return results;
} //Ends Get Results
My IActionResult for the api is:
[HttpGet]
public async Task<IActionResult> Get([FromQuery] RequestS_Model values)
{
SV_Results Results = new SV_Results();
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
RS_Model model = new RS_Model();
model.dblr[0] = await Results.Get_Results(values);
values.Parm1 = 2;
model.dblr[1] = await Results.Get_Results(values);
values.Parm1 = 3;
model.dblr[2] = await Results.Get_Results(values);
values.Parm1 = 4;
model.dblr[3] = await Results.Get_Results(values);
values.Parm1 = 5;
model.dblr[4] = await Results.Get_Results(values);
values.Parm1 = 6;
model.dblr[5] = await Results.Get_Results(values);
values.Parm1 = 7;
model.dblr[6] = await Results.Get_Results(values);
//int[] results = await Task.WhenAll(new Task<int>[] { task1, task2 });
return new OkObjectResult(model);
} //IActionResults
I know that I've forced them into a synchronous call be doing what I am, but I can't seem to make the 7 calls asynchronous and then wait for them to be all done before I build my final model. The final response needs to be in Json, but I haven't even gotten that far yet.

Instead of this:
model.dblr[0] = await Results.Get_Results(values);
model.dblr[1] = await Results.Get_Results(values);
model.dblr[2] = await Results.Get_Results(values);
model.dblr[3] = await Results.Get_Results(values);
model.dblr[4] = await Results.Get_Results(values);
model.dblr[5] = await Results.Get_Results(values);
model.dblr[6] = await Results.Get_Results(values);
Create a list of tasks and await them as a group:
var tasks = Enumerable.Range(0,7).Select( i => Results.Get_Results(values) ).ToList();
await Task.WhenAll(tasks);
for (int i=0; i<7; i++) model.dblr[i] = tasks[i].Result;

Moment of multiple tasks end while they return different objects

I've got 5 tasks like this:
private async Task<List<Places>> GetPlaces()
{
Stopwatch sw = new Stopwatch();
sw.Start();
var list5 = await GetPlacesTask();
sw.Stop();
Testowybox3.Text = sw.Elapsed.ToString()+" get places times";
return list5;
}
private async Task<List<Categories>> GetCategories()
{
Stopwatch sw = new Stopwatch();
sw.Start();
var list5 = await GetCategoriesTask();
sw.Stop();
Testowybox2.Text = sw.Elapsed.ToString()+" get categories times";
return list5;
}
They differs by returned value for e.g. my second task:
private async Task<EventsInstance[]> GetDetailedData()
{
Stopwatch sw = new Stopwatch();
sw.Start();
list4 = await GetDetailedDataTask();
List<EventsListPrint> lista = new List<EventsListPrint>();
foreach (var VARIABLE in list4)
{
lista.Add(new EventsListPrint() { Date = string.Format("{0:dd.MM.yyyy}", VARIABLE.startDate.Date), Time = VARIABLE.startDate.TimeOfDay.ToString(), Category = VARIABLE.categoryId.ToString(), Where = VARIABLE.organizer.designation, What = VARIABLE.name });
}
fruitdatagrid2.ItemsSource = lista;
sw.Stop();
Testowybox.Text = sw.Elapsed.ToString()+" get detailed data times";
return list4;
}
private async Task<List<Organizers>> GetOrganizers()
{
Stopwatch sw = new Stopwatch();
sw.Start();
var list5 = await GetOrganizersTask();
sw.Stop();
Testowybox4.Text = sw.Elapsed + " get orgzanizers times";
return list5;
}
They are parsing Jsons from web. I'm collecting data from them like this:
var DetailedDataList = await GetDetailedData();
var CategoriesList = await GetCategories();
var PlacesList = await GetPlaces();
var OrganizersList = await GetOrganizers();
var EventsList = await GetEvents();
How can I assure that they're all done? I tried:
Task.WhenAll(GetDetailedData(), GetCategories(), GetPlaces(), GetOrganizers(), GetEvents());
but it seems that I cannot retrieve values then. What I really want to achieve is to create method which would be aware of successful completion of tasks and then operarate on this data in my WPF GUI. I could maybe chain them in one, but I just don't know how to do so. Everything works fine as far as I'm using one of the returned list. When trying to do linq on one list using another list, there is a chance that the another one isn't prepared yet and it simply crashes my app.

When you await Task.WhenAll, when the line resumes, you can be assured that the tasks that were supplied will be in a completed state.
var detailedDataTask = GetDetailedData();
var categoriesTask = GetCategories();
//etc
await Task.WhenAll(detailedDataTask, categoriesTask); //etc
However, the variables that contain the tasks (detailedDataTask) are still Task<T>. Not the type that they wrap, T.
How do we unwrap the completed tasks?
We can either (unintuitively, but preferably) await the completed task:
var detailedData = await detailedDataTask;
or (because we know that the tasks are complete) we can go directly for the .Result property without fear of blocking...
var detailedData = detailedDataTask.Result;
Because of the additional magic that happens when the runtime untangles your asynchronous exceptions, you should probably prefer awaiting all the way.

Replacing foreach with Threading / Parallelism

I'm trying to improve the execution time for a operation.
I get a list of users, and by the user id I'm executing a method from a service which calculates some data. After i get the data by the user id, I set it to every entity, and than just save the context.
Currently with this simple code it take about 1 hour. Approximately 21 - 24 seconds per 100 users. But it depends on each and every user.
using (var ctx = new MyContext())
{
var users = ctx.Users.Where(u => u.Status != -1).ToList();
foreach (var user in users)
{
user.ProfilePercentage = await UserAlgorithm.CalculatePercentage(user.Id);
}
await ctx.SaveChangesAsync();
}
I also tried like this, but takes even longer:
Parallel.ForEach(
users,
new ParallelOptions { MaxDegreeOfParallelism = -1 },
user => {
user.ProfilePercentage = await UserAlgorithm.CalculatePercentage(user.Id); }
);
**CalculatePercentage **
public async Task<decimal> CalculatePercentage (int userId)
{
decimal completeness = 0;
bool? hasEducationsTags = null; //--> nullable boolea will handle 3 cases on SP side
bool? hasPosition = null; //--> nullable boolea will handle 3 cases on SP side
///-- check if user has any education
if(await SchoolService.HasEducations(userId).ConfigureAwait(false))
{
hasEducationsTags = await SchoolService.HasAnyFieldsOfStudy(user=>user.Id == userId);
}
///-- check if user can set position
if (await UserInfoService.CanSetLocation(userId).ConfigureAwait(false))
{
///-- check if position was set
string accountName = await UserInfoService.GetAccountName(userId).ConfigureAwait(false);
hasPosition = await HasUserSetPosition(accountName).ConfigureAwait(false);
}
///-- call store procedure
using (SqlConnection sqlConnection = ConnectionFactory.Create())
{
await sqlConnection.OpenAsync().ConfigureAwait(false);
using (SqlCommand sqlCommand = sqlConnection.CreateCommand())
{
sqlCommand.CommandType = CommandType.StoredProcedure;
sqlCommand.CommandText = "ProfileCompletionAlgorithm";
sqlCommand.Parameters.AddWithValue("#USER_ID", userId);
sqlCommand.Parameters.AddWithValue("#HAS_POSITION", hasPosition);
sqlCommand.Parameters.AddWithValue("#HAS_SCHOOL_TAGS", hasEducationsTags);
///-- not using OUTPUT parameter but by getting the return result
SqlParameter sqlParameter = sqlCommand.Parameters.Add("#RESULT", SqlDbType.Decimal);
sqlParameter.Direction = ParameterDirection.ReturnValue;
sqlCommand.ExecuteNonQuery();
completeness = Convert.ToDecimal(sqlParameter.Value);
}
}
return completeness;
}
Some of the links that I've checked:
https://msdn.microsoft.com/en-us/library/system.threading.tasks.paralleloptions.maxdegreeofparallelism.aspx
https://msdn.microsoft.com/en-us/library/system.threading.tasks.parallel.foreach(v=vs.110).aspx
How can I optimize this logic in order to get the best time?
If you chose to downvote please explain
Thanks.

Since your code is I/O-bound (not CPU-bound), throwing more threads (or parallelism) at the problem isn't going to help. What you want is asynchronous concurrency (Task.WhenAll), not parallel concurrency (Parallel):
using (var ctx = new MyContext())
{
var users = ctx.Users.Where(u => u.Status != -1);
var tasks = users.Select(UpdateUserProfilePercentageAsync);
await Task.WhenAll(tasks);
await ctx.SaveChangesAsync();
}
private async Task UpdateUserProfilePercentageAsync(User user)
{
user.ProfilePercentage = await UserAlgorithm.CalculatePercentage(user.Id);
}
You may find out that your database or other services don't appreciate being hammered so hard, in which case you could do some throttling. E.g., for 20 at a time:
private SemaphoreSlim _throttle = new SemaphoreSlim(20);
private async Task UpdateUserProfilePercentageAsync(User user)
{
await _throttle.WaitAsync();
try
{
user.ProfilePercentage = await UserAlgorithm.CalculatePercentage(user.Id);
}
finally { _throttle.Release(); }
}

Multithreading and TPL do not speed up execution C#

I have a program that makes use of SQL Server to pull information from a database and then perform a series of insertions into other tables and also send an email with the data that was retireved.
The program takes around 3 and a half minutes to execute and there is only 5 rows of data in the database. I am trying to reduce this time in any way I can and have tried multithreading which seems to slow it down further, and TPL which neither increases nor reduces the time. Does anyone know why I am not seeing performance improvements?
I am using an Intel Core i5 which I know has 2 cores so using more than 2 cores I understand will reduce performance. Here is how I am incorporating the use of tasks:
private static void Main(string[] args)
{
Util util = new Util(); //Util object
List<Data> dataList = new List<Data>(); //List of Data Objects
//Reads each row of data and creates Data obj for each
//Then adds each object to the list
dataList = util.getData();
var stopwatch = Stopwatch.StartNew();
var tasks = new Task[dataList.Count];
int i = 0; //Count
foreach (Data data in dataList)
{
//Perform insertions and send email with data
tasks[i++] = Task.Factory.StartNew(() => util.processData(data));
}
Task.WaitAll(tasks); //Wait for completion
Console.WriteLine("DONE: {0}", stopwatch.ElapsedMilliseconds);
}
Util Class:
class Util
{
// create and open a connection object
SqlConnection conn = new SqlConnection("**Connection String**");
//Gets all results from table, and adds object to list
public List<Data> getData()
{
conn.Open();
SqlCommand cmd = new SqlCommand("REF.GET_DATA", conn);
cmd.CommandType = CommandType.StoredProcedure;
SqlDataReader reader = cmd.ExecuteReader();
List<Data> dataList = new List<Data>();
while (reader.Read())
{
//** Take data from table and assigns them to variables
//** Removed for simplicity
Data data= new Data(** pass varaibles here **);
dataList.Add(data); //Add object to datalist
}
return dataList;
}
public void processData(Data data)
{
//** Perform range of trivial operations on data
//** Removed for simplicity
byte[] results = data.RenderData(); //THIS IS WHAT TAKES A LONG TIME TO COMPLETE
data.EmailFile(results);
return;
} //END handleReport()
}
Am I using tasks in the wrong place? Should I instead be making use of parellelism in the util.processData() method? I also tried using await and async around the util.processData(data) call in the main method with no improvements.
EDIT:
Here is the renderData function:
//returns byte data of report results which will be attatched to email.
public byte[] RenderData(string format, string mimeType, ReportExecution.ParameterValue[] parameters)
{
ReportExecutionService res = new ReportExecutionService();
res.Credentials = System.Net.CredentialCache.DefaultCredentials;
res.Timeout = 600000;
//Prepare Render arguments
string historyID = null;
string deviceInfo = String.Empty;
string extension = String.Empty;
string encoding = String.Empty;
ReportExecution.Warning[] warnings = null;
string[] streamIDs = null;
byte[] results = null;
try
{
res.LoadReport(reportPath, historyID);
res.SetExecutionParameters(parameters, "en-gb"); //"/LSG Reporting/Repossession Sales (SAL)/SAL004 - Conveyancing Matter Listing"
results = res.Render(format, deviceInfo, out extension, out mimeType, out encoding, out warnings, out streamIDs);
break;
}
catch (Exception ex)
{
Console.WriteLine(ex.StackTrace)
}
return results;
}

Using the async Datareader

We are pretty new to this, but we're trying to make use of the new async datareader
we have been following the sample from http://blogs.msdn.com/b/adonet/archive/2012/07/15/using-sqldatareader-s-new-async-methods-in-net-4-5-beta-part-2-examples.aspx
but experiencing problems.
What we are trying to achieve is :-
The stored proc returns 2 tables, using a datareader, we are trying to populate 2 models asynchronously, ie the 2 models will be created at the same time and not have to wait for the first model to be built before the second one can begin
(If that makes sense?)
Below is what we have at the moment :-
public async Task<UsertResultsModel> GetResults(string user_ID)
{
UsertResultsModel rm = new UsertResultsModel();
List<TableColumn> ltc = new List<TableColumn>();
List<List<TableColumn>> lltc = new List<List<TableColumn>>();
var col = 0;
try
{
using (SqlConnection con = new SqlConnection(Config.DatabaseStringCDA))
{
await con.OpenAsync();
SqlCommand com = new SqlCommand(#"USP_GET_USER_RESULTS", con);
com.CommandType = CommandType.StoredProcedure;
using (SqlDataReader rdr = await com.ExecuteReaderAsync())
{
col = rdr.FieldCount;
while (await rdr.ReadAsync())
{
for (int i = 0; i < rdr.FieldCount; i++)
{
string colName = rdr.GetName(i).ToSafeString();
ltc.Add(new TableColumn(colName, rdr[i].ToSafeString(), 50, EnumColumnType.String, colName));
}
lltc.Add(ltc);
}
rm.Summary = new TableViewModel { Grid = lltc };
await rdr.NextResultAsync();
lltc = new List<List<TableColumn>>();
while (rdr.Read())
{
for (int i = 0; i < rdr.FieldCount; i++)
{
string colName = rdr.GetName(i).ToSafeString();
ltc.Add(new TableColumn(colName, rdr[i].ToSafeString(), 50, EnumColumnType.String, colName));
}
lltc.Add(ltc);
}
rm.Trend = new TableViewModel { Grid = lltc };
}
}
}
catch (Exception ex)
{
log.Error(MethodBase.GetCurrentMethod(), ex);
return null;
}
await Task.WhenAll();
return rm;
// return lltc;
}
however, stepping through the code, as soon as we hit any await, The application simply loads, and no further code is hit..
Any advice or recommendations for this would be greatly appreciated.

The async part is the DB I/O, meaning that while you are awaiting an open, or a read, your app is free to do other things rather than blocking on the DB.
await/async allows you to write code in a sequential manner even though parts may take a while. The code flows exactly as you would expect in sequence, the line after each await doesn't run until the awaited item has finished, but the current thread is unwound back up the stack so it can get on with other things.
If you want to do two things at once using await/async, then you need to wrap each one in a function that returns a Task just like traditional TPL. That way you can keep hold of the tasks WITHOUT awaiting their results until you want to.