I'm using Google Analytics and that service has limit of 10 concurrent requests. I had to limit my API somehow, so I decided to use a semaphore, but it seems it doesn't work. All requests are triggered simultaneously. I can't find the problem in my code.
public async Task<SiteAnalyticsDTO> Handle(GetSiteAnalyticsParameter query)
{
var todayVisits = _googleAnalyticsService.GetTodayVisitsNumber();
var todayTraffic = _googleAnalyticsService.GetTodayTraffic();
var newAndReturningUsers = _googleAnalyticsService.GetNewAndReturningUsersNumber();
var averageSessionDuration = _googleAnalyticsService.GetAverageSessionDuration();
var deviceCategory = _googleAnalyticsService.GetSessionNumberByDeviceCategory();
var topPages = _googleAnalyticsService.GetTodaysTopPages();
var guestsAndRegisteredUsers = _googleAnalyticsService.GetGuestsVsRegisteredUsers();
var averageNumberOfSessionsPerDay = _googleAnalyticsService.GetAverageSessionsNumber();
var visitsPerWeekday = _googleAnalyticsService.GetTrafficByWeekday();
var visitsByHours = _googleAnalyticsService.GetTrafficByTimeOfDay();
var usersByPrefectures = _googleAnalyticsService.GetUsersByPrefectures();
var usersByCountry = _googleAnalyticsService.GetUsersByCountry();
var tasks = new List<Task>()
{
todayVisits, todayTraffic, newAndReturningUsers,
averageSessionDuration, deviceCategory, topPages,
guestsAndRegisteredUsers, averageNumberOfSessionsPerDay, visitsPerWeekday,
visitsByHours, usersByPrefectures, usersByCountry
};
var throttler = new SemaphoreSlim(MaxRequests, MaxRequests);
foreach(var task in tasks)
{
await throttler.WaitAsync();
try
{
await task;
await Task.Delay(1000); // It's important due to limits of Google Analytics requests (10 queries per second per IP address)
}
finally
{
throttler.Release();
}
}
await Task.WhenAll(tasks);
return new SiteAnalyticsDTO()
{
TodayVisits = await todayVisits,
TodayTraffic = await todayTraffic,
NewAndReturningUsers = await newAndReturningUsers,
AverageSessionDuration = await averageSessionDuration,
DeviceCategory = await deviceCategory,
TopPages = await topPages,
GuestsAndRegisteredUsers = await guestsAndRegisteredUsers,
AverageNumberOfSessionsPerDay = await averageNumberOfSessionsPerDay,
VisitsPerWeekday = await visitsPerWeekday,
VisitsByHours = await visitsByHours,
UsersByPrefectures = await usersByPrefectures,
UsersByCountry = await usersByCountry
};
}
And here is some example method for Google Analytics call:
public async Task<int> GetTodayVisitsNumber(List<long> listingIds = null)
{
string filter = GetFilter(listingIds);
var getReportsRequest = GetReportsRequestModel(GetTodayDateRange(), "ga:sessionCount", "ga:sessions", _configuration.MainViewId, filter);
var response = await _service.Reports.BatchGet(getReportsRequest).ExecuteAsync();
Console.WriteLine(response);
var data = response.Reports.FirstOrDefault();
return Convert.ToInt32(data?.Data.Totals[0].Values[0]);
}
All requests are triggered simultaneously.
Let's take a look here
var todayVisits = _googleAnalyticsService.GetTodayVisitsNumber();
var todayTraffic = _googleAnalyticsService.GetTodayTraffic();
var newAndReturningUsers = _googleAnalyticsService.GetNewAndReturningUsersNumber();
var averageSessionDuration = _googleAnalyticsService.GetAverageSessionDuration();
var deviceCategory = _googleAnalyticsService.GetSessionNumberByDeviceCategory();
var topPages = _googleAnalyticsService.GetTodaysTopPages();
var guestsAndRegisteredUsers = _googleAnalyticsService.GetGuestsVsRegisteredUsers();
var averageNumberOfSessionsPerDay = _googleAnalyticsService.GetAverageSessionsNumber();
var visitsPerWeekday = _googleAnalyticsService.GetTrafficByWeekday();
var visitsByHours = _googleAnalyticsService.GetTrafficByTimeOfDay();
var usersByPrefectures = _googleAnalyticsService.GetUsersByPrefectures();
var usersByCountry = _googleAnalyticsService.GetUsersByCountry();
You are storing the result of each of these methods. When you use the parenthesis tokens such as "methodName();" you invoke the method and store the result in var.
You then store the results of these methods in a list, then await each of them with a Semaphore to limit the number of tasks that can be awaited at once.
The issue is: each await finishes immediately because you already waited(synchronously) them when you initially invoked them above.
This leads you to believe that the SemaphoreSlim isn't working, because if each Task immediately returns when awaited(because they've already been invoked) then there is no time between them.
Store the async methods for later instead of invoking them all at once.
You can't store a delegate like those in a var you have to store them in an explicitly typed variable Func<TResult>.
For example:
Func<Task<object>> todayVisits = _googleAnalyticsService.GetTodayVisitsNumber;
editors note, I have no clue what these methods return I substituted object to be as general as possible
Now - it would be pretty cumbersome if we stored each one in a variable, so instead of storing them in individual variables let's just put them directly in a list like this:
var awaitableTasks = new List<Func<Task<object>>>()
{
_googleAnalyticsService.GetTodayVisitsNumber,
_googleAnalyticsService.GetTodayTraffic,
_googleAnalyticsService.GetNewAndReturningUsersNumber,
_googleAnalyticsService.GetAverageSessionDuration,
_googleAnalyticsService.GetSessionNumberByDeviceCategory,
_googleAnalyticsService.GetTodaysTopPages,
_googleAnalyticsService.GetGuestsVsRegisteredUsers,
_googleAnalyticsService.GetAverageSessionsNumber,
_googleAnalyticsService.GetTrafficByWeekday,
_googleAnalyticsService.GetTrafficByTimeOfDay,
_googleAnalyticsService.GetUsersByPrefectures,
_googleAnalyticsService.GetUsersByCountry
};
Because these new objects themselves aren't tasks but rather methods that return a Task we have to change how we store and invoke them, for this we'll use a local method so I'll go over each of the changes I made.
Let's create that Semaphore and create somewhere we can put tasks to keep track of them.
Let's also create somewhere we can store the results of each of those tasks when we await them.
var throttler = new SemaphoreSlim(MaxRequests, MaxRequests);
var tasks = new List<Task>();
ConcurrentDictionary<string, object> results = new();
Let's create a local method with a couple of responsibilities
Accept a Func<Task<object>> as a param
Await the method
Put the result of that method somewhere we can get it later
Release the Semphore even if it encounters an error
async Task Worker(Func<Task<object>> awaitableFunc)
{
try
{
resultDict.TryAdd(awaitableFunc.GetMethodInfo().Name, await awaitableFunc());
}
finally
{
throttler.Release();
}
}
Editors note: You can pull the same thing off with a lambda expression, but I prefer to use a local method for clarity and formatting.
Start the workers and store the tasks they return.
That way.. if they aren't done by the time the last couple are created, then we can wait for them to finish before creating the final object(since we are going to need all of the results they provide to create the final object).
foreach (var task in awaitableTasks)
{
await throttler.WaitAsync();
tasks.Add(Task.Run(() => Worker(task)));
}
// wait for the tasks to finish
await Task.WhenAll(tasks);
Create the final object then return it.
return new SiteAnalyticsDTO()
{
TodayVisits = resultDict[nameof(_googleAnalyticsService.GetTodayVisitsNumber)],
TodayTraffic = resultDict[nameof(_googleAnalyticsService.GetTodayTraffic)],
NewAndReturningUsers = resultDict[nameof(_googleAnalyticsService.GetNewAndReturningUsersNumber)],
AverageSessionDuration = resultDict[nameof(_googleAnalyticsService.GetAverageSessionDuration)],
DeviceCategory = resultDict[nameof(_googleAnalyticsService.GetSessionNumberByDeviceCategory)],
TopPages = resultDict[nameof(_googleAnalyticsService.GetTodaysTopPages)],
GuestsAndRegisteredUsers = resultDict[nameof(_googleAnalyticsService.GetGuestsVsRegisteredUsers)],
AverageNumberOfSessionsPerDay = resultDict[nameof(_googleAnalyticsService.GetAverageSessionsNumber)],
VisitsPerWeekday = resultDict[nameof(_googleAnalyticsService.GetTrafficByWeekday)],
VisitsByHours = resultDict[nameof(_googleAnalyticsService.GetTrafficByTimeOfDay)],
UsersByPrefectures = resultDict[nameof(_googleAnalyticsService.GetUsersByPrefectures)],
UsersByCountry = resultDict[nameof(_googleAnalyticsService.GetUsersByCountry)]
};
Wrap it all together and I think we have something that might work, or at least be easy to modify to meet your needs.
public static async Task<SiteAnalyticsDTO> Handle(GetSiteAnalyticsParameter query)
{
// store these methods so we can iterate and execute them later
var awaitableTasks = new List<Func<Task<object>>>()
{
_googleAnalyticsService.GetTodayVisitsNumber,
_googleAnalyticsService.GetTodayTraffic,
_googleAnalyticsService.GetNewAndReturningUsersNumber,
_googleAnalyticsService.GetAverageSessionDuration,
_googleAnalyticsService.GetSessionNumberByDeviceCategory,
_googleAnalyticsService.GetTodaysTopPages,
_googleAnalyticsService.GetGuestsVsRegisteredUsers,
_googleAnalyticsService.GetAverageSessionsNumber,
_googleAnalyticsService.GetTrafficByWeekday,
_googleAnalyticsService.GetTrafficByTimeOfDay,
_googleAnalyticsService.GetUsersByPrefectures,
_googleAnalyticsService.GetUsersByCountry
};
// create a way to limit the number of concurrent requests
var throttler = new SemaphoreSlim(MaxRequests, MaxRequests);
// create a place to store the tasks we create
var finalTasks = new List<Task>();
// make sure we have some where to put our results
ConcurrentDictionary<string, object> resultDict = new();
// make a worker that accepts one of those methods, invokes it
// then adds the result to the dict
async Task Worker(Func<Task<object>> awaitableFunc)
{
try
{
resultDict.TryAdd(awaitableFunc.GetMethodInfo().Name, await awaitableFunc());
}
finally
{
// make sure even if we encounter an error we still release the semphore
throttler.Release();
}
}
// iterate over the tasks, wait for the sempahore
// when we get a slot, create a worker and send it to the background
foreach (var task in awaitableTasks)
{
await throttler.WaitAsync();
finalTasks.Add(Task.Run(() => Worker(task)));
}
// wait for any remaining tasks to finish up in the background if they are still running
await Task.WhenAll(finalTasks);
// create the return object from the results of the dictionary
return new SiteAnalyticsDTO()
{
TodayVisits = resultDict[nameof(_googleAnalyticsService.GetTodayVisitsNumber)],
TodayTraffic = resultDict[nameof(_googleAnalyticsService.GetTodayTraffic)],
NewAndReturningUsers = resultDict[nameof(_googleAnalyticsService.GetNewAndReturningUsersNumber)],
AverageSessionDuration = resultDict[nameof(_googleAnalyticsService.GetAverageSessionDuration)],
DeviceCategory = resultDict[nameof(_googleAnalyticsService.GetSessionNumberByDeviceCategory)],
TopPages = resultDict[nameof(_googleAnalyticsService.GetTodaysTopPages)],
GuestsAndRegisteredUsers = resultDict[nameof(_googleAnalyticsService.GetGuestsVsRegisteredUsers)],
AverageNumberOfSessionsPerDay = resultDict[nameof(_googleAnalyticsService.GetAverageSessionsNumber)],
VisitsPerWeekday = resultDict[nameof(_googleAnalyticsService.GetTrafficByWeekday)],
VisitsByHours = resultDict[nameof(_googleAnalyticsService.GetTrafficByTimeOfDay)],
UsersByPrefectures = resultDict[nameof(_googleAnalyticsService.GetUsersByPrefectures)],
UsersByCountry = resultDict[nameof(_googleAnalyticsService.GetUsersByCountry)]
};
}
The problem of your setup is that all tasks are started at the same time, and only their awaiting is throttled. Throttling the awaiting has no useful effect. Only your continuations are delayed. The target service receives all the requests in bulk.
My suggestion is to use a dedicated class to encapsulate the throttling logic. It seems that you need to limit both the concurrency and the rate of sending the requests, and each one of these limitations can be achieved by using a separate SemaphoreSlim. Here is a simple implementation:
public class ThrottledExecution
{
private readonly SemaphoreSlim _concurrencySemaphore;
private readonly SemaphoreSlim _delaySemaphore;
private readonly TimeSpan _delay;
public ThrottledExecution(int concurrencyLimit, TimeSpan rateLimitTime,
int rateLimitCount)
{
// Arguments validation omitted
_concurrencySemaphore = new SemaphoreSlim(concurrencyLimit, concurrencyLimit);
_delaySemaphore = new SemaphoreSlim(rateLimitCount, rateLimitCount);
_delay = rateLimitTime;
}
public async Task<TResult> Run<TResult>(Func<Task<TResult>> action)
{
await _delaySemaphore.WaitAsync();
ScheduleDelaySemaphoreRelease();
await _concurrencySemaphore.WaitAsync();
try { return await action().ConfigureAwait(false); }
finally { _concurrencySemaphore.Release(); }
}
private async void ScheduleDelaySemaphoreRelease()
{
await Task.Delay(_delay).ConfigureAwait(false);
_delaySemaphore.Release();
}
}
And here is how you could use it:
public async Task<SiteAnalyticsDTO> Handle(GetSiteAnalyticsParameter query)
{
var throttler = new ThrottledExecution(MaxRequests, TimeSpan.FromSeconds(1), 1);
var todayVisits = throttler.Run(() => _service.GetTodayVisitsNumber());
var todayTraffic = throttler.Run(() => _service.GetTodayTraffic());
var newAndReturningUsers = throttler.Run(() => _service.GetNewAndReturningUsersNumber());
var averageSessionDuration = throttler.Run(() => _service.GetAverageSessionDuration());
var deviceCategory = throttler.Run(() => _service.GetSessionNumberByDeviceCategory());
var topPages = throttler.Run(() => _service.GetTodaysTopPages());
var guestsAndRegisteredUsers = throttler.Run(() => _service.GetGuestsVsRegisteredUsers());
var averageNumberOfSessionsPerDay = throttler.Run(() => _service.GetAverageSessionsNumber());
var visitsPerWeekday = throttler.Run(() => _service.GetTrafficByWeekday());
var visitsByHours = throttler.Run(() => _service.GetTrafficByTimeOfDay());
var usersByPrefectures = throttler.Run(() => _service.GetUsersByPrefectures());
var usersByCountry = throttler.Run(() => _service.GetUsersByCountry());
var tasks = new List<Task>()
{
todayVisits, todayTraffic, newAndReturningUsers,
averageSessionDuration, deviceCategory, topPages,
guestsAndRegisteredUsers, averageNumberOfSessionsPerDay, visitsPerWeekday,
visitsByHours, usersByPrefectures, usersByCountry
};
await Task.WhenAll(tasks);
return new SiteAnalyticsDTO()
{
TodayVisits = await todayVisits,
TodayTraffic = await todayTraffic,
NewAndReturningUsers = await newAndReturningUsers,
AverageSessionDuration = await averageSessionDuration,
DeviceCategory = await deviceCategory,
TopPages = await topPages,
GuestsAndRegisteredUsers = await guestsAndRegisteredUsers,
AverageNumberOfSessionsPerDay = await averageNumberOfSessionsPerDay,
VisitsPerWeekday = await visitsPerWeekday,
VisitsByHours = await visitsByHours,
UsersByPrefectures = await usersByPrefectures,
UsersByCountry = await usersByCountry,
};
}
It seems that a partially successful result is not useful to you, so you could consider adding some auto-cancellation logic inside the ThrottledExecution class. In case a task fails, all pending and subsequent asynchronous operations should be canceled.
Related
One question about parallel programming.
I need to POST 2 APIs (one after another) to get a Order list. First one is for getting token..
After I got the list, I need to POST 3 APIs (one after another) to integrate these Orders.
These 3 APIs don't accept arrays, so I need to send one by one. I cannot send batch data. With 1 thread it only integrates 10 orders in 1 minute. I need more performance. How can I run foreach part in parallel?
using System.Net.Http;
using System.Text;
using System.Text.Json;
namespace Order_Integrator
{
public class Program
{
static readonly HttpClient client = new HttpClient();
static async Task Main()
{
//Auth
var connectResponse = await client.PostAsync(connectUrl, connectContent);
var connectResponseString = await connectResponse.Content.ReadAsStringAsync();
var connect = JsonSerializer.Deserialize<connectResponse>(connectResponseString);
var token = connect.Token;
//Get Order List
var orderResponse = await client.PostAsync(orderUrl, orderContent);
var orderResponseString = await orderResponse.Content.ReadAsStringAsync();
var orders = JsonSerializer.Deserialize<orderResponse>(orderResponseString);
foreach (var order in orders)
{
//Get Order Details
//Generate getOrderDetailsContent with order
var getOrderDetailsResponse = await client.PostAsync(getOrderDetailsUrl, getOrderDetailsContent);
var getOrderDetailsResponseString = await getOrderDetailsResponse.Content.ReadAsStringAsync();
var getOrderDetails = JsonSerializer.Deserialize<getOrderDetailsResponse>(getOrderDetailsResponseString);
//Create Order
//Generate createOrderContent with GetOrderDetails
var createOrderResponse = await client.PostAsync(createOrderUrl, createOrderContent);
var createOrderResponseString = await createOrderResponse.Content.ReadAsStringAsync();
var createOrder = JsonSerializer.Deserialize<createOrderResponse>(createOrderResponseString);
//Create Log
//Generate createLogContent with CreateOrderResponse
var createLogResponse = await client.PostAsync(createLogUrl, createLogContent);
var createLogResponseString = await createLogResponse.Content.ReadAsStringAsync();
var createLog = JsonSerializer.Deserialize<createLogResponse>(createLogResponseString);
}
}
}
}
You can use Parallel.ForEachAsync
var options = new ParallelOptions()
{
MaxDegreeOfParallelism = 20
};
await Parallel.ForEachAsync(orders, options, async (OrderNumber, ct) => {
var getOrderDetailsResponse = await client.PostAsync(getOrderDetailsUrl, getOrderDetailsContent);
});
Since you're using asynchronous code, you should be using asynchronous concurrency, not (multithreaded) parallelism. Parallel.ForEachAsync will do both, but it's not available on older runtimes.
You can do just asynchronous concurrency by using Select and then Task.WhenAll:
var tasks = orders.Select(async order =>
{
//Get Order Details
//Generate getOrderDetailsContent with order
var getOrderDetailsResponse = await client.PostAsync(getOrderDetailsUrl, getOrderDetailsContent);
var getOrderDetailsResponseString = await getOrderDetailsResponse.Content.ReadAsStringAsync();
var getOrderDetails = JsonSerializer.Deserialize<getOrderDetailsResponse>(getOrderDetailsResponseString);
//Create Order
//Generate createOrderContent with GetOrderDetails
var createOrderResponse = await client.PostAsync(createOrderUrl, createOrderContent);
var createOrderResponseString = await createOrderResponse.Content.ReadAsStringAsync();
var createOrder = JsonSerializer.Deserialize<createOrderResponse>(createOrderResponseString);
//Create Log
//Generate createLogContent with CreateOrderResponse
var createLogResponse = await client.PostAsync(createLogUrl, createLogContent);
var createLogResponseString = await createLogResponse.Content.ReadAsStringAsync();
var createLog = JsonSerializer.Deserialize<createLogResponse>(createLogResponseString);
return createLog;
}).ToList();
await Task.WhenAll(tasks);
My code returns Task<List<...>>.
There is a type conversion error from Task<List<...>> to List<...>.
Tell me, please, where I did not finish await ?
public async Task<List<DepartamentsResponse>> Handle(GetDepartmentsRequest token, CancellationToken cancellationToken)
{
var departments = await _departmentApiClient.GetReception(token.accessToken, OdpgDepartmentType.Reception);
var result = departments.ConvertAll(async d => new DepartamentsResponse
{
FederalDistrict = GetFederalDistrictCode(d.FederalRegion.DistrictCode),
SubjectName = d.Name,
Supervisor = GetDirector(d.Users.Where(u => u.InOdpgRole(OdpgUserRole.Director)).FirstOrDefault()),
ContactsSupervisor = GetContacts(d.Users.Where(u => u.InOdpgRole(OdpgUserRole.Director)).FirstOrDefault()),
Schedule = "C 9:00 18:00",
ReceptionContacts = await GetAddressAsync(d.Addresses.FirstOrDefault(d => d.AddressType == DepartmentAddressType.Reception), token)
});
return result;
}
private async Task<string> GetAddressAsync(DepartmentAddressDto? address, GetDepartmentsRequest token)
{
if (address != null)
{
var fullAddress = await _fiasApiClient.GetFullAddress(token.accessToken,
new ESOG.Fias.Api.Model.GetFullAddressRequest
{ BaseAddressId = address.FiasId, Building = address.Building, Room = address.Room });
//var res = JsonConvert.DeserializeObject<DepartmentAddress>(fullAddress);
return fullAddress;
}
return "";
}
GetFederalDistrictCode, GetDirector, GetContacts - these methods are not asynchronous
It should just return a List<>, not Task<List<>>
Your call to departments.ConvertAll is returning a List<Task<DepartamentsResponse>>, but you want a Task<List<DepartamentsResponse>>.
Look at the Task.WhenAll method for how you would convert a collection of Tasks to a single Task that returns a collection. Then await that single Task and build your final list.
I am using the AsyncEnumerator library by #Serge Semenov, and after getting the orders, creating the ConcurrentBag and setting count to 0, nothing happens - can anyone see what's missing here? It never hits the processedOrders.ToList() line when debugging.
private async Task<List<OrderDto>> ProcessOrderEvents(DateTime cob)
{
var orders = await _client.GetOrderEvents(cob);
var processedOrders = new ConcurrentBag<OrderDto>();
var count = 0;
await orders.ParallelForEachAsync(async order =>
{
var orderDto = new FidessaOrderDto {Cob = cob};
var orderId = order.Ids.PrimaryId;
Debug.WriteLine($"Processing OrderId: {orderId} #= {++count}");
var instrumentDetails = await GetInstrumentDetails(order);
if (instrumentDetails.Results.Any())
{
//...
}
var tradeDetails = await _client.GetTrade(tradeId: fill, cob);
processedOrders.Add(orderDto);
}, maxDegreeOfParallelism:2);
return processedOrders.ToList();
}
Update: Adding Test root.
[TestMethod]
public async Task Test_20_07_2022()
{
var service = new DataService(new ApiClient());
var data = await service.ProcessData(new DateTime(2022, 07, 20), ReportDataType.Orders);
Assert.AreEqual(920, data.Count);
}
public async Task<List<OrderDto>> ProcessData(DateTime cob,
ReportDataType dataType)
{
await PopulateRouteEvents(cob);
var reportData = new List<OrderDto>();
if (dataType == ReportDataType.Orders)
reportData = await ProcessOrderEventsTpl(cob);
return reportData;
}
How to get the individual API call status success response in C#.
I am creating a mobile application using Xamarin Forms,
In my application, I need to prefetch certain information when app launches to use the mobile application.
Right now, I am calling the details like this,
public async Task<Response> GetAllVasInformationAsync()
{
var userDetails = GetUserDetailsAsync();
var getWageInfo = GetUserWageInfoAsync();
var getSalaryInfo = GetSalaryInfoAsync();
await Task.WhenAll(userDetails,
getWageInfo,
getSalaryInfo,
);
var resultToReturn = new Response
{
IsuserDetailsSucceeded = userDetails.Result,
IsgetWageInfoSucceeded = getWageInfo.Result,
IsgetSalaryInfoSucceeded = getSalaryInfo.Result,
};
return resultToReturn;
}
In my app I need to update details based on the success response. Something like this (2/5) completed. And the text should be updated whenever we get a new response.
What is the best way to implement this feature? Is it possible to use along with Task.WhenAll. Because I am trying to wrap everything in one method call.
In my app I need to update details based on the success response.
The proper way to do this is IProgress<string>. The calling code should supply a Progress<string> that updates the UI accordingly.
public async Task<Response> GetAllVasInformationAsync(IProgress<string> progress)
{
var userDetails = UpdateWhenComplete(GetUserDetailsAsync(), "user details");
var getWageInfo = UpdateWhenComplete(GetUserWageInfoAsync(), "wage information");
var getSalaryInfo = UpdateWhenComplete(GetSalaryInfoAsync(), "salary information");
await Task.WhenAll(userDetails, getWageInfo, getSalaryInfo);
return new Response
{
IsuserDetailsSucceeded = await userDetails,
IsgetWageInfoSucceeded = await getWageInfo,
IsgetSalaryInfoSucceeded = await getSalaryInfo,
};
async Task<T> UpdateWhenComplete<T>(Task<T> task, string taskName)
{
try { return await task; }
finally { progress?.Report($"Completed {taskName}"); }
}
}
If you also need a count, you can either use IProgress<(int, string)> or change how the report progress string is built to include the count.
So here's what I would do in C# 8 and .NET Standard 2.1:
First, I create the method which will produce the async enumerable:
static async IAsyncEnumerable<bool> TasksToPerform() {
Task[] tasks = new Task[3] { userDetails, getWageInfo, getSalaryInfo };
for (i = 0; i < tasks.Length; i++) {
await tasks[i];
yield return true;
}
}
So now you need to await foreach on this task enumerable. Every time you get a return, you know that a task has been finished.
int numberOfFinishedTasks = 0;
await foreach (var b in TasksToPerform()) {
numberOfFinishedTasks++;
//Update UI here to reflect the finished task number
}
No need to over-complicate this. This code will show how many of your tasks had exceptions. Your await task.whenall just triggers them and waits for them to finish. So after that you can do whatever you want with the tasks :)
var task = Task.Delay(300);
var tasks = new List<Task> { task };
var faultedTasks = 0;
tasks.ForEach(t =>
{
t.ContinueWith(t2 =>
{
//do something with a field / property holding ViewModel state
//that your view is listening to
});
});
await Task.WhenAll(tasks);
//use this to respond with a finished count
tasks.ForEach(_ => { if (_.IsFaulted) faultedTasks++; });
Console.WriteLine($"{tasks.Count() - faultedTasks} / {tasks.Count()} completed.");
.WhenAll() will allow you to determine if /any/ of the tasks failed, they you just count the tasks that have failed.
public async Task<Response> GetAllVasInformationAsync()
{
var userDetails = GetUserDetailsAsync();
var getWageInfo = GetUserWageInfoAsync();
var getSalaryInfo = GetSalaryInfoAsync();
await Task.WhenAll(userDetails, getWaitInfo, getSalaryInfo)
.ContinueWith((task) =>
{
if(task.IsFaulted)
{
int failedCount = 0;
if(userDetails.IsFaulted) failedCount++;
if(getWaitInfo.IsFaulted) failedCount++;
if(getSalaryInfo.IsFaulted) failedCount++;
return $"{failedCount} tasks failed";
}
});
var resultToReturn = new Response
{
IsuserDetailsSucceeded = userDetails.Result,
IsgetWageInfoSucceeded = getWageInfo.Result,
IsgetSalaryInfoSucceeded = getSalaryInfo.Result,
};
return resultToReturn;
}
I have a list of items, and for each item, I need to execute series of tasks.
For accessing data layer, I am using the following code:
public async Task<ExampleResult> GetExampleResultAsync(Parameter parameter, CancellationToken cancellationToken = default(CancellationToken))
{
GetCustomerResult result = null;
OracleConnection connection = this.Database.GetOracleConnection();
bool needClose = false;
if (connection.State != ConnectionState.Open)
{
await connection.OpenAsync(cancellationToken);
needClose = true;
}
try
{
using (OracleCommand cmd = connection.CreateCommand())
{
.... do the work
}
}
finally
{
if (needClose)
connection.Close();
}
return result;
}
however, this leads to concurrency and thus I am getting
the connection was not closed error.
The possible workaround for this that I ended up is to change the way of connecting to the database. I am thinking to use new connection instance for every request and surround these instances with using statement.
This would lead me lots of work and I would like to get an idea about best practices in the field for handling concurrency issues.
EDIT:
my caller function is below for your reference
public async Task<DomainResult<IList<MbRiskDto>>> QueryAsync(Action<MbrAccountAutoMatcherQueryParameter> parameter,
CancellationToken cancellationToken = default(CancellationToken))
{
parameter(_parameter);
var nonDeclaredMbrAccounts = await _nonDeclaredMbrAccountsQuery.QueryAsync(param => param.TransactionDate = _parameter.TransactionDate, cancellationToken);
if (nonDeclaredMbrAccounts.IsFailed)
nonDeclaredMbrAccounts.Errors.ForEach(error => _errors.Add(error));
var taskList = new List<Task<MbRiskDto>>();
var throttler = new SemaphoreSlim(initialCount: 10);
foreach (var nonDeclaredAccount in nonDeclaredMbrAccounts.Result)
{
await throttler.WaitAsync(cancellationToken);
var account = (AccountDto) nonDeclaredAccount.Clone();
var firstAccountHolder = Convert.ToInt32(account.AccountHolders.FirstOrDefault());
var task = Task.Run(async () =>
{
MbRiskDto result;
try
{
result = new MbRiskDto
{
KimNo = await GetKimNo(firstAccountHolder),
HesNo = account.AccountNo,
FinCode = await GetFinanceCode(account, firstAccountHolder, cancellationToken),
Unvan = account.SMA.Substring(0, Math.Min(account.SMA.Length, 54))
};
}
finally
{
throttler.Release();
}
return result;
}, cancellationToken);
taskList.Add(task);
}
var taskResult = await Task.WhenAll(taskList);
return DomainResult<IList<MbRiskDto>>.Success(taskResult);
}
According to the comments I removed Task.Run() clause and replaced with the code below. Right now, I believe that my concurrency problems are solved.
Func<Task<MbRiskDto>> mbRiskTask = async () => new MbRiskDto
{
KimNo = await GetKimNo(firstAccountHolder, cancellationToken),
HesNo = account.AccountNo,
FinCode = await GetFinanceCode(account, firstAccountHolder, cancellationToken),
Unvan = account.SMA.Substring(0, Math.Min(account.SMA.Length, 54))
};
taskList.Add(mbRiskTask.Invoke());
Thus, Resulting code can be seen below
public async Task<DomainResult<IList<MbRiskDto>>> QueryAsync(Action<MbrAccountAutoMatcherQueryParameter> parameter,
CancellationToken cancellationToken = default(CancellationToken))
{
parameter(_parameter);
var nonDeclaredMbrAccounts = await _nonDeclaredMbrAccountsQuery.QueryAsync(param => param.TransactionDate = _parameter.TransactionDate, cancellationToken);
if (nonDeclaredMbrAccounts.IsFailed)
nonDeclaredMbrAccounts.Errors.ForEach(error => _errors.Add(error));
var taskList = new List<Task<MbRiskDto>>();
foreach (var nonDeclaredAccount in nonDeclaredMbrAccounts.Result)
{
var account = (AccountDto) nonDeclaredAccount.Clone();
var firstAccountHolder = Convert.ToInt32(account.AccountHolders.FirstOrDefault());
Func<Task<MbRiskDto>> mbRiskTask = async () => new MbRiskDto
{
KimNo = await GetKimNo(firstAccountHolder, cancellationToken),
HesNo = account.AccountNo,
FinCode = await GetFinanceCode(account, firstAccountHolder, cancellationToken),
Unvan = account.SMA.Substring(0, Math.Min(account.SMA.Length, 54))
};
//var task = Task.Run();
taskList.Add(mbRiskTask.Invoke());
}
var taskResult = await Task.WhenAll(taskList);
return DomainResult<IList<MbRiskDto>>.Success(taskResult);
}
EDIT:
With this refactoring, the operation takes approximately 4minutes to finish. Do you have any suggestion in order to make it work faster?