Develop Reference

Task.WhenAll() not working as expected without visual studio debugger attached

I have a question about async programming and Task.WhenAll(). I have a code snippet that downloads a folder from google drive and it works as expected when i am debugging my code. However when I run the application without debugger the download function takes atleast 4x the time it takes when it runs with the debugger. I also get crash logs with TaskCancelled exceptions (System.Threading.Tasks.TaskCanceledException: A task was canceled.) which do not happen with the debugger attached. What needs to be changed for the code to work as expected without debugger attached. NB this snippet downloads +- 1000 files in about 22-25 seconds with debugger and 2min+ without debugger.
public static async Task<bool> DownloadFolder(CloudDataModel.File file, string path, params string[] exclude)
{
try
{
if (file != null && !string.IsNullOrEmpty(file.id))
{
List<string> toExclude = new List<string>();
if(exclude != null)
{
toExclude = exclude.ToList();
}
List<Task> downloadFilesTask = new List<Task>();
var files = await file.GetFiles();
foreach (var f in files)
{
var task = f.Download(path);
downloadFilesTask.Add(task);
}
var folders = await file.GetFoldersAsync();
foreach (var folder in folders)
{
if (toExclude.Contains(folder.name))
{
continue;
}
Task task = null;
if (path.Equals(Statics.ProjectFolderName))
{
task = DownloadFolder(folder, folder.name);
}
else
{
task = DownloadFolder(folder, Path.Combine(path, folder.name));
}
downloadFilesTask.Add(task);
}
var array = downloadFilesTask.ToArray();
await Task.WhenAll(array);
return true;
}
}
catch (Exception e)
{
Crashes.TrackError(e);
}
return false;
}
Edit
after some more trial and error the fault has been identified.
the downloading of the file was the cause of the unexpected behaviour
public static async Task<StorageFile> DownloadFile(CloudDataModel.File file, string destinationFolder)
{
try
{
if (file != null)
{
Debug.WriteLine($"start download {file.name}");
if (file.mimeType == Statics.GoogleDriveFolderMimeType)
{
Debug.WriteLine($"did not download resource, resource was folder instead of file. mimeType: {file.mimeType}");
return null;
}
var endpoint = Path.Combine(DownloadFileEndpoint, $"{file.id}?alt=media");
// this would cause the unexpected behaviour
HttpResponseMessage response = await client.GetAsync(endpoint);
StorageFile downloadedFile;
using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
{
var memstream = new MemoryStream();
StreamReader reader = new StreamReader(streamToReadFrom);
streamToReadFrom.Position = 0;
await streamToReadFrom.CopyToAsync(memstream);
downloadedFile = await fileHandler.SaveDownloadedCloudFile(memstream, file, destinationFolder);
Debug.WriteLine($"download finished {file.name}");
}
return downloadedFile;
}
return null;
}
catch (Exception e)
{
Crashes.TrackError(e);
return null;
}
}
After setting a timeout to the client (System.Net.Http.HttpClient) the code executed as expected.
client.Timeout = new TimeSpan(0,0,5);

So after some trial and error the downloading of the file was identified as the cause of the problem.
var task = f.Download(path);
the Download function implements a System.Net.Http.HttpClient wich was initialized without any kind of timeout. when one of the tasks had a timeout, all other tasks would also not execute as expected. to fix this issue a timeout of 5 seconds was added to the HttpClient object.
private static void InitializeClient()
{
HttpClientHandler handler = new HttpClientHandler { AllowAutoRedirect = true };
client = new HttpClient(handler);
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", AccountManager.LoggedInAccount.Token);
client.Timeout = new TimeSpan(0,0,5);
}
After this change the code would execute as expected.

My guess is that without the debugger the parallelism increases, and so the remote server is overflowed by requests and can't perform optimally. Your code does not have any provision for limiting the degree of concurrency/parallelism. One possible way to solve this problem is to use a SemaphoreSlim, as shown in this question (among many others).

The write operation doesn't update the characteristic value for iOS in xamarin forms

I'm creating a BLE application using xamarin forms. Everything is working fine in Android I'm able to read and write GATT characteristics. In iOS I'm able to read successfully but the write operation doesn't update the characteristics value. There is no error in the write operation as well it is executing but the characteristics value is not changing. I tried iOS native application called light blue there its working fine the characteristic value is updated I'm facing issue only in Xamarin forms app. This is my code
private async Task<string> ProcessDeviceInformationService(IService deviceInfoService)
{
try
{
await adapter.ConnectToDeviceAsync(device);
var sb = new StringBuilder("Getting information from Device Information service: \n");
var characteristics = deviceInfoService.GetCharacteristicsAsync();
var characteristic = await deviceInfoService.GetCharacteristicAsync(Guid.Parse("00002a2b-0000-1000-8000-00805F9B34FB"));
try
{
if (characteristic != null)
{
var sbnew = new StringBuilder("BLE Characteristics\n");
byte[] senddata = Encoding.UTF8.GetBytes(string.IsNullOrEmpty(SendMessageLabel.Text) ? "12" : SendMessageLabel.Text);
characteristic.ValueUpdated += (o, args) =>
{
var bytes = characteristic.Value;
};
await characteristic.WriteAsync(senddata);
string str = Encoding.UTF8.GetString(senddata);
sbnew.AppendLine($"Characteristics found on this device: {string.Join(", ", str.ToString())}");
CharactericsLabel.Text = sbnew.ToString();
}
}
catch (Exception ex)
{
//return ex.Message;
DisplayAlert("Notice", ex.Message.ToString(), "OK");
}
I tried delay and I also tried to get write without response from peripheral but it doesn't work. This is my peripheral code
// Current Time characteristic
BluetoothGattCharacteristic currentTime = new BluetoothGattCharacteristic(CURRENT_TIME,
//Read-only characteristic, supports notifications
BluetoothGattCharacteristic.PROPERTY_READ | BluetoothGattCharacteristic.PROPERTY_NOTIFY | BluetoothGattCharacteristic.PROPERTY_WRITE_NO_RESPONSE,
BluetoothGattCharacteristic.PERMISSION_READ | BluetoothGattCharacteristic.PROPERTY_WRITE);
BluetoothGattDescriptor configDescriptor = new BluetoothGattDescriptor(CLIENT_CONFIG,
//Read/write descriptor
BluetoothGattDescriptor.PERMISSION_READ | BluetoothGattDescriptor.PERMISSION_WRITE);
currentTime.addDescriptor(configDescriptor);
// Local Time Information characteristic
BluetoothGattCharacteristic localTime = new BluetoothGattCharacteristic(LOCAL_TIME_INFO,
//Read-only characteristic
BluetoothGattCharacteristic.PROPERTY_READ,
BluetoothGattCharacteristic.PERMISSION_READ);
BluetoothGattCharacteristic sampleText = new BluetoothGattCharacteristic sampleText = new BluetoothGattCharacteristic(SAMPLE_TEXT,
//Read-only characteristic
BluetoothGattCharacteristic.PROPERTY_WRITE_NO_RESPONSE | BluetoothGattCharacteristic.PROPERTY_READ | BluetoothGattCharacteristic.PROPERTY_NOTIFY,
BluetoothGattCharacteristic.PERMISSION_WRITE | BluetoothGattCharacteristic.PERMISSION_READ);
I have no clue how to fix this any suggestions.I tried even Semaphore but it didn't help as you can see in my code
private static async Task<string> WriteAndWaitForResponseAsync(
ICharacteristic characteristic,
byte[] senddata)
{
var semaphore = new SemaphoreSlim(0, 1);
string result = null;
characteristic.ValueUpdated += (o, args) =>
{
var bytes = characteristic.Value;
result = Encoding.UTF8.GetString(bytes); // Note I don't know if this is your intended behaviour with the values you get back, you can decide what to actually do with the response.
// Notify a value has been received.
semaphore.Release();
};
await characteristic.WriteAsync(senddata,new CancellationToken(true)).ConfigureAwait(false);
// Wait until we receive a notification.
await semaphore.WaitAsync(); // I strongly suggest you look in to CancellationTokens but I am not going in to that now.
return result;
}

I suspect that threading is part of the problem here and that you subscribe for the event, call WriteAsync but then you ultimately leave the method before the data/event has been received. I suggest you need to try blocking the method from leaving before you receive your data.
Try something like:
private static async Task<string> WriteAndWaitForResponseAsync(
ICharacteristic characteristic,
byte[] senddata)
{
var semaphore = new SemaphoreSlim(0, 1);
string result = null;
characteristic.ValueUpdated += (o, args) =>
{
// Make sure you read from the new value in event not the existing characteristic.
var bytes = args.Characteristic.Value;
result = Encoding.UTF8.GetString(bytes); // Note I don't know if this is your intended behaviour with the values you get back, you can decide what to actually do with the response.
// Notify a value has been received.
semaphore.Release();
};
await characteristic.WriteAsync(senddata);
// Wait until we receive a notification.
await semaphore.WaitAsync(); // I strongly suggest you look in to CancellationTokens but I am not going in to that now.
return result;
}
Then you can call this method from within your current one like:
string str = await WriteAndWaitForResponseAsync(characteristic, senddata);

PostAsync() and WaitAll() break instantly

I'm working on an API that enables C# to communicate and manage GNS projects easily. Now I'm looking for an efficient way to run and stop the projects. You can do it pretty much easily by simple POST requests synchronously. However, this process take some time so I'm trying to make it asynchronous since PostAsync let you do if I'm not mistaken.
Sadly, when I try to run my code, this breaks so bad. This is part where all the problem comes up:
// Change the status of the project (start or stop)
private void ChangeProjectStatus(string status){
// First part of the URL
string URLHeader = $"http://{host}:{port}/v2/projects/{projectID}/nodes";
// Number of nodes
int numNodes = nodes.Length;
// Pack the content we will send
string content = JsonConvert.SerializeObject(new Dictionary<string, string> { { "-d", "{}" } });
ByteArrayContent byteContent = new ByteArrayContent(System.Text.Encoding.UTF8.GetBytes(content));
byteContent.Headers.ContentType = new MediaTypeHeaderValue("application/json");
if (status.Equals("start"))
Console.WriteLine("Activating all the nodes in the project...");
else
Console.WriteLine("Deactivating all the nodes in the project...");
Task<System.Net.Http.HttpResponseMessage>[] tasks = new Task<System.Net.Http.HttpResponseMessage>[nodes.Length];
for (ushort i = 0; i < nodes.Length; i++){
try{
tasks[i] = HTTPclient.PostAsync($"{URLHeader}/{nodes[i].ID}/{status}", byteContent);
} catch(Exception err){
Console.Error.WriteLine("Impossible to {2} node {0}: {1}", nodes[i].Name, err.Message, status);
}
}
Task.WaitAll(tasks);
Console.WriteLine("...ok");
}
The error I get (from the WaitAll() block actually) is:
An item with the same key has already been added. Key: Content-Length
Any idea on how to fix it?

How properly to throttle access to DocumentDb from WebJobs

I have an Azure WebKob with blob and queue triggers to save data to Azure DocumentDb.
From time to time I'm getting an error:
Microsoft.Azure.Documents.RequestRateTooLargeException: Message: {"Errors":["Request rate is large"]}
Currently I throttle requests using this code. A WebJob function:
public async Task ParseCategoriesFromCsv(...)
{
double find = 2.23, add = 5.9, replace = 10.67;
double requestCharge = Math.Round(find + Math.Max(add, replace));
await categoryProvider.SaveCategories(requestCharge , categories);
}
Category provider to manipulate document db client:
public async Task<ResourceResponse<Document>[]> SaveCategories(double requestCharge, Category[] categories)
{
var requestDelay = TimeSpan.FromSeconds(60.0 / (collectionOptions.RequestUnits / requestCharge));
var scheduler = new IntervalTaskScheduler(requestDelay, Scheduler.Default); // Rx
var client = new DocumentClient(endpoint, authorizationKey,
new ConnectionPolicy
{
ConnectionMode = documentDbOptions.ConnectionMode,
ConnectionProtocol = documentDbOptions.ConnectionProtocol
});
return await Task.WhenAll(documents.Select(async d =>
await scheduler.ScheduleTask(
() => client.PutDocumentToDb(collectionOptions.CollectionLink, d.SearchIndex, d))));
}
Task scheduler to throttle/measure/synchronize requests:
private readonly Subject<Action> _requests = new Subject<Action>();
private readonly IDisposable _observable;
public IntervalTaskScheduler(TimeSpan requestDelay, IScheduler scheduler)
{
_observable = _requests.Select(i => Observable.Empty<Action>()
.Delay(requestDelay)
.StartWith(i))
.Concat()
.ObserveOn(scheduler)
.Subscribe(action => action());
}
public Task<T> ScheduleTask<T>(Func<Task<T>> request)
{
var tcs = new TaskCompletionSource<T>();
_requests.OnNext(async () =>
{
try
{
T result = await request();
tcs.SetResult(result);
}
catch (Exception ex)
{
tcs.SetException(ex);
}
});
return tcs.Task;
}
So it's basically a number of constants from ResourceResponse<Document>.RequestCharge but:
When I have 1 queue triggered it works fine but when 8 queue it throws an error.
If increase request charge in 8 times then 8 queues work fine but just 1 works 8 times slower than it could.
What a throttling/measuring/synchronization mechanism could work here well?

Starting .NET SDK 1.8.0, we automatically handle the Request Rate too large exceptions to a reasonable extent(will retry 9 times by default and honor the retry after returned from server for next retry).
In case you need greater control, you can configure the RetryOptions on the ConnectionPolicy instance that you pass in to the DocumentClient objectand we would override the default retry policy with it.
So you no longer need to add any custom logic for handling 429 exceptions in your application code like above.

When getting a 429 (Request rate too large) the response tells you how long to wait. There is a header x-ms-retry-after. This has a value. Wait for that time period in ms.
catch (AggregateException ex) when (ex.InnerException is DocumentClientException)
{
DocumentClientException dce = (DocumentClientException)ex.InnerException;
switch ((int)dce.StatusCode)
{
case 429:
Thread.Sleep(dce.RetryAfter);
break;
default:
Console.WriteLine(" Failed: {0}", ex.InnerException.Message);
throw;
}
}

It seems to me that you should be able to do this with your SaveCategories method to make it work nicely with Rx:
public IObservable<ResourceResponse<Document>[]> SaveCategories(double requestCharge, Category[] categories)
{
var requestDelay = TimeSpan.FromSeconds(60.0 / (collectionOptions.RequestUnits / requestCharge));
var client = new DocumentClient(endpoint, authorizationKey,
new ConnectionPolicy
{
ConnectionMode = documentDbOptions.ConnectionMode,
ConnectionProtocol = documentDbOptions.ConnectionProtocol
});
return
Observable.Interval(requestDelay)
.Zip(documents, (delay, doc) => doc)
.SelectMany(doc => Observable.FromAsync(() => client.PutDocumentToDb(collectionOptions.CollectionLink, doc.SearchIndex, doc)))
.ToArray();
}
This totally gets rid of your IntervalTaskScheduler class and ensures that you limit the request rate to one request per the requestDelay time span, but allows the response to take as long as needed. To .ToArray() call turns the IObservable<ResourceResponse<Document>> that returns many values into an IObservable<ResourceResponse<Document>[]> that returns a single array of values when the observable completes.
I couldn't test your code, so I tested a sample which I think simulates your code:
var r = new Random();
var a = Enumerable.Range(0, 1000);
var i = Observable.Interval(TimeSpan.FromSeconds(2.0));
var sw = Stopwatch.StartNew();
var query =
i.Zip(a, (ii, aa) => aa)
.SelectMany(aa => Observable.Start(() =>
{
var x = sw.Elapsed.TotalMilliseconds;
Thread.Sleep(r.Next(0, 5000));
return x;
}))
.Select(x => new
{
started = x,
ended = sw.Elapsed.TotalMilliseconds
});
I got this kind of result which shows that the requests were throttled:
4026.2983 5259.7043
2030.1287 6940.2326
6027.0439 9664.1045
8027.9993 10207.0579
10028.1762 12301.4746
12028.3190 12711.4440
14040.7972 17433.1964
16040.9267 17574.5924
18041.0529 19077.5545

limit # of web service requests simultaneously

I have an Excel Add-In written in C#, .NET 4.5. It will send many web service requests to a web server to get data. E.g. it sends 30,000 requests to web service server. When data of a request comes back, the addin will plot the data in Excel.
Originally I did all the requests asynchronously, but sometime I will get OutOfMemoryException
So I changed, sent the requests one by one, but it is too slow, takes long time to finish all requests.
I wonder if there is a way that I can do 100 requests at a time asynchronously, once the data of all the 100 requests come back and plot in Excel, then send the next 100 requests.
Thanks
Edit
On my addin, there is a ribbon button "Refresh", when it is clicked, refresh process starts.
On main UI thread, ribbon/button is clicked, it will call web service BuildMetaData,
once it is returned back, in its callback MetaDataCompleteCallback, another web service call is sent
Once it is returned back, in its callback DataRequestJobFinished, it will call plot to plot data on Excel. see below
RefreshBtn_Click()
{
if (cells == null) return;
Range firstOccurence = null;
firstOccurence = cells.Find(functionPattern, null,
null, null,
XlSearchOrder.xlByRows,
XlSearchDirection.xlNext,
null, null, null);
DataRequest request = null;
_reportObj = null;
Range currentOccurence = null;
while (!Helper.RefreshCancelled)
{
if(firstOccurence == null ||IsRangeEqual(firstOccurence, currentOccurence)) break;
found = true;
currentOccurence = cells.FindNext(currentOccurence ?? firstOccurence);
try
{
var excelFormulaCell = new ExcelFormulaCell(currentOccurence);
if (excelFormulaCell.HasValidFormulaCell)
{
request = new DataRequest(_unityContainer, XLApp, excelFormulaCell);
request.IsRefreshClicked = true;
request.Workbook = Workbook;
request.Worksheets = Worksheets;
_reportObj = new ReportBuilder(_unityContainer, XLApp, request, index, false);
_reportObj.ParseParameters();
_reportObj.GenerateReport();
//this is necessary b/c error message is wrapped in valid object DataResponse
//if (!string.IsNullOrEmpty(_reportObj.ErrorMessage)) //Clear previous error message
{
ErrorMessage = _reportObj.ErrorMessage;
Errors.Add(ErrorMessage);
AddCommentToCell(_reportObj);
Errors.Remove(ErrorMessage);
}
}
}
catch (Exception ex)
{
ErrorMessage = ex.Message;
Errors.Add(ErrorMessage);
_reportObj.ErrorMessage = ErrorMessage;
AddCommentToCell(_reportObj);
Errors.Remove(ErrorMessage);
Helper.LogError(ex);
}
}
}
on Class to GenerateReport
public void GenerateReport()
{
Request.ParseFunction();
Request.MetacompleteCallBack = MetaDataCompleteCallback;
Request.BuildMetaData();
}
public void MetaDataCompleteCallback(int id)
{
try
{
if (Request.IsRequestCancelled)
{
Request.FormulaCell.Dispose();
return;
}
ErrorMessage = Request.ErrorMessage;
if (string.IsNullOrEmpty(Request.ErrorMessage))
{
_queryJob = new DataQueryJob(UnityContainer, Request.BuildQueryString(), DataRequestJobFinished, Request);
}
else
{
ModifyCommentOnFormulaCellPublishRefreshEvent();
}
}
catch (Exception ex)
{
ErrorMessage = ex.Message;
ModifyCommentOnFormulaCellPublishRefreshEvent();
}
finally
{
Request.MetacompleteCallBack = null;
}
}
public void DataRequestJobFinished(DataRequestResponse response)
{
Dispatcher.Invoke(new Action<DataRequestResponse>(DataRequestJobFinishedUI), response);
}
public void DataRequestJobFinished(DataRequestResponse response)
{
try
{
if (Request.IsRequestCancelled)
{
return;
}
if (response.status != Status.COMPLETE)
{
ErrorMessage = ManipulateStatusMsg(response);
}
else // COMPLETE
{
var tmpReq = Request as DataRequest;
if (tmpReq == null) return;
new VerticalTemplate(tmpReq, response).Plot();
}
}
catch (Exception e)
{
ErrorMessage = e.Message;
Helper.LogError(e);
}
finally
{
//if (token != null)
// this.UnityContainer.Resolve<IEventAggregator>().GetEvent<DataQueryJobComplete>().Unsubscribe(token);
ModifyCommentOnFormulaCellPublishRefreshEvent();
Request.FormulaCell.Dispose();
}
}
on plot class
public void Plot()
{
...
attributeRange.Value2 = headerArray;
DataRange.Value2 = ....
DataRange.NumberFormat = ...
}

OutOfMemoryException is not about the too many requests sent simultaneously. It is about freeing your resources right way. In my practice there are two main problems when you are getting such exception:
Wrong working with immutable structures or System.String class
Not disposing your disposable resources, especially graphic objects and WCF requests.
In case of reporting, for my opinion, you got a second one type of a problem. DataRequest and DataRequestResponse are good point to start the investigation for the such objects.
If this doesn't help, try to use the Tasks library with async/await pattern, you can find good examples here:
// Signature specifies Task<TResult>
async Task<int> TaskOfTResult_MethodAsync()
{
int hours;
// . . .
// Return statement specifies an integer result.
return hours;
}
// Calls to TaskOfTResult_MethodAsync
Task<int> returnedTaskTResult = TaskOfTResult_MethodAsync();
int intResult = await returnedTaskTResult;
// or, in a single statement
int intResult = await TaskOfTResult_MethodAsync();
// Signature specifies Task
async Task Task_MethodAsync()
{
// . . .
// The method has no return statement.
}
// Calls to Task_MethodAsync
Task returnedTask = Task_MethodAsync();
await returnedTask;
// or, in a single statement
await Task_MethodAsync();
In your code I see a while loop, in which you can store your Task[] of size of 100, for which you can use the WaitAll method, and the problem should be solved. Sorry, but your code is huge enough, and I can't provide you a more straight example.

I'm having a lot of trouble parsing your code to figure out is being iterated for your request but the basic template for batching asynchronously is going to be something like this:
static const int batchSize = 100;
public async Task<IEnumerable<Results>> GetDataInBatches(IEnumerable<RequestParameters> parameters) {
if(!parameters.Any())
return Enumerable.Empty<Result>();
var batchResults = await Task.WhenAll(parameters.Take(batchSize).Select(doQuery));
return batchResults.Concat(await GetDataInBatches(parameters.Skip(batchSize));
}
where doQuery is something with the signature
Task<Results> async doQuery(RequestParameters parameters) {
//.. however you do the query
}
I wouldn't use this for a million requests since its recursive, but your case should would generate a callstack only 300 deep so you'll be fine.
Note that this also assumes that your data request stuff is done asynchronously and returns a Task. Most libraries have been updated to do this (look for methods with the Async suffix). If it doesn't expose that api you might want to create a separate question for how to specifically get your library to play nice with the TPL.