Develop Reference

Task.WhenAll() not working as expected without visual studio debugger attached

I have a question about async programming and Task.WhenAll(). I have a code snippet that downloads a folder from google drive and it works as expected when i am debugging my code. However when I run the application without debugger the download function takes atleast 4x the time it takes when it runs with the debugger. I also get crash logs with TaskCancelled exceptions (System.Threading.Tasks.TaskCanceledException: A task was canceled.) which do not happen with the debugger attached. What needs to be changed for the code to work as expected without debugger attached. NB this snippet downloads +- 1000 files in about 22-25 seconds with debugger and 2min+ without debugger.
public static async Task<bool> DownloadFolder(CloudDataModel.File file, string path, params string[] exclude)
{
try
{
if (file != null && !string.IsNullOrEmpty(file.id))
{
List<string> toExclude = new List<string>();
if(exclude != null)
{
toExclude = exclude.ToList();
}
List<Task> downloadFilesTask = new List<Task>();
var files = await file.GetFiles();
foreach (var f in files)
{
var task = f.Download(path);
downloadFilesTask.Add(task);
}
var folders = await file.GetFoldersAsync();
foreach (var folder in folders)
{
if (toExclude.Contains(folder.name))
{
continue;
}
Task task = null;
if (path.Equals(Statics.ProjectFolderName))
{
task = DownloadFolder(folder, folder.name);
}
else
{
task = DownloadFolder(folder, Path.Combine(path, folder.name));
}
downloadFilesTask.Add(task);
}
var array = downloadFilesTask.ToArray();
await Task.WhenAll(array);
return true;
}
}
catch (Exception e)
{
Crashes.TrackError(e);
}
return false;
}
Edit
after some more trial and error the fault has been identified.
the downloading of the file was the cause of the unexpected behaviour
public static async Task<StorageFile> DownloadFile(CloudDataModel.File file, string destinationFolder)
{
try
{
if (file != null)
{
Debug.WriteLine($"start download {file.name}");
if (file.mimeType == Statics.GoogleDriveFolderMimeType)
{
Debug.WriteLine($"did not download resource, resource was folder instead of file. mimeType: {file.mimeType}");
return null;
}
var endpoint = Path.Combine(DownloadFileEndpoint, $"{file.id}?alt=media");
// this would cause the unexpected behaviour
HttpResponseMessage response = await client.GetAsync(endpoint);
StorageFile downloadedFile;
using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
{
var memstream = new MemoryStream();
StreamReader reader = new StreamReader(streamToReadFrom);
streamToReadFrom.Position = 0;
await streamToReadFrom.CopyToAsync(memstream);
downloadedFile = await fileHandler.SaveDownloadedCloudFile(memstream, file, destinationFolder);
Debug.WriteLine($"download finished {file.name}");
}
return downloadedFile;
}
return null;
}
catch (Exception e)
{
Crashes.TrackError(e);
return null;
}
}
After setting a timeout to the client (System.Net.Http.HttpClient) the code executed as expected.
client.Timeout = new TimeSpan(0,0,5);

So after some trial and error the downloading of the file was identified as the cause of the problem.
var task = f.Download(path);
the Download function implements a System.Net.Http.HttpClient wich was initialized without any kind of timeout. when one of the tasks had a timeout, all other tasks would also not execute as expected. to fix this issue a timeout of 5 seconds was added to the HttpClient object.
private static void InitializeClient()
{
HttpClientHandler handler = new HttpClientHandler { AllowAutoRedirect = true };
client = new HttpClient(handler);
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", AccountManager.LoggedInAccount.Token);
client.Timeout = new TimeSpan(0,0,5);
}
After this change the code would execute as expected.

My guess is that without the debugger the parallelism increases, and so the remote server is overflowed by requests and can't perform optimally. Your code does not have any provision for limiting the degree of concurrency/parallelism. One possible way to solve this problem is to use a SemaphoreSlim, as shown in this question (among many others).

One or more errors occurred. (Exception of type 'System.OutOfMemoryException' was thrown.)

I'm running into below mentioned error when a list of more than 30k items is created, via Lambda function.
Error:
System.Collections.Generic.List`1.set_Capacity(Int32 value) at System.Collections.Generic.List`1.EnsureCapacity(Int32 min) at System.Collections.Generic.List`1.AddWithResize(T item) at
at System.Collections.Generic.List`1.set_Capacity(Int32 value)
at System.Collections.Generic.List`1.EnsureCapacity(Int32 min)
at System.Collections.Generic.List`1.AddWithResize(T item)
at AWSLambda3.Function.ListS3ObjectsAsync(String bucketName, String filestoDwnld, IAmazonS3 client)
Code:
public async Task<List<string>> ListS3ObjectsAsync(string bucketName, string filestoDwnld, IAmazonS3 client)
{
//int fileCount = 0;
List<string> pdfFiles = new List<string>();
ListObjectsRequest request = new ListObjectsRequest
{
BucketName = bucketName,
Prefix = filestoDwnld,
};
ListObjectsResponse response = await client.ListObjectsAsync(request);
do
{
foreach (S3Object entry in response.S3Objects)
pdfFiles.Add(entry.Key);
if (response.IsTruncated)
{
request.Marker = response.NextMarker;
}
else
request = null;
} while (request != null);
return pdfFiles;
}
I've also tried increasing list capacity, but that doesn't help as well. Please assist.

The reason for the OutOfMemoryException is in the infinite loop that is triggered when response.IsTruncated == true In this case the request is never set to null and the loop doesn't stop. The code restart a new loop loading again the same set of elements into the list pdfFiles and so on until you have no more memory.
I don't know exactly how your service backend works but I could immagine that you need to change just one line of code inserting the request to the service call inside the loop
do
{
// This inside the loop will be executed at least one time and
// eventually again until the IsTruncated property is set to true.
ListObjectsResponse response = await client.ListObjectsAsync(request);
foreach (S3Object entry in response.S3Objects)
pdfFiles.Add(entry.Key);
if (response.IsTruncated)
{
request.Marker = response.NextMarker;
}
else
request = null;
} while (request != null);
In this way, after the first loop you ask again a new set of elements from the service backend pointing at the NextMarker and eventually you will reach a point in which IsTruncated will be false ending the loop

Forcing an asynchronous parallel loop to stop immediately- C#

Here is my method for requesting information from Azures FaceAPI.
I finally realized that in order to make my application work best, a bunch of my security camera frames must be grabbed in parallel, then sent away to the Neural Network to be analyzed.
(Note: it's an Alexa custom skill, so it times out around 8-10 seconds)
Because I grab a bunch of frames in parallel, and asynchronously, there is no way to know which of the images will return a decent face detection. However, when a good detection is found, and the request return Face Data, there is no way to stop the rest of the information from returning.
This happens because the Security camera images were sent in Parallel, they are gone, the info is coming back no matter what.
You'll see that I'm able to use "thread local variables" to capture information and send it back to the function scoped "imageAnalysis" variable to serialize and allow Alexa to describe people in the security image. BUT, because the loop is in Parallel, it doesn't break right away.
It may only take a second or two, but I'm on a time limit thanks to Alexas strict time-out policies.
There doesn't seem to be a way to break the parallel loop immediately...
Or is there?
The more time is spent collecting the "imageAnalysis" Data, the longer Alexa has to wait for a response. She doesn't wait long, and it's important to try and send as many possible images for analysis as possible before Alexa times-out, and also keeping under the Azure FaceAPI limits.
public static async Task<List<Person>> DetectPersonAsync()
{
ConfigurationDto config = Configuration.Configuration.GetSettings();
string imageAnalysis;
using (var cameraApi = new SecurityCameraApi())
{
byte[] imageData = cameraApi.GetImageAsByte(config.SecurityCameraUrl +
config.SecurityCameraStaticImage +
DateTime.Now);
//Unable to get an image from the Security Camera
if (!imageData.Any())
{
Logger.LogInfo(Logger.LogType.Info, "Unable to aquire image from Security Camera \n");
return null;
}
Logger.LogInfo(Logger.LogType.Info, "Attempting Image Face Detection...\n");
Func<string, bool> isEmptyOrErrorAnalysis = s => s.Equals("[]") || s.Contains("error");
imageAnalysis = "[]";
List<byte[]> savedImageList = cameraApi.GetListOfSavedImagesAsByte();
if (savedImageList.Any())
{
Parallel.ForEach(savedImageList, new ParallelOptions
{
MaxDegreeOfParallelism = 50
},
async (image, loopState) =>
{
string threadLocalImageAnalysis = "[]";
if (!loopState.IsStopped)
threadLocalImageAnalysis = await GetImageAnalysisAsync(image, config);
if (!isEmptyOrErrorAnalysis(threadLocalImageAnalysis))
{
imageAnalysis = threadLocalImageAnalysis;
loopState.Break();
}
});
}
// Don't do to many image analysis - or Alexa will time-out.
Func<List<byte[]>, int> detectionCount =
savedImageListDetections => savedImageListDetections.Count > 5 ? 0 : 16;
//Continue with detections of current Camera image frames
if (isEmptyOrErrorAnalysis(imageAnalysis))
{
Parallel.For(0, detectionCount(savedImageList), new ParallelOptions
{
MaxDegreeOfParallelism = 50
},
async (i, loopState) =>
{
imageData = cameraApi.GetImageAsByte(config.SecurityCameraUrl +
config.SecurityCameraStaticImage +
DateTime.Now);
string threadLocalImageAnalysis = "[]";
if (!loopState.IsStopped)
threadLocalImageAnalysis = await GetImageAnalysisAsync(imageData, config);
if (!isEmptyOrErrorAnalysis(threadLocalImageAnalysis))
{
imageAnalysis = threadLocalImageAnalysis;
loopState.Break();
}
});
}
}
try
{
//Cognitive sense has found elements(people) in the image
return new NewtonsoftJsonSerializer().DeserializeFromString<List<Person>>(imageAnalysis);
}
catch (Exception ex)
{
//No elements(people) detected from the camera stream
Logger.LogInfo(Logger.LogType.Info,
string.Format(
"Microsoft Cognitive Sense Face Api Reports: \n{0} \nNo people in the CCTV Camera Image.\n",
ex.Message));
return new List<Person>(); //Empty
}
}
private static async Task<string> GetImageAnalysisAsync(byte[] image, ConfigurationDto config)
{
string json;
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key",
config.MicrosoftCognitiveSenseApiKey);
// Request parameters.
const string queryString =
"returnFaceId=true" +
"&returnFaceLandmarks=false" +
"&returnFaceAttributes=age,gender,accessories,hair";
const string uri =
"https://westus.api.cognitive.microsoft.com/face/v1.0/detect?" + queryString;
using (var content = new ByteArrayContent(image))
{
content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
HttpResponseMessage response = await client.PostAsync(uri, content);
json = await response.Content.ReadAsStringAsync();
try
{
Logger.LogInfo(Logger.LogType.Info, json);
}
catch
{
}
}
}
return json;
}

Catching exception (and ignoring) in called method, but need to throw it in caller

I have a method that tries to get a web page. I want to attempt to get it several times so I built a wrapper around it to retry several times. In the method called I catch and then ignore the exception returning null. Therefore, after the first attempt the retries will occur. Here is the called method:
internal static async Task<string> WebClientAsync(string URI, NetworkCredential Creds = null, Dictionary.FantasySite Site = Dictionary.FantasySite.Other)
{
if (Creds == null)
{
try
{ //attempt to get the web page
HttpClient client = new HttpClient(); //create client
HttpResponseMessage response = await client.GetAsync(URI); //get response
response.EnsureSuccessStatusCode(); //ensure the response is good (or throw Exception)
return await response.Content.ReadAsStringAsync(); //return the string back
}
catch (HttpRequestException)
{
//MessageBox.Show(string.Format("\nHttpRequestException Caught!\nMessage :{0} for URI {1}.", e.Message, URI));
return null; //Catch the exception because we wrapped this and are trying again (null is the indicator it didn't work)
}
catch (Exception)
{
//MessageBox.Show(string.Format("\nException Caught!\nMessage :{0} for URI {1}.", e.Message, URI)); //TODO - THis hasn't happened, but remove it for production
return null; //Catch the exception because we wrapped this and are trying again (null is the indicator it didn't work)
}
}
}
If this still fails after all the retries then I want to throw the exception, but since I threw it away, I can't. Here is the calling method.
internal static async Task<string> WebClientRetryAsync(string URI, NetworkCredential Creds = null, Dictionary.FantasySite Site = Dictionary.FantasySite.Other)
{
string webPage = null;
for (int i = 0; i < Dictionary.WEB_PAGE_ATTEMPTS; i++) //Attempt to get the webpage x times
{
System.Diagnostics.Debug.Print(string.Format("WebClientRetryAsync attempt {0} for {1}", i + 1, URI));
//wait some time before retrying just in case we are too busy
//Start wait at 0 for first time and multiply with each successive failure to slow down process by multiplying by i squared
Thread.Sleep(Wait.Next(i * i * Dictionary.RETRY_WAIT_MS));
webPage = await WebClientAsync(URI, Creds, Site);
if (webPage != null) { break; } //don't attempt again if success
}
/*TODO - If webPage is null we didn't have success and need to throw an exception.
* This is done in the calls to this method and should be done here, move code over */
return webPage;
}
Can someone suggest if this is a bad approach and how I could refactor the code to throw the exception after failing too many times? Should I pass the exception to the calling method and ignore it until the retries have run out?

Yup. You should not throw away exceptions that you wish to rethrow. One possible approach is the following (trying to make a minimal amount of modifications to your current code):
internal static async Task<string> WebClientAsync(string URI, NetworkCredential Creds = null, Dictionary.FantasySite Site = Dictionary.FantasySite.Other)
{
// If (Creds == null) removed, you must return a task or throw an exception.
//attempt to get the web page
HttpClient client = new HttpClient(); //create client
HttpResponseMessage response = await client.GetAsync(URI); //get response
response.EnsureSuccessStatusCode(); //ensure the response is good (or throw Exception)
return await response.Content.ReadAsStringAsync(); //return the string back
}
internal static async Task<string> WebClientRetryAsync(string URI, NetworkCredential Creds = null, Dictionary.FantasySite Site = Dictionary.FantasySite.Other)
{
// assumes you have .NET 4.5, otherwise save exception.
// uses System.Runtime.ExceptionServices;
ExceptionDispatchInfo exceptionDispatchInfo = null;
for (int i = 0; i < Dictionary.WEB_PAGE_ATTEMPTS; i++) //Attempt to get the webpage x times
{
System.Diagnostics.Debug.Print(string.Format("WebClientRetryAsync attempt {0} for {1}", i + 1, URI));
try
{
var webPage = await WebClientAsync(URI, Creds, Site);
return webPage;
}
catch (Exception ex)
{
// save exception so it can be rethrown.
exceptionDispatchInfo = ExceptionDispatchInfo.Capture(ex);
}
// Edit: also need to do an async wait (no thread.sleep):
if (i < Dictionary.WEB_PAGE_ATTEMPTS - 1)
{
//wait some time before retrying just in case we are too busy
//Start wait at 0 for first time and multiply with each successive failure to slow down process by multiplying by i squared
await Task.Delay(Wait.Next(i * i * Dictionary.RETRY_WAIT_MS));
}
}
Debug.Assert(exceptionDispatchInfo != null); // shouldn't be null if we get here.
exceptionDispatchInfo.Throw();
}

Access is Denied when Saving a Object in in local storage Windows 8 Metro App

I'm attempting to store a List (where Show is my class that implements IXmlSerializable) to the local isolated storage. I'm using the code from this page:
http://metrostoragehelper.codeplex.com/
I've implemented the change suggested in the Issues section.
I am using the following code to add a Show object when it is clicked from an item list.
private async void addShowButton_Click_1(object sender, RoutedEventArgs e)
{
var isoStorage = new StorageHelper<List<Show>>(StorageType.Local);
List<Show> currentShows = await isoStorage.LoadASync("myShowsEx");
if(currentShows == null) {
currentShows = new List<Show>();
}
currentShows.Add(currentShow);
isoStorage.SaveASync(currentShows, "myShowsEx");
//Read it back, for debugging to check if it has been added properly.
List<Show> currentShowsRB = await isoStorage.LoadASync("myShowsEx"); //Exception here
}
The first show is added perfectly fine and it shows up in the currentShowsRB List. When a second item is clicked and the method above invoked an exception occurs on the last LoadAsync call: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
How can I get around this to access the local data store for multiple calls?
Below is also the relevant code from the StorageHelper:
public async void SaveASync(T Obj, string FileName)
{
FileName = FileName + ".xml";
try
{
if (Obj != null)
{
StorageFile file = null;
StorageFolder folder = GetFolder(storageType);
file = await folder.CreateFileAsync(FileName, CreationCollisionOption.ReplaceExisting);
using (var writeStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
Stream outStream = Task.Run(() => writeStream.AsStreamForWrite()).Result;
serializer.Serialize(outStream, Obj);
//writeStream.Dispose(); //Added and we get UnauthorizedAccessException
// outStream.Dispose(); //Added ObjectDisposedException caught in catch statement below
}
}
}
catch (Exception)
{
throw;
}
}
public async Task<T> LoadASync(string FileName)
{
FileName = FileName + ".xml";
try
{
StorageFile file = null;
StorageFolder folder = GetFolder(storageType);
file = await folder.GetFileAsync(FileName);
using (var readStream = await file.OpenAsync(FileAccessMode.Read))
{
Stream inStream = Task.Run(() => readStream.AsStreamForRead()).Result;
inStream.Position = 0;
return (T)serializer.Deserialize(inStream);
}
}
catch (FileNotFoundException)
{
//file not existing is perfectly valid so simply return the default
return default(T);
//throw;
}
catch (Exception)
{
//Unable to load contents of file
throw;
}
}
The writeStream.Dispose() line I added in, but even when this is included I get the same error message of Access is Denied. If I also include the outStream.Dispose() line then I get a ObjectDisposedException being caught in the catch statement right below. Is there something else I should be doing?

You are not waiting for the SaveAsync to finish, trying to Load when the Save is still in progress. Change it to:
//isoStorage.SaveASync(currentShows, "myShowsEx");
await isoStorage.SaveASync(currentShows, "myShowsEx");
List<Show> currentShowsRB = await isoStorage.LoadASync("myShowsEx");
Edit, awaiting on void is a standard problem.
The quick fix is :
await TaskEx.Run(() => isoStorage.SaveASync(currentShows, "myShowsEx"));
But you can also move the TaskEx.Run() inside SaveASync(). And given a name that ends with Async, it should not be void but:
Task SaveASyncT Obj, string FileName)
{
return TaskEx.Run() => { .... }
}
I don't believe there is an async version of Serialize, so it stays at TaskEx.Run() .

This block can be shortened to:
using (var readStream = await file.OpenAsync(FileAccessMode.Read))
{
return (T)serializer.Deserialize(readStream);
}
as you can directly read/write from the stream in the XmlSerializer class.
You may also find that CreationCollisionOption.ReplaceExisting can cause issues and OpenIfExists will cause the file to be overwritten by the serializer as required.