I have a question about async programming and Task.WhenAll(). I have a code snippet that downloads a folder from google drive and it works as expected when i am debugging my code. However when I run the application without debugger the download function takes atleast 4x the time it takes when it runs with the debugger. I also get crash logs with TaskCancelled exceptions (System.Threading.Tasks.TaskCanceledException: A task was canceled.) which do not happen with the debugger attached. What needs to be changed for the code to work as expected without debugger attached. NB this snippet downloads +- 1000 files in about 22-25 seconds with debugger and 2min+ without debugger.
public static async Task<bool> DownloadFolder(CloudDataModel.File file, string path, params string[] exclude)
{
try
{
if (file != null && !string.IsNullOrEmpty(file.id))
{
List<string> toExclude = new List<string>();
if(exclude != null)
{
toExclude = exclude.ToList();
}
List<Task> downloadFilesTask = new List<Task>();
var files = await file.GetFiles();
foreach (var f in files)
{
var task = f.Download(path);
downloadFilesTask.Add(task);
}
var folders = await file.GetFoldersAsync();
foreach (var folder in folders)
{
if (toExclude.Contains(folder.name))
{
continue;
}
Task task = null;
if (path.Equals(Statics.ProjectFolderName))
{
task = DownloadFolder(folder, folder.name);
}
else
{
task = DownloadFolder(folder, Path.Combine(path, folder.name));
}
downloadFilesTask.Add(task);
}
var array = downloadFilesTask.ToArray();
await Task.WhenAll(array);
return true;
}
}
catch (Exception e)
{
Crashes.TrackError(e);
}
return false;
}
Edit
after some more trial and error the fault has been identified.
the downloading of the file was the cause of the unexpected behaviour
public static async Task<StorageFile> DownloadFile(CloudDataModel.File file, string destinationFolder)
{
try
{
if (file != null)
{
Debug.WriteLine($"start download {file.name}");
if (file.mimeType == Statics.GoogleDriveFolderMimeType)
{
Debug.WriteLine($"did not download resource, resource was folder instead of file. mimeType: {file.mimeType}");
return null;
}
var endpoint = Path.Combine(DownloadFileEndpoint, $"{file.id}?alt=media");
// this would cause the unexpected behaviour
HttpResponseMessage response = await client.GetAsync(endpoint);
StorageFile downloadedFile;
using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
{
var memstream = new MemoryStream();
StreamReader reader = new StreamReader(streamToReadFrom);
streamToReadFrom.Position = 0;
await streamToReadFrom.CopyToAsync(memstream);
downloadedFile = await fileHandler.SaveDownloadedCloudFile(memstream, file, destinationFolder);
Debug.WriteLine($"download finished {file.name}");
}
return downloadedFile;
}
return null;
}
catch (Exception e)
{
Crashes.TrackError(e);
return null;
}
}
After setting a timeout to the client (System.Net.Http.HttpClient) the code executed as expected.
client.Timeout = new TimeSpan(0,0,5);
So after some trial and error the downloading of the file was identified as the cause of the problem.
var task = f.Download(path);
the Download function implements a System.Net.Http.HttpClient wich was initialized without any kind of timeout. when one of the tasks had a timeout, all other tasks would also not execute as expected. to fix this issue a timeout of 5 seconds was added to the HttpClient object.
private static void InitializeClient()
{
HttpClientHandler handler = new HttpClientHandler { AllowAutoRedirect = true };
client = new HttpClient(handler);
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", AccountManager.LoggedInAccount.Token);
client.Timeout = new TimeSpan(0,0,5);
}
After this change the code would execute as expected.
My guess is that without the debugger the parallelism increases, and so the remote server is overflowed by requests and can't perform optimally. Your code does not have any provision for limiting the degree of concurrency/parallelism. One possible way to solve this problem is to use a SemaphoreSlim, as shown in this question (among many others).
Related
Hello I have a wrapservice which calls sub services in the project. I am trying to add timeout value as a second but there is too many variables so it makes thing hard to implement. I am adding my code snippet and open to ideas thank you
public Stream Posnet(Stream request)
{
if (!Globals.IsConsumerAuthorizedForWasService)
{
throw new FaultException("consumer is not authorized", new FaultCode("3001"));
}
var streamReader = new StreamReader(request);
string streamString = streamReader.ReadToEnd();
streamReader.Close();
PosnetResponse resp = _merchantLegacyAPIWrapper.WrapService(streamString); // I want to add timeout here where I call WrapService;
// I want to add timeout here where I call WrapService
string responseXml = SerializeResponse(resp);
return GenerateStreamFromString(responseXml);
}
and I tried this code snippet below to implement but not succeed:
using System.Threading.Tasks;
var task = Task.Run(() => SomeMethod(input));
if (task.Wait(TimeSpan.FromSeconds(10)))
return task.Result;
else
throw new Exception("Timed out");
Here is my method for requesting information from Azures FaceAPI.
I finally realized that in order to make my application work best, a bunch of my security camera frames must be grabbed in parallel, then sent away to the Neural Network to be analyzed.
(Note: it's an Alexa custom skill, so it times out around 8-10 seconds)
Because I grab a bunch of frames in parallel, and asynchronously, there is no way to know which of the images will return a decent face detection. However, when a good detection is found, and the request return Face Data, there is no way to stop the rest of the information from returning.
This happens because the Security camera images were sent in Parallel, they are gone, the info is coming back no matter what.
You'll see that I'm able to use "thread local variables" to capture information and send it back to the function scoped "imageAnalysis" variable to serialize and allow Alexa to describe people in the security image. BUT, because the loop is in Parallel, it doesn't break right away.
It may only take a second or two, but I'm on a time limit thanks to Alexas strict time-out policies.
There doesn't seem to be a way to break the parallel loop immediately...
Or is there?
The more time is spent collecting the "imageAnalysis" Data, the longer Alexa has to wait for a response. She doesn't wait long, and it's important to try and send as many possible images for analysis as possible before Alexa times-out, and also keeping under the Azure FaceAPI limits.
public static async Task<List<Person>> DetectPersonAsync()
{
ConfigurationDto config = Configuration.Configuration.GetSettings();
string imageAnalysis;
using (var cameraApi = new SecurityCameraApi())
{
byte[] imageData = cameraApi.GetImageAsByte(config.SecurityCameraUrl +
config.SecurityCameraStaticImage +
DateTime.Now);
//Unable to get an image from the Security Camera
if (!imageData.Any())
{
Logger.LogInfo(Logger.LogType.Info, "Unable to aquire image from Security Camera \n");
return null;
}
Logger.LogInfo(Logger.LogType.Info, "Attempting Image Face Detection...\n");
Func<string, bool> isEmptyOrErrorAnalysis = s => s.Equals("[]") || s.Contains("error");
imageAnalysis = "[]";
List<byte[]> savedImageList = cameraApi.GetListOfSavedImagesAsByte();
if (savedImageList.Any())
{
Parallel.ForEach(savedImageList, new ParallelOptions
{
MaxDegreeOfParallelism = 50
},
async (image, loopState) =>
{
string threadLocalImageAnalysis = "[]";
if (!loopState.IsStopped)
threadLocalImageAnalysis = await GetImageAnalysisAsync(image, config);
if (!isEmptyOrErrorAnalysis(threadLocalImageAnalysis))
{
imageAnalysis = threadLocalImageAnalysis;
loopState.Break();
}
});
}
// Don't do to many image analysis - or Alexa will time-out.
Func<List<byte[]>, int> detectionCount =
savedImageListDetections => savedImageListDetections.Count > 5 ? 0 : 16;
//Continue with detections of current Camera image frames
if (isEmptyOrErrorAnalysis(imageAnalysis))
{
Parallel.For(0, detectionCount(savedImageList), new ParallelOptions
{
MaxDegreeOfParallelism = 50
},
async (i, loopState) =>
{
imageData = cameraApi.GetImageAsByte(config.SecurityCameraUrl +
config.SecurityCameraStaticImage +
DateTime.Now);
string threadLocalImageAnalysis = "[]";
if (!loopState.IsStopped)
threadLocalImageAnalysis = await GetImageAnalysisAsync(imageData, config);
if (!isEmptyOrErrorAnalysis(threadLocalImageAnalysis))
{
imageAnalysis = threadLocalImageAnalysis;
loopState.Break();
}
});
}
}
try
{
//Cognitive sense has found elements(people) in the image
return new NewtonsoftJsonSerializer().DeserializeFromString<List<Person>>(imageAnalysis);
}
catch (Exception ex)
{
//No elements(people) detected from the camera stream
Logger.LogInfo(Logger.LogType.Info,
string.Format(
"Microsoft Cognitive Sense Face Api Reports: \n{0} \nNo people in the CCTV Camera Image.\n",
ex.Message));
return new List<Person>(); //Empty
}
}
private static async Task<string> GetImageAnalysisAsync(byte[] image, ConfigurationDto config)
{
string json;
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key",
config.MicrosoftCognitiveSenseApiKey);
// Request parameters.
const string queryString =
"returnFaceId=true" +
"&returnFaceLandmarks=false" +
"&returnFaceAttributes=age,gender,accessories,hair";
const string uri =
"https://westus.api.cognitive.microsoft.com/face/v1.0/detect?" + queryString;
using (var content = new ByteArrayContent(image))
{
content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
HttpResponseMessage response = await client.PostAsync(uri, content);
json = await response.Content.ReadAsStringAsync();
try
{
Logger.LogInfo(Logger.LogType.Info, json);
}
catch
{
}
}
}
return json;
}
I am writing a universal app primarily targeting Windows Phone using SQLite-net.
During the course of operation, the user is presented with an option to download multiple files. At the end of each file download, I need to mark the file in the db as completed. I am using BackgroundDownloader in order to download files - the WP8.0 app used Background Transfer Service and that worked great. Files can be huge (some 200+ mbs, user content) and i am not looking forward to wrapping the downloads in the HttpClient or WebClient.
However, it seems that the progress callback doesn't work with awaits unless I actually breakpoint in the method.
The following are listings from a sample app i quickly put together that demonstrates the behaviour:
Model:
public class Field
{
[PrimaryKey]
[AutoIncrement]
public int Id { get; set; }
public bool Done { get; set; }
}
MainPage codebehind (i am creating a db here only for the purposes of this example!):
private async void Button_Click(object sender, RoutedEventArgs e)
{
using (var db = new SQLiteConnection(Windows.Storage.ApplicationData.Current.LocalFolder.Path + "//Main.db"))
{
db.CreateTable<Field>();
db.Commit();
}
this.DbConnection = new SQLiteAsyncConnection(Windows.Storage.ApplicationData.Current.LocalFolder.Path + "//My.db");
var dl = new BackgroundDownloader();
dl.CostPolicy = BackgroundTransferCostPolicy.Always;
var transferUri = new Uri("http://192.168.1.4/hello.world", UriKind.Absolute);
var folder = await ApplicationData.Current.LocalFolder.CreateFolderAsync(
"Content",
CreationCollisionOption.OpenIfExists);
var localFile = await folder.CreateFileAsync("cheesecakes.file", CreationCollisionOption.ReplaceExisting);
var d = dl.CreateDownload(transferUri, localFile);
d.Priority = BackgroundTransferPriority.High;
var progressCallback = new Progress<DownloadOperation>(this.DownloadProgress);
await d.StartAsync().AsTask(progressCallback);
}
private async void DownloadProgress(DownloadOperation download)
{
Debug.WriteLine("Callback");
if (download.Progress.Status == BackgroundTransferStatus.Completed)
{
var f = new Field();
f.Done = true;
await this.DbConnection.InsertAsync(f);
Debug.WriteLine("DONE");
}
}
If i breakpoint inside the DownloadProgress and then press F5 i get both Debug messages, and my db gets a new record.
However, if i just let the code execute, i never see "DONE" printed to me and neither is my db updated.
I tried wrapping the code in a new task:
await Task.Run(
async () =>
{
Debug.WriteLine("taskrun");
.... OTHER CODE FROM ABOVE...
});
But again, i only get to see 'taskrun' if i breakpoint in the callback.
UPDATE I actually think this is more related to checking the status. E.g. the statements outside of the check are executed, but only once, whereas anything inside the check is not executed.
Is there any way to force that callback to be invoked when the download is completed?
private async void DownloadProgress(DownloadOperation download)
{
Debug.WriteLine("Callback");
var value = download.Progress.BytesReceived * 100 download.Progress.TotalBytesToReceive;
new System.Threading.ManualResetEvent(false).WaitOne(1000);
if (download.Progress.Status == BackgroundTransferStatus.Completed )
{
var f = new Field();
f.Done = true;
await this.DbConnection.InsertAsync(f);
Debug.WriteLine("DONE");
}
}
I had this problem too, and I solved this by sleeping for 1000 ms, which worked really well for me.
Not sure what is causing this, but I was able to get the sample app to work reliably by manually checking the bytes to download as opposed to relying on the DownloadOperation.Progress.Status:
private async void DownloadProgress(DownloadOperation download)
{
Debug.WriteLine("Callback");
var value = download.Progress.BytesReceived * 100 / download.Progress.TotalBytesToReceive;
if (download.Progress.Status == BackgroundTransferStatus.Completed || value >= 100)
{
var f = new Field();
f.Done = true;
await this.DbConnection.InsertAsync(f);
Debug.WriteLine("DONE");
}
This gets me to 'DONE' every time.
I have a windows form that upload file to Amazon S3. I tried to implement built in async method but seems not working fine so I think the best way would be to implement System.Threading.Tasks.
My actual code looks like this:
public void UploadFileAsync(string bucketName, CloudDocument doc, bool publicRead)
{
config = new AmazonS3Config();
config.CommunicationProtocol = Protocol.HTTP;
client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKeyID, secretAccessKeyID, config);
// Load stream from file location
FileMode mode = FileMode.Open;
using (FileStream fs = new FileStream(doc.FullName, mode, FileAccess.Read))
{
// Create put object request
TransferUtilityUploadRequest objectRequest = new TransferUtilityUploadRequest();
objectRequest.InputStream = fs;
objectRequest.BucketName = bucketName;
if (publicRead) objectRequest.CannedACL = S3CannedACL.PublicRead;
objectRequest.Key = doc.KeyName + doc.FileName.Replace(' ', '_');
objectRequest.UploadProgressEvent += new EventHandler<UploadProgressArgs>(UploadProgressEvent);
transferUtility = new TransferUtility(client);
IAsyncResult asyncResult = transferUtility.BeginUpload(objectRequest, new AsyncCallback(UploadCallBack), results);
waitHandles.Add(asyncResult.AsyncWaitHandle);
// Wait till all the requests that were started are completed.
WaitHandle.WaitAll(waitHandles.ToArray());
}
client.Dispose();
}
}
private void UploadProgressEvent(object sender, UploadProgressArgs e)
{
if (UploadProgressChanged != null)
UploadProgressChanged(this, e);
}
private void UploadCallBack(IAsyncResult result)
{
Results results = result.AsyncState as Results;
try
{
// If there was an error during the put attributes operation it will be thrown as part of the EndPutAttributes method.
transferUtility.EndUpload(result);
results.Successes++;
}
catch (Exception e)
{
Console.WriteLine(e.Message);
results.Errors++;
}
}
Have anyone tried to implement Await / Async Task in order to upload async to amazon s3?
You can find instructions for wrapping APM and EAP members into TAP members on MSDN.
In summary, you use Task.Factory.FromAsync to wrap Begin/End method pairs, something like this:
public static Task UploadAsync(this TransferUtility #this, TransferUtilityUploadRequest request)
{
return Task.Factory.FromAsync(#this.BeginUpload, #this.EndUpload, request, null);
}
Then you can use it as such:
var task = transferUtility.UploadAsync(objectRequest);
tasks.Add(task);
await Task.WhenAll(tasks);
You may also be interested in the Developer Preview of the AWS SDK 2.0 for .NET. The source for the async extension class for the TransferUtility can be found here. Let us know if you have any feedback on the preview!
Thanks!
I'm attempting to store a List (where Show is my class that implements IXmlSerializable) to the local isolated storage. I'm using the code from this page:
http://metrostoragehelper.codeplex.com/
I've implemented the change suggested in the Issues section.
I am using the following code to add a Show object when it is clicked from an item list.
private async void addShowButton_Click_1(object sender, RoutedEventArgs e)
{
var isoStorage = new StorageHelper<List<Show>>(StorageType.Local);
List<Show> currentShows = await isoStorage.LoadASync("myShowsEx");
if(currentShows == null) {
currentShows = new List<Show>();
}
currentShows.Add(currentShow);
isoStorage.SaveASync(currentShows, "myShowsEx");
//Read it back, for debugging to check if it has been added properly.
List<Show> currentShowsRB = await isoStorage.LoadASync("myShowsEx"); //Exception here
}
The first show is added perfectly fine and it shows up in the currentShowsRB List. When a second item is clicked and the method above invoked an exception occurs on the last LoadAsync call: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
How can I get around this to access the local data store for multiple calls?
Below is also the relevant code from the StorageHelper:
public async void SaveASync(T Obj, string FileName)
{
FileName = FileName + ".xml";
try
{
if (Obj != null)
{
StorageFile file = null;
StorageFolder folder = GetFolder(storageType);
file = await folder.CreateFileAsync(FileName, CreationCollisionOption.ReplaceExisting);
using (var writeStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
Stream outStream = Task.Run(() => writeStream.AsStreamForWrite()).Result;
serializer.Serialize(outStream, Obj);
//writeStream.Dispose(); //Added and we get UnauthorizedAccessException
// outStream.Dispose(); //Added ObjectDisposedException caught in catch statement below
}
}
}
catch (Exception)
{
throw;
}
}
public async Task<T> LoadASync(string FileName)
{
FileName = FileName + ".xml";
try
{
StorageFile file = null;
StorageFolder folder = GetFolder(storageType);
file = await folder.GetFileAsync(FileName);
using (var readStream = await file.OpenAsync(FileAccessMode.Read))
{
Stream inStream = Task.Run(() => readStream.AsStreamForRead()).Result;
inStream.Position = 0;
return (T)serializer.Deserialize(inStream);
}
}
catch (FileNotFoundException)
{
//file not existing is perfectly valid so simply return the default
return default(T);
//throw;
}
catch (Exception)
{
//Unable to load contents of file
throw;
}
}
The writeStream.Dispose() line I added in, but even when this is included I get the same error message of Access is Denied. If I also include the outStream.Dispose() line then I get a ObjectDisposedException being caught in the catch statement right below. Is there something else I should be doing?
You are not waiting for the SaveAsync to finish, trying to Load when the Save is still in progress. Change it to:
//isoStorage.SaveASync(currentShows, "myShowsEx");
await isoStorage.SaveASync(currentShows, "myShowsEx");
List<Show> currentShowsRB = await isoStorage.LoadASync("myShowsEx");
Edit, awaiting on void is a standard problem.
The quick fix is :
await TaskEx.Run(() => isoStorage.SaveASync(currentShows, "myShowsEx"));
But you can also move the TaskEx.Run() inside SaveASync(). And given a name that ends with Async, it should not be void but:
Task SaveASyncT Obj, string FileName)
{
return TaskEx.Run() => { .... }
}
I don't believe there is an async version of Serialize, so it stays at TaskEx.Run() .
This block can be shortened to:
using (var readStream = await file.OpenAsync(FileAccessMode.Read))
{
return (T)serializer.Deserialize(readStream);
}
as you can directly read/write from the stream in the XmlSerializer class.
You may also find that CreationCollisionOption.ReplaceExisting can cause issues and OpenIfExists will cause the file to be overwritten by the serializer as required.