I have a windows form that upload file to Amazon S3. I tried to implement built in async method but seems not working fine so I think the best way would be to implement System.Threading.Tasks.
My actual code looks like this:
public void UploadFileAsync(string bucketName, CloudDocument doc, bool publicRead)
{
config = new AmazonS3Config();
config.CommunicationProtocol = Protocol.HTTP;
client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKeyID, secretAccessKeyID, config);
// Load stream from file location
FileMode mode = FileMode.Open;
using (FileStream fs = new FileStream(doc.FullName, mode, FileAccess.Read))
{
// Create put object request
TransferUtilityUploadRequest objectRequest = new TransferUtilityUploadRequest();
objectRequest.InputStream = fs;
objectRequest.BucketName = bucketName;
if (publicRead) objectRequest.CannedACL = S3CannedACL.PublicRead;
objectRequest.Key = doc.KeyName + doc.FileName.Replace(' ', '_');
objectRequest.UploadProgressEvent += new EventHandler<UploadProgressArgs>(UploadProgressEvent);
transferUtility = new TransferUtility(client);
IAsyncResult asyncResult = transferUtility.BeginUpload(objectRequest, new AsyncCallback(UploadCallBack), results);
waitHandles.Add(asyncResult.AsyncWaitHandle);
// Wait till all the requests that were started are completed.
WaitHandle.WaitAll(waitHandles.ToArray());
}
client.Dispose();
}
}
private void UploadProgressEvent(object sender, UploadProgressArgs e)
{
if (UploadProgressChanged != null)
UploadProgressChanged(this, e);
}
private void UploadCallBack(IAsyncResult result)
{
Results results = result.AsyncState as Results;
try
{
// If there was an error during the put attributes operation it will be thrown as part of the EndPutAttributes method.
transferUtility.EndUpload(result);
results.Successes++;
}
catch (Exception e)
{
Console.WriteLine(e.Message);
results.Errors++;
}
}
Have anyone tried to implement Await / Async Task in order to upload async to amazon s3?
You can find instructions for wrapping APM and EAP members into TAP members on MSDN.
In summary, you use Task.Factory.FromAsync to wrap Begin/End method pairs, something like this:
public static Task UploadAsync(this TransferUtility #this, TransferUtilityUploadRequest request)
{
return Task.Factory.FromAsync(#this.BeginUpload, #this.EndUpload, request, null);
}
Then you can use it as such:
var task = transferUtility.UploadAsync(objectRequest);
tasks.Add(task);
await Task.WhenAll(tasks);
You may also be interested in the Developer Preview of the AWS SDK 2.0 for .NET. The source for the async extension class for the TransferUtility can be found here. Let us know if you have any feedback on the preview!
Thanks!
Related
I have a question about async programming and Task.WhenAll(). I have a code snippet that downloads a folder from google drive and it works as expected when i am debugging my code. However when I run the application without debugger the download function takes atleast 4x the time it takes when it runs with the debugger. I also get crash logs with TaskCancelled exceptions (System.Threading.Tasks.TaskCanceledException: A task was canceled.) which do not happen with the debugger attached. What needs to be changed for the code to work as expected without debugger attached. NB this snippet downloads +- 1000 files in about 22-25 seconds with debugger and 2min+ without debugger.
public static async Task<bool> DownloadFolder(CloudDataModel.File file, string path, params string[] exclude)
{
try
{
if (file != null && !string.IsNullOrEmpty(file.id))
{
List<string> toExclude = new List<string>();
if(exclude != null)
{
toExclude = exclude.ToList();
}
List<Task> downloadFilesTask = new List<Task>();
var files = await file.GetFiles();
foreach (var f in files)
{
var task = f.Download(path);
downloadFilesTask.Add(task);
}
var folders = await file.GetFoldersAsync();
foreach (var folder in folders)
{
if (toExclude.Contains(folder.name))
{
continue;
}
Task task = null;
if (path.Equals(Statics.ProjectFolderName))
{
task = DownloadFolder(folder, folder.name);
}
else
{
task = DownloadFolder(folder, Path.Combine(path, folder.name));
}
downloadFilesTask.Add(task);
}
var array = downloadFilesTask.ToArray();
await Task.WhenAll(array);
return true;
}
}
catch (Exception e)
{
Crashes.TrackError(e);
}
return false;
}
Edit
after some more trial and error the fault has been identified.
the downloading of the file was the cause of the unexpected behaviour
public static async Task<StorageFile> DownloadFile(CloudDataModel.File file, string destinationFolder)
{
try
{
if (file != null)
{
Debug.WriteLine($"start download {file.name}");
if (file.mimeType == Statics.GoogleDriveFolderMimeType)
{
Debug.WriteLine($"did not download resource, resource was folder instead of file. mimeType: {file.mimeType}");
return null;
}
var endpoint = Path.Combine(DownloadFileEndpoint, $"{file.id}?alt=media");
// this would cause the unexpected behaviour
HttpResponseMessage response = await client.GetAsync(endpoint);
StorageFile downloadedFile;
using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
{
var memstream = new MemoryStream();
StreamReader reader = new StreamReader(streamToReadFrom);
streamToReadFrom.Position = 0;
await streamToReadFrom.CopyToAsync(memstream);
downloadedFile = await fileHandler.SaveDownloadedCloudFile(memstream, file, destinationFolder);
Debug.WriteLine($"download finished {file.name}");
}
return downloadedFile;
}
return null;
}
catch (Exception e)
{
Crashes.TrackError(e);
return null;
}
}
After setting a timeout to the client (System.Net.Http.HttpClient) the code executed as expected.
client.Timeout = new TimeSpan(0,0,5);
So after some trial and error the downloading of the file was identified as the cause of the problem.
var task = f.Download(path);
the Download function implements a System.Net.Http.HttpClient wich was initialized without any kind of timeout. when one of the tasks had a timeout, all other tasks would also not execute as expected. to fix this issue a timeout of 5 seconds was added to the HttpClient object.
private static void InitializeClient()
{
HttpClientHandler handler = new HttpClientHandler { AllowAutoRedirect = true };
client = new HttpClient(handler);
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", AccountManager.LoggedInAccount.Token);
client.Timeout = new TimeSpan(0,0,5);
}
After this change the code would execute as expected.
My guess is that without the debugger the parallelism increases, and so the remote server is overflowed by requests and can't perform optimally. Your code does not have any provision for limiting the degree of concurrency/parallelism. One possible way to solve this problem is to use a SemaphoreSlim, as shown in this question (among many others).
I'm busy with writing a program that writes JSON serialized objects to a JSON-file.
Now I've a problem with it. I want to write each object inside a list seperately to the JSON-file. Ofcourse in real scenario the program should add all objects to a list and hen write the list to the JSON file at once, but for now I want to do it one by one. The way I'm trying to do that introduces I/O errors and it's not clear for me why.
readonly BaseStore<TestObject> _store2 = new BaseStore<TestObject>("test", Settings.DatabaseBasePath);
private async void testToolStripMenuItem_Click(object sender, EventArgs e)
{
await Write();
}
private async Task Write()
{
for (int i = 0; i < 1000; i++)
{
await _store2.Save(new TestObject
{
Property = "value"
});
}
}
Code of the Save method:
public async Task Save(T obj)
{
var collection = _database.Load<T>(_collection);
// do some collection changes
await _database.Save(collection, _collection);
}
Above method will call the following method:
public async Task Save(object obj, string collectionName)
{
string fileName = string.Format("{0}{1}", collectionName, ".json");
var json = JsonConvert.SerializeObject(obj, Formatting.None);
byte[] encodedText = Encoding.Unicode.GetBytes(json);
using (FileStream sourceStream = new FileStream(Path.Combine(_basePath, fileName),
FileMode.Append, FileAccess.Write, FileShare.None,
bufferSize: 4096, useAsync: true))
{
await sourceStream.WriteAsync(encodedText, 0, encodedText.Length);
};
}
Above code throws the following errors:
Exception has been thrown by the target of an invocation.
The requested operation cannot be performed on a file with a user-mapped section open.
I think maybe the problem is that the Save method inside the for-loop tries to access the JSON-file while it is opened by the previous Save call, but I don't know how to solve that.
Can somebody point me into the right direction please?
I am writing a universal app primarily targeting Windows Phone using SQLite-net.
During the course of operation, the user is presented with an option to download multiple files. At the end of each file download, I need to mark the file in the db as completed. I am using BackgroundDownloader in order to download files - the WP8.0 app used Background Transfer Service and that worked great. Files can be huge (some 200+ mbs, user content) and i am not looking forward to wrapping the downloads in the HttpClient or WebClient.
However, it seems that the progress callback doesn't work with awaits unless I actually breakpoint in the method.
The following are listings from a sample app i quickly put together that demonstrates the behaviour:
Model:
public class Field
{
[PrimaryKey]
[AutoIncrement]
public int Id { get; set; }
public bool Done { get; set; }
}
MainPage codebehind (i am creating a db here only for the purposes of this example!):
private async void Button_Click(object sender, RoutedEventArgs e)
{
using (var db = new SQLiteConnection(Windows.Storage.ApplicationData.Current.LocalFolder.Path + "//Main.db"))
{
db.CreateTable<Field>();
db.Commit();
}
this.DbConnection = new SQLiteAsyncConnection(Windows.Storage.ApplicationData.Current.LocalFolder.Path + "//My.db");
var dl = new BackgroundDownloader();
dl.CostPolicy = BackgroundTransferCostPolicy.Always;
var transferUri = new Uri("http://192.168.1.4/hello.world", UriKind.Absolute);
var folder = await ApplicationData.Current.LocalFolder.CreateFolderAsync(
"Content",
CreationCollisionOption.OpenIfExists);
var localFile = await folder.CreateFileAsync("cheesecakes.file", CreationCollisionOption.ReplaceExisting);
var d = dl.CreateDownload(transferUri, localFile);
d.Priority = BackgroundTransferPriority.High;
var progressCallback = new Progress<DownloadOperation>(this.DownloadProgress);
await d.StartAsync().AsTask(progressCallback);
}
private async void DownloadProgress(DownloadOperation download)
{
Debug.WriteLine("Callback");
if (download.Progress.Status == BackgroundTransferStatus.Completed)
{
var f = new Field();
f.Done = true;
await this.DbConnection.InsertAsync(f);
Debug.WriteLine("DONE");
}
}
If i breakpoint inside the DownloadProgress and then press F5 i get both Debug messages, and my db gets a new record.
However, if i just let the code execute, i never see "DONE" printed to me and neither is my db updated.
I tried wrapping the code in a new task:
await Task.Run(
async () =>
{
Debug.WriteLine("taskrun");
.... OTHER CODE FROM ABOVE...
});
But again, i only get to see 'taskrun' if i breakpoint in the callback.
UPDATE I actually think this is more related to checking the status. E.g. the statements outside of the check are executed, but only once, whereas anything inside the check is not executed.
Is there any way to force that callback to be invoked when the download is completed?
private async void DownloadProgress(DownloadOperation download)
{
Debug.WriteLine("Callback");
var value = download.Progress.BytesReceived * 100 download.Progress.TotalBytesToReceive;
new System.Threading.ManualResetEvent(false).WaitOne(1000);
if (download.Progress.Status == BackgroundTransferStatus.Completed )
{
var f = new Field();
f.Done = true;
await this.DbConnection.InsertAsync(f);
Debug.WriteLine("DONE");
}
}
I had this problem too, and I solved this by sleeping for 1000 ms, which worked really well for me.
Not sure what is causing this, but I was able to get the sample app to work reliably by manually checking the bytes to download as opposed to relying on the DownloadOperation.Progress.Status:
private async void DownloadProgress(DownloadOperation download)
{
Debug.WriteLine("Callback");
var value = download.Progress.BytesReceived * 100 / download.Progress.TotalBytesToReceive;
if (download.Progress.Status == BackgroundTransferStatus.Completed || value >= 100)
{
var f = new Field();
f.Done = true;
await this.DbConnection.InsertAsync(f);
Debug.WriteLine("DONE");
}
This gets me to 'DONE' every time.
I am trying to parse data from a json file but I am getting variable outputs(Sometimes right, othertimes nothing). I am pretty much sure it is related with the time needed to parse the file, but having trouble finding out where. Here it is-
public class HspitalVM
{
List<Hspital> hspitalList=null;
public List<KeyedList<string, Hspital>> GroupedHospitals
{
get
{
getJson();
var groupedHospital =
from hspital in hspitalList
group hspital by hspital.Type into hspitalByType
select new KeyedList<string, Hspital>(hspitalByType);
return new List<KeyedList<string, Hspital>>(groupedHospital);
}
}
public async void getJson()
{
StorageFolder localFolder = ApplicationData.Current.LocalFolder;
try
{
StorageFile textFile = await localFolder.GetFileAsync(m_HospFileName);
using (IRandomAccessStream textStream = await textFile.OpenReadAsync())
{
using (DataReader textReader = new DataReader(textStream))
{
uint textLength = (uint)textStream.Size;
await textReader.LoadAsync(textLength);
string jsonContents = textReader.ReadString(textLength);
hspitalList = JsonConvert.DeserializeObject<IList<Hspital>>(jsonContents) as List<Hspital>;
}
}
}
catch (Exception ex)
{
string err = "Exception: " + ex.Message;
MessageBox.Show(err);
}
}
}
You are not await-ing result of getJson() call, so as expected most of the times you'll get no information because actual call to GetFileAsync is not completed yet.
Now since getJson method returns void you can't really await on it. Potential fix by using Result to turn asynchronous code into synchronous for get:
public List<KeyedList<string, Hspital>> GroupedHospitals
{
get
{
hspitalList= getJson().Result;
...
}
}
...
public async Task<IList<Hspital>> getJson()
{
....
return JsonConvert.DeserializeObject<IList<Hspital>>(jsonContents) as List<Hspital>;
}
Note: It is generally bad idea to make get methods to perform long operations, also calling async method in synchronous way via Wait/Result can cause deadlocks in your code -await vs Task.Wait - Deadlock?.
I'm attempting to store a List (where Show is my class that implements IXmlSerializable) to the local isolated storage. I'm using the code from this page:
http://metrostoragehelper.codeplex.com/
I've implemented the change suggested in the Issues section.
I am using the following code to add a Show object when it is clicked from an item list.
private async void addShowButton_Click_1(object sender, RoutedEventArgs e)
{
var isoStorage = new StorageHelper<List<Show>>(StorageType.Local);
List<Show> currentShows = await isoStorage.LoadASync("myShowsEx");
if(currentShows == null) {
currentShows = new List<Show>();
}
currentShows.Add(currentShow);
isoStorage.SaveASync(currentShows, "myShowsEx");
//Read it back, for debugging to check if it has been added properly.
List<Show> currentShowsRB = await isoStorage.LoadASync("myShowsEx"); //Exception here
}
The first show is added perfectly fine and it shows up in the currentShowsRB List. When a second item is clicked and the method above invoked an exception occurs on the last LoadAsync call: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
How can I get around this to access the local data store for multiple calls?
Below is also the relevant code from the StorageHelper:
public async void SaveASync(T Obj, string FileName)
{
FileName = FileName + ".xml";
try
{
if (Obj != null)
{
StorageFile file = null;
StorageFolder folder = GetFolder(storageType);
file = await folder.CreateFileAsync(FileName, CreationCollisionOption.ReplaceExisting);
using (var writeStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
Stream outStream = Task.Run(() => writeStream.AsStreamForWrite()).Result;
serializer.Serialize(outStream, Obj);
//writeStream.Dispose(); //Added and we get UnauthorizedAccessException
// outStream.Dispose(); //Added ObjectDisposedException caught in catch statement below
}
}
}
catch (Exception)
{
throw;
}
}
public async Task<T> LoadASync(string FileName)
{
FileName = FileName + ".xml";
try
{
StorageFile file = null;
StorageFolder folder = GetFolder(storageType);
file = await folder.GetFileAsync(FileName);
using (var readStream = await file.OpenAsync(FileAccessMode.Read))
{
Stream inStream = Task.Run(() => readStream.AsStreamForRead()).Result;
inStream.Position = 0;
return (T)serializer.Deserialize(inStream);
}
}
catch (FileNotFoundException)
{
//file not existing is perfectly valid so simply return the default
return default(T);
//throw;
}
catch (Exception)
{
//Unable to load contents of file
throw;
}
}
The writeStream.Dispose() line I added in, but even when this is included I get the same error message of Access is Denied. If I also include the outStream.Dispose() line then I get a ObjectDisposedException being caught in the catch statement right below. Is there something else I should be doing?
You are not waiting for the SaveAsync to finish, trying to Load when the Save is still in progress. Change it to:
//isoStorage.SaveASync(currentShows, "myShowsEx");
await isoStorage.SaveASync(currentShows, "myShowsEx");
List<Show> currentShowsRB = await isoStorage.LoadASync("myShowsEx");
Edit, awaiting on void is a standard problem.
The quick fix is :
await TaskEx.Run(() => isoStorage.SaveASync(currentShows, "myShowsEx"));
But you can also move the TaskEx.Run() inside SaveASync(). And given a name that ends with Async, it should not be void but:
Task SaveASyncT Obj, string FileName)
{
return TaskEx.Run() => { .... }
}
I don't believe there is an async version of Serialize, so it stays at TaskEx.Run() .
This block can be shortened to:
using (var readStream = await file.OpenAsync(FileAccessMode.Read))
{
return (T)serializer.Deserialize(readStream);
}
as you can directly read/write from the stream in the XmlSerializer class.
You may also find that CreationCollisionOption.ReplaceExisting can cause issues and OpenIfExists will cause the file to be overwritten by the serializer as required.