I Have endpoint which use handlers of 2 others endpoints it's probably not best practice, but it's not the point. In this methods I use a lot of MemoryStreams, ZipStream and stuff like that. Of course I dispose all of them. And everything works good till I run all tests together, then tests throw errors like: “Input string was not in a correct format.”, "Cannot read Zip file" or other weird messages. This are also test of this 2 handlers which I use in previous test.
Solution what I found is to add "Thread.Sleep(1);" at the end of the "Handle" method, just before return. It looks like something need more time to dispose, but why?. Have you any ideas why this 1ms sleep help with this?
ExtractFilesFromZipAndWriteToGivenZipArchive is an async method.
public async Task<MemoryStream> Handle(MultipleTypesExportQuery request, CancellationToken cancellationToken)
{
var stepwiseData = await HandleStepwise(request.RainmeterId, request.StepwiseQueries, cancellationToken);
var periodicData = await HandlePeriodic(request.RainmeterId, request.PeriodicQueries, cancellationToken);
var data = new List<MemoryStream>();
data.AddRange(stepwiseData);
data.AddRange(periodicData);
await using (var ms = new MemoryStream())
using (var archive = new ZipArchive(ms, ZipArchiveMode.Create,false))
{
int i = 0;
foreach (var d in data)
{
d.Open();
d.Position = 0;
var file = ZipFile.Read(d);
ExtractFilesFromZipAndWriteToGivenZipArchive(file, archive, i, cancellationToken);
i++;
file.Dispose();
d.Dispose();
}
//Thread.Sleep(100);
return ms;
}
}
ExtractFilesFromZipAndWriteToGivenZipArchive() is an asynchronous function which means, in this case, that you need to await it:
await ExtractFilesFromZipAndWriteToGivenZipArchive(file, archive, i, cancellationToken);
Otherwise, the execution will keep going without waiting the function to return.
Related
In my Asp.Net Core WebApi Controller, I'm receiving a IFormFile[] files. I need to convert this to of List<DocumentData>. I first used foreach. It was working fine. But later decided to change to Parallel.ForEach as I'm receiving many(>5) files.
Here is my DocumentData Class:
public class DocumentData
{
public byte[] BinaryData { get; set; }
public string FileName { get; set; }
}
Here is my Parallel.ForEach Logic:
var documents = new ConcurrentBag<DocumentData>();
Parallel.ForEach(files, async (currentFile) =>
{
if (currentFile.Length > 0)
{
using (var ms = new MemoryStream())
{
await currentFile.CopyToAsync(ms);
documents.Add(new DocumentData
{
BinaryData = ms.ToArray(),
FileName = currentFile.FileName
});
}
}
});
For Example, even for two files as input, documents always gives one file as output. Am I missing something?
I initially had List<DocumentData>. I found that it's not thread safe and changed to ConcurrentBag<DocumentData>. But still I'm getting unexpected results. Please assist on where I'm wrong?
I guess it is because, Parallel.Foreach doesn't support async/await. It only takes Action as input and executes it for each item. And in case of async delegates it will execute them in a fire-and-forget manner.
In that case passed lambda will be considered as async void function and async void can't be awaited.
If there were overload which takes Func<Task> then it would work.
I suggest you to create Tasks with the help of Select and use Task.WhenAll for executing them at the same time.
For example:
var tasks = files.Select(async currentFile =>
{
if (currentFile.Length > 0)
{
using (var ms = new MemoryStream())
{
await currentFile.CopyToAsync(ms);
documents.Add(new DocumentData
{
BinaryData = ms.ToArray(),
FileName = currentFile.FileName
});
}
}
});
await Task.WhenAll(tasks);
Additionally you can improve that code with just returning DocumentData instance from that method, and in such case there is no need to modify documents collection. Task.WhenAll has overload which takes IEnumerable<Task<TResult> as input and produces Task of TResult array. So, the result will be so:
var tasks = files.Select(async currentFile =>
{
if (currentFile.Length > 0)
{
using (var ms = new MemoryStream())
{
await currentFile.CopyToAsync(ms);
return new DocumentData
{
BinaryData = ms.ToArray(),
FileName = currentFile.FileName
};
}
}
return null;
});
var documents = (await Task.WhenAll(tasks)).Where(d => d != null).ToArray();
You had the right idea with a concurrent collection, but misused a TPL method.
In short you need to be very careful about async lambdas, and if you are passing them to an Action or Func<Task>
Your problem is because Parallel.For / ForEach is not suited for the async and await pattern or IO bound tasks. They are suited for cpu bound workloads. Which means they essentially have Action parameters and let's the task scheduler create the tasks for you
If you want to run mutple tasks at the same time use Task.WhenAll , or a TPL Dataflow ActionBlock which can deal effectively with both CPU bound and IO bound works loads, or said more directly, they can deal with tasks which is what an async method is.
The fundimental issue is when you call an async lambda on an Action, you are essentially creating an async void method, which will run as a task unobserved. That's to say, your TPL method is just creating a bunch of tasks in parallel to run a bunch of unobserved tasks and not waiting for them.
Think of it like this, you ask a bunch of friends to go and get you some groceries, they in turn tell someone else to get your groceries, yet your friends report back to you and say thier job is done. It obviously isn't and you have no groceries.
Given the following method signature in a WCF service:
public string Query(string request)
{
using (Log.BeginTimedOperation("Persist request"))
{
var messageCorrelationId = Guid.NewGuid().ToString();
var payloadURI = PayloadHelper.GetFullPath(messageCorrelationId);
PayloadHelper.PersistPayloadWithPath(request, payloadURI);
Log.Information("My service request {MessageCorrelationId} {RequestPayloadPath}", messageCorrelationId, payloadURI);
}
// DoWork here, code removed for brevity
return result;
}
and the corresponding extension methods:
public static string GetFullPath(string messageCorrelationId)
{
var folderDate = DateTime.Now.ToString("yyyyMMdd");
var folderHour = DateTime.Now.ToString("HH");
var logFolder = Path.Combine(ConfigurationManager.AppSettings["NetworkFiler"], "Payloads", folderDate, folderHour);
Directory.CreateDirectory(logFolder);
var fileName = $"{messageCorrelationId}-{"MyWCFService"}-{$"{DateTime.Now:yyyy-MM-dd-HH-mm-ss-fff}-{Guid.NewGuid():N}"}.{"xml"}";
return Path.Combine(logFolder, fileName);
}
public static void PersistPayloadWithPath(string text, string path)
{
var task = WriteFileAsync(path, text);
task.GetAwaiter().GetResult();
}
private static async Task WriteFileAsync(string path, string text)
{
try
{
byte[] buffer = Encoding.Unicode.GetBytes(text);
using (var stream = new FileStream(path, FileMode.Create, FileAccess.Write, FileShare.None, bufferSize: 4096, useAsync: true))
{
await stream.WriteAsync(buffer, 0, buffer.Length).ConfigureAwait(false);
}
}
catch (Exception ex)
{
Serilog.Log.Error(ex, "WriteFileAsync");
}
}
This code will block, if for instance the file is being interrogated by anti-virus (guess) or IO slowness to the filer.
So here is the great debate, calling a asynchronous method from a synchronous method in ASP.NET. To this day I still don't know if there is a reliable way to create a fire and forget mechanism. It's not that I don't care about the failure, I do, that 'should be' handled by the catch statement and the static Serilog instance.
While writing this version of the post, it dawned on me that maybe one of the problems might be in fact the logger and it's async wrapper around the File Sink.. will test that in a few.
Any help is appreciated with this bugger.
Thank you,
Stephen
UPDATE-ASYNC
public async Task<string> QueryAsync(string request)
{
using (Log.BeginTimedOperation("PersistAsync-Request"))
{
var messageCorrelationId = Guid.NewGuid().ToString();
var payloadURI = PayloadHelper.GetFullPath(messageCorrelationId);
await PayloadHelper.PersistPayloadWithPathAsync(request, payloadURI).ConfigureAwait(false);
Log.Information("My service request {MessageCorrelationId} {RequestPayloadPath}", messageCorrelationId, payloadURI);
}
// DoWork here, code removed for brevity
return result;
}
public static string GetFullPath(string messageCorrelationId)
{
var folderDate = DateTime.Now.ToString("yyyyMMdd");
var folderHour = DateTime.Now.ToString("HH");
var logFolder = Path.Combine(ConfigurationManager.AppSettings["NetworkFiler"], "Payloads", folderDate, folderHour);
Directory.CreateDirectory(logFolder);
var fileName = $"{messageCorrelationId}-MyWCFService-{DateTime.Now:yyyy-MM-dd-HH-mm-ss-fff}-{Guid.NewGuid():N}.xml";
return Path.Combine(logFolder, fileName);
}
public static async Task PersistPayloadWithPathAsync(string text, string path)
{
await WriteFileAsync(path, text).ConfigureAwait(false);
}
private static async Task WriteFileAsync(string path, string text)
{
try
{
byte[] buffer = Encoding.Unicode.GetBytes(text);
using (var stream = new FileStream(path, FileMode.Create, FileAccess.Write, FileShare.None, bufferSize: 4096, useAsync: true))
{
await stream.WriteAsync(buffer, 0, buffer.Length).ConfigureAwait(false);
}
}
catch
{
// ignored
}
}
Still blocking randomly, lick every 20-30 requests
So it's blocking on the new FileStream(). Looking at the source code, the constructor calls a method called Init(), which actually ends up opening the file. So it is doing I/O in the constructor, which really it shouldn't be since you can't await it.
Setting useAsync should make it run async, if it can. But maybe it can't open the file from a network drive asynchronously.
So your best bet is to just run it inside Task.Run() to guarantee it doesn't block.
The following only applies to .NET Core (unfortunately):
You would be better off using File.WriteAllTextAsync, which would actually make your life easier:
await File.WriteAllTextAsync(path, text);
The documentation of that doesn't really explain anything for some reason, but it works the same as File.WriteAllText (it's just async):
Creates a new file, write the contents to the file, and then closes the file. If the target file already exists, it is overwritten.
So, exactly what you're doing in your code, but this way you can await the whole operation (including opening the file).
I think it waits becuse you use
task.GetAwaiter().GetResult();
If you don't need the result, just remove this line. It should work fine.
If you need the result you should make PersistPayloadWithPath function also async.
The problem is your WriteFileAsync method. It actually runs synchronously until it hits the first await (that's how async/await works). I believe it hangs at new FileStream(...).
If you want to just fire-end-forget form synchronous code, this should be enough:
public static void PersistPayloadWithPath(string text, string path)
{
Task.Run(async () => await WriteFileAsync(path, text));
}
The code above should help you when you don't have async alternatives. However as Gabriel Luci suggested in his answer you should go with await File.WriteAllTextAsync(path, text); as it's probably optimized for async work.
You can still use Task.Run(...); with await File.WriteAllTextAsync(path, text); in fire-and-forget scenario.
Just beware that exceptions from inside the task (WriteFileAsync) won't propagate to the calling thread. This won't be the problem in your case since the whole WriteFileAsync method's body is inside try-catch block, where you log the exception.
EDIT
To illustrate how the threads behave in async/await methods, play around with the following example (try all 3 ways the Bar function is ran):
using System;
using System.Net;
using System.Threading;
using System.Threading.Tasks;
namespace ConsoleApp2
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine($"Main thread: {Thread.CurrentThread.ManagedThreadId}");
Task.Run(async () => await Bar());
// Task.Run(() => Bar());
// Bar();
Console.ReadLine();
}
static async Task Bar()
{
Console.WriteLine($"Bar thread before await: {Thread.CurrentThread.ManagedThreadId}");
await Foo();
Console.WriteLine($"Bar thread after await: {Thread.CurrentThread.ManagedThreadId}");
}
static async Task Foo()
{
Console.WriteLine($"Foo thread before await: {Thread.CurrentThread.ManagedThreadId}");
var c = new WebClient();
var source = await c.DownloadStringTaskAsync("http://google.com");
Console.WriteLine($"Foo thread after await: {Thread.CurrentThread.ManagedThreadId}");
}
}
}
The root caused was determined to be an incorrectly configured StealthAudit, a tool that we use to audit our filers. Once the configuration was adjusted everything works as expected.
I need to read a numeric value (a version number) from a text file in my resources.
I compare this version number to the version number of an installed component.
If the version number in the resources is higher than the installed version, I copy the new component (a database) from my resources to a local directory where the user can use it.
I need to do this synchronously because my application can't work without the database.
However, I don't see any way to do it synchronously.
MS forces me to do it with an async task like this:
private async Task<string> ResourcesReadTextFile(string uFileName)
{
string sRet = "";
try
{
StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(new Uri(cstAssets + uFileName));
using (var inputStream = await file.OpenReadAsync())
using (var classicStream = inputStream.AsStreamForRead())
using (var streamReader = new StreamReader(classicStream))
{
while (streamReader.Peek() >= 0)
{
sRet = streamReader.ReadLine();
}
}
}
catch (Exception ex)
{
Debug.Assert(false);//check here
}
return sRet;
}
Now I've encountered a situation where the app started before the database was copied over to the local directory as the copying also needs to be done asynchronously, there simply isn't any way to do it synchronous.
There is no such function as StorageFile.Copy().
What I'm therefore using is:
private async void pCopyFromResourcesToLocal(string uFileName)
{
// Cant await inside catch, but this works anyway
try
{
StorageFile storfile = await StorageFile.GetFileFromApplicationUriAsync(new Uri(cstAssets + uFileName));
await storfile.CopyAsync(ApplicationData.Current.LocalFolder);
}
catch (Exception ex)
{
Debug.WriteLine("");
}
}
This drives me crazy.
People write that Async should be embraced and hugged and appreciated, but in my case it causes nothing but trouble.
I don't see any way of making this thing synchronous, and I wonder why MS forces me to do it that way.
Any help very much appreciated.
Thank you.
Code as a screenshot:
Edit: I've added the top methods here:
public static async Task<DB> InitAppDb()
{
IFileHelper helper = DependencyService.Get<IFileHelper>();
string path = await helper.GetFilePathAndCopyFromResourcesIfNotPresent("tablet.db");
return (_dbApp = new DB(path));
}
public async Task CopyDatabaseIfNotExists(string uFileName)
{
IsolatedStorageFile nExpectedFolder = IsolatedStorageFile.GetUserStoreForApplication();
bool bCopyNewDB = false;
Task<bool> datatask = pResourceIsNewer(uFileName);
bCopyNewDB = await datatask;
if (! bCopyNewDB)
{
try
{
await ApplicationData.Current.LocalFolder.GetFileAsync(uFileName); //nExpectedFolder.GetFileAsync(dbPath);/// ApplicationData.Current.LocalFolder.GetFileAsync("preinstalledDB.db");
// No exception means it exists
return;
}
catch (System.IO.FileNotFoundException)
{
// The file obviously doesn't exist
}
}
pCopyFromResourcesToLocal(uFileName);
}
private async Task<bool>pResourceIsNewer(string uPath)
{
string sFileNameAppDBVersion =uPath + ".txt";
if (IsolatedStorageFileExist(sFileNameAppDBVersion))
{
int iAppDBVersionInstalled = Convert.ToInt32(IsolatedStorageReadTextFile(sFileNameAppDBVersion));
Task<string> datatask = ResourcesReadTextFile(sFileNameAppDBVersion);
string s = await datatask;
int iAppDBResources = Convert.ToInt32(s);
bool b = (iAppDBResources > iAppDBVersionInstalled);
return b;
}
else
{
return true;
}
}
All you have to do is:
//private async void pCopyFromResourcesToLocal(string uFileName) { ... }
private async Task pCopyFromResourcesToLocal(string uFileName) { ... }
and then you can await it:
//pCopyFromResourcesToLocal(uFileName);
await pCopyFromResourcesToLocal(uFileName);
and it will all be completed before you call return (_dbApp = new DB(path));
Nothing in this async/await chain can happen out of order.
When you say that your app can't work without the database, remember that using an await keyword does just that, so the following line of code will not execute until after the async call returns. You can structure your code such that it is responsive while you wait for the DB to come back online.
However, you can force your function to be synchronous by using a construct such as:
StorageFile.GetFileFromApplicationUriAsync(new Uri(cstAssets + uFileName)).GetAwaiter().GetResult();
Or, even better, have a look at the JoinableTaskFactory.
Any asynchronous API method can be made synchronous by simply tagging .GetAwaiter().GetResult() on the end, as #pm_2 says.
In the case of Task<T> results, you can simply use .Result, and for Task results, .Wait().
You can either write an asynchronous function to read your file, then wait for its result at the top level, or you can write a synchronous method and wait for the result of every call it makes to async functions.
So Microsoft is not forcing you into anything: it's just providing a simpler API with the lowest common denominator of both asynchronous and synchronous workflows.
I've been reading examples for a long time now, but unfortunately I've been unable to apply the solutions to the code I'm working with. Some quick Facts/Assorted Info:
1) I'm new to C#
2) The code posted below is modified from Amazon Web Services (mostly stock)
3) Purpose of code is to compare server info to offline already downloaded info and create a list of need to download files. This snip is for the list made from the server side, only option with AWS is to call async, but I need this to finish before moving forward.
public void InitiateSearch()
{
UnityInitializer.AttachToGameObject(this.gameObject);
//these are the access key and secret access key for credentials
BasicAWSCredentials credentials = new BasicAWSCredentials("secret key", "very secret key");
AmazonS3Config S3Config = new AmazonS3Config()
{
ServiceURL = ("url"),
RegionEndpoint = RegionEndpoint.blahblah
};
//Setting the client to be used in the call below
AmazonS3Client Client = new AmazonS3Client(credentials, S3Config);
var request = new ListObjectsRequest()
{
BucketName = "thebucket"
};
Client.ListObjectsAsync(request, (responseObject) =>
{
if (responseObject.Exception == null)
{
responseObject.Response.S3Objects.ForEach((o) =>
{
int StartCut = o.Key.IndexOf(SearchType) - 11;
if (SearchType == o.Key.Substring(o.Key.IndexOf(SearchType), SearchType.Length))
{
if (ZipCode == o.Key.Substring(StartCut + 12 + SearchType.Length, 5))
{
AWSFileList.Add(o.Key + ", " + o.LastModified);
}
}
}
);
}
else
{
Debug.Log(responseObject.Exception);
}
});
}
I have no idea how to apply await to the Client.ListObjectsAsync line, I'm hoping you all can give me some guidance and let me keep my hair for a few more years.
You can either mark your method async and await it, or you can call .Wait() or .Result() on the Task you're given back.
I have no idea how to apply await to the Client.ListObjectsAsync line
You probably just put await in front of it:
await Client.ListObjectsAsync(request, (responseObject) => ...
As soon as you do this, Visual Studio will give you an error. Take a good look at the error message, because it tells you exactly what to do next (mark InitiateSearch with async and change its return type to Task):
public async Task InitiateSearchAsync()
(it's also a good idea to add an Async suffix to follow the common pattern).
Next, you'd add an await everywhere that InitiateSearchAsync is called, and so on.
I'm assuming Client.ListObjectsAsync returns a Task object, so a solution for your specific problem would be this:
public async void InitiateSearch()
{
//code
var collection = await Client.ListObjectsAsync(request, (responseObject) =>
{
//code
});
foreach (var item in collection)
{
//do stuff with item
}
}
the variable result will now be filled with the objects. You may want to set the return type of InitiateSearch() to Task, so you can await it too.
await InitiateSearch(); //like this
If this method is an event handler of some sort (like called by the click of a button), then you can keep using void as return type.
A simple introduction from an unpublished part of the documentation for async-await:
Three things are needed to use async-await:
The Task object: This object is returned by a method which is executed asynchronous. It allows you to control the execution of the method.
The await keyword: "Awaits" a Task. Put this keyword before the Task to asynchronously wait for it to finish
The async keyword: All methods which use the await keyword have to be marked as async
A small example which demonstrates the usage of this keywords
public async Task DoStuffAsync()
{
var result = await DownloadFromWebpageAsync(); //calls method and waits till execution finished
var task = WriteTextAsync(#"temp.txt", result); //starts saving the string to a file, continues execution right await
Debug.Write("this is executed parallel with WriteTextAsync!"); //executed parallel with WriteTextAsync!
await task; //wait for WriteTextAsync to finish execution
}
private async Task<string> DownloadFromWebpageAsync()
{
using (var client = new WebClient())
{
return await client.DownloadStringTaskAsync(new Uri("http://stackoverflow.com"));
}
}
private async Task WriteTextAsync(string filePath, string text)
{
byte[] encodedText = Encoding.Unicode.GetBytes(text);
using (FileStream sourceStream = new FileStream(filePath, FileMode.Append))
{
await sourceStream.WriteAsync(encodedText, 0, encodedText.Length);
}
}
Some thing to note:
You can specify a return value from an asynchronous operations with Task. The await keyword waits till the execution of the method finishes, and returns the string.
the Task object contains the status of the execution of the method, it can be used as any other variable.
if an exception is thrown (for example by the WebClient) it bubbles up at the first time the await keyword is used (in this example at the line string result (...))
It is recommended to name methods which return the Task object as MethodNameAsync
For more information about this take a look at http://blog.stephencleary.com/2012/02/async-and-await.html.
While on my local box the following code works:
public async Task<GameStatistic> LoadReport()
{
var folder = await ApplicationData.Current.LocalFolder.CreateFolderAsync(FolderName, CreationCollisionOption.OpenIfExists);
var file = await folder.GetFileAsync(FileName);
GameStatistic returnValue;
using (var inStream = await file.OpenSequentialReadAsync())
{
var serializer = new DataContractJsonSerializer(typeof (GameStatistic));
returnValue = serializer.ReadObject(inStream.AsStreamForRead()) as GameStatistic;
}
return returnValue;
}
Code that calls the above method:
public GameStatistic GetReportData()
{
var repo = new GameRepository();
var gameStatTask = repo.LoadReport(); //(awaitable) Task<GameStatistic>
gameStatTask.Wait(); //this seems to make no difference
return gameStatTask.Result;
}
But When I move to code to my Surface Pro and run the application (no debugger), the folder.GetFileAsync(FileName) fails because the async call to get the folder hasn't returned yet.
When I debug the application on my Surface Pro (via Remote Machine) and slowly walk the debugger past the first line of code and wait a few seconds, and then step again, everything works.
I don't like the idea of trying to put a thread to sleep for an arbitrary length of time, but I am not sure what else I can do here.
Is there something I am doing wrong or something I should be doing that I am not doing at all?
Is there a common practice that would really wait until the CreateFolderAsync returns so that when I call folder.GetFileAsync that I could be sure the preceding line was complete?
Thanks for any help you may be able to provide.
As #J.B points out you need to use await instead of wait. Also, any function that calls an async method should itself be async (there is at least one exception to this). So almost your entire call stack up to the UI must be changed to be some variation of async Task<...>...:
async public Task<GameStatistic> GetReportData()
{
var repo = new GameRepository();
return await repo.LoadReport(); //(awaitable) Task<GameStatistic>
}
Caller of above (just an arbitrary method):
async public Task<MyResultClass> GetReportAndResult()
{
var gameStat = await GetReportData();
return ReportDataToMyResult(gameStat);
}
Top of call chain (event handler) must be async void:
async void GetReportData_ButtonClick(...)
{
var result = await GetReportAndResult();
// do something with result
// ...
}