Develop Reference

Parallel.ForEach not adding items as expected in ConcurrentBag in C#

In my Asp.Net Core WebApi Controller, I'm receiving a IFormFile[] files. I need to convert this to of List<DocumentData>. I first used foreach. It was working fine. But later decided to change to Parallel.ForEach as I'm receiving many(>5) files.
Here is my DocumentData Class:
public class DocumentData
{
public byte[] BinaryData { get; set; }
public string FileName { get; set; }
}
Here is my Parallel.ForEach Logic:
var documents = new ConcurrentBag<DocumentData>();
Parallel.ForEach(files, async (currentFile) =>
{
if (currentFile.Length > 0)
{
using (var ms = new MemoryStream())
{
await currentFile.CopyToAsync(ms);
documents.Add(new DocumentData
{
BinaryData = ms.ToArray(),
FileName = currentFile.FileName
});
}
}
});
For Example, even for two files as input, documents always gives one file as output. Am I missing something?
I initially had List<DocumentData>. I found that it's not thread safe and changed to ConcurrentBag<DocumentData>. But still I'm getting unexpected results. Please assist on where I'm wrong?

I guess it is because, Parallel.Foreach doesn't support async/await. It only takes Action as input and executes it for each item. And in case of async delegates it will execute them in a fire-and-forget manner.
In that case passed lambda will be considered as async void function and async void can't be awaited.
If there were overload which takes Func<Task> then it would work.
I suggest you to create Tasks with the help of Select and use Task.WhenAll for executing them at the same time.
For example:
var tasks = files.Select(async currentFile =>
{
if (currentFile.Length > 0)
{
using (var ms = new MemoryStream())
{
await currentFile.CopyToAsync(ms);
documents.Add(new DocumentData
{
BinaryData = ms.ToArray(),
FileName = currentFile.FileName
});
}
}
});
await Task.WhenAll(tasks);
Additionally you can improve that code with just returning DocumentData instance from that method, and in such case there is no need to modify documents collection. Task.WhenAll has overload which takes IEnumerable<Task<TResult> as input and produces Task of TResult array. So, the result will be so:
var tasks = files.Select(async currentFile =>
{
if (currentFile.Length > 0)
{
using (var ms = new MemoryStream())
{
await currentFile.CopyToAsync(ms);
return new DocumentData
{
BinaryData = ms.ToArray(),
FileName = currentFile.FileName
};
}
}
return null;
});
var documents = (await Task.WhenAll(tasks)).Where(d => d != null).ToArray();

You had the right idea with a concurrent collection, but misused a TPL method.
In short you need to be very careful about async lambdas, and if you are passing them to an Action or Func<Task>
Your problem is because Parallel.For / ForEach is not suited for the async and await pattern or IO bound tasks. They are suited for cpu bound workloads. Which means they essentially have Action parameters and let's the task scheduler create the tasks for you
If you want to run mutple tasks at the same time use Task.WhenAll , or a TPL Dataflow ActionBlock which can deal effectively with both CPU bound and IO bound works loads, or said more directly, they can deal with tasks which is what an async method is.
The fundimental issue is when you call an async lambda on an Action, you are essentially creating an async void method, which will run as a task unobserved. That's to say, your TPL method is just creating a bunch of tasks in parallel to run a bunch of unobserved tasks and not waiting for them.
Think of it like this, you ask a bunch of friends to go and get you some groceries, they in turn tell someone else to get your groceries, yet your friends report back to you and say thier job is done. It obviously isn't and you have no groceries.

I cannot understand how exactly does await/async work

I started to look into Task, async/await concepts is c# and I'm having big problems understanding it, well at least i don't know how to implement it. I started rewriting an older test program i had written before, but now instead of threading i want to use these new concepts. Basically the layout is as it follows:
I have a simple class where i download the HTML content of a web page.
I process that in another class where i basically just parse the page to my model. Later on i want to display that to my UI.
The problem is that my program is not responsive, it blocks the UI while I'm processing the info.
I started learning this 2 days ago, i have read a lot of stuff online, including MSDN and some blogs but yet I'm unable to figure it out. Maybe someone can provide a look as well
HtmlDOwnloadCOde:
public async Task<string> GetMangaDescriptionPage(string detailUrl)
{
WebClient client = new WebClient();
Stream data = await client.OpenReadTaskAsync(detailUrl);
StreamReader reader = new StreamReader(data);
string s = reader.ReadToEnd();
data.Dispose();
reader.Dispose();
data.Close();
reader.Close();
return s;
}
My parse class code:
public async Task<MangaDetailsModel> ParseMangaDescriptionPage()
{
ParseOneManga pom = new ParseOneManga();
string t1 = await pom.GetMangaDescriptionPage(selectedManga.url);
HtmlDocument htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(t1);
var divs = htmlDoc.DocumentNode.Descendants("div").Where(x => x.Attributes.Contains("id") &&
x.Attributes["id"].Value.Contains("title")).ToArray();
mangaDetails.mangaName = divs[0].Element("h1").InnerText;
mangaDetails.description = divs[0].Descendants("p").Single().InnerText ?? "DSA";
var tds = divs[0].Descendants("td");
int info = 0;
var chapters = htmlDoc.DocumentNode.Descendants("div").Where(x => x.Attributes.Contains("id") &&
x.Attributes["id"].Value.Contains("chapters")).ToArray();
var chapterUi = chapters[0].Descendants("ul").Where(x => x.Attributes.Contains("class") &&
x.Attributes["class"].Value.Contains("chlist"));
foreach (var li in chapterUi)
{
var liChapter = li.Descendants("li");
foreach (var h3tag in liChapter)
{
var chapterH3 = h3tag.Descendants("a").ToArray();
SingleManagFox chapterData = new SingleManagFox();
chapterData.name = chapterH3[1].InnerHtml;
chapterData.url = chapterH3[1].GetAttributeValue("href", "0");
mangaDetails.chapters.Add(chapterData);
}
};
return mangaDetails;
}
UI code:
private async void mainBtn_Click(object sender, RoutedEventArgs e)
{
if (mangaList.SelectedItem != null)
{
test12((SingleManagFox)mangaList.SelectedItem);
}
}
private async void test12(SingleManagFox selectedManga)
{
selectedManga = (SingleManagFox)mangaList.SelectedItem;
MangaDetails mangaDetails = new MangaDetails(selectedManga);
MangaDetailsModel mdm = await mangaDetails.ParseMangaDescriptionPage();
txtMangaArtist.Text = mdm.artisName;
txtMangaAuthor.Text = mdm.authorName;
chapterList.ItemsSource = mdm.chapters;
}
Sorry if its trivial but i cannot figure it out myself.

When going async you need to try to go async all the way and avoid mixing blocking calls with async calls.
You are using async void in the event handler with no await.
Try to avoid async void unless it is an event handler. test12 should be updated to return Task and awaited in the event handler mainBtn_Click.
private async void mainBtn_Click(object sender, RoutedEventArgs e) {
if (mangaList.SelectedItem != null) {
await test12((SingleManagFox)mangaList.SelectedItem);
}
}
private async Task test12(SingleManagFox selectedManga) {
selectedManga = (SingleManagFox)mangaList.SelectedItem;
MangaDetails mangaDetails = new MangaDetails(selectedManga);
MangaDetailsModel mdm = await mangaDetails.ParseMangaDescriptionPage();
txtMangaArtist.Text = mdm.artisName;
txtMangaAuthor.Text = mdm.authorName;
chapterList.ItemsSource = mdm.chapters;
}
Also consider updating the web call to use HttpClient if available.
class ParseOneManga {
public async Task<string> GetMangaDescriptionPageAsync(string detailUrl) {
using (var client = new HttpClient()) {
string s = await client.GetStringAsync(detailUrl);
return s;
}
}
}
Reference: - Async/Await - Best Practices in Asynchronous Programming

Quite often people think that async-await means that multiple threads are processing your code at the same time. This is not the case, unless you explicitly start a different thread.
A good metaphore that helped me a lot explaining async-await is the restauran metaphor used in this interview with Eric Lippert. Search somewhere in the middle for async-await.
Eric Lipperts compares async-await processing with a cook who has to wait for his water to boil. Instead of waiting, he looks around if he can do other things instead. When finished doing the other thing, he comes back to see if the water is boiling and starts processing the boiling water.
The same is with your process. There is only one thread busy (at a time). This thread keeps processing until he has to await for something. This something is usually a fairly long process that is processed without using your CPU core, like writing a file to disk, loading a web page, or querying information from an external database.
Your thread can only do one thing at a time. So while it is busy calculating something, if can't react on operator input and your UI freezes, until the calculations are done. Async await will only help if there are a lot of times your thread would be waiting for other processes to complete
If you call an async function, you are certain that somewhere in that function is an await. In fact, if you declare your function async, and your forget to await in it, your compiler will warn you.
When your call meets the await in the function, your thread goes up its call stack to see if it can do other things. If you are not awaiting, you can continue processing, until you have to await. The thread goes up its call stack again to see if one of the callers is not awaiting etc.
async Task ReadDataAsync()
{
// do some preparations
using (TextReader textReader = ...)
{
var myReadTask = textReader.ReadToEndAsync();
// while the textReader is waiting for the information to be available
// you can do other things
ProcessSomething();
// after a while you really need the results from the read data,
// so you await for it.
string text = await MyReadTask;
// after the await, the results from ReatToEnd are available
Process(text);
...
There are some rules to follow:
an async function should return Task instead of void and Task<TResult> instead of TResult
There is one exception: the async event handler returns void instead of Task.
Inside your async function you should await somehow. If you don't await, it is useless to declare your function async
The result of await Task is void, and the result of await Task<TResult> is TResult
If you call an async function, see if you can do some processing instead of waiting for the results of the call
Note that even if you call several async functions before awaiting for them, does not mean that several threads are running these functions synchronously. The statement after your first call to the async function is processed after the called function starts awaiting.
async Task DoSomethingAsync()
{
var task1 = ReadAsync(...);
// no await, so next statement processes as soon as ReadAsync starts awaiting
DoSomeThingElse();
var task2 = QueryAsync(...);
// again no await
// now I need results from bothtask1, or from task2:
await Task.WhenAll(new Task[] {task1, task2});
var result1 = Task1.Result;
var result2 = Task2.Result;
Process(result1, result2);
...
Usually all your async functionality is performed by the same context. In practice this means that you can program as if your program is single threaded. This makes the look of your program much easier.
Another article that helped me a lot understanding async-await is Async-Await best practices written by the ever so helpful Stephen Cleary

Async Task in WPF

I tried to run a task for a WPF UI Code Behind . but it doesn't work and noting happend when I use Task.Factory but it works when I use the code without Task.Factory
public MainWindow()
{
InitializeComponent();
GetNews();
}
private void GetNews()
{
Task.Factory.StartNew(() =>
{
FeedReader reader = new FeedReader();
var news = reader.RetrieveFeed("http://www.bbc.com/feed/");
foreach (var item in news)
{
textBlock.Text = item.Title;
}
});
}
How can I use async/await or anything that prevent block main thread?

Nothing happens because an exception is thrown as you are trying to modify UI element outside of UI thread. You should either use textBlock.Dispatcher.Invoke method or make FeedReader asynchronous and use async/await in your method, which is preferable.
So the prefered solution would be (provided RetrieveFeed is async method):
private async void GetNews()
{
FeedReader reader = new FeedReader();
var news = await reader.RetrieveFeed("http://www.bbc.com/feed/");
foreach (var item in news)
{
textBlock.Text = item.Title;
}
}
Alternatively you could wrap feed retrieval in a task, which will work with your current implementation:
private async void GetNews()
{
FeedReader reader = new FeedReader();
var news = await Task.Run(() => reader.RetrieveFeed("http://www.bbc.com/feed/"));
foreach (var item in news)
{
textBlock.Text = item.Title;
}
}
This approach will work because await will capture SynchronizationContext of UI thread and will set value to the textbox in UI thread.
And the caveat here is that exceptions thrown in async void method are never observed and are lost. So you should wrap the code inside GetNews method in try/catch block and at least log the exception so you are aware of it.

You can use the async version to retrieve the feed
SyndicationFeed feed = await client.RetrieveFeedAsync(uri).AsTask(ct);
DisplayResults(feed);
Check the example from msdn.
It calls the Windows Runtime method, RetrieveFeedAsync, and applies a .NET Framework extension method, AsTask, to the returned IAsyncOperation instance. AsTask represents the instance as a Task, so that you can await it.

Async/Await - not understanding why one way is Blocking and another is not

Still trying to wrap my head around async/await. I have the following method for drag/drop loading:
private async void p_DragDrop(object sender, DragEventArgs e)
{
string[] files = (string[])e.Data.GetData(DataFormats.FileDrop);
List<string> CurvesToLoad = new List<string>();
List<string> TestsToLoad = new List<string>();
foreach (string file in files)
{
if (file.ToUpper().EndsWith(".CCC"))
CurvesToLoad.Add(file);
else if (file.ToUpper().EndsWith(".TTT"))
TestsToLoad.Add(file);
}
//SNIPPET IN BELOW SECTION
foreach (string CurvePath in CurvesToLoad)
{
Curve c = new Curve(CurvePath);
await Task.Run(() =>
{
c.load();
c.calculate();
});
AddCurveControls(c);
}
//END SNIPPET
foreach (string TestPath in TestsToLoad)
{
Test t = new Test(TestPath);
await Task.Run(() =>
{
t.load();
});
AddTestControls(t);
}
}
It is non-blocking as I expected - I am able to navigate between tabs of the TabControl as multiple items are loaded and I can see each tab pop up as it complete loading.
I then tried to convert to this:
private Task<Curve> LoadAndCalculateCurve(string path)
{
Curve c = new Curve(path);
c.load();
c.calculate();
return Task.FromResult(c);
}
And then replace the marked snippet from the first code block with:
foreach (string CurvePath in CurvesToLoad)
{
Curve c = await LoadAndCalculateCurve(CurvePath);
AddCurveControls(c);
}
And it becomes blocking - I can't navigate through tabs as it's loading, and then all of the loaded items appear at once when they are completed. Just trying to learn and understand the differences at play here - many thanks in advance.
EDIT:
Updated LoadAndCalculateCurve():
private async Task<Curve> LoadAndCalculateCurve(string path)
{
Curve c = new Curve(path);
await Task.Run(() => {
c.load();
c.calculate();
});
return c;
}

Async methods do not execute in a different thread, await does not start a thread. async merely enables the await keyword and await waits for something that already runs.
All of that code is running on the UI thread.

So basically this is what is happening to my knowledge.
In your first code you write
foreach (string CurvePath in CurvesToLoad)
{
Curve c = new Curve(CurvePath);
await Task.Run(() =>
{
c.load();
c.calculate();
});
AddCurveControls(c);
}
this does the async flow as expected because you are using the Task.Run which adds the work to the ThreadPool queue.
In your second try you don't do this and you are using the same task. Try to use the same logic of (Task.Run) in the second try and I think it will work

The implementation of your LoadAndCalculateCurve method is to synchronously create a Curve, synchronously load it and perform the calculation, and then return the result wrapped in a Task. Nothing about this is asynchronously. When you await it it will invoke the method, do all of the synchronous work to get the (already completed) task, and then add a continuation to that (already completed) task that will fire immediately.
When you instead use Task.Run you're scheduling those long running operations to take place in another thread and immediately returning a (not yet completed) Task that you can use to run code when it does eventually finish its work.

Async/Await in foreach with HTTPClient

I have a webservice that loads up some plugins (dlls) and calls their Process method. One of the plugins takes a list of members and ensures that they are all included in a MailChimp list.
Here is the code that adds the users to the MailChimp group.
private async Task AddMCUsers(List<Member> _memberList)
{
using (var http = new HttpClient())
{
var creds = Convert.ToBase64String(Encoding.ASCII.GetBytes("user:password");
http.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", creds);
string memberURI = string.Format(#"{0}lists/{1}/members", _baseURI, _memberGroupId);
var jss = new JavaScriptSerializer();
foreach (var user in _memberlist)
{
var _addStatus = "";
try
{
var content = jss.Serialize(new MCPost()
{
email_address = user.Email,
status = "subscribed",
merge_fields = new MCMergeFields()
{
FNAME = user.Firstname,
LNAME = user.Lastname
}
});
using(var result = await http.PostAsync(memberURI, new StringContent(content,Encoding.UTF8, "application/json")))
{
var resultText = await result.Content.ReadAsStringAsync();
if(result.IsSuccessStatusCode)
{
_addStatus = "Success";
var _returnedUser = jss.Deserialize<MCMember>(resultText);
//Store new user's id
user.ServiceId = _returnedUser.id;
}
else
{
_addStatus = "Fail";
}
}
}
catch {
_addStatus = "Error";
}
LogEvent("Add User - " + _addStatus, string.Format("Id: {0} - {1} {2} (Account: {3}) : {4}", user.Id, user.Firstname, user.Lastname, user.AccountId, user.Email));
}
}
}
In normal procedural code, this wouldn't be a problem. However, the only Post method available on the httpClient was PostAsync. Being fairly new to the async/await stuff, I'm not sure the ramifications on the rest of my code ... particularly as it relates to my attempt to reuse the httpClient instead of instantiating a new one for each http call.
I'm not sure what happens with await when its wrapped in a foreach like I have. Will I run into issues with reusing the httpClient to make repeated calls when running asynchronously?
My other question is, what is actually going to be returned. IOW, my understanding is that await returns a Task. However, here, I'm looping through the list and making multiple calls to await PostAsync. My method returns a Task. But which task gets returned? If my calling method needs to wait for completion before moving on, what does its call look like?
private void Process()
{
//Get List
var task = AddMCUsers(list);
task.Wait();
//Subsequent processing
}
I've read that you should use Async all the way. Does this mean my calling method should look more like this?
public async Task Process()
{
//Get list
...
await AddMCUsers(list);
//Other processing
}
Thanks to whatever help you can offer on this.

In normal procedural code, this wouldn't be a problem.
The whole point of async/await is to write asynchronous code in a way that looks practically identical to "normal" synchronous code.
Being fairly new to the async/await stuff, I'm not sure the ramifications on the rest of my code ... particularly as it relates to my attempt to reuse the httpClient instead of instantiating a new one for each http call.
HttpClient was intended to be reused; in fact, it can be used for any number of calls simultaneously.
I'm not sure what happens with await when its wrapped in a foreach like I have.
One way to think of it is that await "pauses" the method until its operation completes. When the operation completes, then the method continues executing. I have an async intro that goes into more detail.
Will I run into issues with reusing the httpClient to make repeated calls when running asynchronously?
No, that's fine.
IOW, my understanding is that await returns a Task.
await takes a Task. It "unwraps" that task and returns the result of the task (if any). If the task completed with an exception, then await raises that exception.
My method returns a Task. But which task gets returned?
The Task returned from an async method is created by the async state machine. You don't have to worry about it. See my intro for more details.
If my calling method needs to wait for completion before moving on, what does its call look like? ... I've read that you should use Async all the way. Does this mean my calling method should look more like this?
Yes, it should look like your second snippet:
public async Task ProcessAsync()
{
//Get list
...
await AddMCUsers(list);
//Other processing
}
The only thing I changed was the Async suffix, which is recommended by the Task-based Asynchronous Pattern.

in your code you should be fine with reusing the HttpClient. What async / await is allow the code to release the execution thread to prevent locking a cpu thread while waiting for the web response. It also releases control back to the caller. When releasing code back to the caller it means that if your Process function does not await your AddMCUsers, Process could finish before AddMCUsers (useful in fire and forget situations to not await a method).
What async/await do not do is affect the logical flow of an individual method. When you await an async web call the execution is paused and then resumed at the same point once the web call returns. There is also thread context tracking and the code resumes in the same context (ie. UI thread or background thread depending on the parent) by default, but this can be changed if needed.
At some point in your code you may want to have a method that blocks until your async code competes and that is where you will want your Task.Wait() call to block execution. If all you use is awaits then it is possible for your program to end before your task competes. See the code example below.
class Program
{
static void Main(string[] args)
{
Task waitForMe = Task.Run(() => waitAsync());
}
static async Task waitAsync()
{
await Task.Delay(5000);
}
}
in the sample with out a Task.Wait call to block the Main method the program will end before the 5 second wait is complete. Having a main method of the following will cause the program to wait for 5 seconds before exiting:
static void Main(string[] args)
{
Task waitForMe = Task.Run(() => waitAsync());
waitForMe.Wait();
}