I have the following code:
Task.Factory.ContinueWhenAll(items.Select(p =>
{
return CreateItem(p);
}).ToArray(), completedTasks => { Console.WriteLine("completed"); });
Is it possible to convert ContinueWhenAll to a synchronous method? I want to switch back between async and sync.
Edit: I should metnion that each of the "tasks" in the continuewhenall method should be executing synchronously.
If you want to leave your existing code intact and have a variable option of executing synchronously you should make these changes:
bool isAsync = false; // some flag to check for async operation
var batch = Task.Factory.ContinueWhenAll(items.Select(p =>
{
return CreateItem(p);
}).ToArray(), completedTasks => { Console.WriteLine("completed"); });
if (!isAsync)
batch.Wait();
This way you can toggle it programmatically instead of by editing your source code. And you can keep the continuation code the same for both methods.
Edit:
Here is a simple pattern for having the same method represented as a synchronous and async version:
public Item CreateItem(string name)
{
return new Item(name);
}
public Task<Item> CreateItemAsync(string name)
{
return Task.Factory.StartNew(() => CreateItem(name));
}
Unless am mistaken this is what you're looking for
Task.WaitAll(tasks);
//continuation code here
i think you can try this.
using TaskContinuationOptions for a simple scenario.
var taskFactory = new TaskFactory(TaskScheduler.Defau
var random = new Random();
var tasks = Enumerable.Range(1, 30).Select(p => {
return taskFactory.StartNew(() => {
var timeout = random.Next(5, p * 50);
Thread.Sleep(timeout / 2);
Console.WriteLine(#" 1: ID = " + p);
return p;
}).ContinueWith(t => {
Console.WriteLine(#"* 2: ID = " + t.Result);
}, TaskContinuationOptions.ExecuteSynchronously);
}).ToArray();
Task.WaitAll(tasks);
or using TPL Dataflow for a complex scenario.
var step2 = new ActionBlock<int>(i => {
Thread.Sleep(i);
Console.WriteLine(#"* 2: ID = " + i);
}, new ExecutionDataflowBlockOptions {
MaxDegreeOfParallelism = 1,
//MaxMessagesPerTask = 1
});
var random = new Random();
var tasks = Enumerable.Range(1, 50).Select(p => {
return Task.Factory.StartNew(() => {
var timeout = random.Next(5, p * 50);
Thread.Sleep(timeout / 2);
Console.WriteLine(#" 1: ID = " + p);
return p;
}).ContinueWith(t => {
Thread.Sleep(t.Result);
step2.Post(t.Result);
});
}).ToArray();
await Task.WhenAll(tasks).ContinueWith(t => step2.Complete());
await step2.Completion;
Related
I have a set of simple blocks which are mostly processed in a serial manner but I have two blocks which I want to process in parallel (processblock1 & processblock2). I just started playing around with TPL datablocks so new to it.
However in the code below, I can see paraellelblock1 is being called as but never parallelblock2 as expected. I was hoping they would both be kicked off in parallel.
class Program
{
static void Main(string[] args)
{
var readBlock = new TransformBlock<int, int>(x => DoSomething(x, "readBlock"),
new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 1 }); //1
var processBlock1 =
new TransformBlock<int, int>(x => DoSomething(x, "processBlock1")); //2
var processBlock2 =
new TransformBlock<int, int>(x => DoSomething(x, "processBlock2")); //3
var saveBlock =
new ActionBlock<int>(
x => Save(x)); //4
readBlock.LinkTo(processBlock1,
new DataflowLinkOptions { PropagateCompletion = true }); //5
readBlock.LinkTo(processBlock2,
new DataflowLinkOptions { PropagateCompletion = true }); //6
processBlock1.LinkTo(
saveBlock); //7
processBlock2.LinkTo(
saveBlock); //8
readBlock.Post(1); //10
Task.WhenAll(
processBlock1.Completion,
processBlock2.Completion)
.ContinueWith(_ => saveBlock.Complete()); //11
readBlock.Complete(); //12
saveBlock.Completion.Wait(); //13
Console.WriteLine("Processing complete!");
Console.ReadLine();
}
private static int DoSomething(int i, string method)
{
Console.WriteLine($"Do Something, callng method : { method}");
return i;
}
private static async Task<int> DoSomethingAsync(int i, string method)
{
DoSomething(i, method);
return i;
}
private static void Save(int i)
{
Console.WriteLine("Save!");
}
}
By default tpl block will only send a message to the first linked block.
Use a BroadcastBlock to send a message to many components.
void Main()
{
var random = new Random();
var readBlock = new TransformBlock<int, int>(x => { return DoSomething(x, "readBlock"); },
new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 1 }); //1
var braodcastBlock = new BroadcastBlock<int>(i => i); // ⬅️ Here
var processBlock1 =
new TransformBlock<int, int>(x => DoSomething(x, "processBlock1")); //2
var processBlock2 =
new TransformBlock<int, int>(x => DoSomething(x, "processBlock2")); //3
var saveBlock =
new ActionBlock<int>(
x => Save(x)); //4
readBlock.LinkTo(braodcastBlock, new DataflowLinkOptions { PropagateCompletion = true });
braodcastBlock.LinkTo(processBlock1,
new DataflowLinkOptions { PropagateCompletion = true }); //5
braodcastBlock.LinkTo(processBlock2,
new DataflowLinkOptions { PropagateCompletion = true }); //6
processBlock1.LinkTo(
saveBlock); //7
processBlock2.LinkTo(
saveBlock); //8
readBlock.Post(1); //10
readBlock.Post(2); //10
Task.WhenAll(
processBlock1.Completion,
processBlock2.Completion)
.ContinueWith(_ => saveBlock.Complete());
readBlock.Complete(); //12
saveBlock.Completion.Wait(); //13
Console.WriteLine("Processing complete!");
}
// Define other methods and classes here
private static int DoSomething(int i, string method)
{
Console.WriteLine($"Do Something, callng method : { method} {i}");
return i;
}
private static Task<int> DoSomethingAsync(int i, string method)
{
DoSomething(i, method);
return Task.FromResult(i);
}
private static void Save(int i)
{
Console.WriteLine("Save! " + i);
}
It appears that you're posting only one item to the graph, and the first consumer to consume it wins. There's no implied 'tee' functionality in the graph you've made--so there's no possible parallelism there.
I am having Parallel For loop for following statements but I want to use async task not Parallel. Any idea how can i use async task on same statements? I don't need fully working code but just an idea about how to replace async task with Parallel. Happy coding
Parallel.For(0, allRequests.Count(), i =>
{
var rand = new Random();
var token = allTokens.ElementAt(rand.Next(allTokens.Count()));
var accessKey = token.AccessKey;
var secretKey = token.SecretKey;
using (var ctx = new db_mytestdb())
{
var firstRequest = allRequests[i];
Console.WriteLine("Started scan for: " + firstRequest.SearchedUser.EbayUsername + " and using token: " + allTokens[i % allTokens.Count].TokenName);
var bulkScannedItems = new ConcurrentBag<BulkScannedItems>();
var userPreferences = ctx.UserPreferences.FirstOrDefault(x => x.UserId == firstRequest.UserId);
var userBrekEven = userPreferences.BreakEven;
var intoPast = DateTime.Now.Subtract(TimeSpan.FromDays(firstRequest.Range));
var filteredProducts = ctx.EbayUserTransactions.Where(x => x.SearchedUserID == firstRequest.SearchedUserID && x.TransactionDate >= intoPast && x.TransactionDate <= firstRequest.SearchedUser.LastUpdatedAt)
.ToList()
.GroupBy(x => x.ItemID).Select(x => new ResultItem()
{
ItemID = x.Key,
SaleNumber = x.Sum(y => y.QuantityPurchased)
})
.Where(x => x.SaleNumber >= firstRequest.MinSales)
.ToList();
var itemSpecifics = ctx.SearchedUserItems.Where(x => x.SearchedUserID == firstRequest.SearchedUserID).ToList();
foreach (var item in itemSpecifics)
{
foreach (var filtered in filteredProducts)
{
if (item.ItemID == filtered.ItemID)
{
if (item.UPC != null)
{
filtered.UPC = item.UPC;
}
else
{
filtered.UPC = "does not apply";
}
if (item.EAN != null)
{
filtered.EAN = item.EAN;
}
else
{
filtered.EAN = "does not apply";
}
if (item.MPN != null)
{
filtered.MPN = item.MPN;
}
else
{
filtered.MPN = "does not apply";
}
}
}
}
var bulkScanner = new BulkScannerAlgorithm();
foreach (var dbItem in filteredProducts)
{
var amazonItem = bulkScanner.Found(dbItem.UPC, dbItem.ItemID, accessKey, secretKey);
if (amazonItem.Found)
{
bulkScanner.InsertAmazonData(firstRequest, bulkScannedItems, userBrekEven, amazonItem);
continue;
}
amazonItem = bulkScanner.Found(dbItem.EAN, dbItem.ItemID, accessKey, secretKey);
if (amazonItem.Found)
{
bulkScanner.InsertAmazonData(firstRequest, bulkScannedItems, userBrekEven, amazonItem);
continue;
}
amazonItem = bulkScanner.Found(dbItem.MPN, dbItem.ItemID, accessKey, secretKey);
if (amazonItem.Found)
{
bulkScanner.InsertAmazonData(firstRequest, bulkScannedItems, userBrekEven, amazonItem);
continue;
}
}
List<BulkScannedItems> filteredCompleteBulk;
if (firstRequest.IsPrimeOnly == true)
{
filteredCompleteBulk = bulkScannedItems.Where(x => x.CalculatedProfit >= firstRequest.MinProfit && x.IsPrime == true && x.EbayPrice >= firstRequest.minPrice && x.EbayPrice <= firstRequest.maxPrice).DistinctBy(x => x.ASIN).ToList();
}
else
{
filteredCompleteBulk = bulkScannedItems.Where(x => x.CalculatedProfit >= firstRequest.MinProfit && x.EbayPrice >= firstRequest.minPrice && x.EbayPrice <= firstRequest.maxPrice).DistinctBy(x => x.ASIN).ToList();
}
EFBatchOperation.For(ctx, ctx.BulkScannedItems).InsertAll(filteredCompleteBulk);
ctx.user_scanReq_update(firstRequest.UserSellerScanRequestId);
Console.WriteLine("Scan complete for user: " + firstRequest.SearchedUser.EbayUsername);
}
});
Parallelism and asynchrony are both forms of concurrency, but parallelism works by dividing the problem among multiple threads, and asynchrony works by freeing up threads. So they're kind of opposites in how they work.
That said, to make the code asynchronous, you'd start from your lowest-level I/O calls, e.g., the EF ToList and presumably also whatever APIs are used in the implementation of InsertAll. Replace those with asynchronous equivalents (e.g., ToListAsync) and call them with await.
Next, you'd need to replace the Parallel.For loop with code that creates a collection of asynchronous tasks and then (asynchronously) waits for them all, something like:
var tasks = allRequests.Select(async request => { ... });
await Task.WhenAll(tasks);
That's the basic pattern for asynchronous concurrency.
If you find that you do need true parallelism (multiple threads) in addition to asynchrony, consider using TPL Dataflow.
A little clarification, whether you are using Parallel.For/Foreach, Tasks, or await/async, they are all using the same thing behind the scenes (albeit slightly differently). You should always pick the one that fits your problem the best.
If you want to replace the parallel.for with a method of return type Task, that is straight forward enough but you would end up waiting for this piece to be done before you continued your processing.
Async/Await is generally used when dealing with UIs and web calls, it doesn't appear to be useful here.
What is it that you are trying to accomplish? Why the need to 'replace async task with Parallel'?
The general way you would off load a method to a task would be
Task<T> task = Task<T>.Factory.StartNew(() =>
{
});
or
public Task<T> task()
{
.....
}
In the following code I expected the result to be 3
Task<int> parent = Task.Factory.StartNew(() =>
{
var sum = 0;
TaskFactory tf = new TaskFactory(TaskCreationOptions.AttachedToParent,
TaskContinuationOptions.ExecuteSynchronously);
tf.StartNew(() => sum++);
tf.StartNew(() => sum++);
tf.StartNew(() => sum++);
return sum;
});
var finalTask = parent.ContinueWith(parentTask => Console.WriteLine(parentTask.Result));
finalTask.Wait();
However the result is 0 which I don't understand. The odd thing is when I change it to use and Array it does seem to be doing the right thing.
Task<Int32[]> parent = Task.Factory.StartNew(() =>
{
var results = new Int32[3];
TaskFactory tf = new TaskFactory(TaskCreationOptions.AttachedToParent,
TaskContinuationOptions.ExecuteSynchronously);
tf.StartNew(() => results[0] = 0);
tf.StartNew(() => results[1] = 1);
tf.StartNew(() => results[2] = 2);
return results;
});
var finalTask = parent.ContinueWith(
parentTask =>
{
foreach (int i in parentTask.Result)
Console.WriteLine(i);
});
finalTask.Wait();
Here the result is as expected:
0
1
2
I guess I am missing something very obvious, what do I need to fix in the first piece of code to have it return 3
Update
I already had a look at this Solution which is why I haven't used Task.Run but it didn't really make a difference
Difference between cases caused by difference in value-type versus reference-type semantic. int is a value type, so it copied on return, and any subsequent changes to sum variable are not seen. Arrays are reference type, so only reference copied on return, so any changes in array made by child tasks will be visible, as it is the same array. To make your first case work, you need to replace int by some reference type:
public class Reference<T> {
public T Value;
public Reference(T value) {
Value=value;
}
}
public static void Test() {
Task<Reference<int>> parent=Task.Factory.StartNew(() => {
var sum=new Reference<int>(0);
TaskFactory tf=new TaskFactory(TaskCreationOptions.AttachedToParent,
TaskContinuationOptions.ExecuteSynchronously);
tf.StartNew(() => sum.Value++);
tf.StartNew(() => sum.Value++);
tf.StartNew(() => sum.Value++);
return sum;
});
var finalTask=parent.ContinueWith(parentTask => Console.WriteLine(parentTask.Result.Value));
finalTask.Wait();
}
I am writing some complex tasks and I can't do success to solve the problem.
I open some tasks for searching some data asynchronously.
At the end I wait for all tasks.
I want to run on all the data and does not stop the tasks until they all finish.
But, when one of the tasks finds some relevant data, I want to continue with the code that is after the wait all tasks, but at the same time, I want to continue searching with my tasks (until all the data will be read).
Here I found a way how to continue the code after the wait all if one of the tasks finds the data, the problem is that it also stops all tasks from running.
Here is my code (this is just code for demonstration of the problem).
private static void RunTasks()
{
TaskCompletionSource<bool> tcs = new TaskCompletionSource<bool>();
Task<bool> t0 = Task.Factory.StartNew<bool>(() => Find(1, 2));
Task<bool> t1 = Task.Factory.StartNew<bool>(() => Find(4, 7));
Task<bool> t2 = Task.Factory.StartNew<bool>(() => Find(13, 14));
t0.ContinueWith(_ =>
{
if (t0.Result)
tcs.TrySetResult(t0.Result);
});
t1.ContinueWith(_ =>
{
if (t1.Result)
tcs.TrySetResult(t1.Result);
});
t2.ContinueWith(_ =>
{
if (t2.Result)
tcs.TrySetResult(t2.Result);
});
tcs.Task.Wait();
Console.WriteLine("Found");
ContinueWork(); //Runs after at least one data found or when all tasks finish.
}
//Just for demonstration...
private static bool Find(int a, int b)
{
Console.WriteLine("a: " + a + " b: " + b);
return a == 4 && b == 7 ? true : false;
}
How can I write it so that when some data is found it will continue to ContinueWork method and also will continue with reading the data with the tasks?
Thanks.
This is not the cleanest way but it will serve your purposes:
TaskCompletionSource<bool> tcs = new TaskCompletionSource<bool>();
Task<bool>[] tasks = new Task<bool>[3];
tasks[0] = Task.Factory.StartNew<bool>(() => Find(1, 2));
tasks[1] = Task.Factory.StartNew<bool>(() => Find(4, 7));
tasks[2] = Task.Factory.StartNew<bool>(() => Find(13, 14));
tasks[0].ContinueWith(_ =>
{
if (tasks[0].Result)
tcs.TrySetResult(tasks[0].Result);
});
tasks[1].ContinueWith(_ =>
{
if (tasks[1].Result)
tcs.TrySetResult(tasks[1].Result);
});
tasks[2].ContinueWith(_ =>
{
if (tasks[2].Result)
tcs.TrySetResult(tasks[2].Result);
});
Task.WaitAny(tasks);
Console.WriteLine("Found");
ContinueWork();
Below I have replicated a real life scenario as a LINQPad script for the sake of simplicity:
var total = 1 * 1000 * 1000;
var cts = new CancellationTokenSource();
var threads = Environment.ProcessorCount;
int capacity = 10;
var edbOptions = new ExecutionDataflowBlockOptions{BoundedCapacity = capacity, CancellationToken = cts.Token, MaxDegreeOfParallelism = threads};
var dbOptions = new DataflowBlockOptions {BoundedCapacity = capacity, CancellationToken = cts.Token};
var gdbOptions = new GroupingDataflowBlockOptions {BoundedCapacity = capacity, CancellationToken = cts.Token};
var dlOptions = new DataflowLinkOptions {PropagateCompletion = true};
var counter1 = 0;
var counter2 = 0;
var delay1 = 10;
var delay2 = 25;
var action1 = new Func<IEnumerable<string>, Task>(async x => {await Task.Delay(delay1); Interlocked.Increment(ref counter1);});
var action2 = new Func<IEnumerable<string>, Task>(async x => {await Task.Delay(delay2); Interlocked.Increment(ref counter2);});
var actionBlock1 = new ActionBlock<IEnumerable<string>>(action1, edbOptions);
var actionBlock2 = new ActionBlock<IEnumerable<string>>(action2, edbOptions);
var batchBlock1 = new BatchBlock<string>(5, gdbOptions);
var batchBlock2 = new BatchBlock<string>(5, gdbOptions);
batchBlock1.LinkTo(actionBlock1, dlOptions);
batchBlock2.LinkTo(actionBlock2, dlOptions);
var bufferBlock1 = new BufferBlock<string>(dbOptions);
var bufferBlock2 = new BufferBlock<string>(dbOptions);
bufferBlock1.LinkTo(batchBlock1, dlOptions);
bufferBlock2.LinkTo(batchBlock2, dlOptions);
var bcBlock = new BroadcastBlock<string>(x => x, dbOptions);
bcBlock.LinkTo(bufferBlock1, dlOptions);
bcBlock.LinkTo(bufferBlock2, dlOptions);
var mainBlock = new TransformBlock<int, string>(x => x.ToString(), edbOptions);
mainBlock.LinkTo(bcBlock, dlOptions);
mainBlock.Dump("Main Block");
bcBlock.Dump("Broadcast Block");
bufferBlock1.Dump("Buffer Block 1");
bufferBlock2.Dump("Buffer Block 2");
actionBlock1.Dump("Action Block 1");
actionBlock2.Dump("Action Block 2");
foreach(var i in Enumerable.Range(1, total))
await mainBlock.SendAsync(i, cts.Token);
mainBlock.Complete();
await Task.WhenAll(actionBlock1.Completion, actionBlock2.Completion);
counter1.Dump("Counter 1");
counter2.Dump("Counter 2");
I have two issues with this code:
Although I limited BoundedCapacity of all appropriate blocks to 10 elements, it seems like I can push all 1,000,000 messages almost at once. Is this expected behavior?
Although the entire network is configured to propagate completion, it seems like all blocks get completed almost immediately after calling mainBlock.Complete(). I expect that both counter1 and counter2 variables to be equal to total. Is there a way to achieve such behavior?
Yes, this is the expected behavior, because of the BroadcastBlock:
Provides a buffer for storing at most one element at time, overwriting each message with the next as it arrives.
This means that if you link BroadcastBlock to blocks with BoundedCapacity, you will lose messages.
To fix that, you could create a custom block that behaves like BroadcastBlock, but guarantees delivery to all targets. But doing that is not trivial, so you might be satisified with a simpler variant (originally from my old answer):
public static ITargetBlock<T> CreateGuaranteedBroadcastBlock<T>(
IEnumerable<ITargetBlock<T>> targets, DataflowBlockOptions options)
{
var targetsList = targets.ToList();
var block = new ActionBlock<T>(
async item =>
{
foreach (var target in targetsList)
{
await target.SendAsync(item);
}
}, new ExecutionDataflowBlockOptions
{
BoundedCapacity = options.BoundedCapacity,
CancellationToken = options.CancellationToken
});
block.Completion.ContinueWith(task =>
{
foreach (var target in targetsList)
{
if (task.Exception != null)
target.Fault(task.Exception);
else
target.Complete();
}
});
return block;
}
Usage in your case would be:
var bcBlock = CreateGuaranteedBroadcastBlock(
new[] { bufferBlock1, bufferBlock2 }, dbOptions);