Develop Reference

How to close all tasks if at least one is over

There is a decimal parameter. Suppose it is equal to 100. There is a Task that reduces it by 0.1 every 100ms. As soon as the parameter becomes equal to 1, the task should end and the parameter should not decrease any more. Works without problems if there is only one Task. But if there are 2, 3, 100... then the parameter will eventually become less than 1. I try to use CancellationToken to end all tasks, but the result is still the same. My code:
class Program
{
static decimal param = 100;
static CancellationTokenSource cancelTokenSource = new CancellationTokenSource();
static CancellationToken token;
static void Main(string[] args)
{
int tasksCount = 16;
token = cancelTokenSource.Token;
Console.WriteLine("Start Param = {0}", param);
Console.WriteLine("Tasks Count = {0}", tasksCount);
var tasksList = new List<Task>();
for (var i = 0; i < tasksCount; i++)
{
Task task = new Task(Decrementation, token);
tasksList.Add(task);
}
tasksList.ForEach(x => x.Start());
Task.WaitAny(tasksList.ToArray());
Console.WriteLine("Result = {0}", param);
Console.Read();
}
private static void Decrementation()
{
while (true)
{
if (token.IsCancellationRequested)
{
break;
}
if (CanTakeMore())
{
Task.Delay(100);
param = param - 0.1m;
}
else
{
cancelTokenSource.Cancel();
return;
}
}
}
private static bool CanTakeMore()
{
if (param > 1)
{
return true;
}
else
{
return false;
}
}
}
The output is different, but it is always less than 1. How to fix ?

Your tasks are checking and modifying the same shared value in parallel, to some degree as allowed by your CPU architecture and/or Operating System.
Any number of your tasks can encounter a CanTakeMore() result of true "at the same time" (when they call it with the shared value being 1.1m), and then all of them that received true from that call will proceed to decrease the shared value.
This problem can usually be avoided by using a lock statement:
private static object _lockObj = new object();
private static void Decrementation()
{
while (true)
{
if (token.IsCancellationRequested)
{
break;
}
lock (_lockObj)
{
if (CanTakeMore())
{
Task.Delay(100); // Note: this needs an `await`, but the method we're in is NOT `async`...!
param = param - 0.1m;
}
else
{
cancelTokenSource.Cancel();
return;
}
}
}
}
Here is a working .NET Fiddle: https://dotnetfiddle.net/BA6tHX

While your code has other issues (such as the fact that you must await Task.Delay), the fundamental problem is that you must either lock your entire read/write operation, or modify your implementation to enable atomic read and writes.
One option is to take your incoming decimal and convert it to a 32 bit integer, multiplying the param by the number of places of precision you need. In this case, it would be 100 * 10 since you have 1 place of precision.
This enables you to use Thread.VolatileRead in conjunction with Interlocked.CompareExchange to produce the behavior you are looking for (working example).
void Main()
{
int tasksCount = 16;
token = cancelTokenSource.Token;
Console.WriteLine("Start Param = {0}", param);
Console.WriteLine("Tasks Count = {0}", tasksCount);
var tasksList = new List<Task>();
for (var i = 0; i < tasksCount; i++)
{
Task task = new Task(Decrementation, token);
tasksList.Add(task);
}
tasksList.ForEach(x => x.Start());
Task.WaitAny(tasksList.ToArray());
Console.WriteLine("Result = {0}", param);
Console.Read();
}
static int param = 1000;
static CancellationTokenSource cancelTokenSource = new CancellationTokenSource();
static CancellationToken token;
private static void Decrementation()
{
while (true)
{
if (token.IsCancellationRequested)
{
break;
}
int temp = Thread.VolatileRead(ref param);
if (temp == 1)
{
cancelTokenSource.Cancel();
return;
}
int updatedValue = temp - 1;
if (Interlocked.CompareExchange(ref param, updatedValue, temp) == temp)
{
// the update was successful. Delay (or do additional work)
// this still does nothing
// You might want to make your method async or switch to a timer
Task.Delay(100);
}
}
}
The advantage of Thread.VolatileRead + Interlocked.CompareExchange over a straight lock is that if there is any significant work being done in the lock, this approach will perform significantly better. When benchmarked against the following reasonable locking implementation using decimal subtraction:
private static object _locker = new object();
private static decimal param2 = 100.0m;
private static void DecrementationLock()
{
while (true)
{
if (token.IsCancellationRequested)
{
break;
}
lock (_locker)
{
if (param2 > 1)
{
param2 = param2 - 0.1m;
Task.Delay(100);
}
else
{
cancelTokenSource.Cancel();
return;
}
}
}
}
even though the Task.Delay is not awaited in either case, the lock code is over 2.5x slower. That said, in cases where no work is being done, there is essentially no execution time difference between the two approaches.

How to keep track of faulted items in TPL pipeline in (thread)safe way

I am using TPL pipeline design together with Stephen Cleary's Try library In short it wraps value/exception and floats it down the pipeline. So even items that have thrown exceptions inside their processing methods, at the end when I await resultsBlock.Completion; have Status=RunToCompletion. So I need other way how to register faulted items. Here is small sample:
var downloadBlock = new TransformBlock<int, Try<int>>(construct => Try.Create(() =>
{
//SomeProcessingMethod();
return 1;
}));
var processBlock = new TransformBlock<Try<int>, Try<int>>(construct => construct.Map(value =>
{
//SomeProcessingMethod();
return 1;
}));
var resultsBlock = new ActionBlock<Try<int>>(construct =>
{
if (construct.IsException)
{
var exception = construct.Exception;
switch (exception)
{
case GoogleApiException gex:
//_notificationService.NotifyUser("OMG, my dear sir, I think I messed something up:/"
//Register that this item was faulted, so we know that we need to retry it.
break;
default:
break;
}
}
});
One solution would be to create a List<int> FaultedItems; where I would insert all faulted items in my Exception handling block and then after await resultsBlock.Completion; I could check if the list is not empty and create new pipeline for faulted items. My question is if I use a List<int> am I at risk of running into problems with thread safety if I decide to play with MaxDegreeOfParallelism settings and I'd be better off using some ConcurrentCollection? Or maybe this approach is flawed in some other way?

I converted a retry-block implementation from an answer to a similar question, to work with Stephen Cleary's Try types as input and output. The method CreateRetryTransformBlock returns a TransformBlock<Try<TInput>, Try<TOutput>>, and the method CreateRetryActionBlock returns something that is practically an ActionBlock<Try<TInput>>.
Three more options are available, the MaxAttemptsPerItem, MinimumRetryDelay and MaxRetriesTotal, on top of the standard execution options.
public class RetryExecutionDataflowBlockOptions : ExecutionDataflowBlockOptions
{
/// <summary>The limit after which an item is returned as failed.</summary>
public int MaxAttemptsPerItem { get; set; } = 1;
/// <summary>The minimum delay duration before retrying an item.</summary>
public TimeSpan MinimumRetryDelay { get; set; } = TimeSpan.Zero;
/// <summary>The limit after which the block transitions to a faulted
/// state (unlimited is the default).</summary>
public int MaxRetriesTotal { get; set; } = -1;
}
public class RetryLimitException : Exception
{
public RetryLimitException(string message, Exception innerException)
: base(message, innerException) { }
}
public static TransformBlock<Try<TInput>, Try<TOutput>>
CreateRetryTransformBlock<TInput, TOutput>(
Func<TInput, Task<TOutput>> transform,
RetryExecutionDataflowBlockOptions dataflowBlockOptions)
{
if (transform == null) throw new ArgumentNullException(nameof(transform));
if (dataflowBlockOptions == null)
throw new ArgumentNullException(nameof(dataflowBlockOptions));
int maxAttemptsPerItem = dataflowBlockOptions.MaxAttemptsPerItem;
int maxRetriesTotal = dataflowBlockOptions.MaxRetriesTotal;
TimeSpan retryDelay = dataflowBlockOptions.MinimumRetryDelay;
if (maxAttemptsPerItem < 1) throw new ArgumentOutOfRangeException(
nameof(dataflowBlockOptions.MaxAttemptsPerItem));
if (maxRetriesTotal < -1) throw new ArgumentOutOfRangeException(
nameof(dataflowBlockOptions.MaxRetriesTotal));
if (retryDelay < TimeSpan.Zero) throw new ArgumentOutOfRangeException(
nameof(dataflowBlockOptions.MinimumRetryDelay));
var internalCTS = CancellationTokenSource
.CreateLinkedTokenSource(dataflowBlockOptions.CancellationToken);
var maxDOP = dataflowBlockOptions.MaxDegreeOfParallelism;
var taskScheduler = dataflowBlockOptions.TaskScheduler;
var exceptionsCount = 0;
SemaphoreSlim semaphore;
if (maxDOP == DataflowBlockOptions.Unbounded)
{
semaphore = new SemaphoreSlim(Int32.MaxValue);
}
else
{
semaphore = new SemaphoreSlim(maxDOP, maxDOP);
// The degree of parallelism is controlled by the semaphore
dataflowBlockOptions.MaxDegreeOfParallelism = DataflowBlockOptions.Unbounded;
// Use a limited-concurrency scheduler for preserving the processing order
dataflowBlockOptions.TaskScheduler = new ConcurrentExclusiveSchedulerPair(
taskScheduler, maxDOP).ConcurrentScheduler;
}
var block = new TransformBlock<Try<TInput>, Try<TOutput>>(async item =>
{
// Continue on captured context after every await
if (item.IsException) return Try<TOutput>.FromException(item.Exception);
var result1 = await ProcessOnceAsync(item);
if (item.IsException || result1.IsValue) return result1;
for (int i = 2; i <= maxAttemptsPerItem; i++)
{
await Task.Delay(retryDelay, internalCTS.Token);
var result = await ProcessOnceAsync(item);
if (result.IsValue) return result;
}
return result1; // Return the first-attempt exception
}, dataflowBlockOptions);
dataflowBlockOptions.MaxDegreeOfParallelism = maxDOP; // Restore initial value
dataflowBlockOptions.TaskScheduler = taskScheduler; // Restore initial value
_ = block.Completion.ContinueWith(_ => internalCTS.Dispose(),
TaskScheduler.Default);
return block;
async Task<Try<TOutput>> ProcessOnceAsync(Try<TInput> item)
{
await semaphore.WaitAsync(internalCTS.Token);
try
{
var result = await item.Map(transform);
if (item.IsValue && result.IsException)
{
ObserveNewException(result.Exception);
}
return result;
}
finally
{
semaphore.Release();
}
}
void ObserveNewException(Exception ex)
{
if (maxRetriesTotal == -1) return;
uint newCount = (uint)Interlocked.Increment(ref exceptionsCount);
if (newCount <= (uint)maxRetriesTotal) return;
if (newCount == (uint)maxRetriesTotal + 1)
{
internalCTS.Cancel(); // The block has failed
throw new RetryLimitException($"The max retry limit " +
$"({maxRetriesTotal}) has been reached.", ex);
}
throw new OperationCanceledException();
}
}
public static ITargetBlock<Try<TInput>> CreateRetryActionBlock<TInput>(
Func<TInput, Task> action,
RetryExecutionDataflowBlockOptions dataflowBlockOptions)
{
if (action == null) throw new ArgumentNullException(nameof(action));
var block = CreateRetryTransformBlock<TInput, object>(async input =>
{
await action(input).ConfigureAwait(false); return null;
}, dataflowBlockOptions);
var nullTarget = DataflowBlock.NullTarget<Try<object>>();
block.LinkTo(nullTarget);
return block;
}
Usage example:
var downloadBlock = CreateRetryTransformBlock(async (int construct) =>
{
int result = await DownloadAsync(construct);
return result;
}, new RetryExecutionDataflowBlockOptions()
{
MaxDegreeOfParallelism = 10,
MaxAttemptsPerItem = 3,
MaxRetriesTotal = 100,
MinimumRetryDelay = TimeSpan.FromSeconds(10)
});
var processBlock = new TransformBlock<Try<int>, Try<int>>(
construct => construct.Map(async value =>
{
return await ProcessAsync(value);
}));
downloadBlock.LinkTo(processBlock,
new DataflowLinkOptions() { PropagateCompletion = true });
To keep things simple, in case that an item has been retried the maximum number of times, the exception preserved is the first one that occurred. The subsequent exceptions are lost. In most cases the lost exceptions are going to be of the same type as the first one anyway.
Caution: The above implementation does not have an efficient input queue. If you feed this block with millions of items, the memory usage will explode.

C# async within an action

I would like to write a method which accept several parameters, including an action and a retry amount and invoke it.
So I have this code:
public static IEnumerable<Task> RunWithRetries<T>(List<T> source, int threads, Func<T, Task<bool>> action, int retries, string method)
{
object lockObj = new object();
int index = 0;
return new Action(async () =>
{
while (true)
{
T item;
lock (lockObj)
{
if (index < source.Count)
{
item = source[index];
index++;
}
else
break;
}
int retry = retries;
while (retry > 0)
{
try
{
bool res = await action(item);
if (res)
retry = -1;
else
//sleep if not success..
Thread.Sleep(200);
}
catch (Exception e)
{
LoggerAgent.LogException(e, method);
}
finally
{
retry--;
}
}
}
}).RunParallel(threads);
}
RunParallel is an extention method for Action, its look like this:
public static IEnumerable<Task> RunParallel(this Action action, int amount)
{
List<Task> tasks = new List<Task>();
for (int i = 0; i < amount; i++)
{
Task task = Task.Factory.StartNew(action);
tasks.Add(task);
}
return tasks;
}
Now, the issue: The thread is just disappearing or collapsing without waiting for the action to finish.
I wrote this example code:
private static async Task ex()
{
List<int> ints = new List<int>();
for (int i = 0; i < 1000; i++)
{
ints.Add(i);
}
var tasks = RetryComponent.RunWithRetries(ints, 100, async (num) =>
{
try
{
List<string> test = await fetchSmthFromDb();
Console.WriteLine("#" + num + " " + test[0]);
return test[0] == "test";
}
catch (Exception e)
{
Console.WriteLine(e.StackTrace);
return false;
}
}, 5, "test");
await Task.WhenAll(tasks);
}
The fetchSmthFromDb is a simple Task> which fetches something from the db and works perfectly fine when invoked outside of this example.
Whenever the List<string> test = await fetchSmthFromDb(); row is invoked, the thread seems to be closing and the Console.WriteLine("#" + num + " " + test[0]); not even being triggered, also when debugging the breakpoint never hit.
The Final Working Code
private static async Task DoWithRetries(Func<Task> action, int retryCount, string method)
{
while (true)
{
try
{
await action();
break;
}
catch (Exception e)
{
LoggerAgent.LogException(e, method);
}
if (retryCount <= 0)
break;
retryCount--;
await Task.Delay(200);
};
}
public static async Task RunWithRetries<T>(List<T> source, int threads, Func<T, Task<bool>> action, int retries, string method)
{
Func<T, Task> newAction = async (item) =>
{
await DoWithRetries(async ()=>
{
await action(item);
}, retries, method);
};
await source.ParallelForEachAsync(newAction, threads);
}

The problem is in this line:
return new Action(async () => ...
You start an async operation with the async lambda, but don't return a task to await on. I.e. it runs on worker threads, but you'll never find out when it's done. And your program terminates before the async operation is complete -that's why you don't see any output.
It needs to be:
return new Func<Task>(async () => ...
UPDATE
First, you need to split responsibilities of methods, so you don't mix retry policy (which should not be hardcoded to a check of a boolean result) with running tasks in parallel.
Then, as previously mentioned, you run your while (true) loop 100 times instead of doing things in parallel.
As #MachineLearning pointed out, use Task.Delay instead of Thread.Sleep.
Overall, your solution looks like this:
using System.Collections.Async;
static async Task DoWithRetries(Func<Task> action, int retryCount, string method)
{
while (true)
{
try
{
await action();
break;
}
catch (Exception e)
{
LoggerAgent.LogException(e, method);
}
if (retryCount <= 0)
break;
retryCount--;
await Task.Delay(millisecondsDelay: 200);
};
}
static async Task Example()
{
List<int> ints = new List<int>();
for (int i = 0; i < 1000; i++)
ints.Add(i);
Func<int, Task> actionOnItem =
async item =>
{
await DoWithRetries(async () =>
{
List<string> test = await fetchSmthFromDb();
Console.WriteLine("#" + item + " " + test[0]);
if (test[0] != "test")
throw new InvalidOperationException("unexpected result"); // will be re-tried
},
retryCount: 5,
method: "test");
};
await ints.ParallelForEachAsync(actionOnItem, maxDegreeOfParalellism: 100);
}
You need to use the AsyncEnumerator NuGet Package in order to use the ParallelForEachAsync extension method from the System.Collections.Async namespace.

Besides the final complete reengineering, I think it's very important to underline what was really wrong with the original code.
0) First of all, as #Serge Semenov immediately pointed out, Action has to be replaced with
Func<Task>
But there are still other two essential changes.
1) With an async delegate as argument it is necessary to use the more recent Task.Run instead of the older pattern new TaskFactory.StartNew (or otherwise you have to add Unwrap() explicitly)
2) Moreover the ex() method can't be async since Task.WhenAll must be waited with Wait() and without await.
At that point, even though there are logical errors that need reengineering, from a pure technical standpoint it does work and the output is produced.
A test is available online: http://rextester.com/HMMI93124

Data Propagation in TPL Dataflow Pipeline with Batchblock.Triggerbatch()

In my Producer-Consumer scenario, I have multiple consumers, and each of the consumers send an action to external hardware, which may take some time. My Pipeline looks somewhat like this:
BatchBlock --> TransformBlock --> BufferBlock --> (Several) ActionBlocks
I have assigned BoundedCapacity of my ActionBlocks to 1.
What I want in theory is, I want to trigger the Batchblock to send a group of items to the Transformblock only when one of my Actionblocks are available for operation. Till then the Batchblock should just keep buffering elements and not pass them on to the Transformblock. My batch-sizes are variable. As Batchsize is mandatory, I do have a really high upper-limit for BatchBlock batch size, however I really don't wish to reach upto that limit, I would like to trigger my batches depending upon the availability of the Actionblocks permforming the said task.
I have achieved this with the help of the Triggerbatch() method. I am calling the Batchblock.Triggerbatch() as the last action in my ActionBlock.However interestingly after several days of working properly the pipeline has come to a hault. Upon checking I found out that sometimes the inputs to the batchblock come in after the ActionBlocks are done with their work. In this case the ActionBlocks do actually call Triggerbatch at the end of their work, however since at this point there is no input to the Batchblock at all, the call to TriggerBatch is fruitless. And after a while when inputs do flow in to the Batchblock, there is no one left to call TriggerBatch and restart the Pipeline. I was looking for something where I could just check if something is infact present in the inputbuffer of the Batchblock, however there is no such feature available, I could also not find a way to check if the TriggerBatch was fruitful.
Could anyone suggest a possible solution to my problem. Unfortunately using a Timer to triggerbatches is not an option for me. Except for the start of the Pipeline, the throttling should be governed only by the availability of one of the ActionBlocks.
The example code is here:
static BatchBlock<int> _groupReadTags;
static void Main(string[] args)
{
_groupReadTags = new BatchBlock<int>(1000);
var bufferOptions = new DataflowBlockOptions{BoundedCapacity = 2};
BufferBlock<int> _frameBuffer = new BufferBlock<int>(bufferOptions);
var consumerOptions = new ExecutionDataflowBlockOptions { BoundedCapacity = 1};
int batchNo = 1;
TransformBlock<int[], int> _workingBlock = new TransformBlock<int[], int>(list =>
{
Console.WriteLine("\n\nWorking on Batch Number {0}", batchNo);
//_groupReadTags.TriggerBatch();
int sum = 0;
foreach (int item in list)
{
Console.WriteLine("Elements in batch {0} :: {1}", batchNo, item);
sum += item;
}
batchNo++;
return sum;
});
ActionBlock<int> _worker1 = new ActionBlock<int>(async x =>
{
Console.WriteLine("Number from ONE :{0}",x);
await Task.Delay(500);
Console.WriteLine("BatchBlock Output Count : {0}", _groupReadTags.OutputCount);
_groupReadTags.TriggerBatch();
},consumerOptions);
ActionBlock<int> _worker2 = new ActionBlock<int>(async x =>
{
Console.WriteLine("Number from TWO :{0}", x);
await Task.Delay(2000);
_groupReadTags.TriggerBatch();
}, consumerOptions);
_groupReadTags.LinkTo(_workingBlock);
_workingBlock.LinkTo(_frameBuffer);
_frameBuffer.LinkTo(_worker1);
_frameBuffer.LinkTo(_worker2);
_groupReadTags.Post(10);
_groupReadTags.Post(20);
_groupReadTags.TriggerBatch();
Task postingTask = new Task(() => PostStuff());
postingTask.Start();
Console.ReadLine();
}
static void PostStuff()
{
for (int i = 0; i < 10; i++)
{
_groupReadTags.Post(i);
Thread.Sleep(100);
}
Parallel.Invoke(
() => _groupReadTags.Post(100),
() => _groupReadTags.Post(200),
() => _groupReadTags.Post(300),
() => _groupReadTags.Post(400),
() => _groupReadTags.Post(500),
() => _groupReadTags.Post(600),
() => _groupReadTags.Post(700),
() => _groupReadTags.Post(800)
);
}

Here is an alternative BatchBlock implementation with some extra features. It includes a TriggerBatch method with this signature:
public int TriggerBatch(int nextMinBatchSizeIfEmpty);
Invoking this method will either trigger a batch immediately if the input queue is not empty, otherwise it will set a temporary MinBatchSize that will affect only the next batch. You could invoke this method with a small value for nextMinBatchSizeIfEmpty to ensure that in case a batch cannot be currently produced, the next batch will occur sooner than the configured BatchSize at the block's constructor.
This method returns the size of the batch produced. It returns 0 in case that the input queue is empty, or the output queue is full, or the block has completed.
public class BatchBlockEx<T> : ITargetBlock<T>, ISourceBlock<T[]>
{
private readonly ITargetBlock<T> _input;
private readonly IPropagatorBlock<T[], T[]> _output;
private readonly Queue<T> _queue;
private readonly object _locker = new object();
private int _nextMinBatchSize = Int32.MaxValue;
public Task Completion { get; }
public int InputCount { get { lock (_locker) return _queue.Count; } }
public int OutputCount => ((BufferBlock<T[]>)_output).Count;
public int BatchSize { get; }
public BatchBlockEx(int batchSize, DataflowBlockOptions dataflowBlockOptions = null)
{
if (batchSize < 1) throw new ArgumentOutOfRangeException(nameof(batchSize));
dataflowBlockOptions = dataflowBlockOptions ?? new DataflowBlockOptions();
if (dataflowBlockOptions.BoundedCapacity != DataflowBlockOptions.Unbounded &&
dataflowBlockOptions.BoundedCapacity < batchSize)
throw new ArgumentOutOfRangeException(nameof(batchSize),
"Number must be no greater than the value specified in BoundedCapacity.");
this.BatchSize = batchSize;
_output = new BufferBlock<T[]>(dataflowBlockOptions);
_queue = new Queue<T>(batchSize);
_input = new ActionBlock<T>(async item =>
{
T[] batch = null;
lock (_locker)
{
_queue.Enqueue(item);
if (_queue.Count == batchSize || _queue.Count >= _nextMinBatchSize)
{
batch = _queue.ToArray(); _queue.Clear();
_nextMinBatchSize = Int32.MaxValue;
}
}
if (batch != null) await _output.SendAsync(batch).ConfigureAwait(false);
}, new ExecutionDataflowBlockOptions()
{
BoundedCapacity = 1,
CancellationToken = dataflowBlockOptions.CancellationToken
});
var inputContinuation = _input.Completion.ContinueWith(async t =>
{
try
{
T[] batch = null;
lock (_locker)
{
if (_queue.Count > 0)
{
batch = _queue.ToArray(); _queue.Clear();
}
}
if (batch != null) await _output.SendAsync(batch).ConfigureAwait(false);
}
finally
{
if (t.IsFaulted)
{
_output.Fault(t.Exception.InnerException);
}
else
{
_output.Complete();
}
}
}, TaskScheduler.Default).Unwrap();
this.Completion = Task.WhenAll(inputContinuation, _output.Completion);
}
public void Complete() => _input.Complete();
void IDataflowBlock.Fault(Exception ex) => _input.Fault(ex);
public int TriggerBatch(Func<T[], bool> condition, int nextMinBatchSizeIfEmpty)
{
if (nextMinBatchSizeIfEmpty < 1)
throw new ArgumentOutOfRangeException(nameof(nextMinBatchSizeIfEmpty));
int count = 0;
lock (_locker)
{
if (_queue.Count > 0)
{
T[] batch = _queue.ToArray();
if (condition == null || condition(batch))
{
bool accepted = _output.Post(batch);
if (accepted) { _queue.Clear(); count = batch.Length; }
}
_nextMinBatchSize = Int32.MaxValue;
}
else
{
_nextMinBatchSize = nextMinBatchSizeIfEmpty;
}
}
return count;
}
public int TriggerBatch(Func<T[], bool> condition)
=> TriggerBatch(condition, Int32.MaxValue);
public int TriggerBatch(int nextMinBatchSizeIfEmpty)
=> TriggerBatch(null, nextMinBatchSizeIfEmpty);
public int TriggerBatch() => TriggerBatch(null, Int32.MaxValue);
DataflowMessageStatus ITargetBlock<T>.OfferMessage(
DataflowMessageHeader messageHeader, T messageValue,
ISourceBlock<T> source, bool consumeToAccept)
{
return _input.OfferMessage(messageHeader, messageValue, source,
consumeToAccept);
}
T[] ISourceBlock<T[]>.ConsumeMessage(DataflowMessageHeader messageHeader,
ITargetBlock<T[]> target, out bool messageConsumed)
{
return _output.ConsumeMessage(messageHeader, target, out messageConsumed);
}
bool ISourceBlock<T[]>.ReserveMessage(DataflowMessageHeader messageHeader,
ITargetBlock<T[]> target)
{
return _output.ReserveMessage(messageHeader, target);
}
void ISourceBlock<T[]>.ReleaseReservation(DataflowMessageHeader messageHeader,
ITargetBlock<T[]> target)
{
_output.ReleaseReservation(messageHeader, target);
}
IDisposable ISourceBlock<T[]>.LinkTo(ITargetBlock<T[]> target,
DataflowLinkOptions linkOptions)
{
return _output.LinkTo(target, linkOptions);
}
}
Another overload of the TriggerBatch method allows to examine the batch that can be currently produced, and decide if it should be triggered or not:
public int TriggerBatch(Func<T[], bool> condition);
The BatchBlockEx class does not support the Greedy and MaxNumberOfGroups options of the built-in BatchBlock.

I have found that using TriggerBatch in this way is unreliable:
_groupReadTags.Post(10);
_groupReadTags.Post(20);
_groupReadTags.TriggerBatch();
Apparently TriggerBatch is intended to be used inside the block, not outside it like this. I have seen this result in odd timing issues, like items from next batch batch being included in the current batch, even though TriggerBatch was called first.
Please see my answer to this question for an alternative using DataflowBlock.Encapsulate: BatchBlock produces batch with elements sent after TriggerBatch()

Retry previous task action TPL

I'd like to implement a retry task that takes the previous failing tasks action and repeat it.
This is what I have so far. However it just repeats the fact that the task is in fault rather than actually firing the action of the task again.
public static async Task<T> Retry<T>(this Task<T> task, int retryCount, int delay, TaskCompletionSource<T> tcs = null)
{
if (tcs == null)
{
tcs = new TaskCompletionSource<T>();
}
await task.ContinueWith(async _original =>
{
if (_original.IsFaulted)
{
if (retryCount == 0)
{
tcs.SetException(_original.Exception.InnerExceptions);
}
else
{
Console.WriteLine("Unhandled exception. Retrying...");
await Task.Delay(delay).ContinueWith(async t =>
{
await Retry(task, retryCount - 1, delay, tcs);
});
}
}
else
tcs.SetResult(_original.Result);
});
return await tcs.Task;
}
I tried to get the action with a little reflection. However it seems that once the task is completed the action is set to null.
var action = task
.GetType()
.GetField("m_action", BindingFlags.NonPublic | BindingFlags.Instance)
.GetValue(task) as Action;
Ideally I would like my implementation to look like this:
try
{
await MakeFailure().Retry(5, 1000);
}
catch (Exception ex)
{
Console.WriteLine("I had an exception");
}
This may not be possible but I'd like to make sure before refactoring the code to a Retry(Func<T> task)

Not completely against it. But it changes the flow of the code to a fault first layout which I don't like
Consider your types. Task represents an asynchronous operation. In the asynchronous world, Task represents an asynchronous operation that has already started. Task is not something you can "retry".
On the other hand, Func<Task> represents an asynchronous operation that can be started. Or restarted. That's what you need to work with.
Once you are using the appropriate type, the code is straightforward:
public static async Task<T> Retry<T>(Func<Task<T>> action, int retryCount, int delay)
{
while (retryCount > 0)
{
try
{
return await action().ConfigureAwait(false);
}
catch (Exception ex)
{
await Task.Delay(delay).ConfigureAwait(false);
--retryCount;
}
}
return await action().ConfigureAwait(false);
}
Like the other answerer, I recommend you use a library that was actually designed for this. The Transient Fault Handling Application Block and Polly are two good examples.

There is a really great library out there that you could leverage without writing your own code. It is called the Transient Fault Application Block. But I would start by evaluating a single library in the block called TransientFaultHandling.Core.
It's used in a way very similar to your code above. Here's a quick example:
using System;
using Microsoft.Practices.TransientFaultHandling;
namespace Stackoverflow
{
class Program
{
internal class MyTransientErrorDetectionStrategy : ITransientErrorDetectionStrategy
{
public bool IsTransient(Exception ex)
{
return true;
}
}
private static void Main(string[] args)
{
const int retryCount = 5;
const int retryIntervalInSeconds = 1;
// define the strategy for retrying
var retryStrategy = new FixedInterval(
retryCount,
TimeSpan.FromSeconds(retryIntervalInSeconds));
// define the policy
var retryPolicy =
new RetryPolicy<MyTransientErrorDetectionStrategy>(retryStrategy);
retryPolicy.Retrying += retryPolicy_Retrying;
for (int i = 0; i < 50; i++)
{
// try this a few times just to illustrate
try
{
retryPolicy.ExecuteAction(SomeMethodThatCanSometimesFail);
// (the retry policy has async support as well)
}
catch (Exception)
{
// if it got to this point, your retries were exhausted
// the original exception is rethrown
throw;
}
}
Console.WriteLine("Press Enter to Exit");
Console.ReadLine();
}
private static void SomeMethodThatCanSometimesFail()
{
var random = new Random().Next(1, 4);
if (random == 2)
{
const string msg = "randomFailure";
Console.WriteLine(msg);
throw new Exception(msg);
}
}
private static void retryPolicy_Retrying(object sender, RetryingEventArgs e)
{
Console.WriteLine("retrying");
}
}
}

The problem that you have is that once you have a Task<T> in flight it can't be undone or retried. You must start with a Func<Task<T>> to be able to retry.
Now you can muck about directly with TPL, but I'd recommend using Microsoft's Reactive Framework to build the functionality you need. It's much much more powerful than TPL.
Given a Func<Task<T>> this is what you need:
Func<Task<T>> taskFactory = ...
int retryCount = 5;
Task<T> retryingTask = Observable.FromAsync(taskFactory).Retry(retryCount).ToTask();
I tested this with this code:
var i = 0;
Func<Task<int>> taskFactory = () => Task.Run(() =>
{
if (i++ == 0)
throw new Exception("Foo");
return i;
});
int retryCount = 5;
Task<int> retryingTask = Observable.FromAsync(taskFactory).Retry(retryCount).ToTask();
Console.WriteLine(retryingTask.Result);
The Reactive Framework will let you do so much more - it's a very powerful library - but it does make this task very simple. You can NuGet "Rx-Main" to get the bits.