Develop Reference

How to use Consul to perform synchronize task on one machine at a time?

I have a system with 10 machines where I need to perform a certain task on each machine one by one in synchronize order. Basically only one machine should do that task at a particular time. We already use Consul for some other purpose but I was thinking can we use Consul to do this as well?
I read more about it and it looks like we can use leader election with consul where each machine will try to acquire lock, do the work and then release the lock. Once work is done, it will release the lock and then other machine will try to acquire lock again and do the same work. This way everything will be synchronized one machine at a time.
I decided to use this C# PlayFab ConsulDotNet library which already has this capability built in looks like but if there is any better option available I am open to that as well. Below Action method in my code base is called on each machine at the same time almost through a watcher mechanism.
private void Action() {
// Try to acquire lock using Consul.
// If lock acquired then DoTheWork() otherwise keep waiting for it until lock is acquired.
// Once work is done, release the lock
// so that some other machine can acquire the lock and do the same work.
}
Now inside that above method I need to do below things -
Try to acquire lock. If you cannot acquire the lock wait for it since other machine might have grabbed it before you.
If lock acquired then DoTheWork().
Once work is done, release the lock so that some other machine can acquire the lock and do the same work.
Idea is all 10 machines should DoTheWork() one at a time in synchronize order. Based on this blog and this blog I decided to modify their example to fit our needs -
Below is my LeaderElectionService class:
public class LeaderElectionService
{
public LeaderElectionService(string leadershipLockKey)
{
this.key = leadershipLockKey;
}
public event EventHandler<LeaderChangedEventArgs> LeaderChanged;
string key;
CancellationTokenSource cts = new CancellationTokenSource();
Timer timer;
bool lastIsHeld = false;
IDistributedLock distributedLock;
public void Start()
{
timer = new Timer(async (object state) => await TryAcquireLock((CancellationToken)state), cts.Token, 0, Timeout.Infinite);
}
private async Task TryAcquireLock(CancellationToken token)
{
if (token.IsCancellationRequested)
return;
try
{
if (distributedLock == null)
{
var clientConfig = new ConsulClientConfiguration { Address = new Uri("http://consul.host.domain.com") };
ConsulClient client = new ConsulClient(clientConfig);
distributedLock = await client.AcquireLock(new LockOptions(key) { LockTryOnce = true, LockWaitTime = TimeSpan.FromSeconds(3) }, token).ConfigureAwait(false);
}
else
{
if (!distributedLock.IsHeld)
{
await distributedLock.Acquire(token).ConfigureAwait(false);
}
}
}
catch (LockMaxAttemptsReachedException ex)
{
//this is expected if it couldn't acquire the lock within the first attempt.
Console.WriteLine(ex.Stacktrace);
}
catch (Exception ex)
{
Console.WriteLine(ex.Stacktrace);
}
finally
{
bool lockHeld = distributedLock?.IsHeld == true;
HandleLockStatusChange(lockHeld);
//Retrigger the timer after a 10 seconds delay (in this example). Delay for 7s if not held as the AcquireLock call will block for ~3s in every failed attempt.
timer.Change(lockHeld ? 10000 : 7000, Timeout.Infinite);
}
}
protected virtual void HandleLockStatusChange(bool isHeldNew)
{
// Is this the right way to check and do the work here?
// In general I want to call method "DoTheWork" in "Action" method itself
// And then release and destroy the session once work is done.
if (isHeldNew)
{
// DoTheWork();
Console.WriteLine("Hello");
// And then were should I release the lock so that other machine can try to grab it?
// distributedLock.Release();
// distributedLock.Destroy();
}
if (lastIsHeld == isHeldNew)
return;
else
{
lastIsHeld = isHeldNew;
}
if (LeaderChanged != null)
{
LeaderChangedEventArgs args = new LeaderChangedEventArgs(lastIsHeld);
foreach (EventHandler<LeaderChangedEventArgs> handler in LeaderChanged.GetInvocationList())
{
try
{
handler(this, args);
}
catch (Exception ex)
{
Console.WriteLine(ex.Stacktrace);
}
}
}
}
}
And below is my LeaderChangedEventArgs class:
public class LeaderChangedEventArgs : EventArgs
{
private bool isLeader;
public LeaderChangedEventArgs(bool isHeld)
{
isLeader = isHeld;
}
public bool IsLeader { get { return isLeader; } }
}
In the above code there are lot of pieces which might not be needed for my use case but idea is same.
Problem Statement
Now in my Action method I would like to use above class and perform the task as soon as lock is acquired otherwise keep waiting for the lock. Once work is done, release and destroy the session so that other machine can grab it and do the work. I am kinda confuse on how to use above class properly in my below method.
private void Action() {
LeaderElectionService electionService = new LeaderElectionService("data/process");
// electionService.LeaderChanged += (source, arguments) => Console.WriteLine(arguments.IsLeader ? "Leader" : "Slave");
electionService.Start();
// now how do I wait for the lock to be acquired here indefinitely
// And once lock is acquired, do the work and then release and destroy the session
// so that other machine can grab the lock and do the work
}
I recently started working with C# so that's why kinda confuse on how to make this work efficiently in production by using Consul and this library.
Update
I tried with below code as per your suggestion and I think I tried this earlier as well but for some reason as soon as it goes to this line await distributedLock.Acquire(cancellationToken);, it just comes back to main method automatically. It never moves forward to my Doing Some Work! print out. Does CreateLock actually works? I am expecting that it will create data/lock on consul (since it is not there) and then try to acquire the lock on it and if acquired, then do the work and then release it for other machines?
private static CancellationTokenSource cts = new CancellationTokenSource();
public static void Main(string[] args)
{
Action(cts.Token);
Console.WriteLine("Hello World");
}
private static async Task Action(CancellationToken cancellationToken)
{
const string keyName = "data/lock";
var clientConfig = new ConsulClientConfiguration { Address = new Uri("http://consul.test.host.com") };
ConsulClient client = new ConsulClient(clientConfig);
var distributedLock = client.CreateLock(keyName);
while (true)
{
try
{
// Try to acquire lock
// As soon as it comes to this line,
// it just goes back to main method automatically. not sure why
await distributedLock.Acquire(cancellationToken);
// Lock is acquired
// DoTheWork();
Console.WriteLine("Doing Some Work!");
// Work is done. Jump out of loop to release the lock
break;
}
catch (LockHeldException)
{
// Cannot acquire the lock. Wait a while then retry
await Task.Delay(TimeSpan.FromSeconds(10), cancellationToken);
}
catch (Exception)
{
// TODO: Handle exception thrown by DoTheWork method
// Here we jump out of the loop to release the lock
// But you can try to acquire the lock again based on your requirements
break;
}
}
// Release and destroy the lock
// So that other machine can grab the lock and do the work
await distributedLock.Release(cancellationToken);
await distributedLock.Destroy(cancellationToken);
}

IMO, LeaderElectionService from those blogs is an overkill in your case.
Update 1
There is no need to do while loop because:
ConsulClient is local variable
No need to check IsHeld property
Acquire will block indefinitely unless
Set LockTryOnce true in LockOptions
Set timeout to CancellationToken
Side note, it is not necessary to invoke Destroy method after you call Release on the distributed lock (reference).
private async Task Action(CancellationToken cancellationToken)
{
const string keyName = "YOUR_KEY";
var client = new ConsulClient();
var distributedLock = client.CreateLock(keyName);
try
{
// Try to acquire lock
// NOTE:
// Acquire method will block indefinitely unless
// 1. Set LockTryOnce = true in LockOptions
// 2. Pass a timeout to cancellation token
await distributedLock.Acquire(cancellationToken);
// Lock is acquired
DoTheWork();
}
catch (Exception)
{
// TODO: Handle exception thrown by DoTheWork method
}
// Release the lock (not necessary to invoke Destroy method),
// so that other machine can grab the lock and do the work
await distributedLock.Release(cancellationToken);
}
Update 2
The reason why OP's code just returns back to Main method is that, Action method is not awaited. You can use async Main if you use C# 7.1, and put await on Action method.
public static async Task Main(string[] args)
{
await Action(cts.Token);
Console.WriteLine("Hello World");
}

how to synchronize this lock across thread in C#

I have the following code:
private static async Task SaveAsync()
{
if (Monitor.TryEnter(_SaveLock, 1000))
{
try
{
Logging.Info(" writing bitmex data to database");
await SomelenghthyDbUpdate1.ConfigureAwait(false);
await SomelenghthyDbUpdate2.ConfigureAwait(false);
await SomelenghthyDbUpdate3.ConfigureAwait(false);
await SomelenghthyDbUpdate4.ConfigureAwait(false);
}
finally
{
Monitor.Exit(_SaveLock);
}
}
}
First, ConfigureAwait false vs true does yield a real speed difference.
So, knowing that the finally part can be executed in another thread than the caller there is an issue when releasing the lock.
I am trying to prevent two save operations to happen at the same time, since they're event driven AND can be skipped if needed as they happen periodically.
another option I was thinking about is to make an array of tasks and, in the caller thread do a Task.WaitAll(tasks). In that scenario, is it guaranteed I would still be on the same thread at exit?
But is there a clean solution to this problem? maybe setting a flag through a lock?

As per Docs "The SemaphoreSlim class doesn't enforce thread or task identity on calls to the Wait, WaitAsync, and Release methods."
private static SemaphoreSlim sem = new SemaphoreSlim(1);
private static async Task SaveAsync()
{
if(await sem.WaitAsync(TimeSpan.FromSeconds(1))) // Can be pimped with cancel token
{
try
{
Logging.Info(" writing bitmex data to database");
await SomelenghthyDbUpdate1.ConfigureAwait(false);
await SomelenghthyDbUpdate2.ConfigureAwait(false);
await SomelenghthyDbUpdate3.ConfigureAwait(false);
await SomelenghthyDbUpdate4.ConfigureAwait(false);
}
finally
{
sem.Release();
}
}
}

Async network operations never finish

I have several asynchronous network operations that return a task that may never finish:
UdpClient.ReceiveAsync doesn't accept a CancellationToken
TcpClient.GetStream returns a NetworkStream that doesn't respect the CancellationToken on Stream.ReadAsync (checking for cancellation only at the start of the operation)
Both wait for a message that may never come (because of packet loss or no response for example). That means I have phantom tasks that never complete, continuations that will never run and used sockets on hold. I know i can use TimeoutAfter, but that will only fix the continuation problem.
So what am I supposed to do?

So i've made an extension method on IDisposable that creates a CancellationToken that disposes the connection on timeout, so the task finishes and everything carries on:
public static IDisposable CreateTimeoutScope(this IDisposable disposable, TimeSpan timeSpan)
{
var cancellationTokenSource = new CancellationTokenSource(timeSpan);
var cancellationTokenRegistration = cancellationTokenSource.Token.Register(disposable.Dispose);
return new DisposableScope(
() =>
{
cancellationTokenRegistration.Dispose();
cancellationTokenSource.Dispose();
disposable.Dispose();
});
}
And the usage is extremely simple:
try
{
var client = new UdpClient();
using (client.CreateTimeoutScope(TimeSpan.FromSeconds(2)))
{
var result = await client.ReceiveAsync();
// Handle result
}
}
catch (ObjectDisposedException)
{
return null;
}
Extra Info:
public sealed class DisposableScope : IDisposable
{
private readonly Action _closeScopeAction;
public DisposableScope(Action closeScopeAction)
{
_closeScopeAction = closeScopeAction;
}
public void Dispose()
{
_closeScopeAction();
}
}

So what am I supposed to do?
In this particular case, I would rather use UdpClient.Client.ReceiveTimeout and TcpClient.ReceiveTimeout to time out a UDP or TCP receive operation gracefully. I'd like to have the time-out error coming from the socket, rather than from any external source.
If in addition to that I need to observe some other cancellation event, like a UI button click, I would just use WithCancellation from Stephen Toub's "How do I cancel non-cancelable async operations?", like this:
using (var client = new UdpClient())
{
UdpClient.Client.ReceiveTimeout = 2000;
var result = await client.ReceiveAsync().WithCancellation(userToken);
// ...
}
To address the comment, in case ReceiveTimeout has no effect on ReceiveAsync, I'd still use WithCancellation:
using (var client = new UdpClient())
using (var cts = CancellationTokenSource.CreateLinkedTokenSource(userToken))
{
UdpClient.Client.ReceiveTimeout = 2000;
cts.CancelAfter(2000);
var result = await client.ReceiveAsync().WithCancellation(cts.Token);
// ...
}
IMO, this more clearly shows my intentions as a developer and is more readable to a 3rd party. Also, I don't need to catch ObjectDisposedException exeception. I still need to observe OperationCanceledException somewhere in my client code which calls this, but I'd be doing that anyway. OperationCanceledException usually stands out from other exceptions, and I have an option to check OperationCanceledException.CancellationToken to observe the reason for cancellation.
Other than that, there's not much difference from #I3arnon's answer. I just don't feel like I need another pattern for this, as I already have WithCancellation at my disposal.
To further address the comments:
I'd only be catching OperationCanceledException in the client code, i.e.:
async void Button_Click(sender o, EventArgs args)
{
try
{
await DoSocketStuffAsync(_userCancellationToken.Token);
}
catch (Exception ex)
{
while (ex is AggregateException)
ex = ex.InnerException;
if (ex is OperationCanceledException)
return; // ignore if cancelled
// report otherwise
MessageBox.Show(ex.Message);
}
}
Yes, I'll be using WithCancellation with each ReadAsync call and I like that fact, for the following reasons. Firstly, I can create an extension ReceiveAsyncWithToken:
public static class UdpClientExt
{
public static Task<UdpReceiveResult> ReceiveAsyncWithToken(
this UdpClient client, CancellationToken token)
{
return client.ReceiveAsync().WithCancellation(token);
}
}
Secondly, in 3yrs from now I may be reviewing this code for .NET 6.0. By then, Microsoft may have a new API, UdpClient.ReceiveAsyncWithTimeout. In my case, I'll simply replace ReceiveAsyncWithToken(token) or ReceiveAsync().WithCancellation(token) with ReceiveAsyncWithTimeout(timeout, userToken). It would not be so obvious to deal with CreateTimeoutScope.

Creating an async webservice method

I've tried to read up on async methods and am now trying to create my own async method. The method is a webservice call that returns a list of error logs. I'm not sure that I've understood correctly so I thought I'd share my code to see if I should do anything different.
All I want the code to do is return a list of errorlogs by calling a method GetAllErrorLogs(), that is a synchronized method. Since it can take a second to fetch all the error logs I want to have the opportunity to do other stuff once I called the GetAllErrorLogs() method. Here is the code.
[WebMethod]
public async Task<List<ErrorLog>> GetAllErrorLogs()
{
List<ErrorLog> errorLogs = new List<ErrorLog>();
await System.Threading.Tasks.Task.Run(() => {
errorLogs = ErrorLogRepository.GetAllErrorLogs();
});
if (errorLogs == null)
return new List<ErrorLog>();
return errorLogs;
}
Thanks!

I recently gave a talk at ThatConference on async on the server side, and I address this issue in the slides.
On the server side, you want to avoid the use of Task.Run and other constructs that queue work to the thread pool. As much as possible, keep thread pool threads available for handling requests.
So, ideally your repository would have an asynchronous method GetAllErrorLogsAsync, which would itself be asynchronous. If GetAllErrorLogs cannot be asynchronous, then you may as well just call it directly (removing the await Task.Run).
Since it can take a second to fetch all the error logs I want to have the opportunity to do other stuff once I called the GetAllErrorLogs() method.
If you have a GetAllErrorLogsAsync available, then this can easily be done using Task.WhenAll. However, if GetAllErrorLogs is synchronous, then you can only do this by doing parallel work in your request (e.g., multiple calls to Task.Run followed by Task.WhenAll).
Parallel code on the server must be approached with great trepidation. It is only acceptable in a very limited set of scenarios. The entire point of async on the server side is to use fewer threads per request, and when you start parallelizing, you're doing the opposite: multiple threads per request. This is only appropriate if you know your user base is very small; otherwise, you'll kill your server scalability.

I found this great codeproject detailed article about how to achieve that
http://www.codeproject.com/Articles/600926/Asynchronous-web-services-call-in-ASP-NET

**This is potentially wrong, read comments or spinoff question at HttpContext.Current after an await
If ErrorLogRepository.GetAllErrorLogs() is not thread-safe, it will cause weird bugs and potentially exception out. Make sure your code is ready for multi-threaded operation before switching to async methods, this is obviously very trivial advice but often overlooked. For example, if you reference HttpContext.Current in your methods, your code will die in the async method, and sometimes even AFTER the await. The reason is that the code within the async block will potentially be run on a separate thread, which will not have access to the same HttpContext.Current thread-static property, and await gets compiled into two methods. All code before an await gets run on one thread, and then calls the code after an await keyword as a continuation, but potentially on yet another thread. So sometimes your code will even work in an async block, only to choke unexpectedly after it gets "out" of the async back to what you think is a synchronous part of your code (but in reality everything after an await keyword is already not guaranteed to be the original thread).

Here is some production code...
using System.Web.Http;
using AysncTask = System.Threading.Tasks.Task;
public class myController : ApiControllerBase
{
[HttpPut]
[Route("api/cleardata/{id}/{requestId}/")]
public async AysncTask ClearData(Guid id, Guid requestId)
{
try
{
await AysncTask.Run(() => DoClearData(id, requestId));
}
catch (Exception ex)
{
throw new Exception("Exception in myController.ClearData", ex);
}
}
}

Handling Async exceptions is also VERY VERY important.. although this is for a windows console app, the same principles should apply.
source: https://blogs.msdn.microsoft.com/ptorr/2014/12/10/async-exceptions-in-c/
using System;
using System.Runtime.CompilerServices;
using System.Threading;
using System.Threading.Tasks;
namespace AsyncAndExceptions
{
class Program
{
static void Main(string[] args)
{
AppDomain.CurrentDomain.UnhandledException += (s, e) => Log("*** Crash! ***", "UnhandledException");
TaskScheduler.UnobservedTaskException += (s, e) => Log("*** Crash! ***", "UnobservedTaskException");
RunTests();
// Let async tasks complete...
Thread.Sleep(500);
GC.Collect(3, GCCollectionMode.Forced, true);
}
private static async Task RunTests()
{
try
{
// crash
// _1_VoidNoWait();
// crash
// _2_AsyncVoidAwait();
// OK
// _3_AsyncVoidAwaitWithTry();
// crash - no await
// _4_TaskNoWait();
// crash - no await
// _5_TaskAwait();
// OK
// await _4_TaskNoWait();
// OK
// await _5_TaskAwait();
}
catch (Exception ex) { Log("Exception handled OK"); }
// crash - no try
// await _4_TaskNoWait();
// crash - no try
// await _5_TaskAwait();
}
// Unsafe
static void _1_VoidNoWait()
{
ThrowAsync();
}
// Unsafe
static async void _2_AsyncVoidAwait()
{
await ThrowAsync();
}
// Safe
static async void _3_AsyncVoidAwaitWithTry()
{
try { await ThrowAsync(); }
catch (Exception ex) { Log("Exception handled OK"); }
}
// Safe only if caller uses await (or Result) inside a try
static Task _4_TaskNoWait()
{
return ThrowAsync();
}
// Safe only if caller uses await (or Result) inside a try
static async Task _5_TaskAwait()
{
await ThrowAsync();
}
// Helper that sets an exception asnychronously
static Task ThrowAsync()
{
TaskCompletionSource tcs = new TaskCompletionSource();
ThreadPool.QueueUserWorkItem(_ => tcs.SetException(new Exception("ThrowAsync")));
return tcs.Task;
}
internal static void Log(string message, [CallerMemberName] string caller = "")
{
Console.WriteLine("{0}: {1}", caller, message);
}
}
}

c# lock and listen to CancellationToken

I want to use lock or a similar synchronization to protect a critical section. At the same time I want to listen to a CancellationToken.
Right now I'm using a mutex like this, but mutex doesn't have as good performance. Can I use any of other synchronization classes (including the new .Net 4.0) instead of the mutex?
WaitHandle.WaitAny(new[] { CancelToken.WaitHandle, _mutex});
CancelToken.ThrowIfCancellationRequested();

Take a look at the new .NET 4.0 Framework feature SemaphoreSlim Class. It provides SemaphoreSlim.Wait(CancellationToken) method.
Blocks the current thread until it can enter the SemaphoreSlim, while
observing a CancellationToken
From some point of view using Semaphore in such simple case could be an overhead because initially it was designed to provide an access for multiple threads, but perhaps you might find it useful.
EDIT: The code snippet
CancellationToken token = new CancellationToken();
SemaphoreSlim semaphore = new SemaphoreSlim(1,1);
bool tokenCanceled = false;
try {
try {
// block section entrance for other threads
semaphore.Wait(token);
}
catch (OperationCanceledException) {
// The token was canceled and the semaphore was NOT entered...
tokenCanceled = true;
}
// critical section code
// ...
if (token.IsCancellationRequested)
{
// ...
}
}
finally {
if (!tokenCanceled)
semaphore.Release();
}

private object _lockObject = new object();
lock (_lockObject)
{
// critical section
using (token.Register(() => token.ThrowIfCancellationRequested())
{
// Do something that might need cancelling.
}
}
Calling Cancel() on a token will result in the ThrowIfCancellationRequested() being invoked as that was what is hooked up to the Register callback. You can put whatever cancellation logic you want in here. This approach is great because you can cancel blocking calls by forcing the conditions that will cause the call to complete.
ThrowIfCancellationRequested throws a OperationCanceledException. You need to handle this on the calling thread or your whole process could be brought down. A simple way of doing this is by starting your task using the Task class which will aggregate all the exceptions up for you to handle on the calling thread.
try
{
var t = new Task(() => LongRunningMethod());
t.Start();
t.Wait();
}
catch (AggregateException ex)
{
ex.Handle(x => true); // this effectively swallows any exceptions
}
Some good stuff here covering co-operative cancellation

You can use Monitor.TryEnter with timeout to wait for the lock and check periodically for cancellation.
private bool TryEnterSyncLock(object syncObject)
{
while(!Monitor.TryEnter(syncObject, TimeSpan.FromMilliseconds(100)))
{
if (cts_.IsCancellationRequested)
return false;
}
return true;
}
Note that I would not recommend this in high contention situations as it can impact performance. I would use it as a safety mechanism against deadlocks in case you cannot use SemaphoreSlim as it has different same thread re-entrancy semantics than Monitor.Enter.
After returning true, lock on syncObject has to be released using Monitor.Exit.