Develop Reference

What is wrong with that optimistic concurrency worker implementation?

I have tried to implement an optimistic concurrency 'worker'.
Goal is to read a batch of data from the same database table (single table with no relations) with multiple parallel 'worker'. This did seem to work so far. I get optimistic concurrency exceptions here and there, catch them and retry.
So far so good, and the function to get the data works stable on my local setup. When moving the application to a test environment however, I get a strange timeout exception, which even if caught, will end the async function (breaks the while loop). Does someone see a flaw in the implementation? What could cause the timeout? What could cause the end of the async function?
public async IAsyncEnumerable<List<WorkItem>> LoadBatchedWorkload([EnumeratorCancellation] CancellationToken token, int batchSize, int runID)
{
DataContext context = null;
try
{
context = GetNewContext(); // create a new dbContext
List<WorkItem> workItems;
bool loadSuccessInner;
while (true)
{
if (token.IsCancellationRequested) break;
loadSuccessInner = false;
context.Dispose();
context = GetNewContext(); // create a new dbContext
RunState currentRunState = context.Runs.Where(a => a.Id == runID).First().Status;
try
{
// Error happens on the following line: Microsoft.Data.SqlClient.SqlException: Timeout
workItems = context.WorkItems.Where(a => a.State == ProcessState.ToProcess).Take(batchSize).ToList();
loadSuccessInner = true;
}
catch (Exception ex)
{
workItems = new List<WorkItem>();
}
if (workItems.Count == 0 && loadSuccessInner)
{
break;
}
//... update to a different RunState
//... if set successful yield the result
//... else cleanup and retry
}
}
finally
{
if (context != null) context.Dispose();
}
}
I verified that EntityFramework (here with MS SQL Server adapter) does full server side query, which
translates to a simple query like this: SELECT TOP 10 field_1, field_2 FROM WorkItems WHERE field_2 = 0
The query usually takes <1ms and the timeout is left on default of
30s
I verified that there are no cancelation requests fired
This happens also when there is only a single worker and no one else is accessing the database. I'm aware that a timeout can happen when the resource is busy or blocked. But until now, I never saw a timeout on any other query yet.

(I'll update this answer whenever more information is being provided.)
Does someone see a flaw in the implementation?
Generally, your code looks fine.
What could cause the end of the async function?
Nothing in the code you showed should normally be an issue. Start by putting another try-catch block inside the loop, to ensure, that no other exceptions are getting thrown anywhere else (especially later in the not shown code):
public async IAsyncEnumerable<List<WorkItem>> LoadBatchedWorkload([EnumeratorCancellation] CancellationToken token, int batchSize, int runID)
{
DataContext context = null;
try
{
context = GetNewContext();
List<WorkItem> workItems;
bool loadSuccessInner;
while (true)
{
try
{
// ... (the inner loop code)
}
catch (Exception e)
{
// TODO: Log the exception here using your favorite method.
throw;
}
}
}
finally
{
if (context != null) context.Dispose();
}
}
Take a look at your log and ensure, that the log does not show any exceptions being thrown. Then additionally log every possible exit condition (break and return) from the loop, to find out how and why the code exits the loop.
If there are no other break or return statements in your code, then the only way the code can exit from the loop is if zero workItems are successfully returned from the database.
What could cause the timeout?
Make sure, that any Task returning/async methods you call are being called using await.
To track down, where the exceptions are actually coming from, you should deploy a Debug release with pdb files to get a full stack trace with source code line references.
You can also implement a DbCommandInterceptor and trace failing commands on your own:
public class TracingCommandInterceptor : DbCommandInterceptor
{
public override void CommandFailed(DbCommand command, CommandErrorEventData eventData)
{
LogException(eventData);
}
public override Task CommandFailedAsync(DbCommand command, CommandErrorEventData eventData, CancellationToken cancellationToken = new CancellationToken())
{
LogException(eventData);
return Task.CompletedTask;
}
private static void LogException(CommandErrorEventData eventData)
{
if (eventData.Exception is SqlException sqlException)
{
// -2 = Timeout error
// See https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2008-r2/cc645611(v=sql.105)?redirectedfrom=MSDN
if (sqlException.Number == -2)
{
var stackTrace = new StackTrace();
var stackTraceText = stackTrace.ToString();
// TODO: Do some logging here and output the stackTraceText
// and other helpful information like the command text etc.
// -->
}
}
}
}
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.UseLoggerFactory(LoggingFactory);
optionsBuilder.UseSqlServer(connectionString);
optionsBuilder.EnableSensitiveDataLogging();
optionsBuilder.EnableDetailedErrors();
// Add the command interceptor.
optionsBuilder.AddInterceptors(new TracingCommandInterceptor());
base.OnConfiguring(optionsBuilder);
}
Additionally logging the command text of the failed command in the interceptor is also a good idea.

Handling domain errors in MassTransit

I'm wondering how I should handle domain exceptions in a proper way?
Does all of my consumer's code should be wrapped into a try, catch block, or I should just thrown an Exception, which will be handled by apropriate FaultConsumer?
Consider this two samples:
Example-1 - whole operation is wrapped into try...catch block.
public async Task Consume(ConsumeContext<CreateOrder> context)
{
try
{
//Consumer that creates order
var order = new Order();
var product = store.GetProduct(command.ProductId); // check if requested product exists
if (product is null)
{
throw new DomainException(OperationCodes.ProductNotExist);
}
order.AddProduct(product);
store.SaveOrder(order);
context.Publish<OrderCreated>(new OrderCreated
{
OrderId = order.Id;
});
}
catch (Exception exception)
{
if (exception is DomainException domainException)
{
context.Publish<CreateOrderRejected>(new CreateOrderRejected
{
ErrorCode = domainException.Code;
});
}
}
}
Example-2 - MassTransit handles DomainException, by pushing message into CreateOrder_error queue. Another service subscribes to this event, and after the event is published on this particular queue, it process it;
public async Task Consume(ConsumeContext<CreateOrder> context)
{
//Consumer that creates order
var order = new Order();
var product = store.GetProduct(command.ProductId); // check if requested product exists
if (product is null)
{
throw new DomainException(OperationCodes.ProductNotExist);
}
order.AddProduct(product);
store.SaveOrder(order);
context.Publish<OrderCreated>(new OrderCreated
{
OrderId = order.Id;
});
}
Which approach should be better?
I know that I can use Request/Response and gets information about error immediately, but in my case, it must be done via message broker.

In your first example, you are handling a domain condition (in your example, a product not existing in the catalog) by producing an event that the order was rejected for an unknown product. This makes complete sense.
Now, if the database query to check the product couldn't connect to the database, that's a temporary situation that may resolve itself, and thus using a retry or scheduled redelivery makes sense - to try again before giving up entirely. Those are exceptions you would want to throw.
But the business exception you'd want to catch, and handle by publishing an event.
public async Task Consume (ConsumeContext<CreateOrder> context) {
try {
var order = new Order ();
var product = store.GetProduct (command.ProductId); // check if requested product exists
if (product is null) {
throw new DomainException (OperationCodes.ProductNotExist);
}
order.AddProduct (product);
store.SaveOrder (order);
context.Publish<OrderCreated> (new OrderCreated {
OrderId = order.Id;
});
} catch (DomainException exception) {
await context.Publish<CreateOrderRejected> (new CreateOrderRejected {
ErrorCode = domainException.Code;
});
}
}

My take on this is that you seem to go to the fire-and-forget commands mess. Of course, it is very context-specific, since there are scenarios, especially integration when you don't have a user on the other side sitting and wondering if their command was eventually executed and what is the outcome.
So, for integration scenarios, I concur with Chris' answer, publishing a domain exception event makes perfect sense.
For the user-interaction scenarios, however, I'd rather suggest using request-response that can return different kinds of response, like a positive and negative response, as described in the documentation. Here is the snippet from the docs:
Service side:
public class CheckOrderStatusConsumer :
IConsumer<CheckOrderStatus>
{
public async Task Consume(ConsumeContext<CheckOrderStatus> context)
{
var order = await _orderRepository.Get(context.Message.OrderId);
if (order == null)
await context.RespondAsync<OrderNotFound>(context.Message);
else
await context.RespondAsync<OrderStatusResult>(new
{
OrderId = order.Id,
order.Timestamp,
order.StatusCode,
order.StatusText
});
}
}
Client side:
var (statusResponse,notFoundResponse) = await client.GetResponse<OrderStatusResult, OrderNotFound>(new { OrderId = id});
// both tuple values are Task<Response<T>>, need to find out which one completed
if(statusResponse.IsCompletedSuccessfully)
{
var orderStatus = await statusResponse;
// do something
}
else
{
var notFound = await notFoundResponse;
// do something else
}

A fast and efficient way to get the needed Inner Exception

In an application using Entity Framework 6, I track the changes made on tables in the database in a specially designed entity. They track the changes on all tables, including their own table.
So if an exception occurs while saving changes in the database, I delete all of the pending tracking entities in order to avoid creating new and new trackers recursively, log the exception and exit the saving method.
However, if the exception is caused due to connection timeout, I try 3 times to resave the changes, while changing the EntityState of the tracking entities to avoid creating unnecessary trackers. In order to accomplish that, I need to catch a DbUpdateException, get the SqlException down the hierarchy of exceptions, and check its number. However, I'm not sure how deep is the SqlException in the hierarchy. To accomplish successfully getting the Sql Exception, I wrote this:
catch (DbUpdateException duEx)
{
var inner = new Exception();
inner = duEx;
do
{
inner = inner.InnerException;
}
while (!(inner is SqlException) && inner != null);
var innerEx = inner as SqlException;
if (innerEx != null && innerEx.Number == -2)
{
//do job here
}
I tested it and it seems to work, however it looks a bit clumsy. So my question is: Is there any way of getting the SqlException, if any, directly?

What I was wondering is whether there is already some 3rd party
extension method which I could use
No, but you can create it yourself:
public static class Helper
{
public static TException GetInnerException<TException>(this Exception ex) where TException : Exception
{
return ex.InnerException != null
? ex.InnerException as TException ?? GetInnerException<TException>(ex.InnerException)
: null;
}
}
And use it :
catch (DbUpdateException duEx)
{
if (duEx.GetInnerException<SqlException>()?.Number == -2)
{
//do job here
}
}

NSubstitute, try catch is not working on async method configured to throw an exception

I'm using NSubstitute for mocking and faking. I'm working with EF6 and would like to setup the SaveChangesAsync-Method of the database context to throw an exception:
context.SaveChangesAsync().Throws(new DbUpdateException("", SqlExceptionHelper.CreateSqlException(2627)));
SaveChangesAsync is called within a method of my data repository like this:
try
{
var fromDatabase = await context.Entries.OfType<Document>().FirstOrDefaultAsync(d => d.Id == doc.Id);
if (fromDatabase == null)
{
fromDatabase = new Document();
context.Entries.Add(fromDatabase);
}
PatchEntity(fromDatabase, doc);
await context.SaveChangesAsync();
}
catch (DbUpdateException ex)
{
var innerException = ex.InnerException as SqlException;
if (innerException != null && innerException.Number == 2627)
{
errors.Add(new DbValidationError(nameof(doc.Name), "A entry with the same Name already exists under the selected parent."));
}
}
And this is the line within my unit test:
var result = await repository.TryAddOrUpdateDocument(doc);
Unfortunately my test keeps failing with the reason, that my test method(!) is throwing the exception, I'm trying to catch. Adding a general exception catch block isn't working either, the exception is not being catched at all. The exception is bubbling up.
My test is declared as "public async Task...", but turning it into simply void and calling .Result on the async method of my repository doesn't help either. What is going on?

I think the problem is that the exception is thrown from the original call instead of from inside the returned Task as described here.
Try something like:
Func<int> throwDbEx = () => {
throw new DbUpdateException("", SqlExceptionHelper.CreateSqlException(2627));
};
context.SaveChangesAsync().Returns(Task.Run(throwDbEx));

I cannot be 100% sure, but you are probably facing the problem that exceptions thrown by async with void return type cannot be caught naturally. Read the section "Avoid Async Void" here:
https://msdn.microsoft.com/en-us/magazine/jj991977.aspx
Even if it won't answer your problem, it is worth reading anyway...

RelayCommand and async method resulting: "'System.Reflection.TargetInvocationException' occurred in mscorlib.dll"

Mvvm light RelayCommand is causing me some headache with async methods. WPF button is tied with the following relayCommand:
private RelayCommand _importDeviceCommand;
/// <summary>
/// Import device button for SelectDeviceView.
/// </summary>
public RelayCommand ImportDeviceCommand
{
get
{
return _importDeviceCommand
?? (_importDeviceCommand = new RelayCommand(async () => await AddDeviceClickExecute(),
() => _selectedCableType != null
&& _selectedAddDevice != null
&& _selectedPointNames != null
&& _selectedPointNames.Any()));
}
}
I'm probably misusing it in some form because keep occasionally encountering the following exception always when the method AddDeviceClickExecute is done.
An unhandled exception of type
'System.Reflection.TargetInvocationException' occurred in mscorlib.dll
What are the possible solutions?
Could it have something to do with the lambda method calls?
Therefore, how can I refactor the relayCommand in such a way that
no lambda is used?
EDIT 1
The called async method, the try/catch is not unfortunately making any difference?
private async Task AddDeviceClickExecute()
{
_linkTheSocket = true;
var deviceImporter = new DeviceImporterAsync2(_projectContext, _deviceContext);
var progress = new Progress<string>(status =>
{
_importDeviceProgress = status;
RaisePropertyChanged("ImportDeviceProgress");
});
try
{
await deviceImporter.InvokeSimpleDeviceImport(UserSelectedSockets, progress);
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString(), "Exception during simple device import", MessageBoxButton.OK, MessageBoxImage.Error);
}
}
EDIT 2
The following exception occurs right after the AddDeviceClickExecute exits.
EDIT 3
Turned out that the way I was utilising async and relayCommand had
nothing to do with my exception. Problem was fixed.

You'll see an unhandled exception any time you have an exception escape an async void method (namely, the lambda you're passing to RelayCommand).
This is normal and expected behavior; it's the same behavior you'd see from a synchronous RelayCommand lambda, except for the TargetInvocationException wrapper. As others have noted, just examine the InnerException to see the actual underlying exception.
If you want to catch these exceptions, you should wrap the entire body of the async void method in a try/catch. This is possible but somewhat awkward within a lambda:
return _importDeviceCommand
?? (_importDeviceCommand = new RelayCommand(async () =>
{ try { await AddDeviceClickExecute(); } catch (Exception ex) { ... } },
() => _selectedCableType != null
&& _selectedAddDevice != null
&& _selectedPointNames != null
&& _selectedPointNames.Any()));
But that's getting really awkward IMO. Better to split it into another method:
private async void ImportDevice()
{
try
{
await AddDeviceClickExecute();
}
catch (Exception ex)
{
...
}
}
return _importDeviceCommand
?? (_importDeviceCommand = new RelayCommand(ImportDevice,
() => _selectedCableType != null
&& _selectedAddDevice != null
&& _selectedPointNames != null
&& _selectedPointNames.Any()));