Develop Reference

How to make a "fire & forget" async FIFO queue in c#?

I'm trying to process documents asynchronously. The idea is that the user sends documents to a service, which takes time, and will look at the results later (about 20-90 seconds per document).
Ideally, I would like to just fill some kind of observable collection that would be emptied by the system as fast as it can. When there is an item, process it and produce the expected output in another object, and when there is no item just do nothing. When the user checks the output collection, he will find the items that are already processed.
Ideally all items would be visible from the start and would have a state (completed, ongoing or in queue), but once I know how to do the first, I should be able to handle the states.
I'm not sure which object to use for that, right now I'm looking at BlockingCollection but I don't think it's suited for the job, as I can't fill it while it's being emptied from the other end.
private BlockingCollection<IDocument> _jobs = new BlockingCollection<IDocument>();
public ObservableCollection<IExtractedDocument> ExtractedDocuments { get; }
public QueueService()
{
ExtractedDocuments = new ObservableCollection<IExtractedDocument>();
}
public async Task Add(string filePath, List<Extra> extras)
{
if (_jobs.IsAddingCompleted || _jobs.IsCompleted)
_jobs = new BlockingCollection<IDocument>();
var doc = new Document(filePath, extras);
_jobs.Add(doc);
_jobs.CompleteAdding();
await ProcessQueue();
}
private async Task ProcessQueue()
{
foreach (var document in _jobs.GetConsumingEnumerable(CancellationToken.None))
{
var resultDocument = await service.ProcessDocument(document);
ExtractedDocuments.Add(resultDocument );
Debug.WriteLine("Job completed");
}
}
This is how I'm handling it right now. If I remove the CompleteAdding call, it hangs on the second attempt. If I have that statement, then I can't just fill the queue, I have to empty it first which defeats the purpose.
Is there a way of having what I'm trying to achieve? A collection that I would fill and the system would process asynchronously and autonomously?
To summarize, I need :
A collection that I can fill, that would be processed gradually and asynchronously. A document or series or document can be added while some are being processed.
An ouput collection that would be filled after the process is complete
The UI thread and app to still be responsive while everything is running
I don't need to have multiple processes in parallel, or one document at a time. Whichever is easiest to put in place and maintain will do (small scale application). I'm assuming one at a time is simpler.

A common pattern here is to have a callback method that executes upon a document state change. With a background task running, it will chew threw documents as fast as it can. Call Dispose to shutdown the processor.
If you need to process the callback on a gui thread, you'll need to synchornize the callback to your main thread some how. Windows forms has methods to do this if that's what you are using.
This example program implements all the necessary classes and interfaces, and you can fine tune and tweak things as you need.
using System;
using System.Collections.Concurrent;
using System.Threading;
using System.Threading.Tasks;
namespace ConsoleApp2
{
class Program
{
private static Task Callback(IExtractedDocument doc, DocumentProcessor.DocState docState)
{
Console.WriteLine("Processing doc {0}, state: {1}", doc, docState);
return Task.CompletedTask;
}
public static void Main()
{
using DocumentProcessor docProcessor = new DocumentProcessor(Callback);
Console.WriteLine("Processor started, press any key to end processing");
for (int i = 0; i < 100; i++)
{
if (Console.KeyAvailable)
{
break;
}
else if (i == 5)
{
// make an error
docProcessor.Add(null);
}
else
{
docProcessor.Add(new Document { Text = "Test text " + Guid.NewGuid().ToString() });
}
Thread.Sleep(500);
}
Console.WriteLine("Doc processor shut down, press ENTER to quit");
Console.ReadLine();
}
public interface IDocument
{
public string Text { get; }
}
public class Document : IDocument
{
public string Text { get; set; }
}
public interface IExtractedDocument : IDocument
{
public IDocument OriginalDocument { get; }
public Exception Error { get; }
}
public class ExtractedDocument : IExtractedDocument
{
public override string ToString()
{
return $"Orig text: {OriginalDocument?.Text}, Extracted Text: {Text}, Error: {Error}";
}
public IDocument OriginalDocument { get; set; }
public string Text { get; set; }
public Exception Error { get; set; }
}
public class DocumentProcessor : IDisposable
{
public enum DocState { Processing, Completed, Error }
private readonly BlockingCollection<IDocument> queue = new BlockingCollection<IDocument>();
private readonly Func<IExtractedDocument, DocState, Task> callback;
private CancellationTokenSource cancelToken = new CancellationTokenSource();
public DocumentProcessor(Func<IExtractedDocument, DocState, Task> callback)
{
this.callback = callback;
Task.Run(() => StartQueueProcessor()).GetAwaiter();
}
public void Dispose()
{
if (!cancelToken.IsCancellationRequested)
{
cancelToken.Cancel();
}
}
public void Add(IDocument doc)
{
if (cancelToken.IsCancellationRequested)
{
throw new InvalidOperationException("Processor is disposed");
}
queue.Add(doc);
}
private void ProcessDocument(IDocument doc)
{
try
{
// do processing
DoCallback(new ExtractedDocument { OriginalDocument = doc }, DocState.Processing);
if (doc is null)
{
throw new ArgumentNullException("Document to process was null");
}
IExtractedDocument successExtractedDocument = DoSomeDocumentProcessing(doc);
DoCallback(successExtractedDocument, DocState.Completed);
}
catch (Exception ex)
{
DoCallback(new ExtractedDocument { OriginalDocument = doc, Error = ex }, DocState.Error);
}
}
private IExtractedDocument DoSomeDocumentProcessing(IDocument originalDocument)
{
return new ExtractedDocument { OriginalDocument = originalDocument, Text = "Extracted: " + originalDocument.Text };
}
private void DoCallback(IExtractedDocument result, DocState docState)
{
if (callback != null)
{
// send callbacks in background
callback(result, docState).GetAwaiter();
}
}
private void StartQueueProcessor()
{
try
{
while (!cancelToken.Token.IsCancellationRequested)
{
if (queue.TryTake(out IDocument doc, 1000, cancelToken.Token))
{
// can chance to Task.Run(() => ProcessDocument(doc)).GetAwaiter() for parallel execution
ProcessDocument(doc);
}
}
}
catch (OperationCanceledException)
{
// ignore, don't need to throw or worry about this
}
while (queue.TryTake(out IDocument doc))
{
DoCallback(new ExtractedDocument { Error = new ObjectDisposedException("Processor was disposed") }, DocState.Error);
}
}
}
}
}

Class Level Error Handler For DAO

I am using Entity Framework. Below is an example of a list method for an Actors context in my ActorsDao class. If you imagine my application is like imdb, there will be CRUD methods for various other contexts such as Movies, Directors, Genres, Reviews, Studios etc.
Regardless of the method or context, I handle errors in the same way. Due to my many methods across many contexts, my catch section is always exactly the same.
Obviously, I could create an error handling class, put the code in there, and just call a method in that class from the catch block.
However, I'm wondering if there a way to omit the TRY...CATCH from each method and set up a global error handler for the methods in my entity framework layer?
I would only want this global error handler to handle these errors and not errors from the rest of the application.
I seem to remember in Java Spring, you could annotate a class or method with the name of a method, and all errors would be passed to that without the need of a TRY...CATCH. I'm wondering if there is something similar for .NET (or a third party library with such functionality)?
public List<Actor> ListActors()
{
List<Actor> actorList = new List<Actor>();
using (var context = new ActorContext())
{
try
{
actorList = context.Actors.ToList<Actor>();
}
catch (Exception e)
{
//Handle error code
}
}
return actorList;
}
EDIT
I did some more research and found this code from here https://stackoverflow.com/a/4851985/1753877
private void GlobalTryCatch(Action action)
{
try
{
action.Invoke();
}
catch (ExpectedException1 e)
{
throw MyCustomException("Something bad happened", e);
}
catch (ExpectedException2 e)
{
throw MyCustomException("Something really bad happened", e);
}
}
public void DoSomething()
{
GlobalTryCatch(() =>
{
// Method code goes here
});
}
Would using a delegate like this be OK? It certainly meets my requirements.

You can create a class like this and extend the controller from this class.
Error Handler class looks like this :
package com.wes.essex.rest;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import java.util.Set;
import javax.validation.ConstraintViolation;
import javax.validation.ConstraintViolationException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ExceptionHandler;
import com.wes.essex.bean.ErrorResponse;
public class SkyNewsController {
private static final Logger LOGGER = LoggerFactory.getLogger(SkyNewsController.class);
#ExceptionHandler(Exception.class)
public ResponseEntity<ErrorResponse> handleError(Exception ex) {
LOGGER.info("start");
LOGGER.error(ex.getMessage(), ex);
ErrorResponse error = new ErrorResponse();
error.setTimestamp(ZonedDateTime.now().format(DateTimeFormatter.ISO_INSTANT));
LOGGER.debug("error : {} ", error);
ResponseEntity<ErrorResponse> response = null;
if (ex instanceof ConstraintViolationException) {
error.setReasonCode(HttpStatus.BAD_REQUEST.value());
ConstraintViolationException constraintException = (ConstraintViolationException) ex;
Set<ConstraintViolation<?>> set = constraintException.getConstraintViolations();
String errorMessage = "Input Validation Failed:";
for (ConstraintViolation<?> constraintViolation : set) {
errorMessage += constraintViolation.getMessageTemplate() + ",";
}
errorMessage = errorMessage.substring(0, errorMessage.length() - 1);
error.setErrorMessage(errorMessage);
response = new ResponseEntity<ErrorResponse>(error, HttpStatus.BAD_REQUEST);
} else {
error.setReasonCode(HttpStatus.INTERNAL_SERVER_ERROR.value());
error.setErrorMessage(ex.getMessage());
response = new ResponseEntity<ErrorResponse>(error, HttpStatus.INTERNAL_SERVER_ERROR);
}
return response;
}
}
This would be the baean class for error response :
package com.wes.essex.bean;
public class ErrorResponse {
private static final long serialVersionUID = 5776681206288518465L;
private String timestamp;
private String errorMessage;
private int reasonCode;
public String getErrorMessage() {
return errorMessage;
}
public void setErrorMessage(String errorMessage) {
this.errorMessage = errorMessage;
}
public int getReasonCode() {
return reasonCode;
}
public void setReasonCode(int reasonCode) {
this.reasonCode = reasonCode;
}
public String getTimestamp() {
return timestamp;
}
public void setTimestamp(String timestamp) {
this.timestamp = timestamp;
}
public static long getSerialversionuid() {
return serialVersionUID;
}
}

Avoid Try Catch Stements with custom ErrorHandler class - C#

I have a class which exposes some functionality,
and I want to ensure exceptions will be handled by a custom ErrorHandler class.
Currently I can achieve this by a try / catch statement per each method, and process the exception by the error handler there.
My question is if there is a better way / design pattern to do it.
Code:
public class BasicErrorHandler
{
public void ProcessException(Exception ex)
{
//Does error handling stuff
}
}
public class Manager
{
BasicErrorHandler _errorHandler;
public Manager()
{
_errorHandler = new BasicErrorHandler();
}
public void MethodA()
{
try
{
//Does Something
}
catch(Exception ex)
{
_errorHandler.ProcessException(ex);
}
}
public void MethodB()
{
try
{
//Does Something Else
}
catch(Exception ex)
{
_errorHandler.ProcessException(ex);
}
}
}

In keeping with DRY principles, you could just wrap your try...catch logic into into own method which takes a predicate of the actual work to do:
public class Manager
{
BasicErrorHandler _errorHandler;
public Manager()
{
_errorHandler = new BasicErrorHandler();
}
public void MethodA()
{
DoWork( () => {
// do something interesting here
});
}
public void MethodB()
{
DoWork( () => {
// do something else interesting here
});
}
private void DoWork(Action action)
{
try
{
action();
}
catch(Exception ex)
{
_errorHandler.ProcessException(ex);
}
}
}

I've crafted this quickly and without thinking too much in the implications, but if you want to avoid all the try/catch blocks, you could do something like:
public class BasicErrorHandler
{
public void ProcessException(Exception ex)
{
//Does error handling stuff
}
public void Do(Action act)
{
try
{
act();
}
catch(Exception ex)
{
ProcessException(ex);
}
}
}
And then use it like:
public class Manager
{
BasicErrorHandler _errorHandler;
public Manager()
{
_errorHandler = new BasicErrorHandler();
}
public void MethodA()
{
_errorHandler.Do(() => {
//Does Something
});
}
public void MethodB()
{
_errorHandler.Do(() => {
//Does Something Else
});
}
}

Design patterns are there to solve a problem. Which problem are you trying to solve? What is wrong with the Try Catch blocks?
Only thing I can imagine is you want to have more clean code. Some answers suggest a helper method with an action. Given the helper methods that encapsulate a delegate: Do consider the impact on your stack trace and debugging sessions using these delegates. It might make logging etc more hard to understand.
If your intend is to do separation of concern, I would say If you can't handle it, just don't catch the exception. Let the class invoking the method handle it. If you insist to have a handler in your class, I would suggest Inversion of Control. That way, your class is not in control of determining which class should handle its exceptions.

Rx .net is for You. Advanced error handling gives You the ability to highly customize Your error handling. Check out the pages about that.
For example:
var source = new Subject<int>();
var result = source.Catch<int, TimeoutException>(tx=>Observable.Return(-1));
result.Dump("Catch");
source.OnNext(1);
source.OnNext(2);
source.OnError(new ArgumentException("Fail!"));
You'll get the following output:
Catch-->1
Catch-->2
Catch failed-->Fail!
The number of retries, the handling of how much time a method can take, everything can be configured.

The following is an Aspect oriented method of soling the problem, this makes use of PostSharp to do the weaving.
[Serializable]
public class HandleExceptionsAttribute : OnExceptionAspect {
/// <summary>
/// Initializes a new instance of the <see cref="HandleExceptionsAttribute"/> class.
/// </summary>
public HandleExceptionsAttribute() {
AspectPriority = 1;
}
public override void OnException(MethodExecutionArgs args) {
//Suppress the current transaction to ensure exception is not rolled back
using (var s = new TransactionScope(TransactionScopeOption.Suppress)) {
//Log exception
using (var exceptionLogContext = new ExceptionLogContext()) {
exceptionLogContext.Set<ExceptionLogEntry>().Add(new ExceptionLogEntry(args.Exception));
exceptionLogContext.SaveChanges();
}
}
}
}
[HandleExceptions]
public class YourClass {
}

AppDomain await async Task prevent SerializationException

I have a windows service, which loads assembly in another AppDomain at runtime. Then it executes them and finally unloads the AppDomain. The problem is the execute method from the plugins are async tasks and I get the SerializationException because Task does not inherit from MarshalByRefObject.
I wrapped the plugin in a proxy which inherits from MarshalByRefObject, but I dont know how to get rid of the SerializationException?
public interface IPlugin : IDisposable
{
Guid GUID { get; }
string Name { get; }
string Description { get; }
Task Execute(PluginPanel panel, string user);
}
The proxy:
[Serializable()]
public class PluginProxy : MarshalByRefObject, IPlugin
{
private IPlugin m_Plugin;
public bool Init(string file)
{
Assembly ass = Assembly.Load(AssemblyName.GetAssemblyName(file));
if (ass == null || ass.GetTypes() == null || ass.GetTypes().Length == 0)
return false;
foreach (Type type in ass.GetTypes())
{
if (type.IsInterface || type.IsAbstract)
continue;
if (type.GetInterface(typeof(IPlugin).FullName) != null)
{
m_Plugin = (IPlugin)Activator.CreateInstance(type);
return true;
}
}
return false;
}
public Guid GUID { get { return m_Plugin.GUID; } }
public string Name { get { return m_Plugin.Name; } }
public string Description { get { return m_Plugin.Description; } }
// I debugged and found out the error happens AFTER m_Plugin.Execute
// so the method runs well, but the return back to the pProxy.Execute is throwing the SerializationException
public async Task Execute(PluginPanel panel, string user) { await m_Plugin.Execute(panel, user); }
}
And the Method which loads the Assembly and gets the SerializationException:
AppDomainSetup setup = new AppDomainSetup();
// some setup stuff
AppDomain dom = AppDomain.CreateDomain(Guid.NewGuid().ToString(), null, setup);
PluginProxy pProxy = (PluginProxy)dom.CreateInstanceFromAndUnwrap(Assembly.GetExecutingAssembly().CodeBase, typeof(PluginProxy).FullName);
pProxy.Init(app.Apppath);
// I await the task later in code, because the user can cancel the execution
try { tExe = pProxy.Execute(panel, user.Username); }
catch (System.Runtime.Serialization.SerializationException e)
{
// runs always in this catch, even if no Exception from the plugin was thrown
}
catch (Exception e) { AddToErrorLog(panel.PanelName, e); }
finally
{
pProxy.Dispose();
AppDomain.Unload(dom);
}
Maybe my whole concept of loading Plugins is wrong?

Thanks to Hamlet Hakobyan and the post from Stephen Toub, I think I was able to solve the problem.
I replaced the line from the caller
try { tExe = pProxy.Execute(panel, user.Username); }
with
tExe = DoWorkInOtherDomain(pProxy, panel, user.Username);
and the method DoWorkInOtherDomain:
private Task DoWorkInOtherDomain(PluginProxy pProxy, PluginPanel panel, string user)
{
var ch = new MarshaledResultSetter<string>();
pProxy.Execute(panel, user, ch);
return ch.Task;
}
and finally the proxy class:
Task.Run(() =>
{
try
{
m_Plugin.Execute(panel, user).Wait();
}
catch (AggregateException e)
{
if (e.InnerExceptions != null)
foreach (Exception ein in e.InnerExceptions)
AddToErrorLog(panel.PanelName, ein);
}
catch (Exception e) { AddToErrorLog(panel.PanelName, e); }
finally { ch.SetResult(AppDomain.CurrentDomain.FriendlyName); }
});
I need to call Wait() in
m_Plugin.Execute(panel, user).Wait();
it catches the Exceptions from the plugin so everything is doing fine. The Wait() call should only blocking the Task.Run and not the other Tasks.
Can anyone tell me if this is a good solution or should I change something? I dont need a result so I just do:
ch.SetResult(AppDomain.CurrentDomain.FriendlyName);
because I dont know how I should do it without a result.

Good practice design pattern for Exception handling

I have exception handling code in every method for the below code for the bottom level methods
throw new Exception("The error that happens");
Is there any way I can avoid writing this code again and again in each method?
I am trying to write my own code and not using any log frameworks
private void TopLevelMethod()
{
try
{
SomeMethod();
}
catch (Exception ex)
{
// Log/report exception/display to user etc.
}
}
private void SomeMethod()
{
TestPartA();
TestPartB();
TestPartC();
TestPartD();
}
private void TestPartA()
{
// Do some testing...
try
{
if (somethingBadHappens)
{
throw new Exception("The error that happens");
}
}
catch (Exception)
{
// Cleanup here. If no cleanup is possible,
// do not catch the exception here, i.e.,
// try...catch would not be necessary in this method.
// Re-throw the original exception.
throw;
}
}
private void TestPartB()
{
// No need for try...catch because we can't do any cleanup for this method.
if (somethingshappens)
{
throw new Exception("The error that happens");
}
}

Only catch errors if you want to do something meaningful to them such as:
Wrapping the exception with a framework exception (e.g. SqlException. ADO.NET never passes you socket-level errors. It passes you a meaningful SQL error code)
Cleanup
Actually responding (e.g. retry, or insert default values)
Logging is almost never appropriate. The top level handler should log. Certainly not every method in the path should log. What a clutter for logs and for the code. Don't do that.
Simply don't swallow error information and let the error bubble out. That way there is no reason left to insert local logging code for errors everywhere.

If you prefer using Functional Programming like code style one way is to use callback error callbacks.
Example :
private void SomeMethod()
{
// do something
}
public bool Execute(Action act, Action<Exception> onErrorCallback)
{
var res = true;
try
{
act();
}
catch (Exception ex)
{
res = false;
onErrorCallback(ex);
}
return res;
}
And use Execute like this:
var successfull = true;
successfull &= Execute(SomeMethod, (ex) => { /* clean up */ });
successfull &= Execute(SomeMethod, (ex) => { /* clean up */ });
successfull &= Execute(SomeMethod, (ex) => { /* clean up */ });
successfull &= Execute(SomeMethod, (ex) => { /* clean up */ });
if (!successfull)
; // show user or something else

Graffito: Would you please give sample code example.Thankyou...
Your code refactored:
private void TopLevelMethod()
{
List<string> errors=new List<string>() ;
if (!SomeMethod(errors)) { /* Log/report errors/display to user etc. */ }
}
private bool SomeMethod(List<string> errors)
{
return TestPartA(errors) && TestPartB(errors) && TestPartC(errors) && TestPartD(errors);
}
private bool TestPartA(List<string> errors)
{
bool result = true ;
try
{
// Do some testing...
if (somethingBadHappens) { result=false; errors.Add("The error that happens"); }
}
catch (Exception ex) { errors.Add("Error in TestPartA: "+Ex.Exception.Message.ToString()) ; }
return result ;
}
private bool TestPartB(List<string> errors)
{
bool result = true ;
// Do some testing...
if (somethingBadHappens) { result = false ; errors.Add("The error that happens"); }
return result ;
}