I have a console application as my webjob to process notifications inside my application. The processes are triggered using a queue. The application interacts with a SQL Azure database using entity framework 6. The Process() method that's being called reads/write data to the database.
I'm getting several errors when the queue messages are processed. They never get to the poison queue since they are reprocessed successfully after 2-3 times. Mainly the errors are the following:
An unhandled exception of type 'System.StackOverflowException' occurred in mscorlib.dll
Error: System.OutOfMemoryException: Exception of type ‘System.OutOfMemoryException’ was thrown.
The default batch size is 16, so the messages are processed in parallel.
My guess is that the Ninject setup for processing messages in parallel is wrong. Therefore, when they are processed at the same time, there are some errors and eventually they are processed successfully.
My question is: Does this setup look ok? Should I use InThreadScope() maybe since I don't know the parallel processing is also multi-threaded.
Here's the code for my application.
Program.cs
namespace NotificationsProcessor
{
public class Program
{
private static StandardKernel _kernel;
private static void Main(string[] args)
{
var module = new CustomModule();
var kernel = new StandardKernel(module);
_kernel = kernel;
var config =
new JobHostConfiguration(AzureStorageAccount.ConnectionString)
{
NameResolver = new QueueNameResolver()
};
var host = new JobHost(config);
//config.Queues.BatchSize = 1; //Process messages in parallel
host.RunAndBlock();
}
public static void ProcessNotification([QueueTrigger("%notificationsQueueKey%")] string item)
{
var n = _kernel.Get<INotifications>();
n.Process(item);
}
public static void ProcessPoison([QueueTrigger("%notificationsQueueKeyPoison%")] string item)
{
//Process poison message.
}
}
}
Here's the code for Ninject's CustomModule
namespace NotificationsProcessor.NinjectFiles
{
public class CustomModule : NinjectModule
{
public override void Load()
{
Bind<IDbContext>().To<DataContext>(); //EF datacontext
Bind<INotifications>().To<NotificationsService>();
Bind<IEmails>().To<EmailsService>();
Bind<ISms>().ToSmsService>();
}
}
}
Code for process method.
public void ProcessMessage(string message)
{
try
{
var notificationQueueMessage = JsonConvert.DeserializeObject<NotificationQueueMessage>(message);
//Grab message and check if it has to be processed
var notification = _context.Set().Find(notificationQueueMessage.NotificationId);
if (notification != null)
{
if (notification.NotificationType == NotificationType.AppointmentReminder.ToString())
{
notificationSuccess = SendAppointmentReminderEmail(notification); //Code that sends email using the SendGrid Api
}
}
}
catch (Exception ex)
{
_logger.LogError(ex + Environment.NewLine + message, LogSources.EmailsService);
throw;
}
}
Update - Added Exception
The exception is being thrown at the Json serializer. Here's the stack trace:
Error: System.OutOfMemoryException: Exception of type ‘System.OutOfMemoryException’ was thrown.
at System.String.CtorCharCount(Char c, Int32 count) at Newtonsoft.Json.JsonTextWriter.WriteIndent() at Newtonsoft.Json.JsonWriter.AutoCompleteClose(JsonContainerType type) at Newtonsoft.Json.JsonWriter.WriteEndObject() at Newtonsoft.Json.JsonWriter.WriteEnd(JsonContainerType type) at Newtonsoft.Json.JsonWriter.WriteEnd() at Newtonsoft.Json.JsonWriter.AutoCompleteAll() at Newtonsoft.Json.JsonTextWriter.Close() at Newtonsoft.Json.JsonWriter.System.IDisposable.Dispose() at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer) at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, Formatting formatting, JsonSerializerSettings settings) at Core.Services.Communications.EmailsService.SendAppointmentReminderEmail(Notificaciones email) in c:\ProjectsGreenLight\EAS\EAS\EAS\Core\Services\Communications\EmailsService.cs:line 489 at Core.Services.Communications.EmailsService.ProcessMessage(String message) in c:\ProjectsGreenLight\EAS\EAS\EAS\Core\Services\Communications\EmailsService.cs:line 124 at Core.Services.NotificacionesService.Process(String message) in c:\ProjectsGreenLight\EAS\EAS\EAS\Core\Services\NotificacionesService.cs:line 56
Since you're receiving OutOfMemoryExceptions and StackOverflowExceptions, i suggest that there may be a recursive or deeply nested method. It would be extremely helpful if you'd have a stacktrace to the exception, sadly that's not the case for StackOverflowExceptions. However, OutOfMemoryException has a stack trace, so you need to log this and see what it says / add it to your question. Also, as far as the StackoverflowException goes, you can also try this.
Scoping
You should not use .InThreadScope() in such a scenario. Why? usually the thread-pool is used. Thread's are reused. That means, that scoped objects live longer than for the processing of a single messages.
Currently you are using .InTransientScope() (that's the default if you don't specify anything else). Since you are using the IDbContext in only one place, that's ok. If you'd wanted to share the same instance across multiple objects, then you'd have to use a scope or pass along the instance manually.
Now what may be problematic in your case is that you may be creating a lot of new IDbContexts but not disposing of them after you've used them. Depending on other factors this may result in the garbage collector taking longer to clean-up memory. See Do i have to call dispose on DbContext.
However i grant that this won't fix your issue. It might just speed up your application.
Here's how you can do it anyway:
What you should do is: Have INotifications derive from IDisposable:
public interface INotifications : IDisposable
{
(...)
}
internal class Notifications : INotifications
{
private readonly IDbContext _context;
public Notifications(IDbContext context)
{
_context = context;
}
(...)
public void Dispose()
{
_context.Dispose();
}
}
and alter your calling code to dispose of INotifications:
public static void ProcessNotification([QueueTrigger("%notificationsQueueKey%")] string item)
{
using(var n = _kernel.Get<INotifications>())
{
n.Process(item);
}
}
Related
I have this static method, which is called somewhere on startup of every WebApi method (using MVC core, WebApi project):
private static object syncObj = new object();
private static string myProp;
public static string MyStaticMethod()
{
lock (syncObj)
{
if (myProp == null)
{
myProp = Utils.GetMyPropFromRegistry();
if (string.IsNullOrWhiteSpace(myProp))
{
throw new MyException("myProp value not set");
}
}
}
return myProp;
}
When the application starts and the first time this method is called, I receive an exception, since I do not have a proper value written in the registry (a call from Utils.GetMyPropFromRegistry returns null). This is all OK. But, if I add value to the registry, while the application is still running (in debug mode), this method does not get called ever again. It actually returns the same exception message through WebApi as the first time, just debugger does not stop on any breakpoint (which it did on application startup). If I examine the stack trace, it shows correctly where this exception occurred, it just doesn't stop there and it doesn't read the new value from the registry.
It seems like C# (or ASP.NET Core), somehow remembers that method throws an exception and it doesn't even execute code?
Is there anything special about static methods that throw an exception on startup? Or maybe it has something to do with the lock statement? I've used static methods before, but never run into such an issue.
Places where this is called:
// 1: Startup.cs
public void ConfigureServices(IServiceCollection services)
{
services
.AddAuthentication(ConfigureJwtBearerAuthenticationScheme)
.AddJwtBearer(options =>
{
var data = MyStaticMethod();
// Do something with data
});
}
// 2: WebApiController.cs:
public async Task<ActionResult<LoginResult>> Login(LoginModel model)
{
var data = MyStaticMethod();
var result = (do something with data)
return result;
}
I'm trying to centrally manage Exception handling for a certain method but I can't seem to get there.
public class ExceptionInterceptor : IInterceptor
{
private readonly Logger _logger;
public ExceptionInterceptor(Logger logger)
{
_logger = logger;
Measure.Configure(new StatsdConfig());
}
public void Intercept(IInvocation invocation)
{
try
{
invocation.Proceed();
//if ((Task<System.Threading.Tasks.VoidTaskReturn>) invocation.ReturnValue.Status == "Failed")
//{
// throw new Exception(invocation.ReturnValue.Exception[0]);
//}
}
catch (Exception e)
{
var errorMessage =
String.Format(
"An error occurred while retrieving fund data. Error Message: {0} Inner Exception: {1}",
e.Message, e.InnerException != null ? e.InnerException.Message : "<None>");
_logger.Log(errorMessage);
Measure.Counter("Exception", 1);
Measure.Event("Exception", errorMessage);
throw;
}
}
I'm wiring this up in a module like so:
builder.RegisterType<DataConsumer>().
As<IConsumer<DataRequest>>().
EnableInterfaceInterceptors().
InterceptedBy(typeof(ExceptionInterceptor));
builder.RegisterType<ExceptionInterceptor>().AsSelf();
var loggingInterceptor = new LoggingInterceptor(Logger);
builder.Register(c => loggingInterceptor);
However when I throw an exception in the method invocation this doesn't bubble up to the interceptor as an exception thrown, so it never enters the catch block.
Is there any way to catch the intercepted method's exception in the interceptor?
I also can't access the invocation.ReturnValue.Status for some reason and as such can't test whether there was a thrown exception in order to re-throw.
Can anyone shed some light into what I might nor be doing right here?
Ta
I'm having a difficult time reproducing your issue due to a bit of incomplete information. For example, you noted that the IConsumer<T> interface is a MassTransit interface, but the interface in MassTransit isn't generic. It also specifically mentions that the interface is supposed to be a marker just for IoC containers, which may have some implications on your wire-up.
First, let's post a working exception handling example. To be self-contained, I'll create an IWorker<T> interface in place of IConsumer<T> and a simple implementation:
public interface IWorker<T>
{
bool DoWork(T message);
}
public class StringWorker : IWorker<string>
{
public bool DoWork(string message)
{
throw new DivideByZeroException();
}
}
Now I'll create a simple exception logger that just pipes info to the console.
public class ExceptionLogger : IInterceptor
{
private readonly TextWriter _output;
public ExceptionLogger(TextWriter output)
{
_output = output;
}
public void Intercept(IInvocation invocation)
{
try
{
invocation.Proceed();
}
catch(Exception ex)
{
_output.WriteLine("Logged Exception: {0}", ex.Message);
throw;
}
}
}
I can then wire it up and see it in action like this:
var builder = new ContainerBuilder();
builder.RegisterInstance(Console.Out).As<TextWriter>();
builder.RegisterType<ExceptionLogger>();
builder.RegisterType<StringWorker>()
.As<IWorker<string>>()
.EnableInterfaceInterceptors()
.InterceptedBy(typeof(ExceptionLogger));
var container = builder.Build();
var worker = container.Resolve<IWorker<string>>();
worker.DoWork("Test!");
When I run this, I see on the console just before the program dies (with the unhandled exception - note my handler didn't swallow it, just logged it):
Logged Exception: Attempted to divide by zero.
So it's working.
I think there's more in your environment that may be causing trouble here. It could be something you think is unrelated but is actually important.
General things to investigate:
Temporarily update your DataConsumer to throw an exception immediately inside one of the interface methods. After building your container, resolve a, IConsumer<DataRequest> and call that interface method. Does it get logged?
Look at the places that you're expecting to see logging occur. Are you resolving and working with IConsumer<DataRequest> or something else? It's wrapping the interface methods, not the object type, so not all methods are covered.
Set a breakpoint in the interceptor and see if any invocation at all is passing through it. It won't catch exceptions if it's not being hit. :)
Check to see if there's any other exception handling policies or code in action. For example, some folks use the Enterprise Library exception handling block to deal with exceptions and that may be interfering with your work here.
I've not used MassTransit, but check to see if there's any other object proxying going on. (Doubtful, but I know I've run into this with products like Glimpse, so you end up with proxies wrapped around proxies, which becomes challenging.)
Is the exception actually happening in the place you think it is? It may be happening and getting handled somewhere that isn't wrapped with the proxy.
Basically, reduce the working pieces to the smallest set possible until you can see it working, then slowly expand until you find the place it breaks down. I don't know if any of these apply to your situation, but these are the things I'd start looking at if I was troubleshooting.
But... exception handling in an AOP fashion using interceptors does work, so it's something else going on that's causing the challenge.
It seems that it isn't possible for the exception thrown in the target proxy to be pushed up to the interceptor, and as such what I was trying to do didn't work. I ended up handling the exceptions at the class they occur.
Disappointed I didn't manage to get this to work the way I was intending to.
I'm using NServiceBus 4.6 and Serilog. I've configured NServiceBus to use Serilog via:
global::NServiceBus.SetLoggingLibrary.Custom(new SeriLoggerFactory());
The factory itself is very simple as well:
public class SeriLoggerFactory : ILoggerFactory
{
public ILog GetLogger(Type type)
{
return new SeriLoggerAdapter(Log.ForContext(type));
}
public ILog GetLogger(string name)
{
var contextLogger = Log.ForContext("SourceContext", name);
return new SeriLoggerAdapter(contextLogger);
}
}
I'm definitely getting log entries related to NServiceBus, but one thing that's missing is the exception details when a message is processed but an exception is thrown. I can see the exception information in the NServiceBus message headers (either directly by viewing the message in the error queue, or via Service Insight), but the message that is logged by NServiceBus is missing most relevant information:
Message with '0d255d19-85f9-4915-a27c-a41000da12ed' id has failed FLR
and will be handed over to SLR for retry attempt 1
or
SLR has failed to resolve the issue with message
0d255d19-85f9-4915-a27c-a41000da12ed and will be forwarded to the
error queue at MYERRORQUEUE
Not having any details about the root exception makes debugging a bit difficult. It requires the developer go open up Service Insight, or open up a tool to view the message in the queue itself. Both are cumbersome, and both lack any extensiblity.
For instance, Serilog allows you to create ILogEventEnricher classes that can log special details about the exception in question - stuff that's not logged by a simple .ToString on the exception. Without NServiceBus actually logging my exceptions, I have no way of extracting these details.
What am I missing here?
NServiceBus has a class named NServiceBus.Faults.ErrorsNotifications which contains the following observables:
MessageSentToErrorQueue
MessageHasFailedAFirstLevelRetryAttempt
MessageHasBeenSentToSecondLevelRetries
You can subscribe to these observables when the endpoint starts, like in the following example which logs an error messeages are sent to first level retry:
public class GlobalErrorHandler : IWantToRunWhenBusStartsAndStops
{
private readonly ILogger _logger;
private readonly BusNotifications _busNotifications;
readonly List<IDisposable> _notificationSubscriptions = new List<IDisposable>();
public GlobalErrorHandler(ILogger logger, BusNotifications busNotifications)
{
_logger = logger;
_busNotifications = busNotifications;
}
public void Start()
{
_notificationSubscriptions.Add(_busNotifications.Errors.MessageHasFailedAFirstLevelRetryAttempt.Subscribe(LogWhenMessageSentToFirstLevelRetry));
}
public void Stop()
{
foreach (var subscription in _notificationSubscriptions)
{
subscription.Dispose();
}
}
private void LogWhenMessageSentToFirstLevelRetry(FirstLevelRetry message)
{
var properties = new
{
MessageType = message.Headers["NServiceBus.EnclosedMessageTypes"],
MessageId = message.Headers["NServiceBus.MessageId"],
OriginatingMachine = message.Headers["NServiceBus.OriginatingMachine"],
OriginatingEndpoint = message.Headers["NServiceBus.OriginatingEndpoint"],
ExceptionType = message.Headers["NServiceBus.ExceptionInfo.ExceptionType"],
ExceptionMessage = message.Headers["NServiceBus.ExceptionInfo.Message"],
ExceptionSource = message.Headers["NServiceBus.ExceptionInfo.Source"],
TimeSent = message.Headers["NServiceBus.TimeSent"]
};
_logger.Error("Message sent to first level retry. " + properties, message.Exception);
}
}
The observable is implemented by using Reactive Extensions, so you will have to install the NuGet package Rx-Core for this to work.
I am creating an application that consumes a SOAP web service in C#. I generated a proxy class for the web service WSDL using the svcutil tool.
I added the proxy class to my code and I am using it to make calls to the web service and get results asynchrounsly.
Everything works pretty fine when the client has an Internet access. However, if I run attempt to access while the application doesn't have Internet access it crashes raising the following exception:
An exception of type 'System.ServiceModel.EndpointNotFoundException' occurred in
System.ServiceModel.Internals.dll but was not handled in user code
I am trying to catch this exception to prevent the application from crashing and provide the user with a more friendly error message, However, since I am doing async web calls, simply surrounding the web service calls by a try- catch does not help.
According to the exception details it happens in the End_FunctionName function that is defined inside the auto-generated proxy file.
Any tips about how to be able to gracefully handle this exception ?
Its pretty difficult to know exactly what is happening; however, I'm going to assume you have a web service like such
[ServiceContract]
public interface IMyService
{
[OperationContract]
String Hello(String Name);
[OperationContract]
Person GetPerson();
}
You probably have a proxy like this :
public class MyPipeClient : IMyService, IDisposable
{
ChannelFactory<IMyService> myServiceFactory;
public MyPipeClient()
{
//This is likely where your culprit will be.
myServiceFactory = new ChannelFactory<IMyService>(new NetNamedPipeBinding(), new EndpointAddress(Constants.myPipeService + #"/" + Constants.myPipeServiceName));
}
public String Hello(String Name)
{
//But this is where you will get the exception
return myServiceFactory.CreateChannel().Hello(Name);
}
public Person GetPerson()
{
return myServiceFactory.CreateChannel().GetPerson();
}
public void Dispose()
{
((IDisposable)myServiceFactory).Dispose();
}
}
If you have an error connecting you will get it not when you try to connect to the channel factory but when you actually try to call a function.
To fix this problem, you can put a try catch around every single function call and handle async calls manually.
Conversely, you can have a function like init() that is called synchronously every single time you instantiate a connection. This way you know that if that call connects that you have a connection.
If you are at risk of a connection dropping at any time I advise you go with the former option.
Anyway here is an example of how you'd fix it:
public class MyPipeClient : IMyService, IDisposable
{
ChannelFactory<IMyService> myServiceFactory;
public MyPipeClient()
{
myServiceFactory = new ChannelFactory<IMyService>(new NetNamedPipeBinding(), new EndpointAddress(Constants.myPipeService + #"/" + Constants.myPipeServiceName + 2) );
}
public String Hello(String Name)
{
try
{
return Channel.Hello(Name);
}
catch
{
return String.Empty;
}
}
public Person GetPerson()
{
try
{
return Channel.GetPerson();
}
catch
{
return null;
}
}
public Task<Person> GetPersonAsync()
{
return new Task<Person>(()=> GetPerson());
}
public Task<String> HelloAsync(String Name)
{
return new Task<String>(()=> Hello(Name));
}
public void Dispose()
{
myServiceFactory.Close();
}
public IMyService Channel
{
get
{
return myServiceFactory.CreateChannel();
}
}
}
I uploaded the source I wrote so that you could download the full source. You can get it here : https://github.com/Aelphaeis/MyWcfPipeExample
PS : This Repository throws the exception you are getting. In order to remove it just go to MyPipeClient and remove the + 2 in the constructor.
If you are using a Duplex, Consider using this repository:
https://github.com/Aelphaeis/MyWcfDuplexPipeExample
I have an legacy System.Web.Services.WebService (not WCF) that I have to maintain.
Ocassionly I run into some wired behaviours that I would describe as race conditions.
Either the service hangs and has to be restarted.
Sometimes I get this exception:
System.NotSupportedException: Multiple simultaneous connections
or connections with different connection strings inside the same
transaction are not currently supported.
at MySql.Data.MySqlClient.MySqlConnection.Open()
...
I know whats the root cause. The service utilizes a lib that talks to mysql and was not designed with webservices in mind. Unfortunatly I cannot change this lib.
One example webmethod looks like this:
[WebMethod(EnableSession = true)]
public void DoSomething()
{
var login = this.Session["login"] as LoginDetails;
ExternalLib.SetLoginData(login.Schema, login.User, login.Pass);
ExternalLib.PerformTask();
}
So the problem here is this:
ExternalLib.SetLoginData just set's some global vars
ExternalLib.PerformTask performs database calls, some inside a transaction.
The process is like 1. Create MySqlConnection or take it from cache 2. Create MySqlCommand 3. Execute Command 4. Dispose command
Client a) calls DoSomething() and I init his connection. Half way done with his job Client b) calls DoSomething() which apparently changes the Login-Data for client a and the next call inside the transaction will use the login from client b) which causes the transaction.
Anyway, I know this is a bad design but my question is how to workaround this.
Currently (since I only have 10 clients) I created a dedicated Website on a differnet port which all point to the same root directory but this is an akward solution.
Maybe there is a possibility to run every session inside its on realm. Any suggestions. If I understand this page correctly for WCF is is the default behaviour: http://msdn.microsoft.com/en-us/magazine/cc163590.aspx
Per-Call Services
Per-call services are the Windows Communication
Foundation default instantiation mode. When the service type is
configured for per-call activation, a service instance, a common
language runtime (CLR) object, exists only while a client call is in
progress. Every client request gets a new dedicated service instance.
Seeing as this is probably a threading issue you can lock the ExternalLib to prevent separate instances from calling the code.
public class ExtenalLibWrapper
{
private static object Locker = new object();
public void DoSomething(LoginDetails details)
{
lock(Locker)
{
ExternalLib.SetLoginData(login.Schema, login.User, login.pass);
ExternalLib.PerformTask();
}
}
}
I already wrapped all my public methods in a neat execute wrapper to provide global exception logging.
This forces my webservice to process one request after another, but like I mentioned, the max. number of simultanious clients is 10
public class MyService : System.Web.Services.WebService
{
[WebMethod(EnableSession = true)]
public static int Add(int value1, int value2)
{
return Execute(() =>
{
var calculator = new Calculator();
return calculator.Add(value1, value2);
});
}
private static Logger logger =
LogManager.GetLogger(typeof(MyService).Name);
private static System.Threading.SemaphoreSlim ss =
new System.Threading.SemaphoreSlim(1, 1);
private void Execute(Action method)
{
ss.Wait();
try { method.Invoke(); }
catch (Exception ex)
{
logger.FatalException(method.Method + " failed", ex); throw;
}
finally { ss.Release(); }
}
private T Execute<T>(Func<T> method)
{
ss.Wait();
try { return method.Invoke(); }
catch (Exception ex)
{
logger.FatalException(method.Method + " failed", ex); throw;
}
finally
{
ss.Release();
}
}
}