Develop Reference

Singleton saga in MassTransit

What I'm trying to do is to create only one saga instance, not updated by subsequent events. So for example if I send EventA at 09:00, with CorrelationId equal to 979158a2-dfa0-45f0-9107-8b8b8028bb9f and then the saga for that event is still alive at 10:00, then if at 10:00 I send another instance of EventA with the same CorrelationId that was sent previously, I don't want it to overwrite the previous saga state but instead I would like this event to be ignored by MT. I kind of achieved that, because my saga state is not overriden (only its version is incremented), but the problem is that MT throws an error when the new event arrives. The exception is (+one log entry above it):
[18:41:12 DBG] SAGA:App.Services.Communicators.Amazon.Dispatcher.Api.Sagas.GetOrdersState:0368a723-a819-450f-b6e5-9211d1d6a3a9 Dupe App.Services.Financial.Contracts.GetOrdersCommand
MongoDB.Driver.MongoWriteException: A write operation resulted in an error.
E11000 duplicate key error collection: Dispatcher.get.orders.states index: _id_ dup key: { _id: BinData(3, 23A7680319A80F45B6E59211D1D6A3A9) }
---> MongoDB.Driver.MongoBulkWriteException`1[App.Services.Communicators.Amazon.Dispatcher.Api.Sagas.GetOrdersState]: A bulk write operation resulted in one or more errors.
E11000 duplicate key error collection: Dispatcher.get.orders.states index: _id_ dup key: { _id: BinData(3, 23A7680319A80F45B6E59211D1D6A3A9) }
at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionBase`1.InsertOneAsync(TDocument document, InsertOneOptions options, Func`3 bulkWriteAsync)
--- End of inner exception stack trace ---
at MongoDB.Driver.MongoCollectionBase`1.InsertOneAsync(TDocument document, InsertOneOptions options, Func`3 bulkWriteAsync)
at MassTransit.MongoDbIntegration.Saga.Context.MongoDbSagaRepositoryContext`2.Insert(TSaga instance)
And the important part of the saga config:
Event(() => GetOrdersIntegrationEventReceived, e =>
e
.CorrelateById(x => x.Message.CorrelationId)
.SelectId(x => x.Message.CorrelationId)
.SetSagaFactory(ctx => new GetOrdersState
{
CorrelationId = ctx.Message.CorrelationId,
PartnerId = ctx.Message.PartnerId,
LastUpdatedBeforeFromRequest = ctx.Message.LastUpdatedBefore
})
.InsertOnInitial = true);
Is what I'm trying to do even possible?

MassTransit will only create one instance of the saga for each CorrelationId, so your reasoning is correct. However, you're approach is a little off and could use some tweaking.
For instance, your event configuration:
Event(() => GetOrdersIntegrationEventReceived, e => e.CorrelateById(x => x.Message.CorrelationId));
That's it, all the extra stuff is why you're seeing messages in the debug log.
Then, after the event is declared:
During(Initial, Created,
When(GetOrdersIntegrationEventReceived)
.Then(context =>
{
context.Instance.PartnerId = context.Message.PartnerId,
context.Instance.LastUpdatedBeforeFromRequest = context.Message.LastUpdatedBefore
})
.TransitionTo(Created)
);
That's it.

Azure Service Bus: AMQP object is closing. Operation 'attach' cannot be performed

I'm using Azure Service Bus Topics with the AMQP protocol in the West Europe datacenter.
This is a schematic way of how the solution implented works:
private Microsoft.Azure.ServiceBus.SubscriptionClient CreateClient() {
string serviceBusConnectionString;
strin serviceBusTopicName;
string subscriptionName;
var subscriptionClient = new Microsoft.Azure.ServiceBus.SubscriptionClient(serviceBusConnectionString, serviceBusTopicName, subscriptionName) {
PrefetchCount = 0
};
return subscriptionClient;
}
public async Task<ISubscriptionClient> CreateSubscriptionClientAsync() {
//Some logic on the subscriptionClient, caching, creating a new one if it doesn't exists, etc.
return CreateClient()
}
private async Task CallbackAsync(Message msg, CancellationToken cancellationToken) {
//Do stuff with the Message
//when you're done Completethe message
}
public async Task<string> OpenAsync(CancellationToken cancellationToken) {
subscriptionClient = await CreateSubscriptionClientAsync().ConfigureAwait(false);
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandlerAsync) {
AutoComplete = false,
MaxAutoRenewDuration = TimeSpan.FromHours(8)
};
subscriptionClient.RegisterMessageHandler(CallbackAsync, messageHandlerOptions);
return string.Empty;
}
But last night I had couple of thousand exception like this one:
Exception: "System.InvalidOperationException: The AMQP object g2b-sessionXXXXXXXX is closing. Operation 'attach' cannot be performed.
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<OnReceiveAsync>d__86.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass64_0.<<ReceiveAsync>b__0>d.MoveNext()
I'm using the Microsoft.Azure.ServiceBus v3.1.0.0,
On line I didn't find anything useful about this "Operation 'attach' cannot be performed".
On the message bus exception list page I didn't find any information about this specific problem. And in the status history site there is no reference about any outage involving Service Bus.
Did anyone experience this problem before?
What is causing the exception?
Do I need to implement any retry logic? How?
Any help is apreciated, thanks.

Lambda encountered an UnobservedTaskException - Cannot access a disposed object

I've been noticing this exception in the CloudWatch logs for an AWS Lambda.
Everything seems to get processed so I think it is an exception within the AWS code (as opposed to something I've written) that is created after the Lambda has finished executing.
Since functionally it works I've been ignoring it but I'm concerned that there might be problems that I haven't noticed.
Lambda encountered an UnobservedTaskException via
'TaskScheduler.UnobservedTaskException' event:
{
"errorType": "AggregateException",
"errorMessage": "A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Cannot access a disposed object.\nObject name: 'System.Net.Sockets.UdpClient'.)",
"cause": {
"errorType": "ObjectDisposedException",
"errorMessage": "Cannot access a disposed object.\nObject name: 'System.Net.Sockets.UdpClient'.",
"stackTrace": [
"at System.Net.Sockets.UdpClient.EndReceive(IAsyncResult asyncResult, IPEndPoint& remoteEP)",
"at System.Net.Sockets.UdpClient.<>c.<ReceiveAsync>b__56_1(IAsyncResult asyncResult)",
"at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)"
]
},
"causes": [ {
"errorType": "ObjectDisposedException",
"errorMessage": "Cannot access a disposed object.\nObject name: 'System.Net.Sockets.UdpClient'.",
"stackTrace": [
"at System.Net.Sockets.UdpClient.EndReceive(IAsyncResult asyncResult, IPEndPoint& remoteEP)",
"at System.Net.Sockets.UdpClient.<>c.<ReceiveAsync>b__56_1(IAsyncResult asyncResult)",
"at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)"
]
}]
}
The lambda code is pretty simple: It adds these SNS messages into an SQL database using Dapper.
I'm thinking that there might be some problem with how I'm doing async in the Fucntion handler. Any ideas?
public class Function
{
private static string _connectionString;
public async Task<IEnumerable<InsertSnsResult>> FunctionHandler(SNSEvent #event, ILambdaContext context)
{
try
{
context.Logger.LogLine("Adding SNS Messages");
_connectionString = _connectionString ?? await DecryptHelper.DecryptEnvironmentVariableAsync("ConnectionString").ConfigureAwait(false);
var handler = new AddSnsMessageHandler(new SnsMessagesRepository(_connectionString, context.Logger));
return await handler.AddSnsEvents(#event).ConfigureAwait(false);
}
catch (Exception e)
{
context.Logger.LogLine(e.Message);
throw;
}
finally
{
context.Logger.LogLine("Finished SNS Adding Messages");
}
}
}
[Edit]
Just be clear here, this exception does not get caught in the try/catch block. If it did it wouldn't be an UnobservedTaskException. That's why I'm having trouble getting to the root of the problem.
And this is the repository code
public async Task<List<InsertSnsResult>> InsertSnsMessages(IEnumerable<SnsEvent> records)
{
using (var connection = new SqlConnection(_connectionString))
{
await connection.OpenAsync().ConfigureAwait(false);
var results = new List<InsertSnsResult>();
foreach (var record in records)
{
try
{
await connection.ExecuteAsync(InsertEventCommand, record).ConfigureAwait(false);
results.Add(new InsertSnsResult(record.CorrelationId, true));
}
catch (Exception ex)
{
_logger.LogLine($"InsertSns failed for {record.Id}. {ex.Message}");
results.Add(new InsertSnsResult(record.CorrelationId, false));
}
}
return results;
}
}

The log messages are straightforward and explain what is happening:
You have an asynchronous task
That asynchronous task is accessing an object that has already been disposed, probably because you have some race condition in your workflow whereby an object in the asynchronous workflow is disposed out-of-order with another portion of the workflow that needs it. This means something is seriously broken in this workflow.
The asynchronous task is never waited on, either asynchronously with await, or (don't do this!) synchronously with Result or Wait. That means that the exceptional continuation path is never taken, and the task notices this when it is collected. Again, something is probably seriously broken in your workflow if you have a task that you never wait for the result. Combine this fact with the fact from the previous point: we now have two pieces of evidence that reinforce each other that there is something seriously broken in this workflow and that it involves a task that is not awaited when it should be to ensure an ordering constraint.
And therefore you get an exception on your finalizer thread, which is really bad.
Since functionally it works I've been ignoring it
I heard once that when a factory catches fire and burns to the ground, on average there were seven different safety systems that people ignored or disabled. Get out of this habit of thinking that it works, so it must be safe. Maybe it's nothing but I would consider these messages to be indicative of a serious problem until I had evidence otherwise.

I too have run into a place where I have a 3rd party library causing errors. I wanted log it outside of CloudWatch. In order to prevent Lambda from logging these I was able to do some evil reflection to reset the event handler.
Here is the code to do this yourself. Please note this is evil code. It is fragile and will break when code is changed in the CLR or even if the compiler does optimizations (which recently happened). However, it was the only way I could find to opt-out of this functionality provided by Lambda
private void ReplaceLambdaDefaultUnobservedTaskException()
{
try
{
var bindingFlags = BindingFlags.NonPublic | BindingFlags.Static;
Type type = typeof(TaskScheduler);
var field = type.GetField("_unobservedTaskException", bindingFlags);
if (field == null)
{
field = type.GetField("UnobservedTaskException", bindingFlags);
}
var handler = new EventHandler<UnobservedTaskExceptionEventArgs>(TaskSchedulerOnUnobservedTaskException);
field.SetValue(null, handler);
}
catch (Exception ex)
{
logger.Warning(ex, "Unable to do evil reflection.");
}
TaskScheduler.UnobservedTaskException += TaskSchedulerOnUnobservedTaskException;
}
private void TaskSchedulerOnUnobservedTaskException(object sender, UnobservedTaskExceptionEventArgs e)
{
e.SetObserved();
logger.Error(e.Exception, "Lambda threw an UnobservedTaskException");
}

IIS is throwing an HTML page error saying "An asynchronous operation cannot be started ..."

I have found a couple examples of this error occurring in the forums but I don't apparently have the technical knowledge to make my case be truly 'async all the way' - if this is even the problem.
The error when caught is:
An asynchronous operation cannot be started at this time. Asynchronous
operations may only be started within an asynchronous handler or
module or during certain events in the Page lifecycle. If this
exception occurred while executing a Page, ensure that the Page is
marked <%# Page Async="true" %>. This exception may also indicate an
attempt to call an "async void" method, which is generally unsupported
within ASP.NET request processing. Instead, the asynchronous method
should return a Task, and the caller should await it.
at System.Web.AspNetSynchronizationContext.OperationStarted() at
System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Create() at
myCompany.System.Turhh.Turhh.<>c__DisplayClass18_0.b__1(Object
sender, GetInfoCompletedEventArgs e) at
myCompany.System.TurhhWebServices.InfoServiceClient.OnGetInfoCompleted(Object
state)
The code:
public void GetInfoAsync(GetInfoRq request, object Infotate)
{
if ((this.onBeginGetInfoDelegate == null))
{
this.onBeginGetInfoDelegate = new BeginOperationDelegate(this.OnBeginGetInfo);
}
if ((this.onEndGetInfoDelegate == null))
{
this.onEndGetInfoDelegate = new EndOperationDelegate(this.OnEndGetInfo);
}
if ((this.onGetInfoCompletedDelegate == null))
{
this.onGetInfoCompletedDelegate = new System.Threading.SendOrPostCallback(this.OnGetInfoCompleted);
}
base.InvokeAsync(this.onBeginGetInfoDelegate, new object[] {
request}, this.onEndGetInfoDelegate, this.onGetInfoCompletedDelegate, Infotate);
}
private BeginOperationDelegate onBeginGetInfoDelegate;
private EndOperationDelegate onEndGetInfoDelegate;
private System.Threading.SendOrPostCallback onGetInfoCompletedDelegate;
public event System.EventHandler<GetInfoCompletedEventArgs> GetInfoCompleted;
public System.IAsyncResult BeginGetInfo(GetInfoRq request, System.AsyncCallback callback, object asyncState)
{
return base.Channel.BeginGetInfo(request, callback, asyncState);
}
public GetInfoRs EndGetInfo(System.IAsyncResult result)
{
return base.Channel.EndGetInfo(result);
}
private System.IAsyncResult OnBeginGetInfo(object[] inValues, System.AsyncCallback callback, object asyncState)
{
return this.BeginGetInfo(rq, callback, asyncState);
}
private object[] OnEndGetInfo(System.IAsyncResult result)
{
GetInfoRs retVal = this.EndGetInfo(result);
return new object[] {retVal};
}
private void OnGetInfoCompleted(object state)
{
if ((this.GetInfoCompleted != null))
{
InvokeAsyncCompletedEventArgs e = ((InvokeAsyncCompletedEventArgs)(state));
var ea = new GetInfoCompletedEventArgs(e.Results, e.Error, e.Cancelled, e.Infotate);
try
{
// Note: Encapsulating this in a Task.Run() hides the issue on the server (here)
// but IIS still dumps its HTML page with an await/async error info.
this.GetInfoCompleted(this, ea);
}
catch (InvalidOperationException ex)
{
// HELP:
// Most of the time, this works fine. About 1 in 10 times, this exception is thrown.
// When this exception is thrown (and caught), IIS still dumps its HTML async/await error
System.Diagnostics.Debug.WriteLine(" **** AWAIT/ASYNC EXCEPTION 3 **** ");
}
}
}
This is the code behind the GetInfoCompleted:
channel.GetInfoCompleted += new EventHandler<GetInfoCompletedEventArgs>(async (sender, e) =>
{
await complete(); // complete contains an 'await Task.Run(async() => { some server work here });' job
});
This exception only occurs about 1 out of 10 times running locally. On production, it occurs almost every time. Essentially, on really fast servers, when the async calls happen synchronously rather than asynchronously, this throws the exception and irritates IIS causing it to dump the HTML error page down to the client. When these calls are made asynchronously, some time after IIS has gracefully given a good result to the client and the client moves on to its next task, everything is fine and this error-prone call on the server successfully accomplishes its mission (whether it is successful or not does not matter to the client - this is why it is threaded out as a post-operation task).

workitem.Save() storing multiple(duplicate) items into TFS

I am using below code to store workitem into TFS, but sometimes when TFS is unavailable/ or some exception, it is storing 4 items with same details into TFS. I am using below code.
private static void SaveWorkItemtoTFS(WorkItem wi, ref int retryCount)
{
try
{
retryCount++;
wi.Save();
}
catch (Exception ex)
{
if (retryCount <= 3)
{
Thread.Sleep(TimeSpan.FromSeconds(5));
wi.SyncToLatest();
SaveWorkItemtoTFS(wi, ref retryCount);
}
else
{
throw ex;
}
}
}
I just want to store unique items only. But it is storing 4 items with same details. Can you please help?
Exception is:
Technical information (for administrator):
The operation has timed out ---> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.SendRequestAndGetResponse(HttpWebRequest webRequest, WebException& webException)
--- End of inner exception stack trace ---
atTfsProj.BusinessLogic.SaveWorkItemtoTFS(WorkItem wi, Int32& retryCount)
atTfsProj.BusinessLogic.SaveWorkItemtoTFS(WorkItem wi, Int32& retryCount)
atTfsProj.BusinessLogic.SaveWorkItemtoTFS(WorkItem wi, Int32& retryCount)
atTfsProj.BusinessLogic.SaveWorkItemtoTFS(WorkItem wi, Int32& retryCount)
atTfsProj.BusinessLogic.AddNewCertificationTaskIntoTFS(RequestModel objRequestModel, String requestId, String clientReqId)`
this error only occurred once, unable to repro this as well.

According to the exception, the error occurs when get the response from TFS Server. When you create a workitem via Save() method, it will send the request to TFS Server first and then waiting for the response to check if the workitem is being created successfully. So in your question, the condition is that the create workitem request is sent successfully and workitem is created successfully but the code didn't get response from TFS Server which cause exception thrown and tried another three times.