I'm using Azure Service Bus Topics with the AMQP protocol in the West Europe datacenter.
This is a schematic way of how the solution implented works:
private Microsoft.Azure.ServiceBus.SubscriptionClient CreateClient() {
string serviceBusConnectionString;
strin serviceBusTopicName;
string subscriptionName;
var subscriptionClient = new Microsoft.Azure.ServiceBus.SubscriptionClient(serviceBusConnectionString, serviceBusTopicName, subscriptionName) {
PrefetchCount = 0
};
return subscriptionClient;
}
public async Task<ISubscriptionClient> CreateSubscriptionClientAsync() {
//Some logic on the subscriptionClient, caching, creating a new one if it doesn't exists, etc.
return CreateClient()
}
private async Task CallbackAsync(Message msg, CancellationToken cancellationToken) {
//Do stuff with the Message
//when you're done Completethe message
}
public async Task<string> OpenAsync(CancellationToken cancellationToken) {
subscriptionClient = await CreateSubscriptionClientAsync().ConfigureAwait(false);
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandlerAsync) {
AutoComplete = false,
MaxAutoRenewDuration = TimeSpan.FromHours(8)
};
subscriptionClient.RegisterMessageHandler(CallbackAsync, messageHandlerOptions);
return string.Empty;
}
But last night I had couple of thousand exception like this one:
Exception: "System.InvalidOperationException: The AMQP object g2b-sessionXXXXXXXX is closing. Operation 'attach' cannot be performed.
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<OnReceiveAsync>d__86.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass64_0.<<ReceiveAsync>b__0>d.MoveNext()
I'm using the Microsoft.Azure.ServiceBus v3.1.0.0,
On line I didn't find anything useful about this "Operation 'attach' cannot be performed".
On the message bus exception list page I didn't find any information about this specific problem. And in the status history site there is no reference about any outage involving Service Bus.
Did anyone experience this problem before?
What is causing the exception?
Do I need to implement any retry logic? How?
Any help is apreciated, thanks.
Related
I've got a dll that relies on a SOAP call to a web service, and I was provided with the WSDL file from the vendor, and after some testing, I've got code that will successfully call the service and get the response with no issue. However, when the vendor moved the service to a new server, my code starting failing.
Specifically, I get a NullRefernceException that, even though it is square in a try/catch block, ignores all of that and crashes anyway.
my call looks like this:
new ChannelFactory<GRChannel>(binding, address).Using(async(factory) =>
{
factory.Credentials.UserName.UserName = "TestUser";
factory.Credentials.UserName.Password = "1234";
var proxy = factory.CreateChannel();
proxy.Open();
var context = new OperationContext((IClientChannel)proxy);
var prevOpContext = OperationContext.Current;
OperationContext.Current = context;
try
{
var results = await proxy.ZBAPI_GOODSMVT_CREATEAsync(payload); //<-- Null happens here
System.Xml.Serialization.XmlSerializer serializer = new System.Xml.Serialization.XmlSerializer(typeof(ZBAPI_GOODSMVT_CREATEResponse));
using (StringWriter text = new StringWriter())
{
serializer.Serialize(text, results.ZBAPI_GOODSMVT_CREATEResponse);
Message = text.ToString();
}
}
catch
{
//Nothing here matters, we never make it to this point
}
});
the Stack Trace is this at the time of failure:
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannelProxy.TaskCreator.<>c__DisplayClass1_0.<CreateGenericTask>b__0(IAsyncResult asyncResult)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at RFCDLL.RFC.<>c__DisplayClass5_0.<<RunRequest>b__0>d.MoveNext() in C:\Users\User01\source\repos\RFCDLL\RFCDLL\RFC.cs:line 46
obviously, having a null failure is fine, and expected when the service isn't running, I just want to make sure the error handling works, and I can not have the whole app crash when that happens.
I am experimenting with a new NServiceBus project utilizing Azure Storage Queues for message transport and JSON serialization using custom message unwrapping logic seen here:
var jsonSerializer = new Newtonsoft.Json.JsonSerializer();
transportExtensions.UnwrapMessagesWith(cloudQueueMessage =>
{
using (var stream = new MemoryStream(cloudQueueMessage.AsBytes))
using (var streamReader = new StreamReader(stream))
using (var textReader = new JsonTextReader(streamReader))
{
try
{
var jObject = JObject.Load(textReader);
using (var jsonReader = jObject.CreateReader())
{
// Try deserialize to a NServiceBus envelope first
var wrapper = jsonSerializer.Deserialize<MessageWrapper>(jsonReader);
if (wrapper.MessageIntent != default)
{
// This was a envelope message
return wrapper;
}
}
// Otherwise this was an EventGrid event
using (var jsonReader = jObject.CreateReader())
{
var #event = jsonSerializer.Deserialize<EventGridEvent>(jsonReader);
var wrapper = new MessageWrapper
{
Id = #event.Id,
Headers = new Dictionary<string, string>
{
{ "NServiceBus.EnclosedMessageTypes", #event.EventType },
{ "NServiceBus.MessageIntent", "Publish" },
{ "EventGrid.topic", #event.Topic },
{ "EventGrid.subject", #event.Subject },
{ "EventGrid.eventTime", #event.EventTime.ToString("u") },
{ "EventGrid.dataVersion", #event.DataVersion },
{ "EventGrid.metadataVersion", #event.MetadataVersion },
},
Body = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(#event.Data)),
MessageIntent = MessageIntentEnum.Publish
};
return wrapper;
}
}
catch
{
logger.Error("Message deserialization failed, sending message to error queue");
throw;
}
}
});
The custom message unwrapping logic works correctly for properly formatted JSON messages and when an improperly formatted JSON message is put into the input queue the custom message unwrapping logic will error out on the first line inside the usings where I create the jObject which is the expected behavior. However, when the custom message unwrapping logic fails the error will get caught by the logic in the MessageRetrieved class which is part of the NServiceBus.Azure.Transports.WindowsAzureStorageQueues NuGet package (v8.2.0) seen below:
public async Task<MessageWrapper> Unwrap()
{
try
{
Logger.DebugFormat("Unwrapping message with native ID: '{0}'", rawMessage.Id);
return unwrapper.Unwrap(rawMessage);
}
catch (Exception ex)
{
await errorQueue.AddMessageAsync(rawMessage).ConfigureAwait(false);
await inputQueue.DeleteMessageAsync(rawMessage).ConfigureAwait(false);
throw new SerializationException($"Failed to deserialize message envelope for message with id {rawMessage.Id}. Make sure the configured serializer is used across all endpoints or configure the message wrapper serializer for this endpoint using the `SerializeMessageWrapperWith` extension on the transport configuration. Please refer to the Azure Storage Queue Transport configuration documentation for more details.", ex);
}
}
The first line of the try catch runs correctly adding the message to the configured error queue, however, when it does that, it appears to be changing the message ID and popreceipt of the raw message as seen here:
Initial Message Values
Updated Message Values
Then when the next line runs attempting to remove the original message from the input queue it is unable to find it as according to this article https://learn.microsoft.com/en-us/rest/api/storageservices/delete-message2#remarks it requires the original message ID and pop reciept which have now changed leading to the following error being thrown:
2020-04-20 14:17:58,603 WARN : Azure Storage Queue transport failed pushing a message through pipeline
Type: Microsoft.WindowsAzure.Storage.StorageException
Message: The remote server returned an error: (404) Not Found.
Source: Microsoft.WindowsAzure.Storage
StackTrace:
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.EndExecuteAsync[T](IAsyncResult result) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 50
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.<>c__DisplayClass7.<CreateCallbackVoid>b__5(IAsyncResult ar) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 121
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.AzureStorageQueues.MessageRetrieved.<Unwrap>d__3.MoveNext() in C:\BuildAgent\work\3c19e2a032c05076\src\Transport\MessageRetrieved.cs:line 40
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NServiceBus.Transport.AzureStorageQueues.MessagePump.<InnerReceive>d__7.MoveNext() in C:\BuildAgent\work\3c19e2a032c05076\src\Transport\MessagePump.cs:line 153
TargetSite: T EndExecuteAsync[T](System.IAsyncResult)
Is this an issue with the NServiceBus package logic, or is something in my custom message unwrapping logic causing these values to change?
This is a bug. When unwrapping is failing, the message is not yet going through the processing pipeline. As a result of that, the normal recoverability is not applicable. The CloudQueueMessage needs to be "cloned" and the clone to be sent to the error queue while the original message used to remove it from the input queue. I've raised a bug issue in GitHub and you can track the process there.
I have windows service that is basically reading messages from customer and we do process and sends them over HTTP/TCP/File etc. Service is written in C#, for database interaction I use NHibernate and TPL task.
For every batch of message service reads and inserts into SQL server database in separate TPL Task and same messages is then fetched from database in another TPL task we send them over HTTP/TCP/File after processing, and we also save this records into database using NHibernate ISession.
piece of code is below
public Func<ISession> GetSession { get; set; }
[Transaction]
public string SaveInMessage(ISession session, string message)
{
try
{
using (var session = GetSession()){
session.Transaction.Begin();
var inMessage = new InMessage();
var task = new Task(()=> InsertToDatabase(session, inMessage));
session.Transaction.Commit();
}
}
catch(Exception ex)
{
session.Transaction.Rollback();
}
}
public void InsertToDatabase(ISession session, InMessage inMessage){
session.SaveOrUpdate(inMessage);
}
[Transaction]
public bool SaveOutMessage()
{
try
{
using (var session = GetSession()){
session.Transaction.Begin();
var inMessage = session.Load<InMessage>();
var outMessage = new OutMessage();
var task = new Task(()=> ConvertMessage(outMessage, inMessage, session));
var task = new Task(()=> SendMessage(outMessage, session, outProtocol));
session.Transaction.Commit();
}
}
catch(Exception ex)
{
session.Transaction.Rollback();
}
}
public void ConvertMessage(OutMessage outMessage, ISession session, Http url)
{
conversion logic goes here;
}
public void SendMessage(OutMessage outMessage,ISession session,Protocol outProtocol)
{
Sending message logic goes here;
session.SaveOrUpdate(inMessage);
}
So, in above I have used Castle.Windsor for IoC and Transaction attribute used in these two methods are from same.
I am keep getting below AggregateException along with OutOfMemoryException within NHibernate and TPL.
System.AggregateException: One or more errors occurred. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.String.ConcatArray(String[] values, Int32 totalLength)
at System.String.Concat(Object[] args)
at NHibernate.Engine.Cascade.CascadeOn(IEntityPersister persister, Object parent, Object anything)
at NHibernate.Event.Default.AbstractFlushingEventListener.CascadeOnFlush(IEventSource session, IEntityPersister persister, Object key, Object anything)
at NHibernate.Event.Default.AbstractFlushingEventListener.PrepareEntityFlushes(IEventSource session)
at NHibernate.Event.Default.AbstractFlushingEventListener.FlushEverythingToExecutions(FlushEvent event)
at NHibernate.Event.Default.DefaultFlushEventListener.OnFlush(FlushEvent event)
at NHibernate.Impl.SessionImpl.Flush()
at NHibernate.Transaction.AdoTransaction.Commit()
at Service.ProcessMessage(Message message, ISession session) in C:\Project\Service\ProcessMessage.cs:line 247
at Service.ProcessMessage.<>c__DisplayClass22_0.<SendMessages>b__0(Task c) in C:\Project\Service\ProcessMessage.cs:line 74
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(CancellationToken cancellationToken)
at Service.ProcessMessage.SendMessages(CancellationToken cancelToken) in C:\Project\Service\ProcessMessage.cs:line 73
---> (Inner Exception #0) System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.String.ConcatArray(String[] values, Int32 totalLength)
at System.String.Concat(Object[] args)
at NHibernate.Engine.Cascade.CascadeOn(IEntityPersister persister, Object parent, Object anything)
at NHibernate.Event.Default.AbstractFlushingEventListener.CascadeOnFlush(IEventSource session, IEntityPersister persister, Object key, Object anything)
at NHibernate.Event.Default.AbstractFlushingEventListener.PrepareEntityFlushes(IEventSource session)
at NHibernate.Event.Default.AbstractFlushingEventListener.FlushEverythingToExecutions(FlushEvent event)
at NHibernate.Event.Default.DefaultFlushEventListener.OnFlush(FlushEvent event)
at NHibernate.Impl.SessionImpl.Flush()
at NHibernate.Transaction.AdoTransaction.Commit()
at Service.ProcessMessage.Process(Message message, ISession session) in C:\Project\Service\ProcessMessage.cs:line 247
at Service.ProcessMessage.<>c__DisplayClass22_0.<SendMessages>b__0(Task c) in C:\Project\Service\ProcessMessage.cs:line 74
at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke()
at System.Threading.Tasks.Task.Execute()<---
LogExtension.Event => LogExtension.Event => LogExtension.LogEvent
Above error is coming after 4-5 hours after windows service is started.
Any hint or help is appreciated.
.Net framework version I use is 4.5
NHibernate version is 4.0.4.4000
Castle.Windsor version is 3.3.0
FluentNhibernate version is 1.3.0.733
Castle.Facilities.NHibernate version is 0.7.1.23602
Castle.Facilities.AutoTx version is 3.2.207.2207
Castle.Core version is 3.3.3
You haven't posted your logic for GetSession(), but I get the impression you are re-using the same session for all of the messages. Sessions are supposed to be short lived. Keeping a single session open will result in the first level cache growing to the point where you get performance degradation and eventually run out of memory.
I have found a couple examples of this error occurring in the forums but I don't apparently have the technical knowledge to make my case be truly 'async all the way' - if this is even the problem.
The error when caught is:
An asynchronous operation cannot be started at this time. Asynchronous
operations may only be started within an asynchronous handler or
module or during certain events in the Page lifecycle. If this
exception occurred while executing a Page, ensure that the Page is
marked <%# Page Async="true" %>. This exception may also indicate an
attempt to call an "async void" method, which is generally unsupported
within ASP.NET request processing. Instead, the asynchronous method
should return a Task, and the caller should await it.
at System.Web.AspNetSynchronizationContext.OperationStarted() at
System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Create() at
myCompany.System.Turhh.Turhh.<>c__DisplayClass18_0.b__1(Object
sender, GetInfoCompletedEventArgs e) at
myCompany.System.TurhhWebServices.InfoServiceClient.OnGetInfoCompleted(Object
state)
The code:
public void GetInfoAsync(GetInfoRq request, object Infotate)
{
if ((this.onBeginGetInfoDelegate == null))
{
this.onBeginGetInfoDelegate = new BeginOperationDelegate(this.OnBeginGetInfo);
}
if ((this.onEndGetInfoDelegate == null))
{
this.onEndGetInfoDelegate = new EndOperationDelegate(this.OnEndGetInfo);
}
if ((this.onGetInfoCompletedDelegate == null))
{
this.onGetInfoCompletedDelegate = new System.Threading.SendOrPostCallback(this.OnGetInfoCompleted);
}
base.InvokeAsync(this.onBeginGetInfoDelegate, new object[] {
request}, this.onEndGetInfoDelegate, this.onGetInfoCompletedDelegate, Infotate);
}
private BeginOperationDelegate onBeginGetInfoDelegate;
private EndOperationDelegate onEndGetInfoDelegate;
private System.Threading.SendOrPostCallback onGetInfoCompletedDelegate;
public event System.EventHandler<GetInfoCompletedEventArgs> GetInfoCompleted;
public System.IAsyncResult BeginGetInfo(GetInfoRq request, System.AsyncCallback callback, object asyncState)
{
return base.Channel.BeginGetInfo(request, callback, asyncState);
}
public GetInfoRs EndGetInfo(System.IAsyncResult result)
{
return base.Channel.EndGetInfo(result);
}
private System.IAsyncResult OnBeginGetInfo(object[] inValues, System.AsyncCallback callback, object asyncState)
{
return this.BeginGetInfo(rq, callback, asyncState);
}
private object[] OnEndGetInfo(System.IAsyncResult result)
{
GetInfoRs retVal = this.EndGetInfo(result);
return new object[] {retVal};
}
private void OnGetInfoCompleted(object state)
{
if ((this.GetInfoCompleted != null))
{
InvokeAsyncCompletedEventArgs e = ((InvokeAsyncCompletedEventArgs)(state));
var ea = new GetInfoCompletedEventArgs(e.Results, e.Error, e.Cancelled, e.Infotate);
try
{
// Note: Encapsulating this in a Task.Run() hides the issue on the server (here)
// but IIS still dumps its HTML page with an await/async error info.
this.GetInfoCompleted(this, ea);
}
catch (InvalidOperationException ex)
{
// HELP:
// Most of the time, this works fine. About 1 in 10 times, this exception is thrown.
// When this exception is thrown (and caught), IIS still dumps its HTML async/await error
System.Diagnostics.Debug.WriteLine(" **** AWAIT/ASYNC EXCEPTION 3 **** ");
}
}
}
This is the code behind the GetInfoCompleted:
channel.GetInfoCompleted += new EventHandler<GetInfoCompletedEventArgs>(async (sender, e) =>
{
await complete(); // complete contains an 'await Task.Run(async() => { some server work here });' job
});
This exception only occurs about 1 out of 10 times running locally. On production, it occurs almost every time. Essentially, on really fast servers, when the async calls happen synchronously rather than asynchronously, this throws the exception and irritates IIS causing it to dump the HTML error page down to the client. When these calls are made asynchronously, some time after IIS has gracefully given a good result to the client and the client moves on to its next task, everything is fine and this error-prone call on the server successfully accomplishes its mission (whether it is successful or not does not matter to the client - this is why it is threaded out as a post-operation task).
I am looking into an issue with some legacy code which is relaying a Http Request to another URL. It is an API Controller that reads the request and forwards it to another address asynchronously.
Very rarely, it throws a "Cannot access a disposed object" exception - full stack trace shown later in this question. It seems to be on the line where the HttpClient is calling the SendAsync method. I think it might be the ResponseHeadersRead option - I suspect it is happening when a large packet is being sent and it gets closed because it has just read the header and quit. Just thought I'd sanity check this with you all, for your thoughts. I'll change the option to be ResponseContentsRead option and see how that goes (but it can take a long time for the error to surface).
Here is the code:
using (var client = new HttpClient())
{
var request = BuildRelayHttpRequest(Request);
await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
}
private static HttpRequestMessage BuildRelayHttpRequest(HttpRequestMessage incomingRequest)
{
var forwardToUrl = new Uri(ConfigurationManager.AppSettings["ForwardFeedURL"]);
var relayRequest = new HttpRequestMessage(incomingRequest.Method, forwardToUrl);
if (incomingRequest.Method != HttpMethod.Get && incomingRequest.Content != null)
{
relayRequest.Content = incomingRequest.Content;
}
//Copies contents
relayRequest.Content = incomingRequest.Content;
return relayRequest;
}
And here is the exception:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Web.Http.WebHost.HttpControllerHandler+LazyStreamContent'.
at System.Net.Http.HttpContent.CheckDisposed()
at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at CHO.Web.Services.ETrains.Controllers.ETrainsApiController.<CopyandForwardFeedAsyn>d__18.MoveNext() in \Controllers\MyAPIController.cs:line 289
Note, line 289 is the "await client.SendAsync" line of code
Odds are there is an error code being set by the server.
Follow your code with a response.EnsureSuccessStatusCode(); that is wrapped in a Try Catch block:
try
{
var request = BuildRelayHttpRequest(Request);
await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
// Handle success
}
catch (HttpRequestException)
{
// Handle failure
}