Develop Reference

How to avoid PopReceiptMismatch errors in Azure durable functions

For my Azure functions application, a fan-out architecture with (sub-)orchestrators and a lot of activities running in parallel is required. Additionally, all these activities have to interact with an Azure blob storage account. When dealing with this high degree of parallelization, I often run into runtime errors like these:
[2023-02-07T14:11:36.859Z] 4c6a215f040141aaaa995f9cfbf1a234:10: An error occurred while processing message [TaskScheduled#46]: DurableTask.AzureStorage.Storage.DurableTaskStorageException: The specified pop receipt did not match the pop receipt for a dequeued message.[2023-02-07T14:11:36.860Z] ---> Microsoft.WindowsAzure.Storage.StorageException: The specified pop receipt did not match the pop receipt for a dequeued message.
[2023-02-07T14:11:36.861Z] at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)[2023-02-07T14:11:36.874Z] at DurableTask.AzureStorage.Storage.AzureStorageClient.WrapFunctionWithReturnType(Func`3 storageRequest, OperationContext context, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 156[2023-02-07T14:11:36.875Z] at DurableTask.AzureStorage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String account, AzureStorageOrchestrationServiceSettings settings, Func`3 operation, AzureStorageOrchestrationServiceStats stats, String clientRequestId)[2023-02-07T14:11:36.876Z] at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force)
[2023-02-07T14:11:36.877Z] Request Information
[2023-02-07T14:11:36.879Z] RequestID:7db00ba3-1e70-42fc-84f5-f6dfb9ec630a
[2023-02-07T14:11:36.880Z] RequestDate:Tue, 07 Feb 2023 15:11:36 GMT
[2023-02-07T14:11:36.896Z] StatusMessage:The specified pop receipt did not match the pop receipt for a dequeued message.
[2023-02-07T14:11:36.897Z] ErrorCode:PopReceiptMismatch
[2023-02-07T14:11:36.898Z] ErrorMessage:The specified pop receipt did not match the pop receipt for a dequeued message.
RequestId:7db00ba3-1e70-42fc-84f5-f6dfb9ec630a
Time:2023-02-07T14:11:36.137Z
[2023-02-07T14:11:36.901Z] --- End of inner exception stack trace ---
These neither trigger try-catch breakpoints nor stop code execution, but are really annoying and cause the orchestrators to restart which leads to data corruption down the road. Is there a proper way of fixing this?

When function projects get created in Visual Studio, the AzureWebJobsStorage value in the local.settings.json file is set to true by default. Configuring Azure functions to use a real storage account for orchestrators can eliminate concurrency issues and faulty state management. Simply replace the value with the connection string of your own storage:
{
...
"Values": {
"AzureWebJobsStorage": "[YOUR CONNECTION STRING HERE]",
...
}
}

ML.net cli -- Resume a cli training after a crash

is there a way to resume an ML.net cli training to where it was before a crash?
I have a lot of data in the folder C:\Users\wwww\AppData\Local\Temp\AutoML-NNI\Experiment-9K67B4 but I do not know how to make mlnet start from there.
Detail:
I used the cli, ie "mlnet classicfiaction...."
I trained for a few days, but I made a mistake which used a lot of memory on my computer, which stopped the mlnet process. I would like to start mlnet to where it left so it can continue from there
Note: I would love to be able to stop earlier and continue/resume a training from the cli.
Thanks
w
2022-08-10 15:03:24.3091 DEBUG System.InvalidOperationException: Event we were waiting on was subject to an exception
---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Array.Resize[T](T[]& array, Int32 newSize)
at Microsoft.ML.Internal.Utilities.ArrayUtils.EnsureSize[T](T[]& array, Int32 min, Int32 max, Boolean keepOld, Boolean& resized)
at Microsoft.ML.Data.CacheDataView.ColumnCache.ImplOne`1.CacheCurrent()
at Microsoft.ML.Data.CacheDataView.Filler(DataViewRowCursor cursor, ColumnCache[] caches, OrderedWaiter waiter)
Relate to
https://github.com/dotnet/machinelearning/issues/6286
https://github.com/dotnet/machinelearning/issues/6287

ERROR NServiceBus.Transports.Msmq.MsmqDequeueStrategy - Error in receiving messages

Event Viewer on my workstation have the following error log:
ERROR NServiceBus.Transports.Msmq.MsmqDequeueStrategy [(null)] - Error in receiving messages.
System.Transactions.TransactionAbortedException: The transaction has aborted. ---> System.Transactions.TransactionManagerCommunicationException: Communication with the underlying transaction manager has failed. ---> System.Runtime.InteropServices.COMException: The Transaction Manager is not available. (Exception from HRESULT: 0x8004D01B)
at System.Transactions.Oletx.IDtcProxyShimFactory.ConnectToProxy(String nodeName, Guid resourceManagerIdentifier, IntPtr managedIdentifier, Boolean& nodeNameMatches, UInt32& whereaboutsSize, CoTaskMemHandle& whereaboutsBuffer, IResourceManagerShim& resourceManagerShim)
at System.Transactions.Oletx.DtcTransactionManager.Initialize()
--- End of inner exception stack trace ---
at System.Transactions.Oletx.OletxTransactionManager.ProxyException(COMException comException)
at System.Transactions.Oletx.DtcTransactionManager.Initialize()
at System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
at System.Transactions.Oletx.OletxTransactionManager.CreateTransaction(TransactionOptions properties)
at System.Transactions.TransactionStatePromoted.EnterState(InternalTransaction tx)
--- End of inner exception stack trace ---
at System.Transactions.TransactionStateAborted.CheckForFinishedTransaction(InternalTransaction tx)
at System.Transactions.Transaction.Promote()
at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction)
at System.Transactions.TransactionInterop.GetDtcTransaction(Transaction transaction)
at System.Messaging.MessageQueue.StaleSafeReceiveMessage(UInt32 timeout, Int32 action, MQPROPS properties, NativeOverlapped* overlapped, ReceiveCallback receiveCallback, CursorHandle cursorHandle, IntPtr transaction)
at System.Messaging.MessageQueue.ReceiveCurrent(TimeSpan timeout, Int32 action, CursorHandle cursor, MessagePropertyFilter filter, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Receive(TimeSpan timeout, MessageQueueTransactionType transactionType)
at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.TryReceiveMessage(Func`1 receive, Message& message) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line 332
Facts:
Distributed Transaction Coordinator (DTC) Component Service ==> DTC Enabled
Distributed Transaction Coordinator (DTC) Service ==> Running (referenced to Answered Stack Oveflow Question)
Realtek Audio Universal Service ==> Disabled (as per reference in Fix for 0x8004d01b)
Can anyone recommend a direction on how to check/troubleshoot the cause of the issue?

After long ardous hours, the issue I encountered boils down to permission issue. The services involved in DTC which uses credentials of LocalSystem do not have enough permission and blocked by company-imposed policy implemented on the server.
Unfortunately, I only managed to find an alternative or workaround by using account that have enough permissions to operate and manage the services involved (including custom services).
#gnud - Thanks a lot for your inputs giving me a direction for investigation and that help me continue move forward on my investigation. I appreciate the help. Thanks.

InstanceOwnerException when running Workflow Foundation 4

I am running Workflow 4 activities, and using SQL Durable Instancing to persist the Workflow when it's suspended. I am intermittently getting an InstanceOwnerException, which is remedied by recreating the instance store. Is there any way of avoiding this exception in the first place?
System.OperationCanceledException: An error processing the current work item has caused the workflow to abort. See the inner exception for details. ---> System.Runtime.DurableInstancing.InstanceOwnerException: The execution of an InstancePersistenceCommand was interrupted because the instance owner registration for owner ID '55413a1f-0e3f-4a0f-b2b0-54c121aa0c52' has become invalid. This error indicates that the in-memory copy of all instances locked by this owner have become stale and should be discarded, along with the InstanceHandles. Typically, this error is best handled by restarting the host.
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.Runtime.DurableInstancing.InstanceStore.EndExecute(IAsyncResult result)
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.OnPersisted(IAsyncResult result)
at System.Runtime.AsyncResult.SyncContinue(IAsyncResult result)
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.Persist()
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.CollectAndMap()
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.Track()
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.InitializeProvider()
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult..ctor(WorkflowApplication instance, TimeSpan timeout, PersistenceOperation operation, Boolean isWorkflowThread, Boolean isInternalPersist, AsyncCallback callback, Object state)
at System.Activities.WorkflowApplication.BeginInternalPersist(PersistenceOperation operation, TimeSpan timeout, Boolean isInternalPersist, AsyncCallback callback, Object state)
at System.Activities.WorkflowApplication.OnBeginPersist(AsyncCallback callback, Object state)
at System.Activities.Runtime.ActivityExecutor.PersistenceWaiter.PersistWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)
--- End of inner exception stack trace ---
The code is running on Azure, using SQL Azure to persist the Workflows. There are roles that are accessing the same instance store.

This actually turned out to be down to attempting to resume a workflow after it had already finished.

IClientChannel antipattern

I've just spend 4 hours (it's 3am in the UK) trying to debug ASP.NET application which caused an exception in a thread managed by Framework (i.e. not my thread). I've just found out that a result from the static method ChannelFactory.CreateChannel can be cast to IClientChannel and explicitly Disposed. I mean that's all fine and nice but why:
1) ChannelFactory.CreateChannel does not return IClientChannel as an out parameter?
2) .Net documentation for CreateChannel does not mention it?
3) .Net documentation does not show a proper usage pattern in examples (no dispose code)?
Don't get me wrong - I love .Net framework. Microsoft (and Krzysztof Cwalina: see Designing Framework Guidelines) has done a really great job. That's why I didn't expect a such disaster. I mean how the hell I should know that my IMyService variable also supports IClientChannel and I should explicitly dispose it?
Here is an ASP.NET log if someone is interested.
Event Type: Error
Event Source: ASP.NET 2.0.50727.0
Event Category: None
Event ID: 1334
Date: 12/08/2009
Time: 01:55:47
User: N/A
Computer: WLGH3GIS
Description:
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1/Root/Maps
Process ID: 3044
Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
StackTrace: at System.Threading.Overlapped.Free(NativeOverlapped* nativeOverlappedPtr)
at System.ServiceModel.Channels.OverlappedContext.Free()
at System.ServiceModel.Channels.PipeConnection.CloseHandle(Boolean abort, String timeoutErrorString, TransferOperation transferOperation)
at System.ServiceModel.Channels.PipeConnection.Close(TimeSpan timeout)
at System.ServiceModel.Channels.BufferedConnection.Close(TimeSpan timeout)
at System.ServiceModel.Channels.ConnectionPool.CloseItem(IConnection item, TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationPool`2.EndpointConnectionPool.CloseItem(TItem item, TimeSpan timeout)
at System.ServiceModel.Channels.IdlingCommunicationPool`2.IdleTimeoutEndpointConnectionPool.CloseItem(TItem item, TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationPool`2.EndpointConnectionPool.CloseIdleConnection(TItem connection, TimeSpan timeout)
at System.ServiceModel.Channels.IdlingCommunicationPool`2.IdleTimeoutEndpointConnectionPool.IdleTimeoutIdleConnectionPool.OnIdle()
at System.ServiceModel.Channels.IdlingCommunicationPool`2.IdleTimeoutEndpointConnectionPool.IdleTimeoutIdleConnectionPool.OnIdle(Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
at System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Karol, I know this is a bit old, but I wanted to say thank you. Your post really helped me out.
The symptom I had was that when I tried to Close a ChannelFactory on the Server, it would always give a Timeout, regardless of the length of timespan I set for OpenTimeout, ReceiveTimeout, SendTimeout, InactivityTimeout, or CloseTimeout.
The solution was actually on the client, to cast the IMyServiceInterface that was returned by ChannelFactory.CreateChannel to ICommunicationObject. Then, it can be passed nicely to the Util.SafeCloseAndDispose(ICommunicationObject) method that you see copy and pasted all over the web.
After I did that on my client, then the Server's ChannelFactory can be closed in only a second or two, with no more timeout.
As far as I know, this insight from Karol's post is one of the only places online where this problem is spelled out.
Thanks again, Karol! :)