Develop Reference

Azure Function HttpClient doesn't give any response and keeps retrying

I have a Function App running in Azure. It is a HttpTrigger, and it is triggered by a Stream Analytics, with a Retry Policy set to Drop. It is coded in C#, and deployed from DevOps through CI/CD. Here are some of the configs.
FUNCTIONS_EXTENSION_VERSION: ~4
FUNCTIONS_WORKER_RUNTIME: dotnet-isolated
WEBSITE_RUN_FROM_PACKAGE: 1
The purpose of the function is to get some data and forward it using a HttpClient (created through a HttpClientFactory, and with a timeout of 15 sec) to various other endpoints on different servers. Everything have worked so far, until recently where a new endpoint on a new server was added. The problem is somehow related to this new server, but I am having a hard time getting any useful logs from my end, and also the whole Function is acting weird when sending data. Here is the end of the code:
log.LogInformation($"Preparing to send...");
var content = new StringContent(jsonContent, Encoding.UTF8, "application/json");
var httpRequestMessage = new HttpRequestMessage
{
Method = HttpMethod.Post,
RequestUri = new Uri(uri),
Headers = {
{ "Authorization", $"Bearer {token}" },
{ "Accept", "*/*" },
},
Content = content
};
var response = await _httpClient.SendAsync(httpRequestMessage);
var responseString = await response.Content.ReadAsStringAsync();
log.LogInformation($"Returned content: {responseString}.");
I use Application Insights to monitor. The uri parameter is different based on where to send the data, and it is working for everywhere except 1 endpoint. When I try to send to this "faulty" endpoint, this is what I see in Application Insights:
01:27:37 PM Trace: Preparing to send...
01:27:37 PM Trace: Start processing HTTP request POST https://... (URI parameter)
01:27:37 PM Trace: Sending HTTP request POST https://... (URI Parameter)
01:27:37 PM Trace: End processing HTTP request after 188.05ms - OK
01:27:37 PM Trace: Received HTTP response after 187.92ms - OK
01:29:17 PM Exception: Exception while executing function: Functions.MyFunction
01:29:17 PM Trace:Executed 'Functions.MyFunction' (Failed, Id=3d9afe5b-c58a-470d-8f54-0d98ca60deb7, Duration=100005ms
01:29:17 PM Exception: Exception while executing function: Functions.MyFunction
When I inspect the exception, it says:
Exception while executing function: Functions.MyFunction Result: Failure
Exception: System.AggregateException: One or more errors occurred. (A task was canceled.)
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Threading.Tasks.Task.GetExceptions(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
at System.Threading.Thread.StartCallback()
--- End of stack trace from previous location ---
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 16
at Microsoft.Azure.Functions.Worker.GrpcWorker.InvocationRequestHandlerAsync(InvocationRequest request, IFunctionsApplication application, IInvocationFeaturesFactory invocationFeaturesFactory, ObjectSerializer serializer, IOutputBindingsInfoProvider outputBindingsInfoProvider) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 167
Stack: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 16
at Microsoft.Azure.Functions.Worker.GrpcWorker.InvocationRequestHandlerAsync(InvocationRequest request, IFunctionsApplication application, IInvocationFeaturesFactory invocationFeaturesFactory, ObjectSerializer serializer, IOutputBindingsInfoProvider outputBindingsInfoProvider) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 167
And then it seems the function retries the Http request, which I don't understand why. The "Preparing to send" log is not present, but the "Start processing" logs appears and fails every ~2 minutes. Since my custom log message is not here, I take it that it is the HttpClient that is retrying somehow and not the entire Function.
Does anyone have a clue what is going on? Why does my code seem to get stuck trying to read the response.Content? How can I further investigate what is going wrong, and why is the HttpClient retrying itself like this? I would expect the Function to crash/stop if the HttpClient fails.
EDIT: Corrected timestamps on log to match duration. There are multiple logs appearing at the same time, and I picked wrong ones.

Hard to help as we don't have access to your environment, but here's what I would try:
It seems to me that the issue with the log is because it's an async operation, this log doesn't represent the inner/real exception. I recommend you switch to a sync operation and send to this faulty endpoint. Once you get the real exception / proper fix, you switch back to async operation.

With tip from Thiago of changing from async, I have received a proper exception, which helped me put the pieces together.
The new error:
The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
I am using 2 different HttpClients, one for my error and another for a timed Token updater. It seems my first issue was me adding the 15 sec Timeout to the wrong client (doh). With this new info, the duration of 100005ms makes much more sense.
Next, the timeout exception from the HttpClient were not caught, and caused the whole function to crash producing the "A task was cancelled" error.
Finally, the reason it keeps restarting seems to be caused by Azure Stream Analytics after all. On a closer look at the error policy docs, it seems despite being set to "Drop", it will still retry in some cases based on the error.
I still have a problem as to why my request times out, but my original question as to what was going on is answered :)

ERROR NServiceBus.Transports.Msmq.MsmqDequeueStrategy - Error in receiving messages

Event Viewer on my workstation have the following error log:
ERROR NServiceBus.Transports.Msmq.MsmqDequeueStrategy [(null)] - Error in receiving messages.
System.Transactions.TransactionAbortedException: The transaction has aborted. ---> System.Transactions.TransactionManagerCommunicationException: Communication with the underlying transaction manager has failed. ---> System.Runtime.InteropServices.COMException: The Transaction Manager is not available. (Exception from HRESULT: 0x8004D01B)
at System.Transactions.Oletx.IDtcProxyShimFactory.ConnectToProxy(String nodeName, Guid resourceManagerIdentifier, IntPtr managedIdentifier, Boolean& nodeNameMatches, UInt32& whereaboutsSize, CoTaskMemHandle& whereaboutsBuffer, IResourceManagerShim& resourceManagerShim)
at System.Transactions.Oletx.DtcTransactionManager.Initialize()
--- End of inner exception stack trace ---
at System.Transactions.Oletx.OletxTransactionManager.ProxyException(COMException comException)
at System.Transactions.Oletx.DtcTransactionManager.Initialize()
at System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
at System.Transactions.Oletx.OletxTransactionManager.CreateTransaction(TransactionOptions properties)
at System.Transactions.TransactionStatePromoted.EnterState(InternalTransaction tx)
--- End of inner exception stack trace ---
at System.Transactions.TransactionStateAborted.CheckForFinishedTransaction(InternalTransaction tx)
at System.Transactions.Transaction.Promote()
at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction)
at System.Transactions.TransactionInterop.GetDtcTransaction(Transaction transaction)
at System.Messaging.MessageQueue.StaleSafeReceiveMessage(UInt32 timeout, Int32 action, MQPROPS properties, NativeOverlapped* overlapped, ReceiveCallback receiveCallback, CursorHandle cursorHandle, IntPtr transaction)
at System.Messaging.MessageQueue.ReceiveCurrent(TimeSpan timeout, Int32 action, CursorHandle cursor, MessagePropertyFilter filter, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Receive(TimeSpan timeout, MessageQueueTransactionType transactionType)
at NServiceBus.Transports.Msmq.MsmqDequeueStrategy.TryReceiveMessage(Func`1 receive, Message& message) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Transports\Msmq\MsmqDequeueStrategy.cs:line 332
Facts:
Distributed Transaction Coordinator (DTC) Component Service ==> DTC Enabled
Distributed Transaction Coordinator (DTC) Service ==> Running (referenced to Answered Stack Oveflow Question)
Realtek Audio Universal Service ==> Disabled (as per reference in Fix for 0x8004d01b)
Can anyone recommend a direction on how to check/troubleshoot the cause of the issue?

After long ardous hours, the issue I encountered boils down to permission issue. The services involved in DTC which uses credentials of LocalSystem do not have enough permission and blocked by company-imposed policy implemented on the server.
Unfortunately, I only managed to find an alternative or workaround by using account that have enough permissions to operate and manage the services involved (including custom services).
#gnud - Thanks a lot for your inputs giving me a direction for investigation and that help me continue move forward on my investigation. I appreciate the help. Thanks.

Unhandled exception of type System.Exception occurred in FM.dll

I'm using WebSync in a .NET application where a client is making calls to connect, subscribe, etc to WebSync. Somewhere along the execution of the code (It's a big app), I get the gray box popup saying the above error message. Because this is a big app, how can I find where in the application this is occurring? The problem is that this application has a lot of threads.
Does this message mean there is a problem in FM.dll (WebSync) or does it mean that I'm missing something in my code that should catch this exception. If the latter, how do I find where in my code this may occur?
Thanks!
Here is the Call Stack to my error. How can I tell if this is an error on my part or if the problem is inside FM.dll (WebSync)?
FM.dll!FM.AsyncException.AsyncThrow.AnonymousMethod__0(object unused) + 0x47 bytes
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) + 0x285 bytes
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) + 0x9 bytes
mscorlib.dll!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() + 0x6f bytes
mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() + 0x1ea bytes
[Native to Managed Transition]
[Appdomain Transition]
[Native to Managed Transition]

If you have 'Break on All Exceptions' turned on then once you start the debugger, when the exception is thrown, you'll be able to see and navigate the stack trace. This should give you an idea of the path the code is on leading up to the exception. You can also interrogate the exception to check to see if the InnerException property has been set. From that information you should be able to determine what is causing the issue. If not post more details from the information you gather and you can get more help.
Instruction for setting up Visual Studio to break on all exceptions: http://msdn.microsoft.com/en-us/library/d14azbfh.aspx

In FM libraries, AsyncException.AsyncThrow is used when an exception is thrown in an async callback. Try wrapping your FM callback code in try/catch blocks to catch exceptions, e.g.:
client.Connect(new ConnectArgs
{
OnSuccess = (e) =>
{
try
{
// your code
}
catch (Exception ex)
{
// handle exception
}
}
});
If you don't wrap your callback code in a try/catch block, then the FM library will push the exception to a thread where it will be fail loudly rather than swallow/hide it.

How to access an object from a different thread without conflict?

I have a thread named fetchImage that runs startFetch() method, which is
for (int i = 0; !stopFetching && i < 2000; i++) {
control.fetchImage().Save("Fetch_Image\\fetchedFile" + fileName + ".JPG", System.Drawing.Imaging.ImageFormat.Jpeg);
fileName++;
//Thread.Sleep(800);
Console.WriteLine("Filename :" + fileName);
control is another class that has fetchImage() method:
return (System.Drawing.Bitmap)mainForm.imageTemp.Clone();
mainForm is a form that has Image variable named imageTemp. Now startFetch() runs fine when I have Thread.Sleep(800) but it gives me that error without the sleep. Here is the stack trace of the error
System.InvalidOperationException was unhandled
Message=Object is currently in use elsewhere.
Source=System.Drawing
StackTrace:
at System.Drawing.Image.Clone()
at KiPNi.Control.fetchImage() in F:\CMPS285\285-fall10-5\Final working code\WindowsFormsApplication7\Control.cs:line 65
at KiPNi.Form4.startFetch() in F:\CMPS285\285-fall10-5\Final working code\WindowsFormsApplication7\Form4.cs:line 80
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
InnerException:
I do not know much about threads (although I tried after encountering this problem).
How do I get access to the image from the fetchImage thread?

Instead of pulling data from the processing thread, see if it's possible to push data from the UI thread when there's a frame update - that way, you can possibly avoid locking; if you can create a clone of the image data, you avoid sharing the object between threads.
You really only should be invoking methods on UI controls from the main thread anyway (Control.BeginInvoke() exists, but it's better trying to modify your design to avoid it, if possible).

IClientChannel antipattern

I've just spend 4 hours (it's 3am in the UK) trying to debug ASP.NET application which caused an exception in a thread managed by Framework (i.e. not my thread). I've just found out that a result from the static method ChannelFactory.CreateChannel can be cast to IClientChannel and explicitly Disposed. I mean that's all fine and nice but why:
1) ChannelFactory.CreateChannel does not return IClientChannel as an out parameter?
2) .Net documentation for CreateChannel does not mention it?
3) .Net documentation does not show a proper usage pattern in examples (no dispose code)?
Don't get me wrong - I love .Net framework. Microsoft (and Krzysztof Cwalina: see Designing Framework Guidelines) has done a really great job. That's why I didn't expect a such disaster. I mean how the hell I should know that my IMyService variable also supports IClientChannel and I should explicitly dispose it?
Here is an ASP.NET log if someone is interested.
Event Type: Error
Event Source: ASP.NET 2.0.50727.0
Event Category: None
Event ID: 1334
Date: 12/08/2009
Time: 01:55:47
User: N/A
Computer: WLGH3GIS
Description:
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1/Root/Maps
Process ID: 3044
Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
StackTrace: at System.Threading.Overlapped.Free(NativeOverlapped* nativeOverlappedPtr)
at System.ServiceModel.Channels.OverlappedContext.Free()
at System.ServiceModel.Channels.PipeConnection.CloseHandle(Boolean abort, String timeoutErrorString, TransferOperation transferOperation)
at System.ServiceModel.Channels.PipeConnection.Close(TimeSpan timeout)
at System.ServiceModel.Channels.BufferedConnection.Close(TimeSpan timeout)
at System.ServiceModel.Channels.ConnectionPool.CloseItem(IConnection item, TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationPool`2.EndpointConnectionPool.CloseItem(TItem item, TimeSpan timeout)
at System.ServiceModel.Channels.IdlingCommunicationPool`2.IdleTimeoutEndpointConnectionPool.CloseItem(TItem item, TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationPool`2.EndpointConnectionPool.CloseIdleConnection(TItem connection, TimeSpan timeout)
at System.ServiceModel.Channels.IdlingCommunicationPool`2.IdleTimeoutEndpointConnectionPool.IdleTimeoutIdleConnectionPool.OnIdle()
at System.ServiceModel.Channels.IdlingCommunicationPool`2.IdleTimeoutEndpointConnectionPool.IdleTimeoutIdleConnectionPool.OnIdle(Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
at System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)
at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Karol, I know this is a bit old, but I wanted to say thank you. Your post really helped me out.
The symptom I had was that when I tried to Close a ChannelFactory on the Server, it would always give a Timeout, regardless of the length of timespan I set for OpenTimeout, ReceiveTimeout, SendTimeout, InactivityTimeout, or CloseTimeout.
The solution was actually on the client, to cast the IMyServiceInterface that was returned by ChannelFactory.CreateChannel to ICommunicationObject. Then, it can be passed nicely to the Util.SafeCloseAndDispose(ICommunicationObject) method that you see copy and pasted all over the web.
After I did that on my client, then the Server's ChannelFactory can be closed in only a second or two, with no more timeout.
As far as I know, this insight from Karol's post is one of the only places online where this problem is spelled out.
Thanks again, Karol! :)