I have a Function App running in Azure. It is a HttpTrigger, and it is triggered by a Stream Analytics, with a Retry Policy set to Drop. It is coded in C#, and deployed from DevOps through CI/CD. Here are some of the configs.
FUNCTIONS_EXTENSION_VERSION: ~4
FUNCTIONS_WORKER_RUNTIME: dotnet-isolated
WEBSITE_RUN_FROM_PACKAGE: 1
The purpose of the function is to get some data and forward it using a HttpClient (created through a HttpClientFactory, and with a timeout of 15 sec) to various other endpoints on different servers. Everything have worked so far, until recently where a new endpoint on a new server was added. The problem is somehow related to this new server, but I am having a hard time getting any useful logs from my end, and also the whole Function is acting weird when sending data. Here is the end of the code:
log.LogInformation($"Preparing to send...");
var content = new StringContent(jsonContent, Encoding.UTF8, "application/json");
var httpRequestMessage = new HttpRequestMessage
{
Method = HttpMethod.Post,
RequestUri = new Uri(uri),
Headers = {
{ "Authorization", $"Bearer {token}" },
{ "Accept", "*/*" },
},
Content = content
};
var response = await _httpClient.SendAsync(httpRequestMessage);
var responseString = await response.Content.ReadAsStringAsync();
log.LogInformation($"Returned content: {responseString}.");
I use Application Insights to monitor. The uri parameter is different based on where to send the data, and it is working for everywhere except 1 endpoint. When I try to send to this "faulty" endpoint, this is what I see in Application Insights:
01:27:37 PM Trace: Preparing to send...
01:27:37 PM Trace: Start processing HTTP request POST https://... (URI parameter)
01:27:37 PM Trace: Sending HTTP request POST https://... (URI Parameter)
01:27:37 PM Trace: End processing HTTP request after 188.05ms - OK
01:27:37 PM Trace: Received HTTP response after 187.92ms - OK
01:29:17 PM Exception: Exception while executing function: Functions.MyFunction
01:29:17 PM Trace:Executed 'Functions.MyFunction' (Failed, Id=3d9afe5b-c58a-470d-8f54-0d98ca60deb7, Duration=100005ms
01:29:17 PM Exception: Exception while executing function: Functions.MyFunction
When I inspect the exception, it says:
Exception while executing function: Functions.MyFunction Result: Failure
Exception: System.AggregateException: One or more errors occurred. (A task was canceled.)
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Threading.Tasks.Task.GetExceptions(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
at System.Threading.Thread.StartCallback()
--- End of stack trace from previous location ---
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 16
at Microsoft.Azure.Functions.Worker.GrpcWorker.InvocationRequestHandlerAsync(InvocationRequest request, IFunctionsApplication application, IInvocationFeaturesFactory invocationFeaturesFactory, ObjectSerializer serializer, IOutputBindingsInfoProvider outputBindingsInfoProvider) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 167
Stack: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 45
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 16
at Microsoft.Azure.Functions.Worker.GrpcWorker.InvocationRequestHandlerAsync(InvocationRequest request, IFunctionsApplication application, IInvocationFeaturesFactory invocationFeaturesFactory, ObjectSerializer serializer, IOutputBindingsInfoProvider outputBindingsInfoProvider) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 167
And then it seems the function retries the Http request, which I don't understand why. The "Preparing to send" log is not present, but the "Start processing" logs appears and fails every ~2 minutes. Since my custom log message is not here, I take it that it is the HttpClient that is retrying somehow and not the entire Function.
Does anyone have a clue what is going on? Why does my code seem to get stuck trying to read the response.Content? How can I further investigate what is going wrong, and why is the HttpClient retrying itself like this? I would expect the Function to crash/stop if the HttpClient fails.
EDIT: Corrected timestamps on log to match duration. There are multiple logs appearing at the same time, and I picked wrong ones.
Hard to help as we don't have access to your environment, but here's what I would try:
It seems to me that the issue with the log is because it's an async operation, this log doesn't represent the inner/real exception. I recommend you switch to a sync operation and send to this faulty endpoint. Once you get the real exception / proper fix, you switch back to async operation.
With tip from Thiago of changing from async, I have received a proper exception, which helped me put the pieces together.
The new error:
The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
I am using 2 different HttpClients, one for my error and another for a timed Token updater. It seems my first issue was me adding the 15 sec Timeout to the wrong client (doh). With this new info, the duration of 100005ms makes much more sense.
Next, the timeout exception from the HttpClient were not caught, and caused the whole function to crash producing the "A task was cancelled" error.
Finally, the reason it keeps restarting seems to be caused by Azure Stream Analytics after all. On a closer look at the error policy docs, it seems despite being set to "Drop", it will still retry in some cases based on the error.
I still have a problem as to why my request times out, but my original question as to what was going on is answered :)
Related
I am trying to load a large amount of data from a server by returning MultiPartContent in the response. However, during the HttpGet method the timeout value I am setting is being ignored. I am trying to set a longer timeout as waiting in this situation is OK.
I have tried various Timeout values between 10 minutes to 2 hours. Realistically the user will only have to wait ~5 minutes the first time they use our app and all other times it will be less.
I have also tried storing a reference of HttpClient to ensure that api.HttpClient isn't re-creating the HttpClient object each time.
var handler = new HttpClientHandler();
var progress = new ProgressMessageHandler(handler);
var api = APIHelpers.GetSession(progress);
progress.HttpReceiveProgress += (e, args) => ProgressChanged.Invoke(e, new SyncEventArgs(GlobalEnums.SyncStage.Downloading, args.ProgressPercentage * 0.01f));
api.HttpClient.Timeout = new TimeSpan(0, 10, 0);
var downloadUri = BuildURI();
var response = await api.HttpClient.GetAsync(downloadUri, HttpCompletionOption.ResponseHeadersRead);
I would expect the application to wait for 10 minutes before throwing a Timeout exception, however the exception is being thrown after 100 seconds (i believe that is the default value?).
Checking the HttpClient.Timeout value after it is set does show that it was set correctly.
The APIHelpers.GetSession() method returns an object of our API with authorisation headers created. The HttpClient object is accessible via an inherited class. This is created by Swashbuckle and Swagger. I have used the api.HttpClient.Timeout = x this way before with success so I don't think this is the issue. It seems to be specific to this scenario in particular.
The exception thrown:
{System.Net.Http.HttpRequestException: An error occurred while sending the request ---> System.Net.WebException: The operation has timed out.
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <a1ab7fc4639d4d84af41d68234158b1c>:0
at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00019] in /Library/Frameworks/Xamarin.iOS.framework/Versions/12.14.0.114/src/Xamarin.iOS/mcs/class/System/System.Net/HttpWebRequest.cs:1200
at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x0000f] in <939d99b14d934342858948926287beba>:0
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.MonoWebRequestHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x003d1] in /Library/Frameworks/Xamarin.iOS.framework/Versions/12.14.0.114/src/Xamarin.iOS/mcs/class/System.Net.Http/MonoWebRequestHandler.cs:499
--- End of inner exception stack trace ---
at System.Net.Http.MonoWebRequestHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x0046a] in /Library/Frameworks/Xamarin.iOS.framework/Versions/12.14.0.114/src/Xamarin.iOS/mcs/class/System.Net.Http/MonoWebRequestHandler.cs:503
at Microsoft.Rest.RetryDelegatingHandler+<>c__DisplayClass11_0.<SendAsync>b__1 () [0x000ad] in <6a6c837cafbb4f1faffaba1ff30ca4e3>:0
at Microsoft.Rest.RetryDelegatingHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00150] in <6a6c837cafbb4f1faffaba1ff30ca4e3>:0
at System.Net.Http.Handlers.ProgressMessageHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00087] in <19af20e76ce04547b3fc150a0f8f2d47>:0
at System.Net.Http.HttpClient.SendAsyncWorker (System.Net.Http.HttpRequestMessage request, System.Net.Http.HttpCompletionOption completionOption, System.Threading.CancellationToken cancellationToken) [0x0009e] in /Library/Frameworks/Xamarin.iOS.framework/Versions/12.14.0.114/src/Xamarin.iOS/mcs/class/System.Net.Http/System.Net.Http/HttpClient.cs:281
at ...
EDIT:
I'm just adding some additional things I have tried in an attempt to solve this.
I have tried using the DependencyService in Xamarin.Forms to get native HttpClientHandlers and HttpClient objects for each platform where I can set the timeouts per platform instead of relying on Xamarin.Forms to translate it for me. This had the exact same result.
public HttpMessageHandler GetHttpHandler(double timeoutSeconds)
{
var sessionConfig = NSUrlSessionConfiguration.DefaultSessionConfiguration;
sessionConfig.TimeoutIntervalForRequest = timeoutSeconds;
sessionConfig.TimeoutIntervalForResource = timeoutSeconds;
sessionConfig.WaitsForConnectivity = true;
var sessionHandler = new NSUrlSessionHandler(sessionConfig);
return sessionHandler;
}
I have tried creating a new HttpClient object just for this API call but with no luck here either. (This replaces Apihelpers.GetSession() in my code)
There was a known issue in both Xamarin.iOS and Xamarin.Android that HttpClient.Timeout values greater than 100 seconds are ignored. This is because the underlying native http clients have a timeout set to 100 seconds, so this times out before the .NET HttpClient times out when the HttpClient.Timeout value > 100 seconds. This should be fixed in all of the latest stable versions, so make sure you are updated. If you are using Visual Studio 2017, you won't have the fix, you will need to get VS 2019 for the fixes.
Had the same issue in my Xamarin Android project. Resolved it by changing HttpClient implementation to Android in the Project properties as below:
I just created simple web job that just writes something to console. This job is supposed to run on demand so when I published it to Azure, I set it to run on demand.
When I run it, it works fine but after a few minutes it fails with the following message.
[12/07/2015 18:07:20 > f78915: ERR ] Command 'cmd /c TestJob.exe' was
aborted due to no output nor CPU activity for 121 seconds. You can
increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or
WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed. cmd /c
TestJob.exe [12/07/2015 18:07:20 > f78915: SYS INFO] Status changed to
Failed
Here's what follows that error:
[12/07/2015 18:07:20 > f78915: SYS ERR ] System.AggregateException:
One or more errors occurred. --->
Kudu.Core.Infrastructure.CommandLineException: Command 'cmd /c
TestJob.exe' was aborted due to no output nor CPU activity for 121
seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or
WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed. cmd /c
TestJob.exe at
Kudu.Core.Infrastructure.IdleManager.WaitForExit(IProcess process)
at Kudu.Core.Infrastructure.ProcessExtensions.d__a.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Kudu.Core.Infrastructure.Executable.d__1d.MoveNext()
--- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean
includeTaskCanceledExceptions) at
System.Threading.Tasks.Task1.GetResultCore(Boolean
waitCompletionNotification) at
System.Threading.Tasks.Task1.get_Result() at
Kudu.Core.Infrastructure.Executable.ExecuteInternal(ITracer tracer,
Func2 onWriteOutput, Func2 onWriteError, Encoding encoding, String
arguments, Object[] args) at
Kudu.Core.Infrastructure.Executable.ExecuteReturnExitCode(ITracer
tracer, Action1 onWriteOutput, Action1 onWriteError, String
arguments, Object[] args) at
Kudu.Core.Jobs.BaseJobRunner.RunJobInstance(JobBase job, IJobLogger
logger, String runId, String trigger, Int32 port)
---> (Inner Exception #0) ExitCode: -1, Output: Command 'cmd /c TestJob.exe' was aborted due to no output nor CPU activity for 121
seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or
WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed., Error: Command
'cmd /c TestJob.exe' was aborted due to no output nor CPU activity for
121 seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting
(or WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed.,
Kudu.Core.Infrastructure.CommandLineException: Command 'cmd /c
TestJob.exe' was aborted due to no output nor CPU activity for 121
seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or
WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed. cmd /c
TestJob.exe at
Kudu.Core.Infrastructure.IdleManager.WaitForExit(IProcess process)
at Kudu.Core.Infrastructure.ProcessExtensions.d__a.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Kudu.Core.Infrastructure.Executable.d__1d.MoveNext()<---
Here's the entire code for this web job.
static void Main()
{
var host = new JobHost();
Console.Out.WriteLine("Testing the test job...");
host.RunAndBlock();
}
I understand that it fails because there's no activity but it's supposed to be a job that runs on demand. Where am I making a mistake?
The fact that it's a job that runs on demand has nothing to do with the error. It's just a job that starts when you tell it to.
Jobs that do not do anything for a period of time (121 seconds according to what you got) will abort.
I am using SignalR to send messages back to a browser about a long running process. I am using the System.Threading.ManualResetEventSlim to run in a separate thread. This is the error:
w3wp.exe Error: 0 : SignalR exception thrown by Task: System.AggregateException: One or
more errors occurred. ---> System.Web.HttpException: The remote host closed the
connection. The error code is 0x80070057.
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean
throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
at System.Web.HttpResponse.Flush(Boolean finalFlush)
at System.Web.HttpResponse.Flush()
at System.Web.HttpResponseWrapper.Flush()
at Microsoft.Owin.Host.SystemWeb.CallStreams.OutputStream.Flush()
at Microsoft.AspNet.SignalR.Owin.ServerResponse.<Flush>b__0()
at Microsoft.AspNet.SignalR.TaskAsyncHelper.FromMethod(Action func)
--- End of inner exception stack trace ---
---> (Inner Exception #0) System.Web.HttpException (0x80070057): The remote host closed
the connection. The error code is 0x80070057.
at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean
throwOnDisconnect)
at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
at System.Web.HttpResponse.Flush(Boolean finalFlush)
at System.Web.HttpResponse.Flush()
at System.Web.HttpResponseWrapper.Flush()
at Microsoft.Owin.Host.SystemWeb.CallStreams.OutputStream.Flush()
at Microsoft.AspNet.SignalR.Owin.ServerResponse.<Flush>b__0()
at Microsoft.AspNet.SignalR.TaskAsyncHelper.FromMethod(Action func)<---
I saw some similar errors on the SignalR issues tracker. There was something about using:
TaskScheduler.UnobservedTaskException += new EventHandler((s, arg) =>
{
arg.SetObserved();
((AggregateException)arg.Exception).Handle(ex =>
{
// Do something with ex (log or ignore...)
return true;
});
}
I am making a pretty long query to Neo4j database using Neo4jClient and getting an exception which occurs pretty randomly. How to fix this?
System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Neo4jClient.GraphClient.<>c__DisplayClass3.<SendHttpRequestAsync>b__2(Task`1 requestTask) in c:\TeamCity\buildAgent\work\f1c4cf3efbf1b05e\Neo4jClient\GraphClient.cs:line 149
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Neo4jClient.GraphClient.<>c__DisplayClass1b`1.<Neo4jClient.IRawGraphClient.ExecuteGetCypherResultsAsync>b__1a(Task`1 responseTask) in c:\TeamCity\buildAgent\work\f1c4cf3efbf1b05e\Neo4jClient\GraphClient.cs:line 745
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Neo4jClient.GraphClient.<>c__DisplayClass3.<SendHttpRequestAsync>b__2(Task`1 requestTask) in c:\TeamCity\buildAgent\work\f1c4cf3efbf1b05e\Neo4jClient\GraphClient.cs:line 149
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
---> (Inner Exception #0) System.Threading.Tasks.TaskCanceledException: A task was canceled.<---
<---
This is being tracked as an issue at https://bitbucket.org/Readify/neo4jclient/issue/70/taskcancelledexception
Diagnosis and eventual 'official' resolution will be posted there.
It took me hours to identify this issue and fix it.
[edit]: Don't use Neo4jClient.GraphClient, use Neo4jClient.BoltGraphClient - both derive from IGraphClient - the BoltGraphClient doesn't use an HttpClient behind the scenes and is way faster and less memory intensive.
var BoltGraphClient = new Neo4jClient.BoltGraphClient(url,username,password);
my old answer + story:
I'm putting a ton of Cypher queries into a List<Task> and executing them via query.ExecuteWithoutResultsAsync(). The Neo4j server can only handle so much at a time and will put the request in a queue.
I've confirmed that the TaskCanceledException gets thrown after 100 seconds, which is the default timeout of HttpClient.
After reading the documentation, I've figured out how to specify an infinite timespan during the initialization of the graphclient. Hope this will save you time.
var httpClientWrapper = new Neo4jClient.HttpClientWrapper(
username,
password,
new System.Net.Http.HttpClient() {
Timeout = System.Threading.Timeout.InfiniteTimeSpan
});
var graphClient = new Neo4jClient.GraphClient(new Uri(url), httpClientWrapper);
I am running Workflow 4 activities, and using SQL Durable Instancing to persist the Workflow when it's suspended. I am intermittently getting an InstanceOwnerException, which is remedied by recreating the instance store. Is there any way of avoiding this exception in the first place?
System.OperationCanceledException: An error processing the current work item has caused the workflow to abort. See the inner exception for details. ---> System.Runtime.DurableInstancing.InstanceOwnerException: The execution of an InstancePersistenceCommand was interrupted because the instance owner registration for owner ID '55413a1f-0e3f-4a0f-b2b0-54c121aa0c52' has become invalid. This error indicates that the in-memory copy of all instances locked by this owner have become stale and should be discarded, along with the InstanceHandles. Typically, this error is best handled by restarting the host.
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.Runtime.DurableInstancing.InstanceStore.EndExecute(IAsyncResult result)
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.OnPersisted(IAsyncResult result)
at System.Runtime.AsyncResult.SyncContinue(IAsyncResult result)
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.Persist()
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.CollectAndMap()
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.Track()
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult.InitializeProvider()
at System.Activities.WorkflowApplication.UnloadOrPersistAsyncResult..ctor(WorkflowApplication instance, TimeSpan timeout, PersistenceOperation operation, Boolean isWorkflowThread, Boolean isInternalPersist, AsyncCallback callback, Object state)
at System.Activities.WorkflowApplication.BeginInternalPersist(PersistenceOperation operation, TimeSpan timeout, Boolean isInternalPersist, AsyncCallback callback, Object state)
at System.Activities.WorkflowApplication.OnBeginPersist(AsyncCallback callback, Object state)
at System.Activities.Runtime.ActivityExecutor.PersistenceWaiter.PersistWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)
--- End of inner exception stack trace ---
The code is running on Azure, using SQL Azure to persist the Workflows. There are roles that are accessing the same instance store.
This actually turned out to be down to attempting to resume a workflow after it had already finished.