The application I've been working on has recently started experiencing intermittent hard crashes which I'm unable to explain. To make this more weird we have both AppDomain.CurrentDomain.UnhandledException and TaskScheduler.UnobservedTaskException set up to catch problems like these.
Usually when something goes wrong the above two work correctly, but we have certain circumstances where they are not catching unhandled exceptions. To make matters even worse the exception that's causing this is a NRE which means we have nothing to go on in terms of tracking down what's going on.
My first thought was that maybe the code we have in the unhandled exception handlers might be throwing exceptions, so we doubled down on exception handling here, but so far that hasn't made any difference. The logs we have also haven't been able to shine any light on this.
This may be a bug with webapi and/or the owin bits we are using to host the application, and I'm going to try bumping the dependencies to see if that helps, but I don't think it will make any difference.
Is there a way to trap unhandled exceptions occurring in the thread pool? Is there some kind of diagnostics we can enable to get more info as to where this is happening?
I'm at my wits end, any help would be greatly appreciated.
System.NullReferenceException: Object reference not set to an instance of an object.
at System.Net.HttpListener.EndGetContext(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at Microsoft.Owin.Host.HttpListener.OwinHttpListener.<ProcessRequestsAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>b__5(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
When using tasks we do not do async void, but we have one case of Task.Run where we do not await or hang on the task. This is:
Task.Run(() => CleanUpAllExpiredErrors());
This method cleans up expired client errors (we capture client side stack traces on unhandled client errors).
By adding extensive tracing and logging we saw that this crash was consistently happening when serving static assets, specifically stylesheets. We white-listed our static assets directories to not require authentication and the crash went away.
We configure certain endpoints to not require authentication by doing the following:
var listener = (HttpListener)app.Properties["System.Net.HttpListener"];
listener.AuthenticationSchemeSelectorDelegate = request =>
{
// Options requests should always be anonymous otherwise preflight cors requests
// will fail in Firefox.
if (request.HttpMethod == "OPTIONS")
{
return AuthenticationSchemes.Anonymous;
}
// here we add additional whitelisted paths
var pathsWithoutAuthentication = KnownPaths.AnonymousWhiteList.Concat(KnownPaths.Assets);
return pathsWithoutAuthentication.Any(a => request.RawUrl.ToLower().StartsWith(a))
? AuthenticationSchemes.Anonymous
: AuthenticationSchemes.IntegratedWindowsAuthentication;
};
Related
I am using a TcpClient with an SslStream. In general everything works fine but we are getting occasional logs of crashes. It seems to be coming from unstable network connections, and I'm unfortunately not able to reproduce it locally despite trying to create "unexpected circumstances":
using Fiddler to replicate a very slow network connection
killing the server application while its sending data
disconnecting network cables
The situation is this:
I call BeginRead like this:
_tcpClientStream.BeginRead(receiveBuffer, 0, receiveBuffer.Length, pfnCallBack, null);
And then EndRead is being called like this:
iRx = _tcpClientStream.EndRead(asyn);
The only place that the iRx is used is to copy the received data to a new array:
var bytes = new byte[iRx];
Array.Copy(receiveBuffer, bytes, iRx);
In the OnDataReceived callback function, I call EndRead on the SslStream, which throws an exception that is handled:
Error in OnDataReceived System.IO.IOException: The read operation failed, see inner exception. ---> System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: size
at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
at System.Net.FixedSizeReader.StartReading()
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.Security.SslStream.EndRead(IAsyncResult asyncResult)
at XXXXXXX.OnDataRecevied(IAsyncResult asyn)
Normally this is not a problem - this exception is caught and the TcpClient is closed and re-initiated to attempt to reconnect.
However, in this case there is a second Exception that is thrown and comes from one of the UnhandledException error handlers:
AppDomain.CurrentDomain.UnhandledException
Current.DispatcherUnhandledException
Current.Dispatcher.UnhandledException
TaskScheduler.UnobservedTaskException
This stack trace is like this:
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Our handling for this type of unhandled error is to display a message to the user that there is a fatal error and the software has to close.
My questions are:
Has anyone ever seen this error before and been able to reproduce the exact out of range exception I'm getting? Despite my attempts to recreate the issue (using the methods described above), I'm not able to ever recreate the exception, it is always the more standard Socket IO exception:
Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
at System.Net.Security.SslStream.EndRead(IAsyncResult asyncResult)
at XXXXXXXXX.OnDataRecevied(IAsyncResult asyn)
Is there any way I can handle this exception in my Socket code or is it always going to appear in one of these UnhandledException events? If so, what is the best practice way of handling this? Basically I need some logic in the event handler to identify that the software should not shut down if this specific error is received?
Have service that receives requests, generates data and saves this in files to AWS S3.
If service receives many requests can try to save up to 20 files (2 servers x 10 workers each) in parallel.
Data generated by requests and saved to S3 can be from few KB to around ~400MB
Problem is that sometimes (seems to be when service is busy/big files to save) S3 fails with the exception below:
We discussed 2 solutions:
1) Implement a retry of S3.UploadAsync() if save fails.
Not sure if will make any difference. Assume S3 already retries internally so maybe is no point to retry. If problem is that files are too big/takes to long to save this won't solve the issue maybe make it worst.
2) Increase the TransferUtilityConfig.DefaultTimeout to, lets say 10min (default is 5min).
If problem is that saving takes more than 5 minutes, this will fix the issue but exception thrown by S3 doesn't indicate is a timeout exception so maybe this will solve anything.
3) Is this an intermittent issue in AWS infrastructure? Could retry help?
Does anyone have experience/solution when this exception happens? Any other ideas?
UPDATE:
TransferUtilityConfig() does not contain DefaultTimeout if using NET 4.5. The functionality has been moved to AmazonS3Config. This offers more parameters to control the upload: Timeout, ReadWriteTimeout, MaxErrorRetry
AmazonS3Config Class
Settings are explained here
AWS Retries and Timeouts
This is the code used by the service to save:
using (var amazonS3Client = new AmazonS3Client(RegionEndpoint.GetBySystemName(_iAwsS3Settings.RegionEndpoint)))
using (var fileTransferUtility = new TransferUtility(amazonS3Client))
using (var memoryStream = new MemoryStream(data))
{
var fileTransferUtilityRequest = new TransferUtilityUploadRequest
{
BucketName = _iAwsS3Settings.BucketName,
InputStream = memoryStream,
StorageClass = S3StorageClass.ReducedRedundancy,
PartSize = 6291456, // 6 MB.
Key = fileLocation,
CannedACL = S3CannedACL.BucketOwnerFullControl
};
await fileTransferUtility.UploadAsync(fileTransferUtilityRequest, ct);
}
This is the exception given when S3 save fails:
System.AggregateException: One or more errors occurred. --->
Amazon.Runtime.AmazonServiceException: A WebException with status
SecureChannelFailure was thrown. ---> System.Net.WebException: The
request was aborted: Could not create SSL/TLS secure channel. at
System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult
asyncResult, TransportContext& context) at
System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult
asyncResult) at
System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult
iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean
requiresSynchronization) --- End of stack trace from previous
location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.HttpHandler1.<InvokeAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
Amazon.Runtime.Internal.HttpHandler1.<InvokeAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.RedirectHandler.d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.Unmarshaller.<InvokeAsync>d__31.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.S3.Internal.AmazonS3ResponseHandler.d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__51.MoveNext()
--- End of inner exception stack trace --- at Amazon.Runtime.Internal.WebExceptionHandler.HandleException(IExecutionContext
executionContext, WebException exception) at
Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext
executionContext, Exception exception) at
Amazon.Runtime.Internal.ErrorHandler.d__51.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.CredentialsRetriever.d__71.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__101.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
Amazon.Runtime.Internal.RetryHandler.d__10`1.MoveNext()
This is kind of an old question but I just had a very similar issue and because I was unable to find an appropriate answer on Google, I wanted to contribute my solution.
I had started implementation of an older API which had the following line:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
I was able to correct the issue by changing SecurityProtocolType.Ssl3 to SecurityProtocolType.Tls12
Further explanation:
SSL3 is not generally supported due to the Poodle vulnerability:
https://security.stackexchange.com/questions/70719/ssl3-poodle-vulnerability/70724#70724
I had not added this line of code myself and as a result, I was having great difficulty finding the source of the issue. However, while researching I had come across this post regarding a similar issue: https://github.com/aws/aws-sdk-net/issues/86
Near the bottom of this discussion, someone suggested adding the following line:
ServicePointManager.CheckCertificateRevocationList = true;
This fix was not able to correct my issue but it did lead me down the correct path. After searching for other references to the ServicePointManager, I was able to find the previously mentioned line regarding the SecurityProtocolType and correct it.
S3 SDK already implements retry logic
By default, an upload is retried 4 times
Created a console application to try to reproduce the error. Console application tried to upload 10-30 files asynchronously. Changing the values in AmazonS3Config for Timeout, ReadWriteTimeout, MaxErrorRetry produced exceptions (System.Net.WebException: The operation has timed out) but not the same we had (Could not create SSL/TLS secure channel).
We hypothesized that the problem could that the service is so busy that can not create the connection, that is why get "Could not create SSL/TLS secure channel"
My exception was:
A WebException with status ConnectFailure was thrown
For me the ServiceUrl was http instead of https and also didn't need a port number. The port number was blocked by the firewall.
Before:
http://s3ws.MyDomain.com:80
After:
https://s3ws.MyDomain.com
var s3Config = new AmazonS3Config
{
RegionEndpoint = Amazon.RegionEndpoint.EUNorth1,
ServiceURL = "https://s3ws.MyDomain.com",
ForcePathStyle = true
};
return new AmazonS3Client(accessKey, secretKey, s3Config);
UPDATE 4 Even More confused
I thought I was onto something, but the same error keeps popping up. This solution seemed promising, but isn't 100%, but thought it might help someone narrow down what's wrong.
Steps:
Restart API and Identity Server IIS Application Pools
Get Access Token Via SoapUI using QA URL
Hit API in QA with SoapUI
Failure - Doesn't work (See below for this stacktrace)
Hit API locally with SoapUI (Using same QA Identity Server URL)
Local API URL returns expected data
Hit API using QA URL using SoapUI (I'm just changing the endpoint in SoapUI)
QA API URL returns expected data
I was able to repeat that 4 times before I would still get the Error when hitting QA. After restarting this process again, I can now get my QA environment to work without having to use the API locally. Any ideas what's going on? Seems to be a setup issue/proxy/certificate issue, but no idea how to debug that.
This is the error that I'm seeing now:
Server Error in '/' Application.
An existing connection was forcibly closed by the remote host
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[SocketException (0x2746): An existing connection was forcibly closed by the remote host]
System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) +8156963
System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) +48
[IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.]
System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult) +8111720
System.Net.TlsStream.EndRead(IAsyncResult asyncResult) +275
System.Net.Connection.ReadCallback(IAsyncResult asyncResult) +45
[WebException: The underlying connection was closed: An unexpected error occurred on a receive.]
System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) +764
System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar) +78
[HttpRequestException: An error occurred while sending the request.]
[AggregateException: One or more errors occurred.]
System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) +4451240
Microsoft.IdentityModel.Protocols.<GetDocumentAsync>d__0.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\HttpDocumentRetriever.cs:53
[IOException: Unable to get document from: https://securityeliqa.twcable.com/core/.well-known/openid-configuration]
Microsoft.IdentityModel.Protocols.<GetDocumentAsync>d__0.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\HttpDocumentRetriever.cs:59
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.IdentityModel.Protocols.<GetAsync>d__0.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\OpenIdConnectConfigurationRetriever.cs:81
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.IdentityModel.Protocols.<GetConfigurationAsync>d__3.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\ConfigurationManager.cs:0
[InvalidOperationException: IDX10803: Unable to create to obtain configuration from: 'https://securityeliqa.twcable.com/core/.well-known/openid-configuration'.]
Microsoft.IdentityModel.Protocols.<GetConfigurationAsync>d__3.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\ConfigurationManager.cs:212
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
Microsoft.IdentityModel.Protocols.<GetConfigurationAsync>d__0.MoveNext() in c:\workspace\WilsonForDotNet45Release\src\Microsoft.IdentityModel.Protocol.Extensions\Configuration\ConfigurationManager.cs:0
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() +31
IdentityServer.WebApi.AccessTokenValidation.<<RetrieveMetadata>b__0>d__4.MoveNext() in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\DiscoveryDocumentIssuerSecurityTokenProvider.cs:123
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +13847892
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +61
System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() +31
IdentityServer.WebApi.AccessTokenValidation.AsyncHelper.RunSync(Func`1 func) in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\AsyncHelper.cs:18
IdentityServer.WebApi.AccessTokenValidation.DiscoveryDocumentIssuerSecurityTokenProvider.RetrieveMetadata() in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\DiscoveryDocumentIssuerSecurityTokenProvider.cs:141
IdentityServer.WebApi.AccessTokenValidation.DiscoveryDocumentIssuerSecurityTokenProvider..ctor(String discoveryEndpoint, IdentityServerBearerTokenAuthenticationOptions options, ILoggerFactory loggerFactory) in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\DiscoveryDocumentIssuerSecurityTokenProvider.cs:43
Owin.IdentityServerBearerTokenValidationAppBuilderExtensions.ConfigureLocalValidation(IdentityServerBearerTokenAuthenticationOptions options, ILoggerFactory loggerFactory) in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\IdentityServerBearerTokenValidationAppBuilderExtensions.cs:129
Owin.IdentityServerBearerTokenValidationAppBuilderExtensions.UseIdentityServerBearerTokenAuthentication(IAppBuilder app, IdentityServerBearerTokenAuthenticationOptions options) in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\IdentityServerBearerTokenValidationAppBuilderExtensions.cs:39
Company.WebApi.waAddressQualification.Startup.Configuration(IAppBuilder app) in e:\Source Code\GitHub\waDemo\Startup.cs:23
[TargetInvocationException: Exception has been thrown by the target of an invocation.]
System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) +0
System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) +128
System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) +146
Owin.Loader.<>c__DisplayClass12.<MakeDelegate>b__b(IAppBuilder builder) +93
Owin.Loader.<>c__DisplayClass1.<LoadImplementation>b__0(IAppBuilder builder) +209
Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(Action`1 startup) +843
Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(Action`1 startup) +51
Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint() +101
System.Threading.LazyInitializer.EnsureInitializedCore(T& target, Boolean& initialized, Object& syncLock, Func`1 valueFactory) +137
Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(HttpApplication context) +172
System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers) +618
System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context) +172
System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context) +402
System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext) +343
[HttpException (0x80004005): Exception has been thrown by the target of an invocation.]
System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +579
System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +112
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +712
Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.6.1055.0
UPDATE 3 Even More confused
At this point I believe it's the server that is cancelling my requests, but even with that I can't be 100% sure. Here are some additional data points:
Service is setup on both a Dev and QA box, Dev box gets the error about 1 out of 100 requests, QA, about 50% of the time.
The QA box has Identity Server setup as https behind a load balancer, Dev is http
The service setup mimics Identity Server where in QA it is https behind a load balancer, Dev is http
A set of three address in QA will usually have at least one address fail, I see the request in the server log for the two that work, shouldn't I see the third? If the third request is not in the server logs does that mean it never got there?
What really gets me is I can't seem to see an error anywhere but on the client, where the HttpClient.GetAsync call is made. If the server is cancelling the request, shouldn't there be a log for each one it cancels?
I've found in the Event log on on the windows server hosting the service errors about cancelling a task, however not all failed requests end up here?
The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at System.Web.Http.Owin.HttpMessageHandlerAdapter.d__13.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Owin.HttpMessageHandlerAdapter.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at IdentityServer.WebApi.AccessTokenValidation.ScopeRequirementMiddleware.d__0.MoveNext() in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\Plumbing\ScopeRequirementMiddleware.cs:line 48 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at IdentityServer.WebApi.AccessTokenValidation.IdentityServerBearerTokenValidationMiddleware.d__6.MoveNext() in e:\Source Code\GitHub\IdentityServer\IdentityServer.WebApi.AccessTokenValidation\IdentityServerBearerTokenValidationMiddleware.cs:line 81 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContext.d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar) at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Original Questions
I'd like to know if I'm call my Demo API correctly, or if the API is causing me an issue. If I send three requests through at the same time I get the issue on at least one of the addresses I'm trying to process.
Update Number one
In the Inner Exception I'm getting the following message, I was able to capture a more complete error as follows: "The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host"
I'm using .NET 4.6.1 for both client and server side
I'm not using a proxy.
I'm able to successfully call the API about 1/3 of the time
I've experimented with the couple of different options in calling the service, the first was just wrapping the DemoApiClient in a lock, the next was to use Task Run with the async and await keywords and finally retrieving the token for authentication using Task Run or just calling it and having ConfigureAwait(false) listed on the await lines. I'm not sure if any of this matters, or if the server is just failing when called. The server isn't that complex, but I'd rather not post that code here. It does validate the token and then process an address, makes a couple of database calls and returns data found from the database.
UPDATE 2
I test with three addresses at a time, when I send them through SoapUI to the server all three are process fine, but with the .Net C# client using the code below only one of the three requests seem to make it to the server. I was expecting to see two stacktraces and one successful request, but the IIS log only shows one request coming in.
The code I'm using to call the API is here:
using Demo.IdentityServer.IdentityModel.Client;
using Newtonsoft.Json;
using PartnerPortal.Model;
using System;
using System.Configuration;
using System.Diagnostics;
using System.Net.Http;
using System.Linq;
using System.Collections.Generic;
using System.Threading.Tasks;
namespace DemoPortal.Client
{
public static class DemoApiClient
{
public readonly static string waDemoUrl = ConfigurationManager.AppSettings["waDemoURL"];
public static async Task<DemoData> GetDemoData(RequestAddress requestAddress)
{
try
{
using (var httpClient = new HttpClient() { MaxResponseContentBufferSize = 10000000, Timeout = TimeSpan.FromMilliseconds(5000) })
{
string reasonPhrase = "";
var demoUri = new Uri(string.Format(waDemoUrl + "api/DemoApp/GetSomeData?trackingId={0}&clientKey={1}&address={2}", requestAddress.TrackingId, requestAddress.ClientKey, requestAddress.Address));
var httpResult = new HttpResponseMessage();
var result = RequestAccessToken.RequestToken().Result;
var accessToken = await Task.Run(() => RequestAccessToken.RequestToken().Result);
//var accessToken = RequestAccessToken.RequestToken().Result;
//which of the above two options should I use? Does it matter?
httpClient.SetBearerToken(accessToken);
HttpResponseMessage response = await httpClient.GetAsync(demoUri);
HttpContent httpContent = response.Content;
if (httpResult.IsSuccessStatusCode)
{
var content = await httpContent.ReadAsStringAsync();
DemoData demoData = null;
demoData = JsonConvert.DeserializeObject<DemoData>(content);
return demoData;
}
else
{
reasonPhrase = httpResult.ReasonPhrase;
if (reasonPhrase.ToUpper() == "UNAUTHORIZED")
{
throw new KeyNotFoundException("Not authorized");
}
}
}
}
catch (Exception ex)
{
Debug.WriteLine("Message is:" + ex);
throw (ex);
}
return null;
}
}
}
And this is how the token is retrieved:
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using System.Web;
using Thinktecture.IdentityModel.Tokens.Http;
namespace Demo.IdentityServer.IdentityModel.Client
{
public static class RequestAccessToken
{
public readonly static string securityUrl = ConfigurationManager.AppSettings["securityUrl"];
public readonly static string clientSecret = ConfigurationManager.AppSettings["clientSecret"];
public static async Task<string> RequestToken()
{
var url = new Uri(securityUrl);
var fields = new Dictionary<string, string>
{
{ OAuth2Constants.GrantType, OAuth2Constants.GrantTypes.ClientCredentials },
{ OAuth2Constants.Scope, "Read"}
};
using (var httpClient = new HttpClient())
{
var cancellationToken = new CancellationToken();
httpClient.DefaultRequestHeaders.Authorization = new BasicAuthenticationHeaderValue("Demo", clientSecret);
var ss = JsonConvert.SerializeObject(fields);
var data = new FormUrlEncodedContent(fields);
var s = data.ReadAsStringAsync();
//var response = await httpClient.PostAsync(url, data, cancellationToken).ConfigureAwait(false);
var response = await httpClient.PostAsync(url, data, cancellationToken);
// Should I use ConfigureAwait(false) or call method with a Task(Run() ... ?
if (response.StatusCode == HttpStatusCode.OK || response.StatusCode == HttpStatusCode.BadRequest)
{
//var content = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
var content = await response.Content.ReadAsStringAsync();
var token = new TokenResponse(content);
var accessToken = token.AccessToken;
return accessToken;
}
else
{
return new TokenResponse(response.StatusCode, response.ReasonPhrase).AccessToken;
}
}
}
}
}
SOLUTION Found!!! The entire issue stemmed from a misconfigured Load Balancer. Something with the SNAT pool. I'm not the network guy, and whatever he did fixed it. Something like the load balancer was referencing the box name directly instead of the SNAT Pool name, or the SNAT Pool had the wrong name. Either way once updated the intermittent successes became always successes.
I can tell you that you are flirting with SNAT port exhaustion at scale. It makes sense to instantiate HttpClient within a using block since it implements IDisposable. However, doing so will lead to HttpClient monopolizing one SNAT port for 240 seconds after it is disposed. See here for a great explanation of this problem.
I am running a winrt program with multiple tasks.
In the middle of running, I get the following exception:
Not enough quota is available to process this command
The stack trace is
at Windows.UI.Core.CoreDispatcher.InvokeAsync(CoreDispatcherPriority priority, InvokedHandler freeThreadedCallback, Object sender, Object context)
at System.Threading.WinRTSynchronizationContext.Post(SendOrPostCallback d, Object state)
at System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.InvokeOrPostAction(Object state)
at System.Threading.Tasks.AwaitTaskContinuation.RunInline(ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.Tasks.AwaitTaskContinuation.<ThrowAsyncIfNecessary>b__1(Object s)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
Then it crashes.
Is this a problem specific to WintRT?
Is there a workaround?
According to the MSDN documentation for that error message, the fix for that error is to either "close some applications" or "increase the size of your paging file".
Given that, you might want to start by looking in Task Manager and see if your application is using a totally unreasonable amount of memory (or if some other process is). If your process has a memory leak, it might lead to this problem.
I have a multi threaded .NET app that uses async I/O and AsyncCallbacks to handle the I/O completions. Rarely, the app will fail with an exception like:
Arithmetic operation resulted in an overflow.
at MyApp.ReadComplete(IAsyncResult ar) in c:\MyApp.cs:line 123
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Line 123 in MyApp.cs is the first executable line of the AsyncCallback and it is inside of a try/catch (Exception ex) but, the catch is NOT being executed.
Is the .NET Framework lying to me about where the exception occurred? Did the exception actually occur out in the async netherworld where I can't catch it? Why can't I catch this exception?
Try to use catch instead of catch(Exception ex). This catches everything incl. COM exceptions.
The actual answer is yes, the .NET Framework was wrong about where the exception occurred but, not because the exception occurred off in the async netherworld. The exception actually occurred in the callback method, many lines after the line indicated in the stack trace. The line where the exception actually occurred was NOT inside of a try/catch block so, the unhandled exception was expected.
I reported the incorrect stack trace to Microsoft but have no idea if it is considered a bug.
I think that it's safe to say that if your stack trace points to the first executable line of an async callback method, you should suspect that it could be wrong.