We started seeing this exception occur intermittently when our Azure App Service was moved to an App Service Environment. We're using Identity Server 4, and the exception occurs during token signing. This can be seen in the call stack below.
The signing cert is never saved to a cert store, it's loaded from a database as a byte array:
new X509Certificate2(rawData, password,
X509KeyStorageFlags.MachineKeySet |
X509KeyStorageFlags.PersistKeySet);
These references appear to be the same issue:
.NET Core X509Certificate2.PrivateKey throws nte_bad_keyset error
https://github.com/dotnet/corefx/issues/2583
Edit:
Initially we thought this was something specific to the App Service Env in Azure, but now we're seeing the exception in standard App Services. As a test we created an Azure web job that loads a certificate from a byte array and creates JWT tokens (which require a private key for signing). This repros the error. Also of note is the Azure Web job won't even run when certain flags are passed to the X509Certificate2 ctor (e.g. X509KeyStorageFlags.UserKeySet).
Stack trace is the same each time:
System.Security.Cryptography.CryptographicException:
at Internal.NativeCrypto.CapiHelper.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeProvHandle()
at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeKeyHandle()
at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 keySize, CspParameters parameters, Boolean useDefaultKeySize)
at System.Security.Cryptography.RSACryptoServiceProvider..ctor(CspParameters parameters)
at Internal.Cryptography.Pal.CertificatePal.<>c.<GetRSAPrivateKey>b__59_0(CspParameters csp)
at Internal.Cryptography.Pal.CertificatePal.GetPrivateKey[T](Func`2 createCsp, Func`2 createCng)
at Internal.Cryptography.Pal.CertificatePal.GetRSAPrivateKey()
at Internal.Cryptography.Pal.CertificateExtensionsCommon.GetPrivateKey[T](X509Certificate2 certificate, Predicate`1 matchesConstraints)
at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKey()
at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_HasPrivateKey()
at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider.HasPrivateKey(SecurityKey key)
at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider..ctor(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateSignatureProvider(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateForSigning(SecurityKey key, String algorithm)
at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.CreateEncodedSignature(String input, SigningCredentials signingCredentials)
at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.WriteToken(SecurityToken token)
at IdentityServer4.Services.DefaultTokenCreationService.CreateJwtAsync(JwtSecurityToken jwt)
at IdentityServer4.Services.DefaultTokenCreationService.<CreateTokenAsync>d__3.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Services.DefaultTokenService.<CreateSecurityTokenAsync>d__9.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at IdentityServer4.ResponseHandling.TokenResponseGenerator.<CreateAccessTokenAsync>d__10.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.ResponseHandling.TokenResponseGenerator.<ProcessTokenRequestAsync>d__8.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.ResponseHandling.TokenResponseGenerator.<ProcessAsync>d__6.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Endpoints.TokenEndpoint.<ProcessTokenRequestAsync>d__6.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Endpoints.TokenEndpoint.<ProcessAsync>d__5.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Hosting.IdentityServerMiddleware.<Invoke>d__3.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Hosting.FederatedSignOutMiddleware.<Invoke>d__6.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Hosting.AuthenticationMiddleware.<Invoke>d__2.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.<Invoke>d__7.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Hosting.BaseUrlMiddleware.<Invoke>d__2.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ApplicationInsights.AspNetCore.ExceptionTrackingMiddleware.<Invoke>d__4.MoveNext()
To resolve the issue we switched from X509 certificates (X509Certificate2) to RSA generated keys (RSACryptoServiceProvider). Identity Server 4 supports both. The exception no longer occurs.
That error means that somewhere after the certificate learned where the key was being stored it got deleted.
Maybe something is calling CngKey.Delete
Maybe something is cloning it into an RSACryptoServiceProvider and setting PersistKeyInCsp to false.
Maybe something is just cleaning up the machine keys directory after the fact.
The other (very rare) known case is where two threads load the PFX in parallel (they don't have to be in the same process) with one of them not having PersistKeySet asserted at import time.
If you're loading and discarding the certificates you really shouldn't set PersistKeySet. Every time you open a PFX another file is created on disk for each private key contained therein... those files get cleaned up normally (as long as the process doesn't abnormally terminate), but PersistKeySet prevents that cleanup from happening.
If nothing in code analysis reveals the two obvious key deletion locations (PersistKeyInCsp=false being the sneaky one) then you'll have to follow the filesystem auditing recommendations in .NET Core X509Certificate2.PrivateKey throws nte_bad_keyset error.
Maybe you didn't give IIS access to your private keys. I had same error, but it was necessary to use X509 Certs instead of keys. I've just granted a permission to read private keys from cert for IIS_IUSRS on MMC. It helps for me.
There's a full answer from #thames:
How to give ASP.NET access to a private key in a certificate in the certificate store?
UPD:
Step-by-step guide:
Create/purchase cert with private key
Import cert to "Local Computer" account (not "Current User"). Check "Allow private key to be exported" in Import Wizard.
Open MMC Console and add Certificates Snap-In. Choose "Computer Account".
Find your cert and right-click on it. Choose "All Tasks" > "Manage Private Keys"
Add Read access to your key for user IIS_IUSRS or IIS AppPool\<AppPoolName> (depends on IIS version and config. It's better to use "Advanced" button to show all possible options)
Restart your application
Related
I developed an azure function inside Visual Studio, which access some graph api endpoints and writes into azure storage some information. When i tested the function on my pc, everything works perfect.
When i deployed the function into azure, everything works but not the azure storage part. What am I doing wrong?
The variables are in DEV-OPS pipeline and I check in azure portal and there are present as well.
Edit 1: As I started to try to log information I was able to see that my functions is just running the first call and not even finishing it. I used before and after each call.
log.Info()
This is the GraphAPI Call, I know it is in preview but this shouldn't be a problem as this code is running fine on my pc in visual studio.
public static async Task<ChatMessage> CreateNewMessageThread(Employee employee)
{
var chatMessage = ChatThreadMainMessage(employee);
var result = await GraphServiceClient
.Teams[ApplicationConfigurationFields.TeamsId]
.Channels[ApplicationConfigurationFields.ChannelId]
.Messages
.Request()
.AddAsync(chatMessage);
return result;
}
Edit 2: I am getting a BadRequest from the call which throws an exception. although i am getting a bad request, the message is still created... Even if I find the problem I would like to understand why is that happening.
This is the stack trace, don't know if it will help someone.
Status Code: BadRequest
Microsoft.Graph.ServiceException: Code: BadRequest
Message: Bad Request
Inner error:
AdditionalData:
request-id: 220c1508-5bbd-xxxxx-xxxxx
date: 2020-04-27T16:02:01
ClientRequestId: 220c1508-5bbd-xxxxxx-xxxxx
at Microsoft.Graph.HttpProvider.<SendAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Graph.BaseRequest.<SendRequestAsync>d__35.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Graph.BaseRequest.<SendAsync>d__31`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MsTeamsNotifications.MsTeamsNotifications.<CreateNewMessageThread>d__19.MoveNext()
Thank you!
I am pretty confident this is caused by Teams engineers screwing something up again, but might be worth a try...
Using Microsoft.Bot.Builder 4.8.0
I's a pretty simple problem actually. When called from Task OnMessageActivityAsync(ITurnContext<IMessageActivity> turnContext, CancellationToken cancellationToken),
the following code:
await turnContext.SendActivityAsync(typing).ConfigureAwait(false);
causes the following exception:
Operation returned an invalid status code 'BadGateway'
with the following stack trace:
at Microsoft.Bot.Connector.Conversations.<ReplyToActivityWithHttpMessagesAsync>d__10.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.Bot.Connector.ConversationsExtensions.<ReplyToActivityAsync>d__17.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Microsoft.Bot.Builder.BotFrameworkAdapter.<SendActivitiesAsync>d__34.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Bot.Builder.TurnContext.<>c__DisplayClass25_0.<<SendActivitiesAsync>g__SendActivitiesThroughAdapter|1>d.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.Bot.Builder.TurnContext.<SendActivityAsync>d__24.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Blabla.<OnMessageActivityAsync>d__4.MoveNext() in path\to\blabla.cs:line xx
Replying with text messages works fine.
Oh and just in case, if you are wondering if I tried to use ShowTypingMiddleware to display "typing" indicator, sure, tried that too, but the indicator is never shown. I did that by adding Use(new ShowTypingMiddleware(1000)); to my BotFrameworkHttpAdapter constructor, which is registered to DI container as singleton. It's just that, it does nothing.
Have a look at the Microsoft Teams: Developer Blog; API outage: Bots posting typing indicators may receive 502s.
As a part of our commitment to customers and Microsoft cloud services continuity during these unprecedented times, we’re making temporary adjustments to select capabilities within Microsoft 365. One of those adjustments is that we are temporarily scaling back typing indicators support in Teams. As a result bots posting typing indicators to users in the Europe region using any of the following values for ServiceUrl may receive errors and users may not see the typing indicators sent by bots.
Currently the guidance in the post above is:
As a bot developer, you may either modify your code to not use typing indicators for now, or ignore the errors encountered while sending typing indicators.
I am using Azure Active Directory login via ASP.NET OWIN Identity for authentication.
I have a load balancer which is doing SSL offloading, so my server itself is running http protocol and not https protocol, but the load balancer is SSL.
I am getting this error:
[WebException: The remote server returned an error: (400) Bad Request.]
System.Net.HttpWebRequest.GetResponse() +1686
Microsoft.IdentityModel.Clients.ActiveDirectory.<GetResponseSyncOrAsync>d__2.MoveNext() +392
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Microsoft.IdentityModel.Clients.ActiveDirectory.<SendPostRequestAndDeserializeJsonResponseAsync>d__0`1.MoveNext() +512
[AdalServiceException: AADSTS70002: Error validating credentials. AADSTS50011: The reply address 'http://intx.mydomain.com/default' does not match the reply address 'https://intx.mydomain.com/default' provided when requesting Authorization code.
Trace ID: d5861c62-9d42-4f2b-886a-06cc63b01f00
Correlation ID: f77d0d3e-9dd2-4af8-a38f-65f7838bc680
Timestamp: 2018-10-28 05:13:16Z]
Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.RunAsyncTask(Task`1 task) +133
CloudmersiveAdminApp.Startup.<ConfigureAuth>b__10_0(AuthorizationCodeReceivedNotification context) in C:\Src\APIs\Account3\CloudmersiveAccount\CloudmersiveAdminApp\App_Start\StartupAuth.cs:109
Microsoft.Owin.Security.OpenIdConnect.<AuthenticateCoreAsync>d__1a.MoveNext() +5428
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
Microsoft.Owin.Security.OpenIdConnect.<AuthenticateCoreAsync>d__1a.MoveNext() +5937
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Microsoft.Owin.Security.Infrastructure.<BaseInitializeAsync>d__0.MoveNext() +817
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Microsoft.Owin.Security.Infrastructure.<Invoke>d__0.MoveNext() +329
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Microsoft.Owin.Security.Infrastructure.<Invoke>d__0.MoveNext() +768
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.<RunApp>d__5.MoveNext() +196
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.<DoFinalWork>d__2.MoveNext() +184
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar) +117
System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +367
System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step) +195
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +128
Note how it is complaining about the protocol being http and not https. The thing is - in the browser, it is https! It is just that the https part is being offloaded by the load balancer.
How can I configure ASP.NET to not throw this error? Any ideas would be greatly appreciated.
Looking at the exception, your problem might be related to reply URL specified in code/configuration.
Make sure that any request to Azure AD still specifies the redirect URL/reply URL to be https://intx.mydomain.com/default.
Usual suspects would be -
a. StartupAuth.cs (around line 109 as per stack trace). Check in that file wherever you have specified RedirectUri
b. Any other place in code where you try to acquire token and specify redirect URL explicitly.
If it still doesn't work out, try adding some more code details to your question about Startup configuration and Token acquire logic.
I am working on an MVC web application that uses Google Natural Language Processing API to parse different input from users.
I have successfully consumed and implemented the API operations and everything works fine as long as I run the application on my local machine. But as soon as I publish a version and upload it on a server I receive the following error on calling the API methods (e.g. AnalyzeSentiment):
Status(StatusCode=Unauthenticated, Detail="Getting metadata from plugin failed with error: Exception occured in metadata credentials plugin.")
With the help of the answers from post: Google Datastore authentication issue - C# I was able to further get details on the error (using gRCP):
An error occurred while sending the request.
Stacktrace: at Google.Apis.Http.ConfigurableMessageHandler.<SendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.Requests.TokenRequestExtenstions.<ExecuteAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.ServiceAccountCredential.<RequestAccessTokenAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.ServiceCredential.<GetAccessTokenForRequestAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.ServiceAccountCredential.<GetAccessTokenForRequestAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Grpc.Auth.GoogleAuthInterceptors.<>c__DisplayClass2_0.<<FromCredential>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Grpc.Core.Internal.NativeMetadataCredentialsPlugin.<GetMetadataAsync>d__11.MoveNext()
This seemed like an authentication issue so I double checked the jsonKey file which is fine. Please note, I have used code to set the credentials in Environment variables:
Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", jsonPath);
and verified it using:
Environment.GetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS")
The call is made as follows:
private AnalyzeSentimentResponse AnalyzeSentiment(string statement)
{
GrpcEnvironment.SetLogger(new MyLogger());
var client = LanguageServiceClient.Create();
var response = client.AnalyzeSentiment(new Document()
{
Content = statement,
Type = Document.Types.Type.PlainText
});
return response;
}
Cannot figure out why it works fine when I run it on my local machine and fails when it is deployed on the server. There is also no restrictions of any kind on the said server.
The result for:
GoogleCredential.GetApplicationDefaultAsync().Result.UnderlyingCredential.GetType()
is:
Google.Apis.Auth.OAuth2.ServiceAccountCredential
Note: The server is our own (Windows Server 2012R2)
With the suggestion given by #JonSkeet, I copied the code into a console application and executed the call. Unfortunately, the issue persisted. What I did next was to move the console application onto another server, it worked there.
So, it was indeed an issue with the server where there maybe some features missing (the firewall is disabled). Network dept is checking it out whereas I have deployed my web application on another server.
Update: There was an issue on the server where some required framework features were not installed. The issue has been resolved by moving the deployment to another server.
I just made an WebApi (C#, .net 4.5.2) and published it to the web. In order to make sure it working good, I started a to test it.
The REST web-service failed the "stress" test. I sent the service 30+- http requests, each second, and got back this typical error message:
System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Net.Http.HttpContentExtensions.<ReadAsAsyncCore>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.ModelBinding.FormatterParameterBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.HttpActionBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()
Same error occurred many times in the log.
When I entered the specific machine, I saw that the CPU is on 100% and the RAM is on 80%. Which mean, the machine gives her best to handle the load.
Questions
According to the StackTrace I provided above, I cannot understand
where exactly was the problem in my code?
My code is scale out automatically (hosted in the cloud). But, new machine created only after 5 minutes (depends on the average CPU > 50%). How should I handle sudden mass of HTTP request? Maybe the server should say: "Hey, try agian in 30 seconds" or something like that? What is the right solution?
System.OperationCanceledException: The operation was canceled.
This usually indicates that the client connecting to the service closed the connection before the service could send a response. In terms of api layer this usually means that you have specified a timeout while establishing the http connection, or the library that you are using for making http calls has a default timeout.
You may want to check the iis request logs to see what is going wrong. Since this is happening on a stress test, my guess is your service is running short of threads (Read this for more details) . You may want to consider async model for your request handlers to improve the scalability and utilize threads better. (Read this)