I am pretty confident this is caused by Teams engineers screwing something up again, but might be worth a try...
Using Microsoft.Bot.Builder 4.8.0
I's a pretty simple problem actually. When called from Task OnMessageActivityAsync(ITurnContext<IMessageActivity> turnContext, CancellationToken cancellationToken),
the following code:
await turnContext.SendActivityAsync(typing).ConfigureAwait(false);
causes the following exception:
Operation returned an invalid status code 'BadGateway'
with the following stack trace:
at Microsoft.Bot.Connector.Conversations.<ReplyToActivityWithHttpMessagesAsync>d__10.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.Bot.Connector.ConversationsExtensions.<ReplyToActivityAsync>d__17.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Microsoft.Bot.Builder.BotFrameworkAdapter.<SendActivitiesAsync>d__34.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Bot.Builder.TurnContext.<>c__DisplayClass25_0.<<SendActivitiesAsync>g__SendActivitiesThroughAdapter|1>d.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.Bot.Builder.TurnContext.<SendActivityAsync>d__24.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Blabla.<OnMessageActivityAsync>d__4.MoveNext() in path\to\blabla.cs:line xx
Replying with text messages works fine.
Oh and just in case, if you are wondering if I tried to use ShowTypingMiddleware to display "typing" indicator, sure, tried that too, but the indicator is never shown. I did that by adding Use(new ShowTypingMiddleware(1000)); to my BotFrameworkHttpAdapter constructor, which is registered to DI container as singleton. It's just that, it does nothing.
Have a look at the Microsoft Teams: Developer Blog; API outage: Bots posting typing indicators may receive 502s.
As a part of our commitment to customers and Microsoft cloud services continuity during these unprecedented times, we’re making temporary adjustments to select capabilities within Microsoft 365. One of those adjustments is that we are temporarily scaling back typing indicators support in Teams. As a result bots posting typing indicators to users in the Europe region using any of the following values for ServiceUrl may receive errors and users may not see the typing indicators sent by bots.
Currently the guidance in the post above is:
As a bot developer, you may either modify your code to not use typing indicators for now, or ignore the errors encountered while sending typing indicators.
Related
I developed an azure function inside Visual Studio, which access some graph api endpoints and writes into azure storage some information. When i tested the function on my pc, everything works perfect.
When i deployed the function into azure, everything works but not the azure storage part. What am I doing wrong?
The variables are in DEV-OPS pipeline and I check in azure portal and there are present as well.
Edit 1: As I started to try to log information I was able to see that my functions is just running the first call and not even finishing it. I used before and after each call.
log.Info()
This is the GraphAPI Call, I know it is in preview but this shouldn't be a problem as this code is running fine on my pc in visual studio.
public static async Task<ChatMessage> CreateNewMessageThread(Employee employee)
{
var chatMessage = ChatThreadMainMessage(employee);
var result = await GraphServiceClient
.Teams[ApplicationConfigurationFields.TeamsId]
.Channels[ApplicationConfigurationFields.ChannelId]
.Messages
.Request()
.AddAsync(chatMessage);
return result;
}
Edit 2: I am getting a BadRequest from the call which throws an exception. although i am getting a bad request, the message is still created... Even if I find the problem I would like to understand why is that happening.
This is the stack trace, don't know if it will help someone.
Status Code: BadRequest
Microsoft.Graph.ServiceException: Code: BadRequest
Message: Bad Request
Inner error:
AdditionalData:
request-id: 220c1508-5bbd-xxxxx-xxxxx
date: 2020-04-27T16:02:01
ClientRequestId: 220c1508-5bbd-xxxxxx-xxxxx
at Microsoft.Graph.HttpProvider.<SendAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Graph.BaseRequest.<SendRequestAsync>d__35.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Graph.BaseRequest.<SendAsync>d__31`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MsTeamsNotifications.MsTeamsNotifications.<CreateNewMessageThread>d__19.MoveNext()
Thank you!
I get below error when attempting to be offline, I began with the starter code as instructed by the Docs and I just keep getting the error.
I followed the details here: https://learn.microsoft.com/en-us/azure/app-service-mobile/app-service-mobile-windows-store-dotnet-get-started-offline-data
Even trying a new download of the starter code doesn't seem to work, for what ever reason the app refuses to acknowledge that it is offline. My offline data is synced and is available, but as soon as I am without connectivity the app crashes with the following exception:
Message: The text associated with this error code could not be found.
The server name or address could not be resolved StackTrace:
{System.Runtime.InteropServices.COMException (0x80072EE7): The text
associated with this error code could not be found. The server name or
address could not be resolved at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
System.Net.Http.HttpHandlerToFilter.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
System.Net.Http.HttpClientHandler.d__86.MoveNext()}
The fact that it is even attempting to resolve is puzzling. What have I not implemented?
I am at a complete loss as to what is causing this and why after having followed the documentation exactly, it does not work.
Any guidance would be appreciated.
I am working on an MVC web application that uses Google Natural Language Processing API to parse different input from users.
I have successfully consumed and implemented the API operations and everything works fine as long as I run the application on my local machine. But as soon as I publish a version and upload it on a server I receive the following error on calling the API methods (e.g. AnalyzeSentiment):
Status(StatusCode=Unauthenticated, Detail="Getting metadata from plugin failed with error: Exception occured in metadata credentials plugin.")
With the help of the answers from post: Google Datastore authentication issue - C# I was able to further get details on the error (using gRCP):
An error occurred while sending the request.
Stacktrace: at Google.Apis.Http.ConfigurableMessageHandler.<SendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.Requests.TokenRequestExtenstions.<ExecuteAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.ServiceAccountCredential.<RequestAccessTokenAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.ServiceCredential.<GetAccessTokenForRequestAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.ServiceAccountCredential.<GetAccessTokenForRequestAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Grpc.Auth.GoogleAuthInterceptors.<>c__DisplayClass2_0.<<FromCredential>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Grpc.Core.Internal.NativeMetadataCredentialsPlugin.<GetMetadataAsync>d__11.MoveNext()
This seemed like an authentication issue so I double checked the jsonKey file which is fine. Please note, I have used code to set the credentials in Environment variables:
Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", jsonPath);
and verified it using:
Environment.GetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS")
The call is made as follows:
private AnalyzeSentimentResponse AnalyzeSentiment(string statement)
{
GrpcEnvironment.SetLogger(new MyLogger());
var client = LanguageServiceClient.Create();
var response = client.AnalyzeSentiment(new Document()
{
Content = statement,
Type = Document.Types.Type.PlainText
});
return response;
}
Cannot figure out why it works fine when I run it on my local machine and fails when it is deployed on the server. There is also no restrictions of any kind on the said server.
The result for:
GoogleCredential.GetApplicationDefaultAsync().Result.UnderlyingCredential.GetType()
is:
Google.Apis.Auth.OAuth2.ServiceAccountCredential
Note: The server is our own (Windows Server 2012R2)
With the suggestion given by #JonSkeet, I copied the code into a console application and executed the call. Unfortunately, the issue persisted. What I did next was to move the console application onto another server, it worked there.
So, it was indeed an issue with the server where there maybe some features missing (the firewall is disabled). Network dept is checking it out whereas I have deployed my web application on another server.
Update: There was an issue on the server where some required framework features were not installed. The issue has been resolved by moving the deployment to another server.
We started seeing this exception occur intermittently when our Azure App Service was moved to an App Service Environment. We're using Identity Server 4, and the exception occurs during token signing. This can be seen in the call stack below.
The signing cert is never saved to a cert store, it's loaded from a database as a byte array:
new X509Certificate2(rawData, password,
X509KeyStorageFlags.MachineKeySet |
X509KeyStorageFlags.PersistKeySet);
These references appear to be the same issue:
.NET Core X509Certificate2.PrivateKey throws nte_bad_keyset error
https://github.com/dotnet/corefx/issues/2583
Edit:
Initially we thought this was something specific to the App Service Env in Azure, but now we're seeing the exception in standard App Services. As a test we created an Azure web job that loads a certificate from a byte array and creates JWT tokens (which require a private key for signing). This repros the error. Also of note is the Azure Web job won't even run when certain flags are passed to the X509Certificate2 ctor (e.g. X509KeyStorageFlags.UserKeySet).
Stack trace is the same each time:
System.Security.Cryptography.CryptographicException:
at Internal.NativeCrypto.CapiHelper.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeProvHandle()
at System.Security.Cryptography.RSACryptoServiceProvider.get_SafeKeyHandle()
at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 keySize, CspParameters parameters, Boolean useDefaultKeySize)
at System.Security.Cryptography.RSACryptoServiceProvider..ctor(CspParameters parameters)
at Internal.Cryptography.Pal.CertificatePal.<>c.<GetRSAPrivateKey>b__59_0(CspParameters csp)
at Internal.Cryptography.Pal.CertificatePal.GetPrivateKey[T](Func`2 createCsp, Func`2 createCng)
at Internal.Cryptography.Pal.CertificatePal.GetRSAPrivateKey()
at Internal.Cryptography.Pal.CertificateExtensionsCommon.GetPrivateKey[T](X509Certificate2 certificate, Predicate`1 matchesConstraints)
at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_PrivateKey()
at Microsoft.IdentityModel.Tokens.X509SecurityKey.get_HasPrivateKey()
at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider.HasPrivateKey(SecurityKey key)
at Microsoft.IdentityModel.Tokens.AsymmetricSignatureProvider..ctor(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateSignatureProvider(SecurityKey key, String algorithm, Boolean willCreateSignatures)
at Microsoft.IdentityModel.Tokens.CryptoProviderFactory.CreateForSigning(SecurityKey key, String algorithm)
at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.CreateEncodedSignature(String input, SigningCredentials signingCredentials)
at System.IdentityModel.Tokens.Jwt.JwtSecurityTokenHandler.WriteToken(SecurityToken token)
at IdentityServer4.Services.DefaultTokenCreationService.CreateJwtAsync(JwtSecurityToken jwt)
at IdentityServer4.Services.DefaultTokenCreationService.<CreateTokenAsync>d__3.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Services.DefaultTokenService.<CreateSecurityTokenAsync>d__9.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
at IdentityServer4.ResponseHandling.TokenResponseGenerator.<CreateAccessTokenAsync>d__10.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.ResponseHandling.TokenResponseGenerator.<ProcessTokenRequestAsync>d__8.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.ResponseHandling.TokenResponseGenerator.<ProcessAsync>d__6.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Endpoints.TokenEndpoint.<ProcessTokenRequestAsync>d__6.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Endpoints.TokenEndpoint.<ProcessAsync>d__5.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Hosting.IdentityServerMiddleware.<Invoke>d__3.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Hosting.FederatedSignOutMiddleware.<Invoke>d__6.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Hosting.AuthenticationMiddleware.<Invoke>d__2.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.<Invoke>d__7.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at IdentityServer4.Hosting.BaseUrlMiddleware.<Invoke>d__2.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ApplicationInsights.AspNetCore.ExceptionTrackingMiddleware.<Invoke>d__4.MoveNext()
To resolve the issue we switched from X509 certificates (X509Certificate2) to RSA generated keys (RSACryptoServiceProvider). Identity Server 4 supports both. The exception no longer occurs.
That error means that somewhere after the certificate learned where the key was being stored it got deleted.
Maybe something is calling CngKey.Delete
Maybe something is cloning it into an RSACryptoServiceProvider and setting PersistKeyInCsp to false.
Maybe something is just cleaning up the machine keys directory after the fact.
The other (very rare) known case is where two threads load the PFX in parallel (they don't have to be in the same process) with one of them not having PersistKeySet asserted at import time.
If you're loading and discarding the certificates you really shouldn't set PersistKeySet. Every time you open a PFX another file is created on disk for each private key contained therein... those files get cleaned up normally (as long as the process doesn't abnormally terminate), but PersistKeySet prevents that cleanup from happening.
If nothing in code analysis reveals the two obvious key deletion locations (PersistKeyInCsp=false being the sneaky one) then you'll have to follow the filesystem auditing recommendations in .NET Core X509Certificate2.PrivateKey throws nte_bad_keyset error.
Maybe you didn't give IIS access to your private keys. I had same error, but it was necessary to use X509 Certs instead of keys. I've just granted a permission to read private keys from cert for IIS_IUSRS on MMC. It helps for me.
There's a full answer from #thames:
How to give ASP.NET access to a private key in a certificate in the certificate store?
UPD:
Step-by-step guide:
Create/purchase cert with private key
Import cert to "Local Computer" account (not "Current User"). Check "Allow private key to be exported" in Import Wizard.
Open MMC Console and add Certificates Snap-In. Choose "Computer Account".
Find your cert and right-click on it. Choose "All Tasks" > "Manage Private Keys"
Add Read access to your key for user IIS_IUSRS or IIS AppPool\<AppPoolName> (depends on IIS version and config. It's better to use "Advanced" button to show all possible options)
Restart your application
I just made an WebApi (C#, .net 4.5.2) and published it to the web. In order to make sure it working good, I started a to test it.
The REST web-service failed the "stress" test. I sent the service 30+- http requests, each second, and got back this typical error message:
System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Net.Http.HttpContentExtensions.<ReadAsAsyncCore>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.ModelBinding.FormatterParameterBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.HttpActionBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()
Same error occurred many times in the log.
When I entered the specific machine, I saw that the CPU is on 100% and the RAM is on 80%. Which mean, the machine gives her best to handle the load.
Questions
According to the StackTrace I provided above, I cannot understand
where exactly was the problem in my code?
My code is scale out automatically (hosted in the cloud). But, new machine created only after 5 minutes (depends on the average CPU > 50%). How should I handle sudden mass of HTTP request? Maybe the server should say: "Hey, try agian in 30 seconds" or something like that? What is the right solution?
System.OperationCanceledException: The operation was canceled.
This usually indicates that the client connecting to the service closed the connection before the service could send a response. In terms of api layer this usually means that you have specified a timeout while establishing the http connection, or the library that you are using for making http calls has a default timeout.
You may want to check the iis request logs to see what is going wrong. Since this is happening on a stress test, my guess is your service is running short of threads (Read this for more details) . You may want to consider async model for your request handlers to improve the scalability and utilize threads better. (Read this)