I just made an WebApi (C#, .net 4.5.2) and published it to the web. In order to make sure it working good, I started a to test it.
The REST web-service failed the "stress" test. I sent the service 30+- http requests, each second, and got back this typical error message:
System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Net.Http.HttpContentExtensions.<ReadAsAsyncCore>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.ModelBinding.FormatterParameterBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.HttpActionBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()
Same error occurred many times in the log.
When I entered the specific machine, I saw that the CPU is on 100% and the RAM is on 80%. Which mean, the machine gives her best to handle the load.
Questions
According to the StackTrace I provided above, I cannot understand
where exactly was the problem in my code?
My code is scale out automatically (hosted in the cloud). But, new machine created only after 5 minutes (depends on the average CPU > 50%). How should I handle sudden mass of HTTP request? Maybe the server should say: "Hey, try agian in 30 seconds" or something like that? What is the right solution?
System.OperationCanceledException: The operation was canceled.
This usually indicates that the client connecting to the service closed the connection before the service could send a response. In terms of api layer this usually means that you have specified a timeout while establishing the http connection, or the library that you are using for making http calls has a default timeout.
You may want to check the iis request logs to see what is going wrong. Since this is happening on a stress test, my guess is your service is running short of threads (Read this for more details) . You may want to consider async model for your request handlers to improve the scalability and utilize threads better. (Read this)
Related
I developed an azure function inside Visual Studio, which access some graph api endpoints and writes into azure storage some information. When i tested the function on my pc, everything works perfect.
When i deployed the function into azure, everything works but not the azure storage part. What am I doing wrong?
The variables are in DEV-OPS pipeline and I check in azure portal and there are present as well.
Edit 1: As I started to try to log information I was able to see that my functions is just running the first call and not even finishing it. I used before and after each call.
log.Info()
This is the GraphAPI Call, I know it is in preview but this shouldn't be a problem as this code is running fine on my pc in visual studio.
public static async Task<ChatMessage> CreateNewMessageThread(Employee employee)
{
var chatMessage = ChatThreadMainMessage(employee);
var result = await GraphServiceClient
.Teams[ApplicationConfigurationFields.TeamsId]
.Channels[ApplicationConfigurationFields.ChannelId]
.Messages
.Request()
.AddAsync(chatMessage);
return result;
}
Edit 2: I am getting a BadRequest from the call which throws an exception. although i am getting a bad request, the message is still created... Even if I find the problem I would like to understand why is that happening.
This is the stack trace, don't know if it will help someone.
Status Code: BadRequest
Microsoft.Graph.ServiceException: Code: BadRequest
Message: Bad Request
Inner error:
AdditionalData:
request-id: 220c1508-5bbd-xxxxx-xxxxx
date: 2020-04-27T16:02:01
ClientRequestId: 220c1508-5bbd-xxxxxx-xxxxx
at Microsoft.Graph.HttpProvider.<SendAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Graph.BaseRequest.<SendRequestAsync>d__35.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Graph.BaseRequest.<SendAsync>d__31`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MsTeamsNotifications.MsTeamsNotifications.<CreateNewMessageThread>d__19.MoveNext()
Thank you!
I am pretty confident this is caused by Teams engineers screwing something up again, but might be worth a try...
Using Microsoft.Bot.Builder 4.8.0
I's a pretty simple problem actually. When called from Task OnMessageActivityAsync(ITurnContext<IMessageActivity> turnContext, CancellationToken cancellationToken),
the following code:
await turnContext.SendActivityAsync(typing).ConfigureAwait(false);
causes the following exception:
Operation returned an invalid status code 'BadGateway'
with the following stack trace:
at Microsoft.Bot.Connector.Conversations.<ReplyToActivityWithHttpMessagesAsync>d__10.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.Bot.Connector.ConversationsExtensions.<ReplyToActivityAsync>d__17.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Microsoft.Bot.Builder.BotFrameworkAdapter.<SendActivitiesAsync>d__34.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Bot.Builder.TurnContext.<>c__DisplayClass25_0.<<SendActivitiesAsync>g__SendActivitiesThroughAdapter|1>d.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.Bot.Builder.TurnContext.<SendActivityAsync>d__24.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at Blabla.<OnMessageActivityAsync>d__4.MoveNext() in path\to\blabla.cs:line xx
Replying with text messages works fine.
Oh and just in case, if you are wondering if I tried to use ShowTypingMiddleware to display "typing" indicator, sure, tried that too, but the indicator is never shown. I did that by adding Use(new ShowTypingMiddleware(1000)); to my BotFrameworkHttpAdapter constructor, which is registered to DI container as singleton. It's just that, it does nothing.
Have a look at the Microsoft Teams: Developer Blog; API outage: Bots posting typing indicators may receive 502s.
As a part of our commitment to customers and Microsoft cloud services continuity during these unprecedented times, we’re making temporary adjustments to select capabilities within Microsoft 365. One of those adjustments is that we are temporarily scaling back typing indicators support in Teams. As a result bots posting typing indicators to users in the Europe region using any of the following values for ServiceUrl may receive errors and users may not see the typing indicators sent by bots.
Currently the guidance in the post above is:
As a bot developer, you may either modify your code to not use typing indicators for now, or ignore the errors encountered while sending typing indicators.
I am using elastic search and nest client in asp.net core application.
I have gateway which is working on https protocol (https://localhost:8080). My gateway doing only redirect all requests.
When I trying to send request to elastic search through gateway throwing exception as following:
Invalid NEST response built from a unsuccessful low level call on GET: /es/_cluster/health
# Audit trail of this API call:
- [1] BadRequest: Node: http://localhost:8080/es/ Took: 00:00:00.1932641
# OriginalException: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The server returned an invalid or unrecognized response
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
at System.Net.Http.WinHttpHandler.d__105.MoveNext()
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.DiagnosticsHandler.d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at System.Net.Http.HttpClient.d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
# Request:
# Response:
var elasticClient = new ElasticClient(connectionSettings);
elasticClient.ClusterHealth();
I get below error when attempting to be offline, I began with the starter code as instructed by the Docs and I just keep getting the error.
I followed the details here: https://learn.microsoft.com/en-us/azure/app-service-mobile/app-service-mobile-windows-store-dotnet-get-started-offline-data
Even trying a new download of the starter code doesn't seem to work, for what ever reason the app refuses to acknowledge that it is offline. My offline data is synced and is available, but as soon as I am without connectivity the app crashes with the following exception:
Message: The text associated with this error code could not be found.
The server name or address could not be resolved StackTrace:
{System.Runtime.InteropServices.COMException (0x80072EE7): The text
associated with this error code could not be found. The server name or
address could not be resolved at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
System.Net.Http.HttpHandlerToFilter.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
System.Net.Http.HttpClientHandler.d__86.MoveNext()}
The fact that it is even attempting to resolve is puzzling. What have I not implemented?
I am at a complete loss as to what is causing this and why after having followed the documentation exactly, it does not work.
Any guidance would be appreciated.
I am working on an MVC web application that uses Google Natural Language Processing API to parse different input from users.
I have successfully consumed and implemented the API operations and everything works fine as long as I run the application on my local machine. But as soon as I publish a version and upload it on a server I receive the following error on calling the API methods (e.g. AnalyzeSentiment):
Status(StatusCode=Unauthenticated, Detail="Getting metadata from plugin failed with error: Exception occured in metadata credentials plugin.")
With the help of the answers from post: Google Datastore authentication issue - C# I was able to further get details on the error (using gRCP):
An error occurred while sending the request.
Stacktrace: at Google.Apis.Http.ConfigurableMessageHandler.<SendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.Requests.TokenRequestExtenstions.<ExecuteAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.ServiceAccountCredential.<RequestAccessTokenAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.ServiceCredential.<GetAccessTokenForRequestAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.ServiceAccountCredential.<GetAccessTokenForRequestAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Grpc.Auth.GoogleAuthInterceptors.<>c__DisplayClass2_0.<<FromCredential>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Grpc.Core.Internal.NativeMetadataCredentialsPlugin.<GetMetadataAsync>d__11.MoveNext()
This seemed like an authentication issue so I double checked the jsonKey file which is fine. Please note, I have used code to set the credentials in Environment variables:
Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", jsonPath);
and verified it using:
Environment.GetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS")
The call is made as follows:
private AnalyzeSentimentResponse AnalyzeSentiment(string statement)
{
GrpcEnvironment.SetLogger(new MyLogger());
var client = LanguageServiceClient.Create();
var response = client.AnalyzeSentiment(new Document()
{
Content = statement,
Type = Document.Types.Type.PlainText
});
return response;
}
Cannot figure out why it works fine when I run it on my local machine and fails when it is deployed on the server. There is also no restrictions of any kind on the said server.
The result for:
GoogleCredential.GetApplicationDefaultAsync().Result.UnderlyingCredential.GetType()
is:
Google.Apis.Auth.OAuth2.ServiceAccountCredential
Note: The server is our own (Windows Server 2012R2)
With the suggestion given by #JonSkeet, I copied the code into a console application and executed the call. Unfortunately, the issue persisted. What I did next was to move the console application onto another server, it worked there.
So, it was indeed an issue with the server where there maybe some features missing (the firewall is disabled). Network dept is checking it out whereas I have deployed my web application on another server.
Update: There was an issue on the server where some required framework features were not installed. The issue has been resolved by moving the deployment to another server.