HttpClient throw a random exception ( maybe 2-3 times out of 10 attempts ) with Azure Container Instance
Exception Info: System.Net.WebException at
System.Net.HttpWebRequest.EndGetResponse(System.IAsyncResult) at
System.Net.Http.HttpClientHandler.GetResponseCallback(System.IAsyncResult)
Exception Info: System.Net.Http.HttpRequestException at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
at
System.Net.Http.HttpClient+\u003cFinishSendAsync\u003ed__58.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSucem.Net.Http.HttpClient+\u003cFinishSendAsync\u003ed__58.MoveNext()
at
SystetHelpers.MSIResourcesAccessInfoHelper+\u003cGetResourcesAccessInfo\u003ed__0.MoveNext()\n.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\n
at System.Runtime.m[[System.__Canon, mscorlib, Version=4.0.0.0,
Culture=neutral,
PublicKeyToken=b77a5c56193ompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.ThreC.Main(System.String[])
On the code side, I am simply firing
await httpClient.GetAsync($"http://xyz.eastus.cloudapp.azure.com/api/controller/{id}");
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsAsync<ResourceAccessInfo>();
The behavior is very random it happens only 2-3 time out of 10 attempts
I have also applied re-try when the first attempt then try the second time but still it fails
Same works fine with VM containers always.
Updated:
Additional Exception Info:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---\u003e System.Net.WebException: The remote name could not be resolved: \u0027cloudbridge1.eastus.cloudapp.azure.com\u0027\r\n at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)\r\n at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)\r\n --- End of inner exception stack trace ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Net.Http.HttpClient.\u003cFinishSendAsync\u003ed__58.MoveNext()\r\n--- End ofstack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at UcClearly.MSIAPIHelpers.MSIResourcesAccessInfoHelper.\u003cGetResourcesAccessInfo\u003ed__0.MoveNext()
Which suggest remote server DNS not resolved, I tried with public IP of the server also but same random behavior, I am not sure why Azure Container instance behave this random way
This behaviour is expected.
Windows containers slow network readiness
On initial creation, Windows containers may have no inbound or outbound connectivity for up to 30 seconds (or longer, in rare cases). If your container application needs an Internet connection, add delay and retry logic to allow 30 seconds to establish Internet connectivity. After initial setup, container networking should resume appropriately.
https://learn.microsoft.com/en-us/azure/container-instances/container-instances-troubleshooting#windows-containers-slow-network-readiness
Related
I get below error when attempting to be offline, I began with the starter code as instructed by the Docs and I just keep getting the error.
I followed the details here: https://learn.microsoft.com/en-us/azure/app-service-mobile/app-service-mobile-windows-store-dotnet-get-started-offline-data
Even trying a new download of the starter code doesn't seem to work, for what ever reason the app refuses to acknowledge that it is offline. My offline data is synced and is available, but as soon as I am without connectivity the app crashes with the following exception:
Message: The text associated with this error code could not be found.
The server name or address could not be resolved StackTrace:
{System.Runtime.InteropServices.COMException (0x80072EE7): The text
associated with this error code could not be found. The server name or
address could not be resolved at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
System.Net.Http.HttpHandlerToFilter.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown --- at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task) at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task) at
System.Net.Http.HttpClientHandler.d__86.MoveNext()}
The fact that it is even attempting to resolve is puzzling. What have I not implemented?
I am at a complete loss as to what is causing this and why after having followed the documentation exactly, it does not work.
Any guidance would be appreciated.
I am working on an MVC web application that uses Google Natural Language Processing API to parse different input from users.
I have successfully consumed and implemented the API operations and everything works fine as long as I run the application on my local machine. But as soon as I publish a version and upload it on a server I receive the following error on calling the API methods (e.g. AnalyzeSentiment):
Status(StatusCode=Unauthenticated, Detail="Getting metadata from plugin failed with error: Exception occured in metadata credentials plugin.")
With the help of the answers from post: Google Datastore authentication issue - C# I was able to further get details on the error (using gRCP):
An error occurred while sending the request.
Stacktrace: at Google.Apis.Http.ConfigurableMessageHandler.<SendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.Requests.TokenRequestExtenstions.<ExecuteAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.ServiceAccountCredential.<RequestAccessTokenAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.ServiceCredential.<GetAccessTokenForRequestAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Google.Apis.Auth.OAuth2.ServiceAccountCredential.<GetAccessTokenForRequestAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Grpc.Auth.GoogleAuthInterceptors.<>c__DisplayClass2_0.<<FromCredential>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Grpc.Core.Internal.NativeMetadataCredentialsPlugin.<GetMetadataAsync>d__11.MoveNext()
This seemed like an authentication issue so I double checked the jsonKey file which is fine. Please note, I have used code to set the credentials in Environment variables:
Environment.SetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS", jsonPath);
and verified it using:
Environment.GetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS")
The call is made as follows:
private AnalyzeSentimentResponse AnalyzeSentiment(string statement)
{
GrpcEnvironment.SetLogger(new MyLogger());
var client = LanguageServiceClient.Create();
var response = client.AnalyzeSentiment(new Document()
{
Content = statement,
Type = Document.Types.Type.PlainText
});
return response;
}
Cannot figure out why it works fine when I run it on my local machine and fails when it is deployed on the server. There is also no restrictions of any kind on the said server.
The result for:
GoogleCredential.GetApplicationDefaultAsync().Result.UnderlyingCredential.GetType()
is:
Google.Apis.Auth.OAuth2.ServiceAccountCredential
Note: The server is our own (Windows Server 2012R2)
With the suggestion given by #JonSkeet, I copied the code into a console application and executed the call. Unfortunately, the issue persisted. What I did next was to move the console application onto another server, it worked there.
So, it was indeed an issue with the server where there maybe some features missing (the firewall is disabled). Network dept is checking it out whereas I have deployed my web application on another server.
Update: There was an issue on the server where some required framework features were not installed. The issue has been resolved by moving the deployment to another server.
I just made an WebApi (C#, .net 4.5.2) and published it to the web. In order to make sure it working good, I started a to test it.
The REST web-service failed the "stress" test. I sent the service 30+- http requests, each second, and got back this typical error message:
System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Net.Http.HttpContentExtensions.<ReadAsAsyncCore>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.ModelBinding.FormatterParameterBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.HttpActionBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()
Same error occurred many times in the log.
When I entered the specific machine, I saw that the CPU is on 100% and the RAM is on 80%. Which mean, the machine gives her best to handle the load.
Questions
According to the StackTrace I provided above, I cannot understand
where exactly was the problem in my code?
My code is scale out automatically (hosted in the cloud). But, new machine created only after 5 minutes (depends on the average CPU > 50%). How should I handle sudden mass of HTTP request? Maybe the server should say: "Hey, try agian in 30 seconds" or something like that? What is the right solution?
System.OperationCanceledException: The operation was canceled.
This usually indicates that the client connecting to the service closed the connection before the service could send a response. In terms of api layer this usually means that you have specified a timeout while establishing the http connection, or the library that you are using for making http calls has a default timeout.
You may want to check the iis request logs to see what is going wrong. Since this is happening on a stress test, my guess is your service is running short of threads (Read this for more details) . You may want to consider async model for your request handlers to improve the scalability and utilize threads better. (Read this)
I am developing a WP8.1 silverlight app, that receives WNS notification. It works fine on the emulator, but on the device (lumia 640), it crashes at the following api call:
var channel = await Windows.Networking.PushNotifications.PushNotificationChannelManager.CreatePushNotificationChannelForApplicationAsync();
I receive the following error:
_exception {System.Exception: Exception from HRESULT: 0x880403E8
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at BC_Menu.App.<UploadChannel>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at BC_Menu.StartUp.FirstPage.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.b__3(Object state)} System.Exception
If I try on another device (lumia 920), it works fine. The immediate difference between the devices are that I have a dummy account on the Lumia 640 and no sim card, but I am able to install and update programs. Which should mean the account is correctly initialized. What else could be the issue?
You're crashing because the Windows.Networking.PushNotifications.PushNotificationChannelManager.CreatePushNotificationChannelForApplicationAsync() call is throwing an exception because the device is not connected to WNS. Make sure you handle that case (e.g. with a try/catch) - your users may not always be connected to the internet, which is required to get a channel.
As for why that device isn't connecting to WNS - if you have no SIM card, the device should connect via Wi-Fi. If you're developing in an enterprise make sure they're not blocking outbound connections (which would cause the device to be unable to connect to WNS). If you have a SIM card installed but it has no data, there is a known bug where the device will still try to connect via cellular data (which of course fails). If that's the case, just remove the data-less SIM or disable cellular.
I am sending 1000 messages in a loop using SendAsync method and get the exception below on this line of code
await _ws.SendAsync(new ArraySegment<byte>(messageBuffer, offset, count), WebSocketMessageType.Text, lastMessage, _cancellationToken);
What am I doing wrong? I know somewhere must be a bottleneck because I have no sleep between the sends. Can I wait for a certain state to provent the exception below?
System.InvalidOperationException was unhandled by user code
HResult=-2146233079
Message=There is already one outstanding 'SendAsync' call for this WebSocket instance. ReceiveAsync and SendAsync can be called simultaneously, but at most one outstanding operation for each of them is allowed at the same time.
Source=System
StackTrace:
at System.Net.WebSockets.WebSocketBase.<SendAsyncCore>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at zzz.Stub.WebSocketWrapper.<SendMessageAsync>d__0.MoveNext() in zzz
You cannot do parallel sending or receiving, keep that in mind. Since you are awaiting it seems fine.
If you have other thread sending ping or keep alive messages it may be the problem.
I use that library for load testing, so I create lot of connections and send lot of messages and it works. It is not great, but it works.