Develop Reference

Apparent delay in Azure KeyVault access

We have an Azure-based ASP.NET Web Service that accesses an Azure KeyVault. We are seeing two instances in which a method "hangs" on a first try, and then works a minute or so later.
In both instances, a KeyVault access occurs. In both instances the problem started when we started using the KeyVault in these methods.
We have done very careful logging in the first instance, and cannot see anything else in our code that could cause the hang. The KeyVault access is the primary suspect.
In addition, if we run the app from our local servers (from Visual Studio), the KeyVault access works fine on the "first try". It only produces the "hang" error when it runs in production on Azure, and only on that "first try".
By "hang" I mean that in one instance, which is triggered by an external API, it takes at least 60 seconds (we can tell that because the external API times out.) In the other instance, which is triggered by a page request, several minutes can pass and the page just spins, at which point we assume the DB request or something else has timed out.
When I say "a minute or so later", that's as fast as we have timed the retry.
Is there some kind of issue or function where the KeyVault needs to be "warmed up" before it works on the first try?
Update: I'm looking at the code more carefully, and I see at least a couple of places where we can insert still more logging to get a more exact picture of where the failure occurs. I'm going to do that, and then I'll report back here.
Update: See answer below - major newbie error, has been corrected.

Found the problem, and the solution.
Key Vault access needs to be called from an async task, because there is a multi-second delay.
private async Task<string> GetKeyVaultSecretValue(varSecretParms) {
I don't understand the underlying technology, however, apparently, if the call is from within a standard code sequence, the server doesn't like to wait, and so the thread is abandoned/halts.

According to your description, it seems that it dues to WebApp that does not enable Always on .
By default, web apps are unloaded if they are idle for some period of time. This lets the system conserve resources. In Basic or Standard mode, you can enable Always On to keep the app loaded all the time
If possible, please have a try to enable Always on and try it again.

timeout exception in asp.net application

I cannot figure out why I am getting timeout exception in asp.net application upon signing into application and when I restart my webserver I don't get timeout exception?
Environment:
Asp.net application(IIS webserver) and DB (Sql server)
both IIS and Sql server are on two different machines.
Sql server timeout:600 seconds
Webconfig setting:
<httpRuntime maxRequestLength="16384" executionTimeout="270"/>
<sessionState mode="InProc" cookieless="false" timeout="60" stateConnectionString="">
Thanks in advance.
Screenshot of error:

Try to add connection timeout in connection string as "Data Source=SQLSERVER;Initial Catalog=MYCATALOG;Integrated Security=True;Connection Timeout=1000".if again "The wait operation timed out" then it is likely you have a database call that took longer than expected. This could be due to any number of things:
Transient network problem
High SQL server load
Problem with SAN, RAID, or storage device
Deadlock or other form of multiprocess contention
You haven't shared enough information to troubleshoot. The way I would manage this would be to check for other occurrences of the problem and see if there is a pattern, e.g. if the problem occurs at a certain time of day.
Certainly increasing the timeout is not a bad idea (if it is currently set pretty low) and may resolve the problem in and of itself.

When executionTimeout is reached, what happens?

If the default is 110 seconds why do I see requests going beyond that (up to 177 seconds)?
I'd expect and hope that once time is reached the request is cancelled and resources reallocated.
I'm seeing these response times in my apm tool (dynatrace) which instruments the code and doesn't likely get the time from the server logs
( referring to In our IIS logs, why do requests last 5 min and longer when executionTimeout is 110 seconds?)
Thank you

Have you considered the requests may be being queued on the server? If you look at perfmon RequestsQueued you might see some queuing going on.
Also look at request wait time to get an indication of how long the last request waited.

Can you send a screenshot of the PurePath showing the Exec Time but also the Elapsed Time column in the tree? Maybe the PurePath itself actually gets aborted by IIS after 110 s but some asynchronous activity in your ASP.NET App is still working and was not interrupted by the IIS Timeout. The PurePath tree should show that as it shows asynchronous subpaths
andi

API Calls with long DB query returning Gateway Timeout out despite extending timeouts

I'm using Entity Framework 6 to access a SQL Server in an API application. The API server and DB server are separate. I have a known long query that takes a little more than a minute to run. I have set the CommandTimeout on my EF DbContext using the following code:
db.Database.CommandTimeout = 20 * 60; // 20 minutes
Unfortunately, this does not seem to have any affect because the query still times out in less than a minute and the server returns a 504 Gateway Timeout error.
Why does this change not have any affect on the command execution time limit?
Is there something else I should be doing to allow this query to run longer?
Some more details:
I have written a desktop app that makes API calls to an API server which in turn makes queries to the db server. On the desktop application side, I have set the HttpClient property:
httpClient = new HttpClient { Timeout = new TimeSpan( 0, 20, 0 ) };
On the API server side, I have set the execution timeout:
<system.web>
<httpRuntime requestPathInvalidCharacters="" maxRequestLength="2097151" executionTimeout="30000" />
</system.web>
And on the API server, I've set the SQL command timeout as above.
At some point, the response always ends after a little less than a minute despite the fact that I've set all the timeouts I can think of to 20 minutes. I assumed the 504 was caused by the db timeout, but I suppose it's just as likely that it's between the desktop app and the API server.
Regardless, I'm running out of ideas so anything will help.

Turns out there was another timeout setting I wasn't thinking about. Because I host the API and database servers on Amazon Web Services using Elastic Beanstalk, there was a Load Balancer between my desktop application and the API server. The default timeout is 60 seconds. Here is a link to increase the Load Balancer's timeout settings:
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/config-idle-timeout.html

IIS ASP.NET Session is lost

I know that this has been asked several times, but in the web config we set the
<sessionState timeout="2880"></sessionState>
the app pool recycle is 1740 minutes, the application pool timeout is also 1740 mins now,
ping and rapid fail protection is turned off, and in the global.asax we log every application start, session start/end, application error.
I see 2 "application start" than 2 "session start" in the logs, nothing changes, a few minutes later, we get a new "application start" (2 of them again) with 2 "session start"-s, with the same sessionid as before (no "session end" in between). The result is a web application that quits every few minutes.

Not sure about this, but how many instance of the application do you allow?
Under IIS>Application Pools>YourAppPool>Advanced Settings;
you'll find relevant settings:
idle time-out (set to 0?)
Disable Overlapped Recycle (this might be your culprit;a new instance starts before the old one is disposed?)
Another thing i can think of is 'Service Throttling' on your binding, or perhaps there are multiple Sessions or Instances allowed?