We are doing http calls with Windows Authentication between asp.net apps (specifically a .net core an and a standard .net framework 4.5.1 app) apps using System.Net.Http.HttpClient like this:
var client = new HttpClient(new HttpClientHandler { Credentials = CredentialCache.DefaultCredentials });
var response= await _winHttpClient.GetAsync(url);
...
This works fine, except for that the first request takes 10 seconds. The requests then go fast for about 40 seconds, and then one request takes 10 seconds. This cycle goes on forever.
Looking at the IIS logs on the receiving end, we can se that every request is denied (401) and then a follow up request goes through, and every so often the delay between these are about 10s. This is all invisible to the client code - it is worked out by the underlying framework.
Example:
2017-03-17 14:19:40 10.241.108.23 GET /person/search/john - 80 - 10.211.37.246 - 401 2 5 31
2017-03-17 14:19:40 10.241.108.23 GET /person/search/john - 80 utv\frank 10.211.37.246 - 200 0 0 93
2017-03-17 14:19:41 10.241.108.23 GET /person/search/johnn - 80 - 10.211.37.246 - 401 2 5 46
2017-03-17 14:19:51 10.241.108.23 GET /person/search/johnn - 80 utv\frank 10.211.37.246 - 200 0 0 281
It seems as if the credentials are somehow cached, and have to be refreshed every 40ish seconds.
It is worth noting that this problem doesn't occur when both applications are run locally, only when they are run in the actual hosting environment.
What's going on?
Is it expected behaviour that the consumer has to do two calls for every request? And why do some of the requests take 10 seconds to authenticate?
Any help would be appreciated.
Related
This question already has answers here:
Trying to run multiple HTTP requests in parallel, but being limited by Windows (registry)
(4 answers)
HttpWebRequest timing out on third try, only two connections allowed HTTP 1.1 [duplicate]
Remove 2 connections HttpWebRequest limitation in C#
(2 answers)
Closed 3 years ago.
I'm trying to open about 25connections to a host at the same time.
My OS is windows 10.
For testing, I bring up a simple website on my local IIS and I response a simple data to the user with a delay of 2 seconds using Thread.Sleep(2000).
Now using this code on the client:
const int len = 25;
for (int i = 0; i < len; i++)
{
new Thread(new ParameterizedThreadStart((idx) =>
{
// start downloading data.
var res = new WebClient().DownloadString("http://192.168.1.101:8090/");
// log index and time when done.
Console.WriteLine($"{Convert.ToInt32(idx).ToString("00")} done at:{ DateTime.Now.ToString("HH:mm:ss:ffff") }");
})).Start(i);
}
I got the following result:
Thread 01 done at 40:8476 ms
Thread 00 done at 40:8476 ms
Thread 03 done at 40:8496 ms
Thread 04 done at 40:8496 ms
Thread 02 done at 40:8506 ms
Thread 05 done at 40:8506 ms
Thread 07 done at 40:8516 ms
Thread 06 done at 40:8516 ms
Thread 08 done at 40:8536 ms
Thread 09 done at 40:8545 ms
Thread 11 done at 42:8510 ms
Thread 10 done at 42:8510 ms
Thread 12 done at 42:8560 ms
Thread 14 done at 42:8560 ms
Thread 15 done at 42:8570 ms
Thread 13 done at 42:8580 ms
Thread 16 done at 42:8590 ms
Thread 17 done at 42:8590 ms
Thread 18 done at 42:8610 ms
Thread 19 done at 42:8610 ms
Thread 21 done at 44:8565 ms
Thread 20 done at 44:8565 ms
Thread 23 done at 44:8634 ms
Thread 24 done at 44:8654 ms
Thread 22 done at 44:8654 ms
The above result tells us that:
1- Thread 0 to 9 got the data at the same time.(second 40)
2- Thread 10 to 19 got the data at the same time 2 seconds later after previous step.(second 42)
3- Thread 20 to 24 got the data at the same time. 2 seconds later after previous step.(second 44)
Now my question is WHO limited me and why it only opens 10 HTTP connections at the same time and how can I set it to unlimited.
If there is any other platform or programming language it will be welcomed.
Who? Your OS and web server manufacturer, being Microsoft.
Why? Because Windows 10 is a client OS, and Microsoft doesn't want you to host any serious web applications on that.
See for example:
Why is IIS allowing only 3 connections at the time?
Are there any connection limits on Windows 7 IIS v7.5?
Maximum number of http-requests on IIS with Windows 7
And from While using signalr, will there be any connection limits on IIS, linking to the SignalR documentation:
When SignalR is hosted in IIS, the following versions are supported. Note that if a client operating system is used, such as for development (Windows 8 or Windows 7), full versions of IIS or Cassini should not be used, since there will be a limit of 10 simultaneous connections imposed, which will be reached very quickly since connections are transient, frequently re-established, and are not disposed immediately upon no longer being used. IIS Express should be used on client operating systems.
There are other posts, like Does windows 10 connection limit apply to self-hosted applications?, mentioning 20 connections, but that's 20 devices (probably recognized by remote IP address) connecting to selected Windows service (SMB, IIS, ...).
The IIS limit is 10, and has been for many years, I think since Windows 7.
So the first half of the answer to this question is:
If you need more than 10 simultaneous incoming HTTP connections while developing on Windows 10, use IIS Express or any other web server than IIS or Cassini.
But there's a second half of this answer. If you need more than 10 (or 2, or ..., depending on the environment) simultaneous outgoing HTTP connections, there's ServicePointManager who manages this. See:
Trying to run multiple HTTP requests in parallel, but being limited by Windows (registry)
How can I programmatically remove the 2 connection limit in WebClient
Microsoft Docs: Network Programming in the .NET Framework - Managing Connections
So change the limit before executing the requests:
System.Net.ServicePointManager.DefaultConnectionLimit = 25;
I have an ASP.NET application running on IIS 8.5 that times out at 60 seconds regardless of what I set. In other words, I can set the connection timeout to 300 seconds or 30 seconds and the page will still time out at 60 seconds. The Request status code is 504 GATEWAY_TIMEOUT.
Site:
Connection timeout = 300 seconds (5 mins)
Session timeout = 20 minutes
Script timeout = 10 minutes
App Pool:
.NET CLR v4.0
Integrated pipeline
64-bit (32-bit not allowed)
Idle timeout = 20 mins (default)
Other things I've done:
Read just about every article I can on this subject
Set the executionTimeout in my web.config
Restarted the site or ran iisreset after each change
Disabled all sites except this one to isolate it
Created a simple ASPX page in which I set a Sleep thread to under or over 60 seconds, and tried every combination with the connection timeout value.
I'm at a loss. Any ideas?
The answer turned out to be a load balancer, which I was unaware existed, that had its own timeout setting.
Check your maxRequestLength, apparently that can cause a 504 if the incoming request exceeds the default limit of 4MB.
504 Error On Server
Can big ViewState content result a HTTP Error 504 - Gateway timeout?
When a page is taking a long time to process in IIS, all other page requests coming in are delayed until the first one either times out, or responds.
This was brought to light by a 3rd party API having high response times. However, I can duplicate the issue by putting a Sleep in any page.
We are using DNN version 7.0.6
For Example:
The page http://www.website.com/foo.aspx has the code System.Threading.Thread.Sleep(10000); in the Page_Load.
While this page is sleeping, http://www.website.com/bar.aspx is requested. Bar.aspx (a page that usually responds right away) will not respond until foo.aspx has completed it's request.
From IIS Logs, you can see this process:
#Fields: date time cs-uri-stem sc-status sc-substatus sc-win32-status time-taken
2016-08-24 19:44:20 /bar.aspx 200 0 0 69
2016-08-24 19:44:24 /foo.aspx 200 0 0 10053
2016-08-24 19:44:24 /bar.aspx 200 0 0 9204
2016-08-24 19:44:26 /bar.aspx 200 0 0 91
I have tried adding additional worker processes, and the problem still exists.
I feel like I'm missing something simple. Am I just overlooking some fundamental way IIS or DNN works? Can anything be done to prevent this from happening?
I have deployed an ASP .net MVC web app to Azure App service.
I do a GET request from my site to some controller method which gets data from DB(DbContext). Sometimes the process of getting data from DB may take more than 4 minutes. That means that my request has no action more than 4 minutes. After that Azure kills the connection - I get message:
500 - The request timed out. The web server failed
to respond within the specified time.
This is a method example:
[HttpGet]
public async Task<JsonResult> LongGet(string testString)
{
var task = Task.Delay(360000);
await task;
return Json("Woke", JsonRequestBehavior.AllowGet);
}
I have seen a lot of questions like this, but I got no answer:
Not working 1
Cant give other link - reputation is too low.
I have read this article - its about Azure Load Balancer which is not available for webapps, but its written that common way of handling my problem in Azure webapp is using TCP Keep-alive. So I changed my method:
[HttpGet]
public async Task<JsonResult> LongPost(string testString)
{
ServicePointManager.SetTcpKeepAlive(true, 1000, 5000);
ServicePointManager.MaxServicePointIdleTime = 400000;
ServicePointManager.FindServicePoint(Request.Url).MaxIdleTime = 4000000;
var task = Task.Delay(360000);
await task;
return Json("Woke", JsonRequestBehavior.AllowGet);
}
But still get same error.
I am using simple GET request like
GET /Home/LongPost?testString="abc" HTTP/1.1
Host: longgetrequest.azurewebsites.net
Cache-Control: no-cache
Postman-Token: bde0d996-8cf3-2b3f-20cd-d704016b29c6
So I am looking for the answer what am I doing wrong and how to increase request timeout time in Azure Web app. Any help is appreciated.
Azure setting on portal:
Web sockets - On
Always On - On
App settings:
SCM_COMMAND_IDLE_TIMEOUT = 3600
WEBSITE_NODE_DEFAULT_VERSION = 4.2.3
230 seconds. That's it. That's the in-flight request timeout in Azure App Service. It's hardcoded in the platform so TCP keep-alives or not you're still bound by it.
Source -- see David Ebbo's answer here:
https://social.msdn.microsoft.com/Forums/en-US/17305ddc-07b2-436c-881b-286d1744c98f/503-errors-with-large-pdf-file?forum=windowsazurewebsitespreview
There is a 230 second (i.e. a little less than 4 mins) timeout for requests that are not sending any data back. After that, the client gets the 500 you saw, even though in reality the request is allowed to continue server side.
Without knowing more about your application it's difficult to suggest a different approach. However what's clear is that you do need a different approach --
Maybe return a 202 Accepted instead with a Location header to poll for the result later?
I just changed my Azure Web Site from Shared Enviroment to Standard, and it works.
I have a problem with an ASP.NET MVC project hosted on IIS. I'm flooding the same request hundreds of times:
function Test(count){
for(var i=0; i<count; i++){
$.ajax({
url: "http://example.com?someparam=sth&test="+i,
context: document.body
}).done(function() {
console.log("done");
});
}
}
Test(500)
Here is the taken time of each request in milliseconds (here are just a part of the sent requests):
221
215
225
429
217
228
227
209
236
355
213
224
257
249
223
211
227
1227
168
181
257
3241
201
244
130
198
283
1714
146
136
177
3304
294
868
772
2750
138
1283
221
775
136
235
792
278
641
1707
880
1711
As you can see there are peaks for some of the requests and the taken time could be more than 10 times of the average of the other requests.
I though it could be a Garbage Collector issue, but I think it's not. I called GB on each request. I had the same result, the delays were still there in the log.
This happens not only for my MVC project but either for an empty MVC project.
I created a new MVC project and sent lots of requests to Home/About. The result was same.
I tried with an action that returns EmprtActionResult... same result.
If anybody knows why this happens and has a solution for the problem or just has a suggestion ... please share the information, I will be really grateful
Also I'm using .NET Memory Profiler, but I can't find out how to track each request and catch exactly the requests with delays. Can I do it with .NET Memory Profiler? If I don't please suggest another profiler that will be working for me.
Thank you!
EDIT: I also tried with an empty WebForms project. There were delays just for the firs 5 requests ... but this is because of IIS warming up for sure. And no delays for the next 1495 requests.
Your testing methodology has no way to identify where your bottleneck might be occurring, only that something is causing your delay.
Also, there is no mention whether this is an isolated server. If you are hitting a production website, you'll be affected when pages are requested by visitors accessing the web site.
At the very least, you'll need to add a control to this. I would start by loading a plain text file from the same web server. Another point to note is that most web browsers will limit the number of concurrent requests to the same web site. Usually, that is two simultaneous requests. Your delay could be a backlog of ajax requests in your client.