I am building a C# console application that sends http requests.
Running the application locally from source, works as expected, the http requests get sent and everything is golden.
Meanwhile when building a container image from the very same source and spinning up an instance from that container locally the http requests frequently time out or return service temporarily unavailable errors (503).
I am using mcr.microsoft.com/dotnet/runtime:6.0 as my base image.
I dont understand how the very same code ran from source code and running inside a docker container have so different behaviour.
Any ideas?
Immediately after posting the question I have found the solution.
I needed to add the following run parameter: --net host
I am curious as to why not having this parameter produced this strange behaviour, as sometimes the requests work and sometimes they dont..
Related
I'm encountering the following problem in production with an ASP.Net Core website that runs InProc on IIS. The installed .Net Core runtime is 2.2.8.
Following a push notification on Signalr, my clients are sending about a hundred requests to a particular URL, which is bound to a controller method that looks like this:
[HttpPost()]
[Route("Api/GetData")]
public string[] GetData()
After serving a few requests, the server suddenly stops responding for a few minutes, before serving all the responses at once. No errors can be seen in the app logs, and the IIS logs shows huge response times for the requests.
I have enabled Failed Request Tracing in IIS to analyse requests that take longer than 9 seconds and I see that the requests that take forever to respond are actual promptly processed by IIS.
E.g:
IIS log: 47952ms
2020-05-12 12:18:15 169.XXX.XXX.XXX POST /Api/GetData - 443 -
195.XXX.XXX.XXX Mozilla/5.0+(Windows+NT+6.1;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko
200 0 0 79479 945 47952
IIS Failed Request Tracing logs:
I haven't used IIS Failed Request Tracing before and I'm not sure how I should interpret this. I first though of a lock in the code, but I haven't found it and I'm not sure how it would also affect requests to other URLs? What's your thoughts?
A maybe important note is that the clients are behind a proxy, so the server sees the same IP for them all. We have never encountered this issue before on another server.
UPDATE: We managed to reproduce the problem with a version of GetData() that simply returns a static array of string. We found that the size of the array mattered. We reproduced the issue with a response of 88KB, but not with a small one (2.5K). Looking at the .net core logs, it seems that the result of the method is computed promptly, but the response itself is hanging (we haven't figured out why). Upgrading to .Net Core 3.1 solved the problem...
I'm keeping the question open in case somebody can provide an explanation.
I have some WCF services hosted in IIS 7.5, with application initialization module installed. Most of the services works perfectly fine expect one.
For that WCF I'm sure that application_start is called during the application pool initiate/recycle process refer to the log, however calling the .svc afterward took around 45s to respond, then the subsequent call is fast, sounds like the warmup is not happening on this WCF. I had no idea what are those 45s are doing.
As that WCF have numbers of .svc, first call to each of them also took around 40s to respond, even the first svc is already called, feels like they are trying to initiate individually.
One more question is that are there any differences between sending a warm up request to /service and /service/service.svc
? I've tried both of them but seem no differences. Cause I'm worried that if I have multiple .svc inside my WCF, do I have to send warmup request to all of them?
Took a look at the svclog and seems the issue is not related to Application Initialization
Solved
After research turn out the whole issue is not related to application initialization, but WCF's metadata exchange. Actually the image above already gave the hint of the issue. Thanks for your help Francesco B. !
http://www.synergex.com/blog/2015/06/25/why-is-that-first-wcf-operation-so-slow/
My project consists of 2 parts Silverlight and WindowsForms.
WindowsForm returns data at localhost:8081 or another port.
Data can easily see through the browser by typing the address, with the parameters on all computers.
If Silverlight is enabled on the local server everything works fine, the problems start when you run it on an external server.
On my computer everything works fine, but on some computers, an error is returned "Async_ExceptionOccurred".
Following the network turned out that Silverlight can't get clinetaccesspolice.xml, request has been sent but nothing happens, no error is returned. Of course directly in the browser can open the file.
After turned Fiddler bug disappears on some computers.
It follows that clinetaccesspolice and cross-domain set correctly. Something is blocking communication between Silverlight from external domain and localhost.
I found a solution. Maybe it's not too professional, but solves the problem.
Fiddler share its own engine for developers. Use it in the program used on PC. The engine will automatically change the proxy settings so the client does not need to know anything ;)
Fiddler core
// Call Startup to tell FiddlerCore to begin listening on the specified port,
// and optionally register as the system proxy and optionally decrypt HTTPS traffic.
Fiddler.FiddlerApplication.Startup(8877, true, true);
// Call Shutdown to tell FiddlerCore to stop listening on the specified port, and
// unregister as the system proxy if attached.
Fiddler.FiddlerApplication.Shutdown();
I am playing with the the Windows Azure emulator running an MVC website with a single controller method that calls Thread.Sleep(5000) before it returns.
On the client I run a loop that sends a POST request to the controller every 1000 ms, receives a reply from the server with the RoleEnvironment.CurrentRoleInstance.Id, and prints it on the screen.
I have 4 instances of my MVC worker role running.
I understand that the connection: keep-alive HTTP header can keep the browser from making a request to a different instance, because an existing connection is open.
But still, even when loading up my site in multiple browser windows, it keeps hanging while waiting for the Thread.Sleep(), and then (most times) continues to get replies from the same instance.
Why doesn't Azure's load balancer send subsequent requests to a non-busy worker role instance? Do I need to manually mark it as busy?
You mentioned using the emulator, which doesn't handle load balancing the same way as Azure's real load balancers. See this post for details about the differences. I don't know what exactly is going on in your case, but... I'd suggest you trying this out in Azure to see if you get the behavior you're expecting.
I have an Azure web role that accesses an external WCF based SOAP web service (port 80) for various bits of data. The response from this service is highly erratic. I routinely get the following error.
There was no endpoint listening at
http://www.myexternalservice.com/service.svc that could accept the message. This is
often caused by an incorrect address or SOAP action.
To isolate the problem I created a simple console app to repetitively call this service in 1 second intervals and log all responses.
using (var svc = new MyExternalService())
{
stopwatch.Start();
var response = svc.CallService();
stopwatch.Stop();
Log(response, stopwatch.ElapsedMilliseconds);
}
If I RDP to one of my Azure web instances and run this app it takes 10 to 20 attempts before it gets a valid response from the external service. These first attempts are always accompanied by the above error. After this "warm up period" it runs fine. If I stop the app and then immediately restart, it has to go back through the same "warm up" period.
However, if I run this same app from any other machine I receive valid responses immediately. I have run this logger app on servers running in multiple data centers (non Azure), desktops on different networks, etc... These test runs are always very stable.
I am not sure why this service would react this way in the Azure environment. Unfortunately, for the short term I am forced to call this service but my users cannot tolerate this inconsistency.
A capture of network traffic on the Azure server indicates a large number of SynReTransmit's in 10 second intervals during the same time I experience the connection errors. Once the "warm up" is complete the SynReTransmit's no longer occur.
The Windows Azure data center region where the Windows Azure application is deployed might not be near the external Web Service. The local machine you're trying (which works fine) might be close to the web service. That’s why there might be huge latency in Azure which would likely cause it to fail.
Success accessing WSDL from a browser in Azure VM might be due to browser caching. Making a function call from browser would tell you if it is actually making a connection.
We found a solution for this problem although I am not completely happy with it. After exhausting all other courses of action we changed the load balancer to Layer-7 Load Balancing from Layer-4 Load Balancing. While this fixed the problem of lost requests I am not sure why this made a difference.