I have an Azure web role that accesses an external WCF based SOAP web service (port 80) for various bits of data. The response from this service is highly erratic. I routinely get the following error.
There was no endpoint listening at
http://www.myexternalservice.com/service.svc that could accept the message. This is
often caused by an incorrect address or SOAP action.
To isolate the problem I created a simple console app to repetitively call this service in 1 second intervals and log all responses.
using (var svc = new MyExternalService())
{
stopwatch.Start();
var response = svc.CallService();
stopwatch.Stop();
Log(response, stopwatch.ElapsedMilliseconds);
}
If I RDP to one of my Azure web instances and run this app it takes 10 to 20 attempts before it gets a valid response from the external service. These first attempts are always accompanied by the above error. After this "warm up period" it runs fine. If I stop the app and then immediately restart, it has to go back through the same "warm up" period.
However, if I run this same app from any other machine I receive valid responses immediately. I have run this logger app on servers running in multiple data centers (non Azure), desktops on different networks, etc... These test runs are always very stable.
I am not sure why this service would react this way in the Azure environment. Unfortunately, for the short term I am forced to call this service but my users cannot tolerate this inconsistency.
A capture of network traffic on the Azure server indicates a large number of SynReTransmit's in 10 second intervals during the same time I experience the connection errors. Once the "warm up" is complete the SynReTransmit's no longer occur.
The Windows Azure data center region where the Windows Azure application is deployed might not be near the external Web Service. The local machine you're trying (which works fine) might be close to the web service. That’s why there might be huge latency in Azure which would likely cause it to fail.
Success accessing WSDL from a browser in Azure VM might be due to browser caching. Making a function call from browser would tell you if it is actually making a connection.
We found a solution for this problem although I am not completely happy with it. After exhausting all other courses of action we changed the load balancer to Layer-7 Load Balancing from Layer-4 Load Balancing. While this fixed the problem of lost requests I am not sure why this made a difference.
Related
Our web application (hosted in a Web App in Azure) experiences spikes in HTTP Queue Length. Each time there is a spike in HTTP Queue Length, the web application crashes and we either have to wait for Azure to restart the web app itself, or we restart the web app ourselves. This happens very often.
The web application does use SignalR, and a Web Job is running that calls a method on the Hub which then broadcasts data to connected clients. There is only ever a handful of users at this stage, so we have not implemented a SignalR backplane.
Here is an example of the spikes in HTTP Queue Length
Note, we tried having the web application in its very own Application Service Plan (P3) and it still exhibited the same behaviour. The memory percentage was much lower than that shown here though, around the 20-40 percent, but still crashed with regular spikes in HTTP Queue Length. Thus, I don't believe it's a memory issue that is causing the issue.
After a while of trying to diagnose this issue, we decided to then host the application (same code) into a VM (still in Azure) and change the URL to point to the VM instead of the web app. The new VM is only very basic, with only 3.5GB Memory.
Since moving to a VM, the application is performing great, no crashes and performs much better than in a Web App with a large dedicated service plan.
So it is difficult to say it is the code, when we running perfmon and other indicators, all memory and queue lengths seem to quickly drop down after serving requests. Whereas in a WebApp, this seemed to continually grow until it crashed.
Just wondering if anyone else has experienced this behaviour with Web Apps? We are going to continue hosting in a VM, but originally preferred hosting within a Web App as PaaS is more appealing.
In case it helps, more information on the tech stack is:
HTML5, C#, Web API 2, Kendo MVVM, SignalR, Azure SQL Server, Web Jobs processing Service Bus Topics.
Kind regards,
Stefan
I've tried a few different ways to do this, but I keep coming up short.
In short, here's what I need to do:
Create a WCF service that acts as a router between client (desktop pc) run diagnostic tools and "widgets" (that also run desktop windows and are have internet connectivety). Since these "widgets" are typically behind some sort of firewall, we've decided to use an IIS hosted WCF service over a tcp connection (port 800, i believe) for callbacks.
Notifications of what the widget is doing need to be sent, asyncronously up through the router to any "connection" clients.
Clients need to be able to syncronously call into the widgets to get diagnostic data or command them to perform a task.
Right now I have a windows service running on the widget that monitors it's status and provided a link to the internal programs to get data.
I also have a light weight diagnostic application running on desktops.
I have created a single callback interface for both status-push and data-pulls that both the widget monitoring program and desktop program implement.
My first attempt was to have the router service keep a list of registered devices and clients and pass messages between them.
Ie: Desktop calls server.getwidgetcolor("widgetid"); and the service calls _widgetlist["widgetId"].getcolor() and returns it.
Similarly the widget monitoring program calls server.notifywidgetcolorchange("widgetid") and the service calls, on all registered client _widgetlist["widgetid"].clients.Notifiycolorchange()
The problem I am running into is that if a wigdet is calling up to the server at the same time the client is calling down to that widget, both calls timeout.
I initially had the server setup as a singleton, and have played with changing the concurancy mode to multiple or re-entrant, but those didn't seem to work.
Conceptually, i'd like to have the service be per-call and persist somehow, that device and client call backs so that when a call comes in, the server wakes up, depersists the call back, sends the message, then goes back to sleep.
With all that said:
Is that ^^ possible (to persist call-back data so that a per-call server can call back on clients)? If not, could I make the service per session (for clients/widgets) but pass the data between service sessions through some other means? Shared memory? File?
Is the over all design possible/recommended? I've looked into the WCF routing library, but that doesn't seem to do what I want, unless I'm reading it wrong?
Are there other technologies I should be using that can do this more easily?
Thanks,
-Bill
I am playing with the the Windows Azure emulator running an MVC website with a single controller method that calls Thread.Sleep(5000) before it returns.
On the client I run a loop that sends a POST request to the controller every 1000 ms, receives a reply from the server with the RoleEnvironment.CurrentRoleInstance.Id, and prints it on the screen.
I have 4 instances of my MVC worker role running.
I understand that the connection: keep-alive HTTP header can keep the browser from making a request to a different instance, because an existing connection is open.
But still, even when loading up my site in multiple browser windows, it keeps hanging while waiting for the Thread.Sleep(), and then (most times) continues to get replies from the same instance.
Why doesn't Azure's load balancer send subsequent requests to a non-busy worker role instance? Do I need to manually mark it as busy?
You mentioned using the emulator, which doesn't handle load balancing the same way as Azure's real load balancers. See this post for details about the differences. I don't know what exactly is going on in your case, but... I'd suggest you trying this out in Azure to see if you get the behavior you're expecting.
I am using web services - not WCF - hosted in an iis web application written in C#/asp.net. I also have a C# winform Desktop application that had originally polled a web method to check for any messages on the server. I found the memory on the client shot up. So, instead of polling this web method I invoke it once, the web method goes into a loop checking for messages. As soon as it finds a message(s) for this client it breaks out of the loop and returns the message(s) to the client. The client in turn will process the message(s) and then re-invoke the same web method waiting for the next message(s).
I run this and the memory on the client desktop and the memory on the web server remain low. I really have 2 questions here.
1). Will the memory escalate on the server when more clients invoke the same web method?
2). Should I avoid this way of doing things?
I know there are callbacks available using WCF and I know I can create a hub using Signal R. what I would like to know is there anything wrong/different to how I am doing it and/or is there a better way of doing it?
Many Thanks.
I am kind of stumped with this one, and was hoping I could find some answers here.
Basically, I have an ASP.NET application that is running across 2 servers. Server A has all of the business logic/data access exposed as web services, and Server B has the website which talks to those services (via WCF, with net.tcp binding).
The problem occurs a few seconds after a recycle of my app pool is initiated by IIS on Server A. The recycle happens after the allotted time (using the default of 29 hours set in IIS).
In the server log (of Server A):
A worker process with process id of
'####' serving application pool
'AppPoolName' has requested a recycle
because the worker process reached its
allowed processing time limit.
I believe that this is normal behavior. The problem is that a few seconds later, I get this exception on Server B:
This channel can no longer be used to
send messages as the output session
was auto-closed due to a
server-initiated shutdown. Either
disable auto-close by setting the
DispatchRuntime.AutomaticInputSessionShutdown
to false, or consider modifying the
shutdown protocol with the remote
server.
This doesn't happen on every recycle; I assume that it happens when someone is hitting the site with a request WHILE the recycle happens.
Furthermore, my application is down until I intervene; this exception continues to occur every time a subsequent request is made to the page. I intervene by editting the web.config (by adding a space or something benign to the end of file) and saving it- I assume that that causes my application to recompile and brings the services back up. I also have experimented with running a batch file that does this for me every time the exception happens ;)
Now, I could barely find any information on this exception, and I've been looking for a while. Most of the information I did find pertains to WCF settings that I am not using.
I already read up on "DispatchRuntime.AutomaticInputSessionShutdown" and I don't think it pertains to this situation. This particular property refers to the service shutting down automatically in response to behavior on the client side, which is not what is happening here. Here, the service is shutdown because of IIS.
I did read this which went through some sort of work around to bring the service back up automatically, but I am really looking to understand what is going on here, not to hack around it!
I have started playing around with the settings in IIS7, specifically turning on/off Overlapped Recycling and increasing the process startup/shutdown times. I am wondering whether it is safe to turn off recycling completely (I believe if I put 0 for the recycling time interval?) But again, I want to know what's going on!
Anyway, if you need more information, let me know. Thanks in advance!
This is probably related to how you open and close WCF connections.
If you open a proxy when your app starts and then continue to use this, a break in the connection, which is caused by a restart on the server side. Results in a error on the client side, since the server that the proxy was talking to is no longer there.
When you restart the client side (changing the web.config) new proxies are created against a server that is running.
The way to fix this is to make sure that you close a WCF connection after you use it.
http://www.codeguru.com/csharp/.net/net_wcf/article.php/c15941/
You should also make sure that you're using the correct SessionMode for your Web Service. I remember having similar trouble with some of my Services until I sorted out the correct mode. This is especially true when you're mixing this with any other authentication mode that is not "None".
This link might have some pointer.
http://msdn.microsoft.com/en-us/library/ms731193.aspx
My suggestion is to simply stop using IIS to host your services. Unless there is something you really need from IIS, I would recommend just writing a standard Windows Service to host your WCF endpoints.
If you can't do that, then by all means turn off recycling. AppPool recycling is mainly there because web developers write crappy code. I know that sounds rather blunt, but if you have enough sense to write code that doesn't leak then there is no reason to have IIS constantly restart your program.