I am hitting into a problem with my company application.
I am going to summarize the system key elements:
My company's system is running since few years on Windows XP and 7 (Home, Pro, Basic) machines.
It has been written in .NET 4.0 and based upon WCF.
It uses the default throttling values (MaxConcurrentSessions = 100 * CPU (4) : enough for our workload).
The main service is hosted by a stand alone deamon process (not IIS).
The main service is configured as Multithraded/PerSession instances.
The protocol is Reliable NET.TCP.
No more than 10 clients access concurrently the service.
The problem is that only on Windows 7, intermittently, I get (I discovered that by the WCF full trace log) a "Server too busy exception" due to an exhausted MaxConcurrentSessions limit (impossible!!!).
Do you have any idea about this strange behaviour?
Thank you and have a Happy New Year!
Antonio
Do all your Clients properly close/dispose connection to service after use ? It's worth to check, "ghost" connections could maybe explain this.
We experienced a similar issue with a self-hosted WCF interface which provided a synchronous request/response web service for an asynchronous (2 one way service calls) backend request. Early in our testing, we noticed that after a somewhat variable amount of time, our service became unresponsive to new requests. After some research, we discovered that whenever the backend service (out of our control) did not send a response, we continued to wait indefinitely and as such we kept our client connection open.
We fixed the issue by providing a “time-to-wait” configuration value so we were sure to respond to the client and close the connection. We used something like the following …
Task processTask = Task.Factory.StartNew(() => Process(message));
bool isProcessSuccess = processTask.Wait(shared.ConfigReader.SyncWebServiceWaitTime);
if (!isProcessSuccess)
{
//handle error …
}
The following link, which provides information regarding WCF Service performance counters, may help further determine if the calls are being closed as expected. http://blogs.microsoft.co.il/blogs/idof/archive/2011/08/11/wcf-scaling-check-your-counters.aspx
Hope this helps.
Regards,
Related
i have thoroughly searched the internet (most of the links sent me to stackoverflow ;)) to try to come up with a solution how to keep a WCF Service alive under IIS (7.5).
Many of the responses here were suggesting to write an application that will periodically send dummy requests to the WCF service in order to keep it alive.
My question is:
what if I create a thread in the WCF which will start when a service is first called (in a static constructor) that will periodically consume the WCF itself?
I mean for example in c#:
while (true)
{
WebClient client = new WebClient();
string returnString = client.DownloadString("http://...");
Thread.Sleep(1000 * 5);
}
assuming that "http://..." is an URI to a provided WebMethod which for example returns some integer.
Would that work?
Basically I need some kind of web service (not particulary a WCF but not a Windows Service) that is running on a server that performs some operations and updates something in a SQL Server database. So if the described approach will not work, what might be the best way to achieve this?
Go to your IIS -> Application Pool (or create new one) -> Advanced settings and set Regular Time Interval=0
See related thread here.
AppFabric allows you to create wcf services which can autostart and be long living - this might be worth checking out as a hosting option (it's just a plugin for IIS)
Auto Start
What you are doing is basically wrong from the outset.
The problem is this:
IIS is basically a stateless request broker for http requests (basic IIS) and a request broker for service requests (IIS w. AppFabric).
What you are asking for is how to turn the inherently stateless IIS into a stateful server, with eternal threads running.
That is not what IIS does, IIS handles requests and its AppDomain is subject to AT ALL TIMES be torn down (destorying all threads).
Which makes the most upvoted answer dangerous, as it teaches you how to affect the recycle process, without controlling the tear-downs (off app-domains and threads) that IIS itself will intermittenly perform.
The requester is "foreign" to the IIS itself.
The internal lifetime of the service though, is entirely managed by IIS (and the configuration of its applications) itself.
So if with "keep alive" you mean: to constantly request some service, then do as Andreas suggest further up (create a schedueled job).
If with "keep alive" you mean: to make sure the same instance of the class handles requests, then you need to look into WCF lifetimes.
If with "keep alive" you mean: to make the code you have created "stateful" and keep f.eks static variables alive and so on, well you are not accepting that IIS is basically a stateless pr. request broker with internal lifetime management.
I suggest you create a small program (console app) that calls the web service. The program should take as arguments the url of the web service. Then you create a windows scheduled task that runs the program. In this way you have a lot of flexibility as compared to the embedded approach you are querying about as the program is just another client to the web service.
Try to avoid using while loop. Maybe http://quartznet.sourceforge.net/ is something you are looking for. On WCF start create Task every 10 minutes which will cal WCF itself.
If this has been asked before my apologies, and this is .NET 2.0 ASMX Web services, again my apologies =D
A .NET Application that only exposes web services. Roughly 10 million messages per day load balanced between multiple IIS Servers. Each incoming messages is XML, and an outgoing message is XML. (XMLElement) (we have beefy servers that run on steroids).
I have a SLA that all messages are processed in under X Seconds.
One function, Linking Methods, in the process is now taking 10-20 seconds, it is required for every transaction, however is not critical that it happens before the web service returns the results. Because of this I made a suggestion to throw it on another thread, but now realize that my words and the eager developers behind them might have not fully thought this through.
The below example shows on the left the current flow. On the right what is being attempted
Effectively what I'm looking for is to have a web service spawn a long running (10-20 second) thread that will execute even after the web service is completed.
This is what, effectively, is going on:
Thread linkThread= new Thread(delegate()
{
Linkmembers(GetContext(), ID1, ID2, SomeOtherThing, XMLOrSomething);
});
linkThread.Start();
Using this we've reduced the time from 19 seconds to 2.1 seconds on our dev boxes, which is quite substantial.
I am worried that with the amount of traffic we get, and if a vendor/outside party decides to throttle us, IIS might decide to recycle/kill those threads before they're done processing. I agree our solution might not be the "best" however we don't have the time to build in a Queue system or another Windows Service to handle this.
Is there a better way to do this? Any caveats that should be considered?
Thanks.
Apart from the issues you've described, I cannot think of any. That being said, there are ways to fix the problem that do not involve building your own solution from scratch.
Use MSMQ with WCF: Create a WCF service with an MSMQ endpoint that is IIS hosted (no need to use a windows service as long as WAS is enabled) and make calls to the service from within your ASMX service. You reap all the benefits of reliable queueing without having to build your own.
Plus, if your MSMQ service fails or throws an exception, it will reprocess automatically. If you use DTC and are hitting a database, you can even have the MSMQ transaction flow to the DB.
I have an Azure web role that accesses an external WCF based SOAP web service (port 80) for various bits of data. The response from this service is highly erratic. I routinely get the following error.
There was no endpoint listening at
http://www.myexternalservice.com/service.svc that could accept the message. This is
often caused by an incorrect address or SOAP action.
To isolate the problem I created a simple console app to repetitively call this service in 1 second intervals and log all responses.
using (var svc = new MyExternalService())
{
stopwatch.Start();
var response = svc.CallService();
stopwatch.Stop();
Log(response, stopwatch.ElapsedMilliseconds);
}
If I RDP to one of my Azure web instances and run this app it takes 10 to 20 attempts before it gets a valid response from the external service. These first attempts are always accompanied by the above error. After this "warm up period" it runs fine. If I stop the app and then immediately restart, it has to go back through the same "warm up" period.
However, if I run this same app from any other machine I receive valid responses immediately. I have run this logger app on servers running in multiple data centers (non Azure), desktops on different networks, etc... These test runs are always very stable.
I am not sure why this service would react this way in the Azure environment. Unfortunately, for the short term I am forced to call this service but my users cannot tolerate this inconsistency.
A capture of network traffic on the Azure server indicates a large number of SynReTransmit's in 10 second intervals during the same time I experience the connection errors. Once the "warm up" is complete the SynReTransmit's no longer occur.
The Windows Azure data center region where the Windows Azure application is deployed might not be near the external Web Service. The local machine you're trying (which works fine) might be close to the web service. That’s why there might be huge latency in Azure which would likely cause it to fail.
Success accessing WSDL from a browser in Azure VM might be due to browser caching. Making a function call from browser would tell you if it is actually making a connection.
We found a solution for this problem although I am not completely happy with it. After exhausting all other courses of action we changed the load balancer to Layer-7 Load Balancing from Layer-4 Load Balancing. While this fixed the problem of lost requests I am not sure why this made a difference.
I am kind of stumped with this one, and was hoping I could find some answers here.
Basically, I have an ASP.NET application that is running across 2 servers. Server A has all of the business logic/data access exposed as web services, and Server B has the website which talks to those services (via WCF, with net.tcp binding).
The problem occurs a few seconds after a recycle of my app pool is initiated by IIS on Server A. The recycle happens after the allotted time (using the default of 29 hours set in IIS).
In the server log (of Server A):
A worker process with process id of
'####' serving application pool
'AppPoolName' has requested a recycle
because the worker process reached its
allowed processing time limit.
I believe that this is normal behavior. The problem is that a few seconds later, I get this exception on Server B:
This channel can no longer be used to
send messages as the output session
was auto-closed due to a
server-initiated shutdown. Either
disable auto-close by setting the
DispatchRuntime.AutomaticInputSessionShutdown
to false, or consider modifying the
shutdown protocol with the remote
server.
This doesn't happen on every recycle; I assume that it happens when someone is hitting the site with a request WHILE the recycle happens.
Furthermore, my application is down until I intervene; this exception continues to occur every time a subsequent request is made to the page. I intervene by editting the web.config (by adding a space or something benign to the end of file) and saving it- I assume that that causes my application to recompile and brings the services back up. I also have experimented with running a batch file that does this for me every time the exception happens ;)
Now, I could barely find any information on this exception, and I've been looking for a while. Most of the information I did find pertains to WCF settings that I am not using.
I already read up on "DispatchRuntime.AutomaticInputSessionShutdown" and I don't think it pertains to this situation. This particular property refers to the service shutting down automatically in response to behavior on the client side, which is not what is happening here. Here, the service is shutdown because of IIS.
I did read this which went through some sort of work around to bring the service back up automatically, but I am really looking to understand what is going on here, not to hack around it!
I have started playing around with the settings in IIS7, specifically turning on/off Overlapped Recycling and increasing the process startup/shutdown times. I am wondering whether it is safe to turn off recycling completely (I believe if I put 0 for the recycling time interval?) But again, I want to know what's going on!
Anyway, if you need more information, let me know. Thanks in advance!
This is probably related to how you open and close WCF connections.
If you open a proxy when your app starts and then continue to use this, a break in the connection, which is caused by a restart on the server side. Results in a error on the client side, since the server that the proxy was talking to is no longer there.
When you restart the client side (changing the web.config) new proxies are created against a server that is running.
The way to fix this is to make sure that you close a WCF connection after you use it.
http://www.codeguru.com/csharp/.net/net_wcf/article.php/c15941/
You should also make sure that you're using the correct SessionMode for your Web Service. I remember having similar trouble with some of my Services until I sorted out the correct mode. This is especially true when you're mixing this with any other authentication mode that is not "None".
This link might have some pointer.
http://msdn.microsoft.com/en-us/library/ms731193.aspx
My suggestion is to simply stop using IIS to host your services. Unless there is something you really need from IIS, I would recommend just writing a standard Windows Service to host your WCF endpoints.
If you can't do that, then by all means turn off recycling. AppPool recycling is mainly there because web developers write crappy code. I know that sounds rather blunt, but if you have enough sense to write code that doesn't leak then there is no reason to have IIS constantly restart your program.
I have been building a client / server app with Silverlight, web services, and polling. Apparently I missed the whole Duplex Communication thing when I was first researching this subject. At any rate, the MSDN article I saw on the subject was promising.
When researching the scalability, it appears as if there's conflicting opinions on the subject.
silverlight.net/forums/t/89970.aspx - This thread seems to indicate that the duplex polling only supports a finite amount of concurrent clients on the server end.
dotnetaddict.dotnetdevelopersjournal.com/sl_polling_duplex.htm - This blog entry shows up in multiple places, so it muddies waters.
silverlight.net/forums/t/108396.aspx - This thread shows that I'm not the only one with this concern, but there are no answers in it.
silverlight.net/forums/t/32858.aspx - Despite all the bad press, this thread seems to have an official response saying the 10 concurrent connections is per machine.
In short, does anyone have facts / benchmarks?
Thanks :)
This is my understanding of this, but I haven't done tests.
There is an inbuilt 10 connection limit on non-server operating systems (XP/Vista/Windows 7).
On IIS 6 (XP) it will reject new connections once there are 10 in progress.
On II7 (Vista/Windows 7) it will queue connections once there are 10 in progress. I think this means that 10 simultaneous connections are out.
On the server OS side (2003/2008), there is no connection limit. However, on IIS6 (2003) each long running connection will take a thread from the threadpool, so you will run into a connection limit pretty quickly. On IIS7 (2008), async threads get suspended in a way that does not use up a thread, so 1000s of connections should be possible.
Scalability of the WCF backend using the protocol in a web farm scenario is discussed at http://tomasz.janczuk.org/2009/09/scale-out-of-silverlight-http-polling.html.
There are WCF built-in limits. However these limits can be very easily extended through configuration. (http://weblogs.asp.net/alexeyzakharov/archive/2009/04/17/how-to-increase-amount-of-silverlight-duplex-clients.aspx)
I'm running into a few issues with the duplex binding. From time to time the channel gets faulted for no apparent reason and has a hard time reconnecting. I'm not aware of any alternatives to implement a push model, short of doing everything yourself (and maybe get even worst results).
Performance of the Silverlight HTTP polling duplex protocol and tuning of a WCF service in IIS is discussed at http://tomasz.janczuk.org/2009/08/performance-of-http-polling-duplex.html.