I have been building a client / server app with Silverlight, web services, and polling. Apparently I missed the whole Duplex Communication thing when I was first researching this subject. At any rate, the MSDN article I saw on the subject was promising.
When researching the scalability, it appears as if there's conflicting opinions on the subject.
silverlight.net/forums/t/89970.aspx - This thread seems to indicate that the duplex polling only supports a finite amount of concurrent clients on the server end.
dotnetaddict.dotnetdevelopersjournal.com/sl_polling_duplex.htm - This blog entry shows up in multiple places, so it muddies waters.
silverlight.net/forums/t/108396.aspx - This thread shows that I'm not the only one with this concern, but there are no answers in it.
silverlight.net/forums/t/32858.aspx - Despite all the bad press, this thread seems to have an official response saying the 10 concurrent connections is per machine.
In short, does anyone have facts / benchmarks?
Thanks :)
This is my understanding of this, but I haven't done tests.
There is an inbuilt 10 connection limit on non-server operating systems (XP/Vista/Windows 7).
On IIS 6 (XP) it will reject new connections once there are 10 in progress.
On II7 (Vista/Windows 7) it will queue connections once there are 10 in progress. I think this means that 10 simultaneous connections are out.
On the server OS side (2003/2008), there is no connection limit. However, on IIS6 (2003) each long running connection will take a thread from the threadpool, so you will run into a connection limit pretty quickly. On IIS7 (2008), async threads get suspended in a way that does not use up a thread, so 1000s of connections should be possible.
Scalability of the WCF backend using the protocol in a web farm scenario is discussed at http://tomasz.janczuk.org/2009/09/scale-out-of-silverlight-http-polling.html.
There are WCF built-in limits. However these limits can be very easily extended through configuration. (http://weblogs.asp.net/alexeyzakharov/archive/2009/04/17/how-to-increase-amount-of-silverlight-duplex-clients.aspx)
I'm running into a few issues with the duplex binding. From time to time the channel gets faulted for no apparent reason and has a hard time reconnecting. I'm not aware of any alternatives to implement a push model, short of doing everything yourself (and maybe get even worst results).
Performance of the Silverlight HTTP polling duplex protocol and tuning of a WCF service in IIS is discussed at http://tomasz.janczuk.org/2009/08/performance-of-http-polling-duplex.html.
Related
I have 30k+ active devices that connect through TCP sticky session from my application. Each request than creates its own threads and start receiving and sending data. Each requests around take 20 mins to send the complete packets over the network. It means that application need to create 30K threads approximately.
Problem is that my application sometimes become unresponsive and doesn't send and receive messages and it has happening randomly. I was reading on HERE that the max limit of threads in each process is 2000. I am wondering if this is the bad design to hold the 30k+ session and wait for new message to send and receive? What can be done to manage such load?
My Assumption/Suggestion:
1: Creating 2000 threads at max would cause the application to slow because of context switching?
2: At any point in time 28000 devices will be on wait because of worker thread limit?
System Information:
64 bit
8 CPU cores
Use fewer threads. Use async IO instead. I would strongly recommend using Kestrel to host this using the "pipelines" model, which would deal with threading, buffer management, and most of the TCP nuances for you, so you can just worry about your protocol and application code. It scales far beyond your needs (although for 30k sockets you might want to start using multiple ports, especially if you're behind a load balancer, due to ephemeral port exhaustion). Kestrel would require .NET Core or above, but the same approach is available on .NET Framework via Pipelines.Sockets.Unofficial. One guide to pipelines and Kestrel in this context is in a 4-part series here - other guides are available.
I am hitting into a problem with my company application.
I am going to summarize the system key elements:
My company's system is running since few years on Windows XP and 7 (Home, Pro, Basic) machines.
It has been written in .NET 4.0 and based upon WCF.
It uses the default throttling values (MaxConcurrentSessions = 100 * CPU (4) : enough for our workload).
The main service is hosted by a stand alone deamon process (not IIS).
The main service is configured as Multithraded/PerSession instances.
The protocol is Reliable NET.TCP.
No more than 10 clients access concurrently the service.
The problem is that only on Windows 7, intermittently, I get (I discovered that by the WCF full trace log) a "Server too busy exception" due to an exhausted MaxConcurrentSessions limit (impossible!!!).
Do you have any idea about this strange behaviour?
Thank you and have a Happy New Year!
Antonio
Do all your Clients properly close/dispose connection to service after use ? It's worth to check, "ghost" connections could maybe explain this.
We experienced a similar issue with a self-hosted WCF interface which provided a synchronous request/response web service for an asynchronous (2 one way service calls) backend request. Early in our testing, we noticed that after a somewhat variable amount of time, our service became unresponsive to new requests. After some research, we discovered that whenever the backend service (out of our control) did not send a response, we continued to wait indefinitely and as such we kept our client connection open.
We fixed the issue by providing a “time-to-wait” configuration value so we were sure to respond to the client and close the connection. We used something like the following …
Task processTask = Task.Factory.StartNew(() => Process(message));
bool isProcessSuccess = processTask.Wait(shared.ConfigReader.SyncWebServiceWaitTime);
if (!isProcessSuccess)
{
//handle error …
}
The following link, which provides information regarding WCF Service performance counters, may help further determine if the calls are being closed as expected. http://blogs.microsoft.co.il/blogs/idof/archive/2011/08/11/wcf-scaling-check-your-counters.aspx
Hope this helps.
Regards,
I have a WCF RESTful service that is hosted in IIS that is hit by several of our applications. The WCF services appear to operate fine for the most part, but sometimes it takes a long time to get a response from the service.
I was seeing if there was a good tutorials or resources to follow on how to best configure WCF RESTful services to be web scale either through the web.config, from IIS, or from our dedicated application pool.
We have gone through our services and used NHibernate profiler to find and optimize any problematic queries and we also have memcached setup to also help with performance. The problem seems to be when many applications are consuming the service in a short period of time or when the service has sat idle for a long period of time.
Thanks for any assistance.
Not sure if its applicable to your scenario, but I read the below mentioned blog post on MSDN a couple of days ago. It's about a problem in the Net IOCP Threadpool which causes long response times for WCF when many requests are issued in short time. Maybe that could help you?
WCF scales up slowly with bursts of work
KB2538826
There is no general advice on heavy load issue, but one of the possible optimizations would be using asynchronous operations on the server side: Scale WCF Application Better with Asynchronous Programming. It's about conserving thread pool resources while making database calls.
As for the idle period issue, check out Configuring Recycling Settings for an Application Pool (IIS 7)
I have a web service slowdown.
My (web) service is in gsoap & managed C++. It's not IIS/apache hosted, but speaks xml.
My client is in .NET
The service computation time is light (<0.1s to prepare reply). I expect the service to be smooth, fast and have good availability.
I have about 100 clients, response time is 1s mandatory.
Clients have about 1 request per minute.
Clients are checking web service presence by tcp open port test.
So, to avoid possible congestion, I turned gSoap KeepAlive to false.
Until there everything runs fine : I bearly see connections in TCPView (sysinternals)
New special synchronisation program now calls the service in a loop.
It's higher load but everything is processed in less 30 seconds.
With sysinternals TCPView, I see that about 1 thousands connections are in TIME_WAIT.
They slowdown the service and It takes seconds for the service to reply, now.
Could it be that I need to reset the SoapHttpClientProtocol connection ?
Someone has TIME_WAIT ghosts with a web service call in a loop ?
Sounds like you aren't closing the connection after the call and opening new connections on each request. Either close the connection or reuse the open connections.
Be very careful with the implementations mentioned above. There are serious problems with them.
The implementation described in yakkowarner.blogspot.com/2008/11/calling-web-service-in-loop.html (COMMENT ABOVE):
PROBLEM: All your work will be be wiped out the next time you regenerate the web service using wsdl.exe and you are going to forget what you did not to mention that this fix is rather hacky relying on a message string to take action.
The implementation described in forums.asp.net/t/1003135.aspx (COMMENT ABOVE):
PROBLEM: You are selecting an endpoint between 5000 and 65535 so on the surface this looks like a good idea. If you think about it there is no way (at least none I can think of) that you could reserve ports to be used later. How can you guarantee that the next port on your list is not currently used? You are sequentially picking up ports to use and if some other application picks a port that is next on your list then you are hosed. Or what if some other application running on your client machine starts using random ports for its connections - you would be hosed at UNPREDICTABLE points in time. You would RANDOMLY get an error message like "remote host can't be reached or is unavailable" - even harder to troubleshoot.
Although I can't give you the right solution to this problem, some things you can do are:
Try to minimize the number of web service requests or spread them out more over a longer period of time
For your type of app maybe web services wasn't the correct architecture - for something with 1ms response time you should be using a messaging system - not a web service
Set your OS's number of connections allowed to 65K using the registry as in Windows
Set you OS's time that sockets remain in TIME_WAIT to some lower number (this presents its own list of problems)
Our server application is listening on a port, and after a period of time it no longer accepts incoming connections. (And while I'd love to solve this issue, it's not what I'm asking about here;)
The strange this is that when our app stops accepting connections on port 44044, so does IIS (on port 8080). Killing our app fixes everything - IIS starts responding again.
So the question is, can an application mess up the entire TCP/IP stack? Or perhaps, how can an application do that?
Senseless detail: Our app is written in C#, under .Net 2.0, on XP/SP2.
Clarification: IIS is not "refusing" the attempted connections. It is never seeing them. Clients are getting a "server did not respond in a timely manner" message (using the .Net TCP Client.)
You may well be starving the stack. It is pretty easy to drain in a high open/close transactions per second environment e.g. webserver serving lots of unpooled requests.
This is exhacerbated by the default TIME-WAIT delay - the amount of time that a socket has to be closed before being recycled defaults to 90s (if I remember right)
There are a bunch of registry keys that can be tweaked - suggest at least the following keys are created/edited
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
TcpTimedWaitDelay = 30
MaxUserPort = 65534
MaxHashTableSize = 65536
MaxFreeTcbs = 16000
Plenty of docs on MSDN & Technet about the function of these keys.
You haven't maxed out the available port handles have you ?
netstat -a
I saw something similar when an app was opening and closing ports (but not actually closing them correctly).
Use netstat -a to see the active connections when this happens. Perhaps, your server app is not closing/disposing of 'closed' connections.
Good suggestions from everyone, thanks for your help.
So here's what was going on:
It turns out that we had several services competing for the same port, and most of the time the "proper" service would get the port. Occasionally a second service would grab the port away, and the first service would try to open a different port. From that time on, the services would keep grabbing new ports every time they serviced a request (since they weren't using their preferred ports) and eventually we would exhaust all available ports.
Of course, the actual question was: "Can an application mess up the entire TCP/IP stack?", and the answer to that question is: Yes. One way to do it is to listen on a whole bunch of ports.
I guess the port number comment from RichS is correct.
Other than that, the TCP/IP stack is just a module in your operating system and, as such, can have bugs that might allow an application to kill it. It wouldn't be the first driver to be killed by a program.
(A tip to the hat towards Andrew Tanenbaum for insisting that operating systems should be modular instead of monolithic.)
I've been in a couple of similar situations myself. A good troubleshooting step is to attempt a connection from the affected machine to good known destination that isn't at that moment experiencing any connectivity issues. If the connection attempt fails, you are very likely to get more interesting details in the error message/code. For example, it could say that there aren't enough handles, or memory.
From a support and sys admin standpoint, I have only seen this on the rarest of occasions (more than once), but it certainly can happen.
When you are diagnosing the problem, you should carefully eliminate the possible causes, rather than blindly rebooting the system at the first sign of trouble. I only say this because many customers I work with are tempted to do that.