Suppose I have a thousand of hardware device(about 20,000) which send the data in real time. I have to capture this data and send all data to database and also show information in application.For this I am thinking to develop a WCF service which poll all the devices and get the data from devices and store in database.
So my question is "Is WCF efficient to do this or I should follow another approach "
I am thinking to develop a WCF service which poll all the devices
A WCF service cannot "poll". A consumer can poll, which is to say it can call a service over and over again, however there is no polling pattern built into WCF client channel. So any polling behavior will need to be programmed.
So my question is "Is WCF efficient to do this or I should follow
another approach "
What I suspect is that you actually want your devices to call the service and pass data, so what you are actually asking is Can a WCF service handle a high volume of calls?, to which the answer is Yes.
However, in order to get the best scalability you should expose a per-call service, and do this over netTcpBinding (basically sockets), which uses optimised encoding and is therefore higher performance.
This would only be available to a consumer running the WCF client stack. If WCF is not available on your devices then the next best option is probably an HTTP endpoint over webHttpBinding, which provides the best interoperability.
Can you please suggest some other technology?
Well, if you are happy using http on your devices you should look at nancyfx for hosting your service - it's a really nice, lightweight http container for .net.
Related
I'll have about 10 computers communicating with the server on a regular basis.
The packets won't be too data intensive. (updates on game data)
The server will be writing to a postgres database
My main concern is concurrent access by the clients. How can I handle the queing in of client requests?
Is WCF a good route? So for instance, rather then opening tcp/ip streams is it feasible to use GET/POST requests & such by WCF over a LAN?
Can anyone suggest any other areas of focus for possible problems?
With WCF you have so much flexibility choosing what to use depends on what to you want to do. Knowing what to choose comes from experimentation and experience.
You can use tools like Fiddler, Wireshark and ServiceTraceViewer to analyze the conversations between clients and the server - they should be used periodically to detect inefficient conversations.
You're producing a game that needs to communicate data via a central server...thus presumably you want that to be as efficient as possible and with the lowest latency.
Thus ideally any requests that come into the server you want to be processed as fast as possible...you want to minimize any overhead.
Do you need to guarantee that the message was received by the server and consequently you are guaranteed to receive a reply (i.e. reliable delivery)?
Do the messages need to be processed in the same order they were sent ?
Then consider does your server need to scale? will you only ever have small number of clients at a time ? You mention 10?.....will this ever increase to a million?
Then consider your environment....you mention a LAN...will clients always be on the same LAN...i.e. Intranet? If so then they can communicate directly with the server using TCP sockets....without going through an IIS stack.
With certain assumptions above with WCF you could choose Endpoints that uses a netTCPBinding with Binary encoding, and with the Service using:
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerSession)]
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Single)].
This would establish sessions (as long as you use a Binding that supports sessions) between the clients and server (doesn't scale because each client gets their own server instance to manage their session), it would allow multiple clients to be handled in parallel, and the Binary encoding "might" reduce data size (you could choose to avoid that as you said your data isn't intensive, and Binary might actually add more overhead to small messages sizes).
If on the other-hand your service needs the ability to infinitely scale and keep response times low, doesn't mind if messages are lost, or delivered in a different order, or have to be deliverable over HTTP, etc, etc, then...then there are other Bindings, InstanceContextModes, message encodings, etc, that you can use. And if you really want to get sophisticated, you can get your Service to expose multiple Endpoints that are configured in different ways, and your client might prefer a particular recipe.
NOTE: (if you host the WCF Service using IIS and you are using netTCPBinding then you have to enable the net.tcp protocol....not needed if you self-host the WCF Service).
http://blogs.msdn.com/b/santhoshonline/archive/2010/07/01/howto-nettcpbinding-on-iis-and-things-to-remember.aspx
So that is "1" way to do it with WCF for a particular scenario.
Experiment with one way...look at its memory use, latency, resilience, etc. and if your needs change, then change the Binding/Encoding that you use...that's the beauty and nightmare of WCF.
If your clients must use HTTP and POST/GET requests then use a Http flavoured binding. If you need guaranteed message delivery, then you need a reliable session.
If your Service needs to infinitely scale then you might start looking at hosting your Service in the Cloud e.g. with Windows Azure.
http://blog.shutupandcode.net/?p=1085
http://msdn.microsoft.com/en-us/library/ms731092.aspx
http://kennyw.com/work/indigo/178
I'm writing a client/server architecture where there are going to be possibly hundreds of clients over multiple virtual machines, mostly on the intranet but some in other locations.
Each client will be gathering data constantly and sending a message to a server every second or so. Each message will probably be about 128 characters or so in length.
My question is, for this architecture where I am writing both client/server in .NET is should I go with WCF or some socket code I've written previously. I need scalability (which the socket code has in mind), reliability and just the ability to handle that many messages.
I would not make final decision without peforming some proof of concept. Create very simple service, host it and use some stress test to get real performance results. Than validate results against your requirements. You have mentioned amount of messages but you didn't mentioned expected response time. There is currently discussed similar question on MSDN forum which complains about slow response time of WCF compared to sockets.
Other requirements are not directly mentioned in your post so I will make some assumption for best performance:
Use netTcpBinding - best performance, binary encoding, requires .NET server / clients. I guess you are going to use Net.Tcp because your other choice was direct socket programming.
Don't use security if you don't have to - reduces performance. Probably not possible for clients outside your intranet.
Reuse proxy on clients if possible. Openning TCP connection is expensive if you reuse the same proxy you will have single connection per proxy. This will affect instancing of you services - by default single service instance will handle all requests from single proxy.
Set service throttling so that your service host is ready for many clients
Also you should make some decisions about load balancing. Load balancing for WCF net.tcp connections requires sticky sessions (session affinity) so that after openning the channel client always calls the service on the same server (bacause instance of that service was created only on single server).
100 requests per second does not sound like much for a WCF service, especially with that little payload. But it should be quite quick to setup a simple setup with a WCF service with one echo method just returning the input and then hook up a client with a bunch of threads and a loop.
If you already have a working socket implementation you might keep it, but otherwise you can pick WCF and spend your precious development time elsewhere.
From my experience with WCF, i can tell you that it's performance on high load is very very nice. Especially you can chose between several bindings to achieve your requirements for the different scenarios (httpBinding for outside communication, netPeerTcpBinding in local network e.g.).
I want to design a new distributed application, but I have a few queries that I need some genius advice on, hopefully from you people:
Scenario
I currently support a legacy application that is starting to fall between the cracks.
It is a distributed Client-Server app implemented using .Net Remoting. I can't explain exactly what it does, because I'm not allowed to.......But let's just say that it does LOTS of MATHS. I want to re-design and re-write the application using WCF.
Pre-requisites
The server side of the implementation will be hosted in a Windows Service.
The client side will be a windows forms application.
The server side will perform lots of memory-intensive processing.
The server will spit this data out to multiple thin clients (20-ish).
The majority of the time the server will be passing data to the clients, but occasionally the clients will be persisting data back to the server.
The speed at which the data is transmitted is highly-important, however I'm well aware that WCF can handle fast distribution of data.
Encryption/Security is not that important as the app will run on a highly protected local network.
Queries
Given the information above:
1)What sort of design pattern am I best going with? - Baring in mind I want the server to continually PUSH the newly calculated information immediately to the clients, as opposed to the current implementation that involves the client pulling from the server continuously.
2)What type of WCF binding should I use to ensure maximum speed of data transfer? (as close to real-time as possible is what I'm after)
3)Should I use a class library to share the common objects between the client and the server applications?
4)What is the best way in which to databind my objects on the client side in order to see live updates continually as data changes?
If I've forgotten anything then feel free to point this out
Help greatly appreciated.
1) What sort of design pattern am I best going with?
Based on your comments, you're wanting to transform the current polling mechanism to an event-based mechanism. That is, instead of the client constantly checking the server for results, have the server notify the client when a new calculation result is available.
I would recommend using Juval Lowy's Publish-Subscribe Framework for this.
(source: microsoft.com)
.
This framework is described in detail in this MSDN article. And you can download the framework's source code for free at Lowy's website, IDesign.net.
Basically, the server logic that performs the calculations inside the Windows service is the Publishing Client in the graphic, and the various WinForm applications are the Subscribing Clients. The Pub/Sub Service lives in your Windows service. It manages the list of subscribing clients and provides a single endpoint for your server to publish calculation results to. In this way, your server performs a calculation and publishes the result once to the Pub/Sub Service endpoint. The Pub/Sub Service is then responsible for publishing the result to the subscribed clients.
2) What type of WCF binding should I use to ensure maximum speed of data transfer?
If all of your WCF communication were on a single machine, you'd want to use the NetNamedPipeBinding. However, since you will be distributed, you want to use the NetTcpBinding.
For WCF binding decisions, I have found this chart useful.
3) Should I use a class library to share the common objects between the client and the server applications?
Since you are in control of both the client and server side, I would highly recommend sharing a class library instead of using Visual Studio's "Add Service Reference" feature. For a detailed discussion of this, refer to this SO question-and-answer.
4) What is the best way in which to databind my objects on the client side in order to see live updates continually as data changes?
I suspect this will depend on what controls you use to display the data. One way that immediately comes to mind would be to have your client fill an in-memory data table as each calculation result is received. This data table could then be bound to a ListBox control, for example, that shows the results in calculation order.
This to me looks like you need to implement the Observer pattern, but distributed. Whereby new calculations are made to the service, and WCF just happens to be the mechanism by which you push your notification back to the client.
Generally speaking, you have your business logic housed in a windows service, whereby a type is a Subject (Observable). You could publish an endpoint for clients to register for notifications. This would be a WCF service, with potentially two operations:
RegisterClient(...)
UnregisterClient(...)
When a client is registered with service, it can receive updates, broadly speaking, the when the service has finished calculating a result, it could iterate through all registered clients and initiate a push. The push being a communication through an endpoint on the client.
A client endpoint might typically by
Notify(Result...);
And your server simply calls that when it has new data...
Typically you'd use TCP to maximise throughput.
This is by no means exactly what you should do, but perhaps its a direction to start in?
Tooday I use ServiceHost for self hosting WCF cervices.
I want to host near to my WCF services my own TCP programm for direct sockets operations (like lien to some sort of broadcasting TCP stream)
I need control over URL namespaces (so I would be able to let my clients to send TCP streams directly into my service using some nice URLs like example.com:port/myserver/stream?id=1 or example.com:port/myserver/stream?id=anything and so that I will not be bothered with Idea of 1 client for 1 socket at one time moment, I realy want to keep my WCF services on the same port as my own server or what it is so to be able to call www.example.com:port/myWCF/stream?id=222... and I want it to work on Any port - not only 80)
Can any body please help me with this?
I am using just WCF now. And I do not enjoy how it works. That is one of many resons why I want to start migration to clear TCP=)
I can not use net-tcp binding or any sort of other cool WS-* binding (tooday I use the simpliest one so that my clients like Flash, AJAX, etc connect to me with ease).
I needed Fast and easy in implemrnting connection protocol like one I created fore use with Sockets for real time hi ammount of data transfering.
So.. Any Ideas? Please - I need help.
Well if you're going to drop down to pure sockets, you could as well make your service act as proxy. Make WCF Services listen on some other port, and your app on the desired port. When you receive request in your app, manually parse header and check weather it is intended for your service or WCF Service. If it's intended for WCF service, open a TCP connection to WCF service and pass the received data to it, and then simply pass back WCF's answer to the client..
On the other hand you could speed up WCF quite a lot by writing your own custom binding. Lots of time WCF looses on serialization that is done using reflection (which is slow), going around this would improve your speed considerably.
If your problem with WCF is performance then you should try the binary net TCP binding to eliminate XML serialization to improve performance.
Some high traffic applications, like realtime games, use UDP for the majority of communication since it cuts through the protocol. With TCP you get ordering and reliability built in, but this comes at the cost of performance because it will implicitly delay packets to wait for out of order packets so that it can hand them to the application in the correct order, or wait for lost packets to be resent. Instead you can use UDP and implement your own scheme for verification of data that is less stringent than TCP.
There are UDP options available for WCF, or you could implement your own. WCF is nothing more than a message pump, and you can replace different steps with whatever you want.
Not sure if this will help you or not, but try turning on your Net TCP Sharing Service.
When I first posted this question I had strong coupling between my web service and application controller where the controller needed to open multiple threads to the service and as it received back data it had to do a lot of processing on the returned data and merge it into one dataset. I did not like the fact that the client had to so much processing and merge the returned data before it was ready to be used and wanted to move that layer to the service and let the service open the asynchronous threads to the suppliers and merge the results before returning them to the client.
One challenge I had was that I could not wait till all threads were complete and results were merged, I had to start receiving data as it was available. That called me to implement an observer pattern on the service so that it would notify my application when new set of results are merged and ready to be used and send them to the application.
I was looking for how to do this using either on ASMX webservices or WCF and so far I have found implementing it using WCF but this thread is always open for suggestions and improvements.
OK the solution to my problem came from WCF
In addition to classic request-reply operation of ASMX web services, WCF supports additional operation types like; one-way calls, duplex callbacks and streaming.
Not too hard to guess, duplex callback was what I was looking for.
Duplex callbacks simply allow the service to do call backs to the client. A callback contract is defined on the server and client is required to provide the callback endpoint on every call. Then it is up to the service to decide when and how many times to use the callback reference.
Only bidirectiona-capable bindings support callback operations. WCF offers the WSDualHttpBinding to support callbacks over HTTP (Callback support also exists by NetNamedPipeBinding and NetTcpBinding as TCP and IPC protocols support duplex communication)
One very important thing to note here is that duplex callbacks are nonstandard and pure Microsoft feature. This is not creating a problem on my current task at hand as both my web service and application are running on Microsoft ASP.NET
Programming WCF Services gave me a good jump start on WCF. Being over 700 pages it delves deep into all WCF consepts and has a dedicated chapter on the Callback and other type of operations.
Some other good resources I found on the net are;
Windows Communication Foundation (WCF) Screencasts
MSDN Webcast: Windows Communication Foundation Top to Bottom
Web Service Software Factory
The Service Factory for WCF
This sounds like a perfect use case for Windows Workflow Foundation. You can easily create a workflow to get information from each supplier, then merge the results when ready. It's much cleaner, and WF will do all the async stuff for you.
I'm not so sure that duplex is needed here... IMO, a standard async call with a callback should be more than sufficient to get notification of data delivery.
What is the biggest problem? If you are talking about async etc, then usually we are talking about the time taken to get the data to the client. Is this due to sheer data volume? or complexity generating the data at the server?
If it is the data volume, then I can think of a number of ways of significantly improving performance - although most of them involve using DTO objects (not DataSet/DataTable, which seemed to be implied in the question). For example, protobuf-net significantly reduces the data volume and processing required to transfer data.
One of the ways to achieve this is by invoking your WS asynchronously (http://www.stardeveloper.com/articles/display.html?article=2001121901&page=1, http://www.ondotnet.com/pub/a/dotnet/2005/08/01/async_webservices.html), and then updating the GUI in the callback.
However, you could have timeout problems if the querying of data takes too long. For example, if one of the supplier's web site is down or very slow, this could mean that the whole query could fail. Maybe it would be better if your business logic on the client side does the merging instead of WS doing it.
Not sure if this solution fits your particular task, but anyway:
Add paging parameters to your WS API (int pageNumber, int pageSize, out int totalPages)
Add a short-living TTL cache that associates request details (maybe a hash value) with output data
When your application asks for the first page, return it as soon as it's ready and put the whole bunch of collected/merged data to cache so when the next page is required you may use what is already prepared.
But note that you won't get the most up-to-date data, configure cache reloading interval cautiously.
The absolute best way to archive in your scenario and technology would be having some kind of token between your web app / library against your web service and your controller needs to have a thread to check if there are new results etc. However please note that you will require to get the complete data back from your WS as it's merge can result in removed items from the initial response.
Or I still think that handling threads would be better from controller with the use of WCF Webservices