Develop Reference

Simplest architecture for service to service data exchange

I have a public server with web services (.Net) that collect data and uploaded files from different mobile apps and I need to synchronise it with an internal intranet server.
The intranet server is deeply protected by firewall and organisation policies.
I think this is a pretty common scenario where messages and brokers could be used, something like Rabbitmq or Nservicebus, but I'm not an expert on it.
As the data is only to be sent from the external server to the intranet one in unidirectional and asynchronous way I was thinking not to add another layer of indirection to the architecture and just use the web services exposed also for server to server communication.
The approach would be like:
An intranet windows serivce would poll regularly and at different scheduled intervals the external web service to know if there is new data to get (maybe from a certain point in time)
The web service would respond with the list of the new data and files
The windows service would iterate with calls to get all the data to be inserted in the intranet and download the uploaded files.
What are the risks of this approach? Would be better that the external web service would respond only a link to a huge zipped file response with all the data and files in it?
Should I use a something like RabbitMq also for a so simple scenario?

If you are literally dealing with files, you might want to think about something even simpler. FTP (more specifically sftp) might fit your needs better, and be FAR simpler to implement.

Synchronous file download using WCF

I have a WCF REST based service which downloads and stores video files in the web server machine. The maximum size of video files is around 1MB and the download is asynchronous at present.
Is it possible to use synchronous download and will a timeout occur if large number or requests are submitted almost simultaneously? If so is there a mechanism to handle the situation?

You could set up your service to act as a singleton. WCF will handle the queueing of multiple calls. A similar question can be found here, and may be of help.

What is the best way to handle a C# server?

I'll have about 10 computers communicating with the server on a regular basis.
The packets won't be too data intensive. (updates on game data)
The server will be writing to a postgres database
My main concern is concurrent access by the clients. How can I handle the queing in of client requests?
Is WCF a good route? So for instance, rather then opening tcp/ip streams is it feasible to use GET/POST requests & such by WCF over a LAN?
Can anyone suggest any other areas of focus for possible problems?

With WCF you have so much flexibility choosing what to use depends on what to you want to do. Knowing what to choose comes from experimentation and experience.
You can use tools like Fiddler, Wireshark and ServiceTraceViewer to analyze the conversations between clients and the server - they should be used periodically to detect inefficient conversations.
You're producing a game that needs to communicate data via a central server...thus presumably you want that to be as efficient as possible and with the lowest latency.
Thus ideally any requests that come into the server you want to be processed as fast as possible...you want to minimize any overhead.
Do you need to guarantee that the message was received by the server and consequently you are guaranteed to receive a reply (i.e. reliable delivery)?
Do the messages need to be processed in the same order they were sent ?
Then consider does your server need to scale? will you only ever have small number of clients at a time ? You mention 10?.....will this ever increase to a million?
Then consider your environment....you mention a LAN...will clients always be on the same LAN...i.e. Intranet? If so then they can communicate directly with the server using TCP sockets....without going through an IIS stack.
With certain assumptions above with WCF you could choose Endpoints that uses a netTCPBinding with Binary encoding, and with the Service using:
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerSession)]
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Single)].
This would establish sessions (as long as you use a Binding that supports sessions) between the clients and server (doesn't scale because each client gets their own server instance to manage their session), it would allow multiple clients to be handled in parallel, and the Binary encoding "might" reduce data size (you could choose to avoid that as you said your data isn't intensive, and Binary might actually add more overhead to small messages sizes).
If on the other-hand your service needs the ability to infinitely scale and keep response times low, doesn't mind if messages are lost, or delivered in a different order, or have to be deliverable over HTTP, etc, etc, then...then there are other Bindings, InstanceContextModes, message encodings, etc, that you can use. And if you really want to get sophisticated, you can get your Service to expose multiple Endpoints that are configured in different ways, and your client might prefer a particular recipe.
NOTE: (if you host the WCF Service using IIS and you are using netTCPBinding then you have to enable the net.tcp protocol....not needed if you self-host the WCF Service).
http://blogs.msdn.com/b/santhoshonline/archive/2010/07/01/howto-nettcpbinding-on-iis-and-things-to-remember.aspx
So that is "1" way to do it with WCF for a particular scenario.
Experiment with one way...look at its memory use, latency, resilience, etc. and if your needs change, then change the Binding/Encoding that you use...that's the beauty and nightmare of WCF.
If your clients must use HTTP and POST/GET requests then use a Http flavoured binding. If you need guaranteed message delivery, then you need a reliable session.
If your Service needs to infinitely scale then you might start looking at hosting your Service in the Cloud e.g. with Windows Azure.
http://blog.shutupandcode.net/?p=1085
http://msdn.microsoft.com/en-us/library/ms731092.aspx
http://kennyw.com/work/indigo/178

WCF or Custom Socket Architecture

I'm writing a client/server architecture where there are going to be possibly hundreds of clients over multiple virtual machines, mostly on the intranet but some in other locations.
Each client will be gathering data constantly and sending a message to a server every second or so. Each message will probably be about 128 characters or so in length.
My question is, for this architecture where I am writing both client/server in .NET is should I go with WCF or some socket code I've written previously. I need scalability (which the socket code has in mind), reliability and just the ability to handle that many messages.

I would not make final decision without peforming some proof of concept. Create very simple service, host it and use some stress test to get real performance results. Than validate results against your requirements. You have mentioned amount of messages but you didn't mentioned expected response time. There is currently discussed similar question on MSDN forum which complains about slow response time of WCF compared to sockets.
Other requirements are not directly mentioned in your post so I will make some assumption for best performance:
Use netTcpBinding - best performance, binary encoding, requires .NET server / clients. I guess you are going to use Net.Tcp because your other choice was direct socket programming.
Don't use security if you don't have to - reduces performance. Probably not possible for clients outside your intranet.
Reuse proxy on clients if possible. Openning TCP connection is expensive if you reuse the same proxy you will have single connection per proxy. This will affect instancing of you services - by default single service instance will handle all requests from single proxy.
Set service throttling so that your service host is ready for many clients
Also you should make some decisions about load balancing. Load balancing for WCF net.tcp connections requires sticky sessions (session affinity) so that after openning the channel client always calls the service on the same server (bacause instance of that service was created only on single server).

100 requests per second does not sound like much for a WCF service, especially with that little payload. But it should be quite quick to setup a simple setup with a WCF service with one echo method just returning the input and then hook up a client with a bunch of threads and a loop.
If you already have a working socket implementation you might keep it, but otherwise you can pick WCF and spend your precious development time elsewhere.

From my experience with WCF, i can tell you that it's performance on high load is very very nice. Especially you can chose between several bindings to achieve your requirements for the different scenarios (httpBinding for outside communication, netPeerTcpBinding in local network e.g.).

WCF communication with several clients without IIS

we're working on a peer to peer comm software that would allow a number of grocery stores to sync their inventory with what we call "headquarters".
To so this, we're thinking WCF+WPF, and no IIS and web services. My experience with WCF is basically zero, so my question is whether a TCP comm solution using WCF would work. The data that's being transferred is quite limited, about 2MB for a compressed plain text file (so we're sending binary data!), and this is done once per day only. So bandwidth/load shouldn't be an issue here.
The idea at this point is to have a WCF "server" running at HQ. Stores make themselves known to that server and then send files back and forth (simliliar to a chat application).
What I'm not sure of: does every store need to have a WCF "server" (or endpoint)? How would the server (=HQ) send a file to one of the clients (=stores)? Every store can send a file to any other store, and the HQ, and every store can also "request" a file from any other store/HQ.
Two limitations: None of the machines/computers involved can run Windows server for budget reasons, and as stated before IIS is a no-go.

If you are only sending files back and forth, I might question whether or not WCF even makes any sense. Have you considered just using a file transfer protocol, like scp or sftp?
Every machine will have to accept connections and have a file drop location setup, and then yuor application will have to monitor that location for new files. I love WCF in general, but a file transfer protocol is going to have a leg up if that is all you want to do.

If you direct all of your traffic via the server then there's no reason why you couldn't achieve this with WCF. The server would host WCF services in IIS with the stores having a client that was able to upload and request files. With this method, stores would not be able to directly transfer fiels to each other, but they would have to do it via the main server, which would suit your needs if you don't have the budget for the other scenario.
If all transfers are made once per day, the requests for files would be made with each client requesting what files they require, followed by each client uploading any files that are required by the server or any other client. The final step would be the server distributing the required files to each client. Obviously, this is a simplified view of it, the actual process may require some more thinking.

You don't need to host WCF in IIS, but is there any particular reason you don't want to do that?
You can host WCF in a ServiceHost, but then you need to build, maintain and deploy a lot of server/service features that IIS provides for free, such as application process recycling, activation-based hosting, etc.
In any case, it almost sounds like you need peer to peer networking. You can do that with WCF using the NetPeerTcpBinding.

If you have an opportunity to redesign your application, I suggest you do. You can throw strings around in WCF but if you can create a data contract you can keep all your communication strongly typed.
If you have access to windows server 2008 then the new IIS can host your WCF even if it isn't using tcp. Otherwise you just need to write an application that opens a service host, which you would usually wrap into a windows service. But as #MArk Seemann pointed out, you get lots of freebies by running your service in IIS.
Don't have any experience with the PeerTcpBinding but I can tell you that the NetTcpBinding is nice and fast plus it comes with all sorts of goodies like encryption and authentication if you want it.