I have a public server with web services (.Net) that collect data and uploaded files from different mobile apps and I need to synchronise it with an internal intranet server.
The intranet server is deeply protected by firewall and organisation policies.
I think this is a pretty common scenario where messages and brokers could be used, something like Rabbitmq or Nservicebus, but I'm not an expert on it.
As the data is only to be sent from the external server to the intranet one in unidirectional and asynchronous way I was thinking not to add another layer of indirection to the architecture and just use the web services exposed also for server to server communication.
The approach would be like:
An intranet windows serivce would poll regularly and at different scheduled intervals the external web service to know if there is new data to get (maybe from a certain point in time)
The web service would respond with the list of the new data and files
The windows service would iterate with calls to get all the data to be inserted in the intranet and download the uploaded files.
What are the risks of this approach? Would be better that the external web service would respond only a link to a huge zipped file response with all the data and files in it?
Should I use a something like RabbitMq also for a so simple scenario?
If you are literally dealing with files, you might want to think about something even simpler. FTP (more specifically sftp) might fit your needs better, and be FAR simpler to implement.
Related
I've inherited a relatively low traffic web application in which the main website is accessible on intranet, however it gets its data from a wcf service that runs on the same server which is only accessible via localhost. It was explained to me that this design was implemented as a security measure - essentially to ensure that no entity external to the server could potentially have access to our service and hence our data. The database however, is usually located on a different server.
It's been ok for a while but I am looking at ways to improve performance and it seems that running queries against the wcf service and having to serialize the response for transmission etc. is a waste of time - I'd like to just access the db directly from my web app.
Is this current design logical? Wouldn't it be better overall (for security, and performance) to have my site access db directly, and beef up the security between the app and the db?
Thanks in advance.
Rusty
I'm not sure if I have the big picture from your description. However, it is a common practise to have web-applications consuming content from more than a service to form a graphical interface to end-user. With the grow of SOA, this allows for fast integration to combine sources (from services) and producing rich output. This is called Mashup
This also, amongst other things, enhances the security of the backend services as they are only accessed by the backend application server, and not directly from front-end client
So, from an architectural perspective, it seems your application is trying to do this.
Having the services on the same server, and only consumed by the local server (localhost) is a choice taken to prevent but the application server running on the same machine from having access. This could be due to lack of better control on network access and network zoning
Your database lives on a different server, where there could be another measure implemented to secure the traffic and access
In general, implementation of security requires a budget, as any other functional and non-functional requirement. This is usually associated with the risk you're exposed to, and the sensitivity of the information. The earlier the security is built into the overall architecture the better.
Accessing the database from your web-application requires best practices to protect against the many risks of intrusions and vulnerabilities. In general, your web-client should NOT access the DB directly, and should always use server-side services and validation to do so, be it through wcf or other means
I have currently built application in which I handle Products, Accounts, Orders, etc.
There are two databases one is a database which I created where I handle Users and Roles and some minor Application specific data. The other database is external and it is the one which holds all the data about Orders, Products and Accounts.
What I am trying to figure out is: how to build a server which runs parallel to the main application and handles all data manipulation with the external database.
Let's say a situation in which this would be helpful: There is an excel file which has to be created based on big amount of data and afterwards it has to be stored in the externalDB as certain type of format and sent as Email to someone. This will surely overload the main thread of the main application, hence we don't want that. Therefore, it would be a good idea to handle those kind of situations outside user's vision.
I am using ASP.NET MVC 5 and was curious what would be a good approach for this situation? I was thinking that I should make Console application, which is working as a service.
You would create a service architecture and have the services communicate with each other through controllers. These could push data back to the main application views, or the views could request data push from the services.
If you wanted to have jobs etc running on that server that send emails, you could easily just create SQL jobs or SSIS jobs or even custom service jobs that did that based on your criteria that are separate from the main view (where view is the main application that the user interacts with)
Services themselves could be configured with a light User Interface that you could call up on the server, perhaps in the service tray or through the Services component of Windows.
SOA architecture
https://www.cleverism.com/how-to-build-service-oriented-architecture-soa/
Micro Service architecture
http://microservices.io/patterns/microservices.html
Hope that helps. Good luck!
There are quite a few ways to do these kinds of things.
If your applications are for Android or iOS devices, it's common practice to have an external server running which handles intensive processing including managing your excel data, sending emails and database communication.
If your applications are cross-platform desktop applications or OS-specific desktop applications, you can still be able to use the external server, although certain other parts of the processing may be done on the client machine.
By using a web service to deal with database communication, it makes the applications more secure in regards to hard-coded database connection information, SMTP server credentials etc.
Using a RESTful web service will mean that you're adhering to the Hypertext Transfer Protocol (HTTP) which is great for exchanging data in the majority of cases.
Using a console application which acts as a TCP client server allows you to use the Transmission Control Protocol (TCP) to communicate in a different way. You're able to pass on data in a custom format if you wish. You'd be able to create a standard for communication which your applications will need to stick to.
The console application would do the same processing as the web service, however you would be in full control of logging requests, responses and data transfer.
I need multiple clients that talk to a WCF service. The WCF service also must be able to connect to any one of the clients also.
So - it sounds like the server and the clients need to have both a WCF server and client built into each one.
Is this correct or is there some way to do this?
I was looking at NetPeerTcpBinding, but that is obsolete. To be fair I'm not sure if that is a valid solution either.
Background:
I plan to have a Windows service installed on hundreds of machines in our network with a WCF service and a WCF client built in.
I will have one Windows service installed on a server with a WCF service and a client built in.
I will have a Windows Forms application
I will have a database
The clients on the network will connect to the service running on the server in order to insert some information on the database.
The user will use the Windows Forms application to connect to the Windows service on the server and this Windows service will connect to the relevant client on the factory floor (to allow remote browsing of files and folders).
Hence I believe the machines on the floor and the server both require a WCF cleint and service built in.
The reason people are recommending wsHttpDualBinding is because it is in itself a secure and interoperable binding that is designed for use with duplex service contracts that allows both services and clients to send and receive messages.
The type of communication mentioned 'duplex' has several variations. Half and Full are the simplest.
Half Duplex: Works like a walkie-talkie, one person may speak at any given time.
Full Duplex: Like a phone, any person may speak at any given time.
Each will introduce a benefit and a problem, they also provide ways to build this communication more effectively based upon your needs.
I'm slightly confused, but I'll attempt to clarify.
You have an assortment of approaches that may occur here, a Windows Communication Foundation (WCF) Service requires the following:
Address
Binding
Contract
Those are essentially the "ABC's" for WCF. The creation of those depicts a picture like this:
As you can see the Service will contain:
Host
Service
Client
The host houses the service which the client will consume so those service methods perform a desired task. An example representation:
As you see Client-1 is going through the Internet (HTTP, HTTPS, etc.) then will hit the Host, which will have the service perform those tasks.
Now Client-n is consuming the service locally, so it is talking over (TCP, etc.) as an example.
The easiest way to remember: One service can be consumed by however many clients require those methods to perform a task. You can create very complex models using a service-oriented architecture (SOA).
All WCF is, is a mean to connect your application to a host or
centralized location you may not have access to.
As you can see in the above image, the Client communicates through a Service to the Host. Which performs a series of task. WCF will talk over an array of protocols. Hopefully this will provide a better understanding of how WCF is structured.
There are a lot of tutorials and even post to get you started. Some excellent books such as "WCF Step by Step".
Essentially your looking for an asynchronous full duplex connection, or a synchronous full duplex service. As mentioned above, your task in essence is the point of a Service.
The question: How does this work best?
It will boil down to your design. There are limitations and structures that you will need to adhere to to truly optimize it for your goal.
Such obstacles may be:
Server Load
Communication Path
Security
Multiple Clients Altering UI / Same Data
Etc.
The list continues and continues. I'd really look up tutorials or a few books on WCF. Here are a few:
WCF Step by Step
WCF Multi-Tier Development
WCF Service Development
They will help you work with the service structure to adhere to your desired goal.
Remember the "ABCs" for the most success with WCF.
Use wsDualHttpBinding if you want your service communicate with your clients.
Read WS Dual HTTP.
You might want to try out creating a WCF service using netTcpBinding. It will work for your requirements. You can use the article How to: Use netTcpBinding with Windows Authentication and Transport Security in WCF Calling from Windows Forms as a start:
Also, there are many examples included within the WCF Samples package which you can use.
i have implemented a fairly simple wcf service which handles the file transfers from my clients to the server the problem is when a client sends a file request.
all of the bandwidth is allocated to that single client and others have to wait until the requested file transfer is completed.
So my question is how to make the service more efficient and let the users share the bandwidth
[ServiceBehavior(IncludeExceptionDetailInFaults = true, InstanceContextMode =InstanceContextMode.PerCall,
ConcurrencyMode=ConcurrencyMode.Multiple)]
I set the InstanceContextMode attribute to PerCall but that didn't do the trick
UPDATE : This Project is similar to mine
http://www.codeproject.com/Articles/33825/WCF-TCP-based-File-Server
WCF does not have proper load balancing, you will have to develop one yourself.
If you are transferring files, lets assume download, you should send packets of data rather than the complete file at once. When doing this, add 'delays/sleeps' to the process to limit the amount of bytes the server sends on each time window, this will make room for other requests.
It's questionable that it's desirable to serve up files through a WCF endpoint. The reasons against doing this are pretty much exactly the problems you have been having. It works for a few clients at a time - but scaling out requires hosting new instances of the service behind a load balancer.
It would be worth considering hosting your files with some kind of storage service and have your WCF service simply return a link or handle to the file. Then the file can be retrieved offline. Microsoft have created Azure Blob Storage for this exact purpose.
Appreciate this does not address your original question, and understand the scope of your requirement may not accommodate a large reworking.
Another option is to use chunking channel if you are transferring large files. Examples: MSDN, codeplex.
Although I agree with #hugh position.
we're working on a peer to peer comm software that would allow a number of grocery stores to sync their inventory with what we call "headquarters".
To so this, we're thinking WCF+WPF, and no IIS and web services. My experience with WCF is basically zero, so my question is whether a TCP comm solution using WCF would work. The data that's being transferred is quite limited, about 2MB for a compressed plain text file (so we're sending binary data!), and this is done once per day only. So bandwidth/load shouldn't be an issue here.
The idea at this point is to have a WCF "server" running at HQ. Stores make themselves known to that server and then send files back and forth (simliliar to a chat application).
What I'm not sure of: does every store need to have a WCF "server" (or endpoint)? How would the server (=HQ) send a file to one of the clients (=stores)? Every store can send a file to any other store, and the HQ, and every store can also "request" a file from any other store/HQ.
Two limitations: None of the machines/computers involved can run Windows server for budget reasons, and as stated before IIS is a no-go.
If you are only sending files back and forth, I might question whether or not WCF even makes any sense. Have you considered just using a file transfer protocol, like scp or sftp?
Every machine will have to accept connections and have a file drop location setup, and then yuor application will have to monitor that location for new files. I love WCF in general, but a file transfer protocol is going to have a leg up if that is all you want to do.
If you direct all of your traffic via the server then there's no reason why you couldn't achieve this with WCF. The server would host WCF services in IIS with the stores having a client that was able to upload and request files. With this method, stores would not be able to directly transfer fiels to each other, but they would have to do it via the main server, which would suit your needs if you don't have the budget for the other scenario.
If all transfers are made once per day, the requests for files would be made with each client requesting what files they require, followed by each client uploading any files that are required by the server or any other client. The final step would be the server distributing the required files to each client. Obviously, this is a simplified view of it, the actual process may require some more thinking.
You don't need to host WCF in IIS, but is there any particular reason you don't want to do that?
You can host WCF in a ServiceHost, but then you need to build, maintain and deploy a lot of server/service features that IIS provides for free, such as application process recycling, activation-based hosting, etc.
In any case, it almost sounds like you need peer to peer networking. You can do that with WCF using the NetPeerTcpBinding.
If you have an opportunity to redesign your application, I suggest you do. You can throw strings around in WCF but if you can create a data contract you can keep all your communication strongly typed.
If you have access to windows server 2008 then the new IIS can host your WCF even if it isn't using tcp. Otherwise you just need to write an application that opens a service host, which you would usually wrap into a windows service. But as #MArk Seemann pointed out, you get lots of freebies by running your service in IIS.
Don't have any experience with the PeerTcpBinding but I can tell you that the NetTcpBinding is nice and fast plus it comes with all sorts of goodies like encryption and authentication if you want it.