I see a ton of questions about uploading multiple files, but none about uploading a single file to multiple servers, so here goes...
I have an ASP.NET app that will be running on two load balanced servers, and I would like to allow users to upload files and have them end up on both servers. What is the cleanest way to do this? I am using IIS 6 btw.
Some ideas that come to mind are:
1) Use a virtual directory that points to some shared location that both servers can access. Will there be any access issues if the application runs at Network Service? I'm assuming the application will need to run as a user account that exists on the shared location machine. How should the permissions be set for this?
2) It would be nice if I could via jQuery post the request to both of my servers, referencing them by their port numbers. Even though the servers are on the same domain, this violates the same origin policy, right?
Is there another solution I'm overlooking? How do other sites do this?
I think you want to consider this problem more carefully - having a pair (or more) of servers means that some of them will be offline some of the time (at least for occasional reboots).
Uploads when not all of the servers are online won't be able to be sent to all servers immediately, so you'd need either an intermediate server (which would be a point of failure unless it was highly available itself) or a queuing system to "remember" which files were where, and to transfer them when the relevant servers were restored.
Also, you'll want a backup system, and some way to add newly provisioned servers to your cluster. You will also want a way to monitor these files are the same in case they get out of sync. Your architecture needs a lot of careful thought. I don't have the answers :)
The cleanest approach is forwarding the files server-side, really. If you force two uploads via JavaScript, not only will you have to worry about working around XSS safeguards, but you'll also force the user to use their very limited upstream bandwidth twice for each file.
You shouldn't be exposing that kind of detail to the client anyway. The browser doesn't need to know where the file ends up, just who to send it to. If you keep that logic server-side, not only do you keep the details hidden (and thus less prone to errors and exploits), but you'll also get more control over the process. You can create a gateway service later that handles a multitude of back end storages and you can handle failing servers better. You can queue failed uploads and retry. All these come at a very low cost if you do them on the server side, but are a pain to be made to work reliably on the client side.
Keep back end logic to your back end. Load balancing should be hidden from the user, so there's no need to tell them where they are sending their files exactly. Make it optional, if you want, but hide the action from them. Just swallow the file on the gateway server (which can be either of the load balancing servers -- in fact, it should probably be load balanced, too, so it should work with either of them in place) and send it to the other servers from there. The transfer from server to server will probably be faster too.
Your best bet is definitely a NAS, if one is available -- a shared file system that is not specifically associated with any machine. Then you can focus on making the NAS highly available via a clustered frontend.
If that's not an option, you can use a virtual directory on each machine that points to one folder on one of the machines, but then you lose redundancy.
I'm faced with this same challenge at my work. My app is small but needs to be highly available, but there's no NAS in sight. So in each machine's web.config I place a list of all the UNC paths that the uploaded file should be stored. After uploading to a temp folder, I copy the file to each machine one by one. It's not perfect -- a machine could go down, in which case when it came up it might not have all the files (and the copy would be slowed by the hunt for the missing machine) -- but in my situation uploads are so infrequent that it's not worth improvement.
As others have mentioned, Javascript is right out. Upload once.
I have seen this problem solved with a NAS, using credentials for the app pool that can read/write files to that NAS. Make sure your NAS is setup for high availability to prevent single point of failure ie:hot swap w/ raid, multiple array controllers, power supplies..etc
You could also put folder monitoring software on the severs that keep certain directories in sync. I don't recommend this solution.
Related
I request to read my problem till end. Someone might find it duplicate.
I have a windows application (Client App) a machine & Web Application (Server App) on another machine in same Network.
Client app is capturing screen 5fps and keeping in a local folder which is shared. I have a Windows service which runs on server machine. It moves client images to server directory from client shared folder. I am using File.Move to move the files along with FastDirectoryEnumerator class. These moved files are used to create videos later and also used to show live streaming.
Questions:
Is there any other (Best/Fastest) option to move these files in real time (transfer as soon as it is created at client side)?
I am also interested in file transfer in real time without shared folder.
Update:
My major concerns.
File Transfer should be faster to allow live streaming through my server app (ASP.Net)
Client should retain it if server/connection is not available & transfer as soon as it comes online.
I do not know why you have the server monitoring the client's share, surely this monitoring or as you say, DirectoryEnumerator procedure takes time.
Since the client knows when the image has been captured why don't you send this information to the server immediately from the client? In this way you do not need to monitor clients from server, you do not need to find / enumerate folders, you simply transfer data from client to server as soon as it is available with a specific WCF / Web Service call which takes a stream of bytes as input.
If you have no control over the client app creating the image files you could have a FileSystemWatcher running on the client machine. Add an event handler which will be called whenever the Created event is raised due to a file being created on the file system.
See https://stackoverflow.com/a/15018082/1730317
Things to consider in gaining the best performance out of this type of activity.
For "Real Time" notifications you may want to look at the FileSystemWatcher class. See the documentation for a complete example. Word of warning on this though, when the Operating system is busy it will drop some of these events and you will not be informed of the new file. It's a good idea to program a sweep up loop make doubly sure you have all the files, I have been caught out with this in the past.
You may want to check your hardware configuration, like network latency between the pc and server, and also the disk speeds at both the PC and Server. A low spec SSD is likely to have a big performance benefit in these kinds of operations over spinning disks.
I have a product, and a front end website where people can purchase the product. Upon purchase, I have a system that creates an A record in my DNS server that points to an IP address. It then creates a new IIS website with the bindings required.
All this works well, but I'm now looking at growing the business and to do this I'll need to handle upgrades of the application.
Currently, I have my application running 40 websites. It's all the same code base and each website uses it's own SQL Server database. Each website is ran in a separate application pool and operate completely independently.
I've looked at using TeamCity to build the application and then have a manual step that runs MSDeploy for each website but this isn't particularly ideal since I'd need to a) purchase a full license and b) always remember to add a new website to the TeamCity build.
How do you handle the upgrade and deployments of the same code base running many different websites and separate SQL Server databases?
First thing, it is possible to have a build configuration in TeamCity that builds and deploys to a specific location...whether a local path or a network drive. I don't remember exactly how but one of the companies I worked with in Perth had exactly the same environment. This assumes that all websites are pointing to the same physical path in the file system.
Now, a word of advice, I don't know how you have it all setup, but if this A record is simply creating a subdomain, I'd shift my approach to a real multi-tenant environment. That is, one single website, one single app pool for all clients and multiple bindings associated to a specific subdomain. This approach is way more scalable and uses way less memory resources...I've done some benchmark profiling in the past and amount of memory each process (apppool) was consuming was a massive waste of resources. There's a catch though, you will need to prepare your app for a multi-tenant architecture to avoid any sort of bleeding such as
Avoiding any per-client singleton component
Avoiding static variables
Cache cannot be global and MUST a client context associated
Pay special attention to how your save client files to the file system
Among other stuff. If you need more details about setting up TeamCity in your current environment, let me know. I could probably find some useful info
I've designed a C# game that makes use of an Access .mdb database file to store variables.
The basic idea is that one of the players will 'host' a game, and the other player will join said game, by connecting to the database, reading and writing items into the database. The database is needed to pass variables to and fro the host and client, and both programs check the database regularly for new messages / variables (yes, yes, I couldn't get TCP/IP Remoting to work).
Now, there, the program works fine (mostly the client, since the host modifies only its local database), for as long as the client can find the database file. Currently, the connection string for the client is located in a .ini file, and for the program to work, it (or at least, the server) must be located in the Shared Documents of an XP machine, or the Public Documents of a Vista / 7 machine.
But then some questions occurred to me:
What if the client user places / installs it into another folder in the Shared / Public Documents? I suppose there's a code out there that could 'drill' into the Shared Documents folders to find the path for the database, share names included... is there?
What about installation? The installer default is always "C:\Program Files\GameName". Could it be set (or locked) to a specific folder into the Shared / Public Documents, by default? And is it even recommended to install it into a Shared Folder, what with problems such as Accidental Deletion?
Say, the default is "C:\Program Files\GameName". I could see that a set installation path will solve any connection issues, since every client will look in the same path of the other computer, and it won't matter if it's XP or Vista. Could "C:\Program Files\GameName" be accessible by a C# program over LAN, without any system modification? Are there any security (UAC?) issues?
P.S. Details that people may find in handy:
I'm using Visual Studio 2005. So is my School.
I'm running .NET framework 2.0. So is my school, and unfortunately, I can't change that.
I hate to tell you this, but this is why the approach is fundamentally flawed. What are you going to do when:
File and print sharing on the server is disabled?
The network client is disabled on a client computer?
The server is configured to only allow authenticated users, and the client doesn't have access?
File and print sharing is blocked because of the firewall?
File and print sharing is completely borked for other who-knows-why reasons?
You need administrator access on the server to share a folder, but the current user is not an administrator?
The Access database engine isn't installed correctly on the server or client?
One of the clients corrupts the database? (This is easier than you might think; all it takes is a network glitch where a client temporarily or permanently disconnects from the server).
These are fundamental issues that you can't resolve with the current solution.
Get it working over normal TCP/UDP ports. You'll have a much easier time. There are libraries to facilitate that, see C# Game Network Library - for example it sounds like Microsoft's XNA SDK has functions for networking in a game. If you use a standard method / standard library to implement networking, your odds of success are much higher. Using a client/server architecture means that the server's state won't be corrupted if a client goofs up.
I realize that this probably involves a significant rewrite. Frankly, I don't think you have a choice if you want this game to be commercially successful on a wide customer base without high support costs, and move beyond being a mere experiment.
Get rid of the access database. The described architecture is already 'user unfriendly'
Use either SQL Express or SQL CE which requires no installed database binaries (they can all be included in the applications folder). This way the db will stay in the applications folder (ideally app_Data) and .Net has included support for databases in this folder.
Please stay away from access you will save yourself a lot of trouble.
http://blogs.msdn.com/b/sqlservercompact/archive/2011/01/12/microsoft-sql-server-compact-4-0-is-available-for-download.aspx
I think that there is little point in using Access unless you wanted to use any of the pretty form-based IO that you can you do with its designer. And I doubt that'll be the case here.
I would encourage you to replace all your file IO with a sound data access layer.
Which will rid you of all the "sharing" issues that you are experiencing, file locks, the overcomplicated installation process, etc.
Alternatively to SQL Server Express as suggested above you could also use SQLite http://code.google.com/p/csharp-sqlite/
looking for examples of what people have done inorder to deploy the same webapp or processes to multiple servers.
The deployment process right now consists of copying the same file multiple times to different servers within our company. There has to be a better way to do this right now I am looking into ms build does anyone have other ideas? Thanks in advance.
Take a look at msdeploy and Web Deploy.
I've done this using a variety of methods. However, I think the best one is what I call a "rolling" deployment.
The following assumes a code only deployment:
Take one or more web servers "offline" by removing them from the load balancing list, let's call this group A. You should keep enough running to keep up with existing traffic, we'll call those group B. Push the code to the offline servers (group A).
Then, put group A back into rotation and pull group B out. Make sure the app is still functional with the new code. If all is good, update group B and put them back in rotation. In the event of a problem, just put group B back in and take A out again.
In the case of a database update there are other factors to consider. If you can take the whole site down for a limited period then do so and perform all necessary updates. This is by far the easiest.
However, if you can't then do a modified "rolling" deployment which requires multiple database servers. Pick a point in time and move a copy of the production database to the second one. Apply your changes. Then pull a group of web servers out, update their code to production and test. If all is good, put those web servers back into rotation and take out group B. Update the code on B while pointing them to the second DB server. Put them back into rotation.
Finally, apply all data changes that occurred on the primary production database to the secondary one.
Note, I don't use Web Deploy or MS Deploy for pushes to production. Quite frankly I want the files ready to be copy/pasted into the correct directory on the server so that the push can run as quickly as possible. Both Web and MS Deploy options involve transferring those files over a network connection; which is typically much slower than simply copy/pasting from one local directory to another.
You can build a simple console app that connects to a fixed sftp download, uncompress
and run all the files in a fixed directory. A meta XML file can be usefull to create rules
such as each machine will run each application, pre-requirements and so on.
You can also use dropbox api to download your files if you don't have a centralized server to unify your apps.
Have a look at kwateeSDCM. It's language and platform agnostic (Windows, Linux, Solaris, MacOS). There's an article dedicated to deployment of a webapp on multiple tomcat servers.
I am scratching my head about this. My scenario are that I need to upload a file to the company server machine(to a folder on c:) from our hosting one(totally different server). I don't know how I should do this. Any of you got tips or code on how this is done.
Thanks Guys
I would set up an FTP server (like the one in IIS or a third-party server) on the Company Server. If security is an issue then you'll want to set up SFTP (secure FTP) rather than vanilla FTP since FTP is not a natively secure transfer protocol. Then create a service on the Hosting Server to pick up the file(s) as they come in and ship them to the company server using C#/.NET's FTP control. Honestly, it should be pretty straightforward.
Update: Reading your question, I am under the strong impression that you will NOT have a web site running on the company server. That is, you do not need a file upload control in your web app (or already know how to implement one given that the control is right in the web page toolbox). Your question, as I understand it, is how to get a file from the web server over to the company server.
Update 2: Added a note about security. Note that this is less of a concern if the servers are on the same subdomain and won't be routed outside of the company network and/or if the data is not sensitive. I didn't think of this at first because I am working a project like this now but our data is not, in any way, sensitive.
Darren Johnstone's File Upload control is as good a solution as you will find anywhere. It has the ability to handle large files without impacting the ASP.NET server memory, and can display file upload progress without requiring a Flash or Silverlight dependency.
http://darrenjohnstone.net/2008/07/15/aspnet-file-upload-module-version-2-beta-1/
There isnt enough info to tell your whole hosting scenario but I have a few suggestions that might get you started in the right direction:
Is your external server owned by another company or group and you cant modify it? If not you might consider hosting the process on the same machine, either in process or as a separate service on the machine. If it cannot be modified, you might consider hosting the service on the destination machine, that way its in the same place as the files need to show up at.
Do the files need to stay in sync with the process? I.e. do they need to be uploaded, moved and verified as a single operation? If not then a separate process is probably the best way to go. The separate process will give you some flexibility, but remember it will be a separate process and a separate set of code to manage and work with
How big is the file(s) that are being uploaded? Do they vary by upload? are the plain files, binaries (zips, executables, etc)? If the files are small you have more options than if they are large. If they are small enough, you can even relay them in line.
Depending on the answers to the above some of these might work for you:
Use MSMQ. This will work for simple messages under about 3MB without too much hassle. Its ideal for messages that can be directly worked with (such as XML).
Use direct HTTP(s) relaying. On the host machine open a HTTP(s) connetion to the destination machine and transfer the file. Again this will work better for smaller files (i.e. only a few KB since it will be done in-line)
If you have access to the host machine, deploy a separate process on the machine which builds or collects the files and uses any of the listed methods to send them to the destination machine.
You can use SCP, FTP (of any form SFTP, etc) on either the host machine (if you have access) or the target machine to host the incoming files and use a batch process to move the files. This will have a lot of issues to address, such as file size, keeping submissions in sync, and timing. I would consider this as a last resort, depending on the situation.
Again depending on message size, you could also use a layer of abstraction such as a DB to act as the intermediate layer between the two machines. This will work as long as the two machines can see the DB (or other storage location) and both act on it. SQL Server Service Broker could be used for this purpose (and most other DB products offer similar products).
You can look at other products like WSO2 ESB or NServiceBus to facilitate messaging between the two apps and do it inline.
Hopefully that will give you some starting points to look into.