I want to write a small c# ftp client class library which basically needs to transfer files to a ftp location.
What I want, is a 100% foolproof code, where I can get some sort of acknowledgement that the ftp file transfer has been either 100% successful or failed.
No resume support is required.
Good to have (but secondary):
Some sort of distributed transaction where only if the file transfer is successful for a file, i update my db for that particular file with 1 (true)... if it is failed, then db to be updated with 0 (false).
But suppose the ftp file transfer was successful, but for whatever reasons the db could not be updated, then the file over ftp should be deleted - I can easily do this using dirty c# code (where i manually try to delete the file if db update failed).
But what I am exactly looking for is file system based transaction over ftp... so both the file transfer as well as db update is not committed until both are successful (hence no need for manual delete).
Any clues?
Having had the "joy" of writing a FTP library myself here is my advice
1) Its NOT gonna be easy, because FTP servers return different return from the same command ( Like directory information,regular ftp commands and pretty much everything).
2) This is gonna take more time then you think
3) The dream about 100% foolproof transfer is not gonna happen, unless you control the FTP server and add a new FTP command so you can compare file hashes.
Pretty much if i was gonna do this again, and my goal was to transfer files ( And not learn from making the library) i would buy a already done library,
.NET has an FTP client you can use. I don't know how robust it is in the face of FTP server quirks; you'll have to test it against your customer's FTP server. As for verifying that the upload was successful, the only tools you have are (1) making sure there was no transport error during the upload, (2) validating the file size when you're done.
The FTP server isn't going to support transactions, so you're going to have to manage that yourself, but this isn't really a complicated scenario. Use a transaction for the DB update; backing out the FTP upload is one call.
try using Ftp with WCF
Related
How can I make sure that a file uploaded through SFTP (in a Linux base system) stays locked during the transfer so an automated system will not read it?
Is there an option on the client side? Or server side?
SFTP protocol supports locking since version 5. See the specification.
You didn't specify, what SFTP server are you using. So I'm assuming the most widespread one, the OpenSSH. The OpenSSH supports SFTP version 3 only, so it does not support locking.
Anyway, even if your server supported file locking, most SFTP clients/libraries won't support SFTP version 5. Or even if they do, they won't support the locking feature. Note that the lock is explicit, the client has to request it.
There are some common workarounds for the problem:
As suggested by #user1717259, you can have the client upload a "done" file, once an upload finishes. Make your automated system wait for the "done" file to appear.
You can have a dedicated "upload" folder and have the client (atomically) move the uploaded file to a "done" folder. Make your automated system look to the "done" folder only.
Have a file naming convention for files being uploaded (".filepart") and have the client (atomically) rename the file after an upload to its final name. Make your automated system ignore the ".filepart" files.
See (my) article Locking files while uploading / Upload to temporary file name for example of implementing this approach.
Also, some SFTP servers have this functionality built-in. For example ProFTPD with its HiddenStores directive (courtesy of #fakedad).
A gross hack is to periodically check for file attributes (size and time) and consider the upload finished, if the attributes have not changed for some time interval.
You can also make use of the fact that some file formats have clear end-of-the-file marker (like XML or ZIP). So you know, when you download an incomplete file.
A typical way of solving this problem is to upload your real file, and then to upload an empty 'done.txt' file.
The automated system should wait for the appearance of the 'done' file before trying to read the real file.
A simple file locking mechanism for SFTP is to first upload a file to a directory (folder) where the read process isn't looking. You can "make" an alternate folder using the sftp> mkdir command. Upload the file to the alternate directory, instead of the ultimate destination directory. Once the SFTP> put command completes, then do a move like this:
SFTP> move alternate_path/filename destination_path/filename. Since the SFTP "move" is just switching the file pointers, it is atomic, so it is an effective lock.
I want to use SQLite database which is on FTP server without downloading it. Is it possible to use this database directly?
No, the FTP protocol is designed to sequentially transfer the entire contents of files. There is no way to perform random reads/writes to a file, which is necessary for SQLite (or any database program) to work.
The connection strings for data providers used by SQLite only supports UNC paths, URL parameters are not supported. You must download the file locally.
I do not now how SQLite works, but if you have your database-files on the server, you can mount the the filesystem over ftp and run a local SQLite-server which talks to the mounted files.
According to how the ftp-protocol is designed; If your database is just one file, the system will download the hole file even if you just want the first row, each time the file are needed(if we not use file-cache). If your database is multiple files each file will be downloaded when they are needed. As Pavel Krymets said, it will be slow, so it is not recommended.
Full edit:
The scenario is that after uploading the file to the server via a secured web service, I'd like to save/create a copy of that file to another server in a LAN or another network.
I'd like to know what possible ways I could use to programmatically copy/create the backup of the file uploaded to the backup server (saving the file to the database would be the last option probably).
Here are a few details:
Files are of different types and sizes mostly text, documents and images that would be around a few KB to a couple of MB's.
Database is SQL Server 2008 R2 and the only way to connect to it is via calls to a secured WCF service.
Servers can be in the same LAN or on separate networks (depends on the client requesting).
The 2nd server is a redundant server and is using the 1st one as it's backup and vice versa.
Took me a while to find this post. Just map the drive to the backup server's shared folder and implement WindowsImpersonationContext.
How to Impersonate a user in managed code?
haven't seen security problems on this and doesn't need to mess with the HTTP/certificates.
Basically I have two parts of my application that I need help with.
I need to send an ini containing some preferences over a TCPClient socket to a client. How might I do this?
I need to scan a folder on the client side and see what is different from the folder on the server, then send the changed files.
Thanks very much,
Christian
EDITS
Potential leads:
Microsoft Sync Service - They mention that this can be used for remote files, but I have not found any explanation or code samples.
FileSync on SourceForge - Looks
good, but does it have network sync?
You could do this with Microsoft sync framework.
Here is a code sample for doing file synchronisation: http://code.msdn.microsoft.com/File-Synchronization-516e3ad7
Here's how I actually fixed this:
Client connects to server (or webserver!)
Client downloads XML with file structure, including MD5 of each file.
Client figures out which files need to be redownloaded.
New files are downloaded from server (or webserver!)
Honestly all of this can be done through a webserver approach, simply host all the files and the XML structure file on a normal webserver and then have the client use these files.
~Christian
I am scratching my head about this. My scenario are that I need to upload a file to the company server machine(to a folder on c:) from our hosting one(totally different server). I don't know how I should do this. Any of you got tips or code on how this is done.
Thanks Guys
I would set up an FTP server (like the one in IIS or a third-party server) on the Company Server. If security is an issue then you'll want to set up SFTP (secure FTP) rather than vanilla FTP since FTP is not a natively secure transfer protocol. Then create a service on the Hosting Server to pick up the file(s) as they come in and ship them to the company server using C#/.NET's FTP control. Honestly, it should be pretty straightforward.
Update: Reading your question, I am under the strong impression that you will NOT have a web site running on the company server. That is, you do not need a file upload control in your web app (or already know how to implement one given that the control is right in the web page toolbox). Your question, as I understand it, is how to get a file from the web server over to the company server.
Update 2: Added a note about security. Note that this is less of a concern if the servers are on the same subdomain and won't be routed outside of the company network and/or if the data is not sensitive. I didn't think of this at first because I am working a project like this now but our data is not, in any way, sensitive.
Darren Johnstone's File Upload control is as good a solution as you will find anywhere. It has the ability to handle large files without impacting the ASP.NET server memory, and can display file upload progress without requiring a Flash or Silverlight dependency.
http://darrenjohnstone.net/2008/07/15/aspnet-file-upload-module-version-2-beta-1/
There isnt enough info to tell your whole hosting scenario but I have a few suggestions that might get you started in the right direction:
Is your external server owned by another company or group and you cant modify it? If not you might consider hosting the process on the same machine, either in process or as a separate service on the machine. If it cannot be modified, you might consider hosting the service on the destination machine, that way its in the same place as the files need to show up at.
Do the files need to stay in sync with the process? I.e. do they need to be uploaded, moved and verified as a single operation? If not then a separate process is probably the best way to go. The separate process will give you some flexibility, but remember it will be a separate process and a separate set of code to manage and work with
How big is the file(s) that are being uploaded? Do they vary by upload? are the plain files, binaries (zips, executables, etc)? If the files are small you have more options than if they are large. If they are small enough, you can even relay them in line.
Depending on the answers to the above some of these might work for you:
Use MSMQ. This will work for simple messages under about 3MB without too much hassle. Its ideal for messages that can be directly worked with (such as XML).
Use direct HTTP(s) relaying. On the host machine open a HTTP(s) connetion to the destination machine and transfer the file. Again this will work better for smaller files (i.e. only a few KB since it will be done in-line)
If you have access to the host machine, deploy a separate process on the machine which builds or collects the files and uses any of the listed methods to send them to the destination machine.
You can use SCP, FTP (of any form SFTP, etc) on either the host machine (if you have access) or the target machine to host the incoming files and use a batch process to move the files. This will have a lot of issues to address, such as file size, keeping submissions in sync, and timing. I would consider this as a last resort, depending on the situation.
Again depending on message size, you could also use a layer of abstraction such as a DB to act as the intermediate layer between the two machines. This will work as long as the two machines can see the DB (or other storage location) and both act on it. SQL Server Service Broker could be used for this purpose (and most other DB products offer similar products).
You can look at other products like WSO2 ESB or NServiceBus to facilitate messaging between the two apps and do it inline.
Hopefully that will give you some starting points to look into.