Develop Reference

Update file, not replace or overwrite

this is more of a question because I am experimenting with this.
All over the internet I see how you can update a .txt file. Well that is all good and well, but lets say I have a .doxc or even an .exe or even a .dll file.
If we make a minor change to a file, do we really have to replace(overwrite) the whole file?
Is it possible to "update" the file so that we don't use too mush data (over the internet).
What I am trying to achieve is to create a FTP client with a FileSystemWatcher. This will monitor a certain folder on the Computer. If anything changes in this folder (even sub directories) then it uploads, deletes, renames, or changes the file. But at the moment I am wondering if I have, lets say, a 20MB .exe file or whatever, if it is possible to change something in that .exe, instead of just overwriting the whole thing... thus, sparing some cap.

In general, it's possible to update the remote file only partially, but not in your case.
What would work:
1) track the file change using a filesystem filter driver, which gives you information about what parts of the file have been updated.
2) use the protocol which allows partial upload or remote modification of the file (eg. SFTP).
As for your scenario:
Step 1 is not possible with FileSystemWatcher.
Step 2 is not possible with FTP protocol which doesn't support modification of file blocks.

Since your are experimenting, I can provide some pointers. But I dont know for sure if the below operations are just updates or replaced newly by the underlysing os calls
Have different cases for each file type. Try with a basic types first, a txt file, then a binary file etc.
You should have the entire copy of the current file somewhere, sine you "should" compare the old file to know what changes
Then when a change is made to the file compare it with the old file e.g) in a text file with 1 MB and the change is only 1 KB you will need to build a format like
[Text][Offset][[operation]
e.g) [Mrs.Y][40][Delete] then [Mr.X][40][Add]
Then your ftp client should be able to implement this format and make changes to the local copy on the client.

No it is not possible to only upload the changes to .exe file.we have to overwrite it.

#Frederik - It would be possible if FTP supports an updating of resource like HTTP's PUT command. Try exploring that angle. Let us know if you find something.

Best way to track files being moved (possibly between disks), VB.NET (or C#)

I am developing a "dynamic shortcutting" application which creates special shortcut files which point to a registry entry rather than an actual file/executable. The registry entry contains the path of the desired file. I want to have a daemon running which watches the linked-to files and updates their registry entries if they are moved or renamed. Renamed I can handle using System.IO.FileSystemWatcher, but what is the best way to handle moved files?
I know this is beyond the basic functions of FSW (despite being a low-level file-system operation). The question is, what is the best way of doing it?
Most posts/articles I have read suggest ways that feel altogether "hacky", which basically involve looking for a delete followed by a create in a new place of a file, and connecting the two by file size, meta-data, time between the delete/create triggers, hashes, etc. This may well be the method I have to resort to, setting up FSWs on all drives. However, I am hoping there might be a better way.
Is it possible to either:
2.1. Listen in to the shell and "hear" move operations?
2.2 Or (even more radical) replace or add something to the shell move operation that either triggers some sort of event or performs the registry-updating task itself, precluding the need for the daemon?
I have a feeling that everyone is going to tell me that 1. is the only course, but I look forward to your suggestions. (answers in VB.NET preferred, but can translate from C# if necessary).

[I'm not sure if this should be appended as an "update" to my original post or posted as a separate answer]
To sum up (all two of) the answers plus my own experimenting (to try to give a definitive answer to this question):
It seems the only high-level (.NET) solution is to use the FileSystemWatcher which does not detect "move" out-of-the-box (despite it being a low-level command). The FSW approach is non-trivial, comparably resource-expensive, sloppy in places (i.e. using timers) and has its limitations and caveats. Nor does it provide a true reflection of "move" - it merely infers it from symptoms that are very likely to be a move (and have the same effect on the file-system in any case) but could theoretically be produced by non-move actions. Also, it appears you have to know what files you want to watch for moves in advance of the move happening, there's no-way of telling as it occurs.
On a lower-level (which would involve C++), one could hook API calls to get a faithful picture of when "moves" are called. This has the advantage that you don't have to decide to watch files in advance, and is also less resource-expensive than listening to "deletes" and "creates" and trying to compare them.
On a systems-programming level (which would involve C++ and could easily break your computer if you didn't know what you were doing) one could build a filesystem filter driver: this would take the concept of detecting moves to a truly anal level, detecting re-allocation of filesystem resources performed even without the kernel.
After some experimenting, here is the general structure of how the FileSystemWatcher approach (or at least the most obvious one to me) works, its quirks and its limitations. [no code atm, it's all pretty integrated into my application and I'm yet to optimise it, but I might add some snippets in here later].
The FileSystemWatcher method (to detect when files are moved or renamed):
.1. FileSystemWatchers.
You will need to create one FSW for each highest-level directory you want to monitor (for example, one for each writable logical drive).
.2. Renamed.
Straightforward renaming of the file is trivially handled.
.3. Moved.
This part is very far from trivial; it basically involves comparing files in three different scenarios.
3.0.1. Deciding if a deleted/moved-from file is the same as a created/moved-to file.
For determining whether a deleted and a created file are a match, filename is useless (can be changed during a move). You could use a mixture of file size and attributes like time created, or even a hash of the entire file. In my particular solution I only needed to watch the movement of specific files "registered" before load-time, so I was able to give these files a unique fingerprint as metadata that I could then use to compare files (this works fine in real-world scenarios, but is easy to break maliciously in testing, which disappoints me as a perfectionist.)
3.0.1.1. When to read filesize/attributes/take hash?
Before I came up with the static fingerprint idea, I was testing my code with a simple filesize + creation date validation check. I quickly realised though that I had to have a note of the filesize and creation date (or hash or whatever else you want to use) of the deleted file BEFORE it signals as "deleted", because you can't check the size of a file that doesn't exist. If (like me) you know the files you want to watch in advance, then you need to read in those values before you enable the FileSystemWatchers; you also need to listen for "change" events on those files to update the values of filesize and creation date, take a new hash etc. This then begs the question: what do you do if you DON'T know what files you are interested in watching to see if they move? What if you only know you are possibly interested in knowing if they've moved when they "delete"? That, unfortunately, is beyond me (it wasn't something I had to deal with.) Unless you can come up with a solution to this problem, there is zero point in continuing with the FileSystemWatcher approach. Furthermore, I would conjecture (though could very easily be wrong) that there is no high-level solution that will meet your needs. If you do however come up with a solution (please post it below/comment on this post/edit it in here on this post), I have made the rest of this compatible.
3.1. Scenario 1: Direct moving of the file itself.
Upon the "delete" of a specific file being detected, you need to start listening for a "create" of a congruous file. Rather than listening indefinitely for the matching "create" of a file that might just have been deleted (which in reality involves inspecting every file created in the directory), you can use a timer to start and stop a "listening" flag (practical, but from a purist point of view a little arbitrary), deciding that after e.g. 1000ms with no appropriately matching create it's likely there won't be one.
3.2.0. A common misconception.
A lot of people seem to be under the impression, after glancing at the docs, that moving or renaming a folder triggers a rename for all their subfiles and subfolders rather than a delete and a create. In actual fact what the docs say is:
If you cut and paste a folder with files into a folder being watched, the FileSystemWatcher object reports only the folder as new, but not its contents because they are essentially only renamed.
(i.e. only the top folder throws rename or create/delete and the subfiles/subfolders throw NOTHING). Meaning if you want to know when and where a certain file is moved, you have to listen out for each and every of its ascendent folders as well.
3.2.1. Scenario 2: Renaming of a containing folder.
In my solution, because I knew all the files I was watching, whenever one of my FileSystemWatchers reported a rename of a folder rather than a file (the portion of the string after the last "/" will contain no ".") I checked each of my watched files to see if their paths were in that directory and if so, changed the beginning of the filepath to the path of the new directory et voila!, I knew where my files had been moved to. If you do not now in advance what files you are looking for, then you will have to recursively search through everything in every folder that throws a "rename".
3.2.2. Scenario 3: Moving of a containing folder.
This one feels like a slap in the face: in order to build your move-detection routine, you have to be able to detect moves. Here folders will throw a "delete" followed by a "create". In my case the solution just recycles the techniques in 3.1 and 3.2.1: when a folder "delete" is detected, I check to see if it contains any of my watched files. If it does, I set a "listen" flag (and a timer to snuff it) and check the subdirectory path of my file in the old folder against every new folder "create" that is detected to see if it points to a file with the desired fingerprint. If it does, I now have the old and new paths of the file and have detected the move. If you don't know what files to watch for, you may have to validate folder moves by comparing size on disk and number of subfiles/subfolders between "deleted" folder and "created" folders to confirm a folder has moved first, then search the folder recursively for the files you're interested in.
3.3. FURTHER COMPLICATION: Cross-drive moving of large files.
This is a problem I fortunately didn't run into (because I was only comparing fingerprint metadata, and didn't need access to files); however moving large files between drives (which transfer in stages, triggering a create event then a series of change events) can cause real headaches.
3.3.1. Headache 1: The "create" fires when the destination file is incomplete.
This means comparing its size to a "deleted" file will produce a false negative. You can't even take a hash of the first part of the file to indicate to your program that this "might" be the deleted file, because the move operation will have the file access permissions locked down. You just have to try and tell if the created file might still be moving and wait for it to finish.
3.3.2. Headache 2: No sure way to "tell" that the created file is still being moved.
Some have suggested checking the file access permissions on the created file, but they might be indistinguishable from those on a file created and still in use by any random application. Others have suggested setting short time-limited listen flags for "changes" on the file, but again this is indistinguishable from a file being modified by an application. In fact if the file happened to be a log file constantly and rapidly being updated by some process, then waiting for "changes" to the file to timeout might never end.
3.3.3. Headache 3: (UNTESTED) possibly these sort of moves "delete" the file after "creating" the destination file*.
It makes sense that this would be the case, though I haven't tested it. [if anyone does know, feel free to edit (or delete) this section appropriately]
3.4. A philosophical quandry: are two identical files the same?
This is a very pedantic and arbitrary thought-experiment, but say you have two drives, each with an identical copy of File.txt. You run a batch file that deletes the copy on the first drive then immediately makes a copy of the file on the second drive into the same folder on the second drive and names it Copy of File.txt. Unless you are using fingerprints, your code will identify a delete and then a create of an identical file and be unable to distinguish what happened from a move (with renaming) of the file from the first drive to the second. The final state of the filesystem is identical in both cases so it shouldn't cause your application to behave unexpectedly, but art thou really content to call that a "move" based purely on isomorphism? (especially when you know the kernel sees it differently)?

Using high-level unrestricted api provided by C# - no, you cant. Use FileSystemWatcher.. On same drive operation of moving file is not "delete and create" - it's "rename".
If you can/want to go into lower-level, then you can hook MoveItem and MoveItems of IFileOperation shell's interface, and MoveFile from Kernel32.dll... It will work with most of apps, but require expansion for security rights for your application, that mostly unacceptable in corporative environment..

The task has two flaws that make it hard to implement: (a) move operation across the disks is actually a sequence of read/write operations followed by deletion rather than move. And during those read/write operations there can be some transformation of data in place ; and (b) moving can be performed not by just a shell.
What you can do is employ a filesystem filter driver to intercept file operations right when they take place. Then you need to detect the sequence of read and write operations performed by the same process over your file. I.e. if your code detects, that the file is read sequentially (NOTE: some copying tools can read the file in multiple threads in parallel) and then write similar blocks of data to the other file AND after reading everything the source file is deleted AND the complete file contents have been written to the other place, then you can guess that you have come over file move operation.

Bump & update: This may well be against the rules of StackOverflow, but I would like to point out to the many people landing on this page (and the myriad similar questions on SO) that I have started a feature request on MicroSoft UserVoice to add MOVE detection to FileSystemWatcher. The best solution in the long term, rather than trying to work around the problem, might be to petition MicroSoft to fix it. If you have come here because you too need a solution to this problem, please consider clicking here and voting for this feature.

Create a Directory using Directory.CreateDirectory( ) that gives a warning message when trying to delete or rename

My application creates several text files that contain important data for its working. I am grouping up all these files in a folder. In Form_load() I gave Directory.CreateDirectory("C:\\xyz"), so that a folder is created in a drive which will later include text files, when the application will be delivered to clients. I want to see that the user does not delete or rename the folder. A dialog box that shows some warning message will be helpful. Is it possible with Directory.CreateDirectory()?

I think you should just warn the user about the importance of this directory. However, you can recreate the directory by performing this kind of check :
if(!Directory.Exists(path))
Directory.CreateDirectory(path);
From this point, you can imagine a mecanism that save the important files in a temporary way, and if the directory is deleted, you recreate it and put the previously saved files in it.

First of all, you might find reading this helpful.
Second: If I understand your needs correctly, what you want is much more painful than you think.
You should really consider doing what Damien_The_Unbeliever told you.
However, if you insist: You will need to use C++ (C# will give you and your users a lot of pain) for it, and you will need to "hook" SHFileOperation through something like ICopyHook. Start by taking a look at ICopyHook. It will be a lot of work and a lot of learning, and I don't think it's worth it, but there's the info. Good luck.

Why isn't my XML file being saved, even though the program sees the updated values next time I start it?

I'm reading the contents of an XML file and parsing that into an object model.
When I modify the values in the object model, then use the following code to save it back to the xml:
XElement optionXml = _panelElement.Elements("options").FirstOrDefault();
optionXml.SetAttributeValue("arming", value.ToString());
_document.Save(_fileName);
This works, as far as I can see, because when I close the application and restart it the values that I had saved are reflected in the object model next time I view it.
However, when I load the actual XML file, the values are still as they were originally.
Why is this? What do I need to do to save the actual XML file with the new values?

You are most likely experiencing file system virtualisation, which was introduced in Windows Vista.
Basically what this means is that you are saving your file, just not where you think you're saving it. For example, you might think that you are saving to C:\Program Files\Your App\yourFile.xml, but what is happening under the hood is that the OS is silently redirecting that to %APPDATA%\Your App\yourFile.xml. When you go to reload it, once again the OS silently redirects from that location.
This is a security measure designed to better encapsulate applications and their data and to prevent unauthorised writes to locations where damage can occur. You can still force a save to %PROGRAMFILES%\Your App, but to do that you either need to relax the ACLs applied to that folder, or you need to elevate the privilege level your application runs at.

I wasn't sure whether to put this as a comment or as an answer, but I think it could be a potential answer. It sounds like the XML file is being saved because the data is being persisted across instances of the application. It may be file system virtualization like slugster mentioned, but it might be a simple as the fact that you are looking at the wrong copy of the XML file. If you are using a relative path, the file may have been copied to the new location. I would suggest you do a quick file search for that file name and see what you get back.

It turns out the file was being copied to and read from the Output Directory. I can see that it's being updated as expected from there.

Periodically finding changed files in a directory using .NET

I'm trying to find the most reliable way of finding new and modified files in a directory using C# and .NET. I'm not looking for a real time solution, I want to check for changes at given times. It could be every 5 minutes or every hour etc.
We have CreationTime and LastWriteTime on the FileInfo object, and this seems to be enough to get new and modified files. But if a file is renamed none of the available dates are changed and the file will be missed if we just look at CreationTime and LastWriteTime.
At the moment i'm maintaining af "snapshot" of the files in the directory including the time of the last check for changes. This enables me to compare all the files in the directory with files in the snapshot, if the snapshot is missing a file it is either new or renamed.
Is this the only way? Rr am I missing something. I'm not going to use FileSystemWatcher as it seems pretty "buggy" and is required to run all the time.
Any suggestions are very welcome.
Merry Christmas!

Use the FileSystemWatcher class, it's the good way. Maybe you could be more specific with the
as it seems pretty "buggy"
EDIT: FileSystemWatcher does support renaming events.

The Microsoft Sync Framework has components for synchronising files.
The framework covers all data types and data storage and the file system component should be more reliable than the FileSystemWatcher. As it says in the MSDN:
It can be used to synchronize files and folders in NTFS, FAT, or SMB file systems. The directories to synchronize can be local or remote; they do not have to be of the same file system. An application can use static filters to exclude or include files either by listing them explicitly or by using wildcard characters (such as *.txt). Or the application can set filters that exclude whole subfolders. An application can also register to receive notification of file synchronization progress
I know you really only want to know when files have changed, but given that you've already dismissed the FileSystemWatcher route it might be the only reliable route (other than doing what you are in maintaining a snapshot yourself).

Your problem looks very much like a Database with no primary key.
If you asssign, say, a GUID to each file in that folder and check for that GUID instead of the filename, your application will be much more reliable.
So that's the theory, in practice, we're talking metadata. Depending on your system, and the files contained in that folder, you could use Alternate Data Streams.
Here is a SO question about it.
It boils down to having information on a file that is not stored within the file, it is merely linked to it.
You can then look it up in a DOS box:
notepade.exe myfile.txt:MYGUID
It requires the system to use NTFS.
HTH.

A very primitive approach would use a command of "dir" and comparing outputs...
Here is some info on params:
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/dir.mspx?mfr=true
Along with you snapshot of dates, you can compare the outputs of dir... its very fast and low on resources consuming...

File system watcher class in .net provides two methods:-
OnChanged
OnRenamed
You can set the EnableRaisingEvents to true and that is!!!! Every thing is simple with Dot Net chill!!!