Develop Reference

Best way to track files being moved (possibly between disks), VB.NET (or C#)

I am developing a "dynamic shortcutting" application which creates special shortcut files which point to a registry entry rather than an actual file/executable. The registry entry contains the path of the desired file. I want to have a daemon running which watches the linked-to files and updates their registry entries if they are moved or renamed. Renamed I can handle using System.IO.FileSystemWatcher, but what is the best way to handle moved files?
I know this is beyond the basic functions of FSW (despite being a low-level file-system operation). The question is, what is the best way of doing it?
Most posts/articles I have read suggest ways that feel altogether "hacky", which basically involve looking for a delete followed by a create in a new place of a file, and connecting the two by file size, meta-data, time between the delete/create triggers, hashes, etc. This may well be the method I have to resort to, setting up FSWs on all drives. However, I am hoping there might be a better way.
Is it possible to either:
2.1. Listen in to the shell and "hear" move operations?
2.2 Or (even more radical) replace or add something to the shell move operation that either triggers some sort of event or performs the registry-updating task itself, precluding the need for the daemon?
I have a feeling that everyone is going to tell me that 1. is the only course, but I look forward to your suggestions. (answers in VB.NET preferred, but can translate from C# if necessary).

[I'm not sure if this should be appended as an "update" to my original post or posted as a separate answer]
To sum up (all two of) the answers plus my own experimenting (to try to give a definitive answer to this question):
It seems the only high-level (.NET) solution is to use the FileSystemWatcher which does not detect "move" out-of-the-box (despite it being a low-level command). The FSW approach is non-trivial, comparably resource-expensive, sloppy in places (i.e. using timers) and has its limitations and caveats. Nor does it provide a true reflection of "move" - it merely infers it from symptoms that are very likely to be a move (and have the same effect on the file-system in any case) but could theoretically be produced by non-move actions. Also, it appears you have to know what files you want to watch for moves in advance of the move happening, there's no-way of telling as it occurs.
On a lower-level (which would involve C++), one could hook API calls to get a faithful picture of when "moves" are called. This has the advantage that you don't have to decide to watch files in advance, and is also less resource-expensive than listening to "deletes" and "creates" and trying to compare them.
On a systems-programming level (which would involve C++ and could easily break your computer if you didn't know what you were doing) one could build a filesystem filter driver: this would take the concept of detecting moves to a truly anal level, detecting re-allocation of filesystem resources performed even without the kernel.
After some experimenting, here is the general structure of how the FileSystemWatcher approach (or at least the most obvious one to me) works, its quirks and its limitations. [no code atm, it's all pretty integrated into my application and I'm yet to optimise it, but I might add some snippets in here later].
The FileSystemWatcher method (to detect when files are moved or renamed):
.1. FileSystemWatchers.
You will need to create one FSW for each highest-level directory you want to monitor (for example, one for each writable logical drive).
.2. Renamed.
Straightforward renaming of the file is trivially handled.
.3. Moved.
This part is very far from trivial; it basically involves comparing files in three different scenarios.
3.0.1. Deciding if a deleted/moved-from file is the same as a created/moved-to file.
For determining whether a deleted and a created file are a match, filename is useless (can be changed during a move). You could use a mixture of file size and attributes like time created, or even a hash of the entire file. In my particular solution I only needed to watch the movement of specific files "registered" before load-time, so I was able to give these files a unique fingerprint as metadata that I could then use to compare files (this works fine in real-world scenarios, but is easy to break maliciously in testing, which disappoints me as a perfectionist.)
3.0.1.1. When to read filesize/attributes/take hash?
Before I came up with the static fingerprint idea, I was testing my code with a simple filesize + creation date validation check. I quickly realised though that I had to have a note of the filesize and creation date (or hash or whatever else you want to use) of the deleted file BEFORE it signals as "deleted", because you can't check the size of a file that doesn't exist. If (like me) you know the files you want to watch in advance, then you need to read in those values before you enable the FileSystemWatchers; you also need to listen for "change" events on those files to update the values of filesize and creation date, take a new hash etc. This then begs the question: what do you do if you DON'T know what files you are interested in watching to see if they move? What if you only know you are possibly interested in knowing if they've moved when they "delete"? That, unfortunately, is beyond me (it wasn't something I had to deal with.) Unless you can come up with a solution to this problem, there is zero point in continuing with the FileSystemWatcher approach. Furthermore, I would conjecture (though could very easily be wrong) that there is no high-level solution that will meet your needs. If you do however come up with a solution (please post it below/comment on this post/edit it in here on this post), I have made the rest of this compatible.
3.1. Scenario 1: Direct moving of the file itself.
Upon the "delete" of a specific file being detected, you need to start listening for a "create" of a congruous file. Rather than listening indefinitely for the matching "create" of a file that might just have been deleted (which in reality involves inspecting every file created in the directory), you can use a timer to start and stop a "listening" flag (practical, but from a purist point of view a little arbitrary), deciding that after e.g. 1000ms with no appropriately matching create it's likely there won't be one.
3.2.0. A common misconception.
A lot of people seem to be under the impression, after glancing at the docs, that moving or renaming a folder triggers a rename for all their subfiles and subfolders rather than a delete and a create. In actual fact what the docs say is:
If you cut and paste a folder with files into a folder being watched, the FileSystemWatcher object reports only the folder as new, but not its contents because they are essentially only renamed.
(i.e. only the top folder throws rename or create/delete and the subfiles/subfolders throw NOTHING). Meaning if you want to know when and where a certain file is moved, you have to listen out for each and every of its ascendent folders as well.
3.2.1. Scenario 2: Renaming of a containing folder.
In my solution, because I knew all the files I was watching, whenever one of my FileSystemWatchers reported a rename of a folder rather than a file (the portion of the string after the last "/" will contain no ".") I checked each of my watched files to see if their paths were in that directory and if so, changed the beginning of the filepath to the path of the new directory et voila!, I knew where my files had been moved to. If you do not now in advance what files you are looking for, then you will have to recursively search through everything in every folder that throws a "rename".
3.2.2. Scenario 3: Moving of a containing folder.
This one feels like a slap in the face: in order to build your move-detection routine, you have to be able to detect moves. Here folders will throw a "delete" followed by a "create". In my case the solution just recycles the techniques in 3.1 and 3.2.1: when a folder "delete" is detected, I check to see if it contains any of my watched files. If it does, I set a "listen" flag (and a timer to snuff it) and check the subdirectory path of my file in the old folder against every new folder "create" that is detected to see if it points to a file with the desired fingerprint. If it does, I now have the old and new paths of the file and have detected the move. If you don't know what files to watch for, you may have to validate folder moves by comparing size on disk and number of subfiles/subfolders between "deleted" folder and "created" folders to confirm a folder has moved first, then search the folder recursively for the files you're interested in.
3.3. FURTHER COMPLICATION: Cross-drive moving of large files.
This is a problem I fortunately didn't run into (because I was only comparing fingerprint metadata, and didn't need access to files); however moving large files between drives (which transfer in stages, triggering a create event then a series of change events) can cause real headaches.
3.3.1. Headache 1: The "create" fires when the destination file is incomplete.
This means comparing its size to a "deleted" file will produce a false negative. You can't even take a hash of the first part of the file to indicate to your program that this "might" be the deleted file, because the move operation will have the file access permissions locked down. You just have to try and tell if the created file might still be moving and wait for it to finish.
3.3.2. Headache 2: No sure way to "tell" that the created file is still being moved.
Some have suggested checking the file access permissions on the created file, but they might be indistinguishable from those on a file created and still in use by any random application. Others have suggested setting short time-limited listen flags for "changes" on the file, but again this is indistinguishable from a file being modified by an application. In fact if the file happened to be a log file constantly and rapidly being updated by some process, then waiting for "changes" to the file to timeout might never end.
3.3.3. Headache 3: (UNTESTED) possibly these sort of moves "delete" the file after "creating" the destination file*.
It makes sense that this would be the case, though I haven't tested it. [if anyone does know, feel free to edit (or delete) this section appropriately]
3.4. A philosophical quandry: are two identical files the same?
This is a very pedantic and arbitrary thought-experiment, but say you have two drives, each with an identical copy of File.txt. You run a batch file that deletes the copy on the first drive then immediately makes a copy of the file on the second drive into the same folder on the second drive and names it Copy of File.txt. Unless you are using fingerprints, your code will identify a delete and then a create of an identical file and be unable to distinguish what happened from a move (with renaming) of the file from the first drive to the second. The final state of the filesystem is identical in both cases so it shouldn't cause your application to behave unexpectedly, but art thou really content to call that a "move" based purely on isomorphism? (especially when you know the kernel sees it differently)?

Using high-level unrestricted api provided by C# - no, you cant. Use FileSystemWatcher.. On same drive operation of moving file is not "delete and create" - it's "rename".
If you can/want to go into lower-level, then you can hook MoveItem and MoveItems of IFileOperation shell's interface, and MoveFile from Kernel32.dll... It will work with most of apps, but require expansion for security rights for your application, that mostly unacceptable in corporative environment..

The task has two flaws that make it hard to implement: (a) move operation across the disks is actually a sequence of read/write operations followed by deletion rather than move. And during those read/write operations there can be some transformation of data in place ; and (b) moving can be performed not by just a shell.
What you can do is employ a filesystem filter driver to intercept file operations right when they take place. Then you need to detect the sequence of read and write operations performed by the same process over your file. I.e. if your code detects, that the file is read sequentially (NOTE: some copying tools can read the file in multiple threads in parallel) and then write similar blocks of data to the other file AND after reading everything the source file is deleted AND the complete file contents have been written to the other place, then you can guess that you have come over file move operation.

Bump & update: This may well be against the rules of StackOverflow, but I would like to point out to the many people landing on this page (and the myriad similar questions on SO) that I have started a feature request on MicroSoft UserVoice to add MOVE detection to FileSystemWatcher. The best solution in the long term, rather than trying to work around the problem, might be to petition MicroSoft to fix it. If you have come here because you too need a solution to this problem, please consider clicking here and voting for this feature.

Is it possible to be sure a certain file can be written to without opening it?

I wrote a custom control for output file name selection with the typical: text box for the filename, a "browse" button, and some other functionality specific to my application.
The text box changes color depending on the filename. If the file location cannot be written to, it turns red. If the file already exist, it turns yellow. Otherwise, it remains the system-assigned color.
To see if a file exists, I use IO.File.Exists; simple enough.
I implemented the "if the file can be written to" as a simple try-catch block where a file is actually opened, something written in it, closed, then deleted. If at any point an exception is thrown, I know the user can't use that filename and I turn the text box red.
This is a catch-all; since I'm doing the actual operation I intend to do, it is fool-proof. However, it seems irresponsible to have software creating and deleting files like crazy just to see if it can.
So my question is, how do I replicate this functionality without creating files? I can see I have to:
Check the path for legality (e.g., 'z:' is not a valid filename). This entails parsing the path and making sure all directories exist.
If the location exists, I have to check for write permissions. (Several answered questions exist to this end.)
Is there anything else?
EDIT
Within minutes I see people are already voting up an answer that criticizes that I'm checking at all that the file is accessible before actual writing to it occurs. While I appreciate experts "standing back" from my question to see whether or not there is a completely different way to achieve it, telling me I shouldn't be doing it is not an answer to my question.
So let me elaborate on my application (I am not expecting hundreds of users at the same time).
I use this file chooser control in data acquisition applications. In many situations the test that you are about to run is "expensive" in one way or another. Therefore it is critical to set things up very carefully. Overwriting data can be very expensive (and for the fearful user I have a checkbox that will append the date and time down to the millisecond to the filename).
So the purpose of my indicator colors is not to provide a surefire way for the software to know the file can be written to (that check is still done at the instant it actually has to), it's to serve as an indicator to the user that at least he has set up the file name correctly so if he goes forward he is guaranteed not to overwrite old data and he's almost sure a last-minute IO error (filename typo) won't let the experiment run unrecorded.

I suggest this - don't check anything before user commits the action. With your current approach, even if you verified the file is okay, it may be locked 5 seconds later when the user actually commits to write to a file. Doing preliminary checks may only give user a false impression of estimated success. Especially consider this point on a terminal server with 100+ simultaneous users.
There is nothing wrong with showing a prompt with Retry/Cancel/etc. if no access, and let user decide.
EDIT:
No offense, but there are standards on how such collisions are handled. Windows standard is to show a prompt to the user. Also consider this - if you suddenly have a deny in write access to the folder, which you are not expected to have, you probably need to hire another system/network administrator.
If the operation is costly, make sure this guy is paid well. C'mon, what if your network goes down during writing? Hard drive? Router? There are many reasons why writing to a file can be interrupted, and you should be prepared for that. If you cannot afford it, make sure you have invested in good infrastructure and good people to support it.
Down on earth, you can increase chances of acquiring a successful lock on the file:
Pick a unique file name, using datetime-based hash as a suffix/prefix.
Write to user's home directory, also known as %UserProfile%, it is likely that you will succeed.

I can understand your problem with not wanting to risk losing "expensive" data because the file couldn't be written and a responsible program will do it's best to avoid the situation.
I would do this by cacheing the results. Before the test is run write a mock result to a file somewhere in the user data space, then leave the file open and write the real result to the file. After this is done write it to the user-specified file. Provide a recovery option that will read the cache file and write it out to the user's file.
Your approach could fail because just because the file was writable at the start doesn't mean it's still writable. The network could have gone down. Someone could have removed the flash drive. Someone else could be doing a large data transfer through a buggy router. (Real world case--it took me a long time to prove it was a network problem and not my program. finally accepted it was their fault when I showed that dir :*.* /s on multiple machines at once would almost certainly cause one or more to fail.)

Prevent EXE being copied to another PC - keep some info inside EXE file (e.g. some flag)

I want to prevent executable being copied to another PC and thus i need to somehow save information inside my EXE file about that it was already used somewhere else on another PC.
Can i embed small piece of information like user's hard drive number into my EXE file so this information would be available when this EXE is copied to another PC?
I thought maybe there is a way to read and write to some resource file embedded in an EXE file but i presume that resource file is read only and if so is there is a place inside EXE file where i could keep information which i need?

You're fighting an uphill battle this way. It's possible to create a home-grown licensing scheme but be prepared to do a lot of work (I did it, so I speak from first-hand experience). Just some problems to solve:
If the hard drive fails and needs to be replaced, your user won't be able to use the program. Every time this happens, you'll get a support call with an angry user.
If the user runs your program inside a virtual machine, the hard drive serial number won't be unique - anyone can clone the virtual machine and now your program can be run on another machine.
Hard drive serial numbers can be changed - they don't come directly from the hardware.
What if the hard drive is a removable drive? Your user can run your program from a removable drive and then keep moving it to different machines.
Even if you get it done, how do you protect the license information from being modified?
If you really want to license your product, look at existing licensing products - they're not cheap but they already did the (considerable amount of) work that's necessary to have any kind of reliability.
Even if you only want to have minimal protection, consider this: you'll have to do a lot of work to get even minimal security of your secret token (whatever that is). If its security is minimal, then what's the point of you even doing all that work? If all you do is force people to put in a meaningless serial number, you'll just annoy your honest customers. If anyone wants to steal something that's not well protected, they will steal it. All a 'simple' protection scheme does is annoys your users and gives you a false sense of protection.
I ended up using Reprise RLM - I'm not associated with this company but I had a good experience with their sales and support people and their product worked well in the testing scenarios.

Ok, I analyzed all the variants that were proposed and decided that in my case it will be better to develop my own copy-protection system, due to the reason that I am an indie developer and not going to work with extra large applications.
Just in case, somebody faces to the same issue - here is the algorithm (well, one of them):
User starts APP1.EXE
APP1.EXE reads itself to some variable and adds HDD serial number to the end of it, e.g. HDD serial number - when you add something to the end it does not break EXE file and you do not have to worry about PE headers
Unfortunately, EXE cannot save itself in runtime so it saves its copy called APP2.EXE with the information about HDD
When APP2.EXE is saved APP1.EXE starts it as a separate process via Process.Start() and terminates itself
Now APP2.EXE is running and has the same content as APP1.EXE + HDD serial number so we simply write all bytes from APP2.EXE back to APP1.EXE, close current process and start APP1.EXE again
From now on APP1.EXE is running and have all needed information about current HDD so each time user starts APP1.EXE it compares HDD number at the end of its content with the actual one on user's PC, if they differ - terminate the process
Delete APP2.EXE so that user would not realize how these files exchange information about his HDD.
Useful info about self-deleting EXE can be found here :
http://www.catch22.net/tuts/self-deleting-executables
http://buffernow.com/selfdelete-executable-in-c/
P. S. I know that it is like a huge hole of security (I will not mention all of them) but implementation of this algorithm took just 20 lines of code in C# and was moved to a separate DLL which I can use everywhere and it works. There is NO any registration in the algorithm above and user can simply take this app and use it and I am sure that ~ 80% of them will not realize how this app is protected from copying.
Link to implementation : https://bitbucket.org/artemiusgreat/examples/src/ef7b60142277?at=master

C# make file unable to edit and delete

Is there any way to make a file totally uneditable and undeleteable ? I am creating simple Anti-Virus program and I want to protect my malware signatures which are saved in files.

The short answer is 'you can't.' The long answer follows. =)
You may implement it via file permissions, but those can be changed if a process have enough privileges.
TMK, the only way to implement this kind of restriction is to keep a process running, with the file open in exclusive mode. That won't prevent an application like Unlocker from killing your main process or deleting the block handles, though.

No, you can't. If a software runs with enough privileges, it will be able to erase them along with your antivirus. This also happens with commercial antivirus software.
What you can do, in order to at least prevent modifications, is store the definitions as compressed, signed and, encrypted. In this way, unless the malware can obtain the criptographic key, it won't be able to meaningfully modify the virus database, but only to delete it. In both cases, your software can detect the intervention and try to react (but if a malware is privileged enough to delete system files, maybe it' already too late)

you cant really do so, but you can try outsmarting malware...
Save a checksum of the file so you know if it was tampered.
Use Async Encryption on the file (somwhat similar to 1.)
make the signatures downloadable through Internet access, and make your software download those...
check the last accessed times of the files.
there are many more tricks like the four above, but they are all NOT boolet proof...
One Crazy idea that i dont really know how to implement... but came to mind is that:
you can create a SATA/IDE Driver and make the a specific file unaccesible...
but again thats my kind of creativity crazy talk :)

The best you can do with C# is to just set the permissions of the file so that only your service has full access, and anyone else doesn't. That don't protects against someone/something that managed to get administrator access, as they always can change permissions.
What many antiviruses do for self-protecting their files and services is to install kernel-mode drivers that block both the critical files and processes, so not even administrators can stop them. Of course C# is unable to create them.

Find out on what place on a harddisk a program / file is installed

Let's say I have a file: test.txt and I save it on my harddisk.
Is there a way to determine on what (physical) spot the file is saved on the hard disk?
For example on vector 12 on track 10 of the hard disk.
I don't know if I got the terminology right of the above, but I hope you get what I mean.
I want to write of program wheer the user can point to a file and the program will find out where the file is on the HDD. Something like the old defrag (it's Windows ;) ) where it shows what parts of the disk is in use.
What is this called and can it be achieved? (I'm not looking for code (although exmaples are ok ofc), but rather whether it is possible)
P.S. The client will be Windows 7 (so think NTFS if it matters).

I'm pretty sure that doing that sort of low-level disk i/o in managed code is going to be...difficult, at best. Here's somebody that's done something like it:
http://codebrainz.ca/index.php/2010/05/23/low-level-disk-io-in-managed-net/
Anything you write to do something like this has to be hardware-dependent: unless you know what hardware you're talking to, you've got no idea how it physically stores data (e.g., a USB memory stick has neither platters, tracks nor sectors, nor does it spin. Yet, for all intents and purposes, it appears to be a disk).
Normally, you'd write some sort of device driver to accomplish this. This link
http://en.wikibooks.org/wiki/Windows_Programming/Device_Driver_Introduction
might help.

In Jeffrey Wall's WebLog you will find Defrag API C# wrappers. His GetFileMap method sems to come close to what you need.

It is possible from C++, so with a little interoping, you should be fine.
Look up FSCTL_GET_RETRIEVAL_POINTERS in the MSDN to get you started.