Develop Reference

FileSystemWatcher unreliable for changes in subdirectory

I am currently implementing file content watchers for OpenFOAM output files. These files get written by OpenFOAM in an Unix environment and consumed by my applications in a Windows environment.
Please consider my first, working watcher for convergence files (these files get updated after each iteration of the solution):
FileSystemWatcher watcher;
watcher = new FileSystemWatcher(WatchPath, "convergenceUp*.out");
watcher.NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.Attributes | NotifyFilters.FileName | NotifyFilters.Size;
watcher.Changed += Watcher_Changed;
watcher.EnableRaisingEvents = true;
private void Watcher_Changed(object sender, FileSystemEventArgs e)
{
Files = Directory.GetFiles(WatchPath, "convergenceUp*.out").OrderBy(x => x).ToList(); // Update List of all files in the directory
ReadFiles(); // Do fancy stuff with the files
}
This works as expected. Everytime a file matching the pattern is changed in the watched directory (Notepad++ does notify me that the file has changed aswell) the files are processed.
Moving on from this simple "all files are in one directory" scenario I started to build a watcher for a different type of file (Force function objects for those familiar with OpenFOAM). These files are saved in a hierarchical folder structure like thus:
NameOfFunctionObject
|_StartTimeOfSolutionSetup#1
| |_forces.dat
|_StartTimeOfSolutionSetup#2
|_forces.dat
My goal is to read all forces.dat from "NameOfFunctionObject" and do some trickery with all the contained data. Additionally I also like to have the chance of reading and watching just one file. So my implementation (which borrows heavily from the above) currently looks like this:
FileSystemWatcher watcher;
if (isSingleFile)
watcher = new FileSystemWatcher(Directory.GetParent(WatchPath).ToString(), Path.GetFileName(WatchPath));
else
watcher = new FileSystemWatcher(WatchPath, "forces.dat");
watcher.IncludeSubdirectories = !isSingleFile;
watcher.NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.Attributes | NotifyFilters.FileName | NotifyFilters.Size | NotifyFilters.DirectoryName | NotifyFilters.LastAccess | NotifyFilters.CreationTime | NotifyFilters.Security;
watcher.Changed += Watcher_Changed;
watcher.Created += Watcher_Created;
watcher.Deleted += Watcher_Deleted;
watcher.Error += Watcher_Error;
watcher.Renamed += Watcher_Renamed;
watcher.EnableRaisingEvents = isWatchEnabled;
So depending on wether I want to watch just one file or multiple files I set up the directory to watch and the file filter. If I watch multiple files I set the watcher to watch subdirectories aswell. Because of vigorous testing I filter for all notifications and catch all watcher events.
If I test the single file option, everything works as expected, changes to the file are reported and processed correctly (again, the check with trusty old Notepad++ works)
On testing the multi-file option though, things get pear shaped.
The file paths are correct, the initial read works as expected. But neither watcher event fires. Here comes the curious bit: Notepad++ beeps still away, saying the file has changed, Windows explorer shows a new file date and a new file size. If I save the file within Notepad++, the watcher gets triggered. If I create a new file matching the pattern insinde the watched directory (top level or below does not matter!) the watcher gets triggered. Even watching for a filter of . to catch creation of temporary files does not trigger, so it is safe to assume that no temporary files are created.
In general, the watcher behaves as expected, it can detect changes to a single file, it can detect creations of files in the root watched folder and its subfolders. It just fails to recognise non-windows-changes to a file once it is located in a subfolder. Is this behaviour by design? And more importantly: how can I work elegantly around it without resorting to using a timer and polling by hand?

I think this might be relevant to you
FileSystemWatcher uses ReadDirectoryChangesW Winapi call with a few relevant flags
When you first call ReadDirectoryChangesW, the system allocates a
buffer to store change information. This buffer is associated with the
directory handle until it is closed and its size does not change
during its lifetime. Directory changes that occur between calls to
this function are added to the buffer and then returned with the next
call. If the buffer overflows, the entire contents of the buffer are
discarded
The analogue in FileSystemWatcher is the FileSystemWatcher.InternalBufferSize property
Remarks You can set the buffer to 4 KB or larger, but it must not
exceed 64 KB. If you try to set the InternalBufferSize property to
less than 4096 bytes, your value is discarded and the
InternalBufferSize property is set to 4096 bytes. For best
performance, use a multiple of 4 KB on Intel-based computers.
The system notifies the component of file changes, and it stores those
changes in a buffer the component creates and passes to the APIs. Each
event can use up to 16 bytes of memory, not including the file name.
If there are many changes in a short time, the buffer can overflow.
This causes the component to lose track of changes in the directory,
and it will only provide blanket notification. Increasing the size of
the buffer can prevent missing file system change events. However,
increasing buffer size is expensive, because it comes from non-paged
memory that cannot be swapped out to disk, so keep the buffer as small
as possible. To avoid a buffer overflow, use the NotifyFilter and
IncludeSubdirectories properties to filter out unwanted change
notifications.
If worse comes to worse, you can use a mix of polling and tracking, it has helped me out of trouble a few times

FileSystemWatcher sort algorithm

I've created a simple FileSystemWatcher service that's running on my PC:
public static void Run()
{
var watcher = new FileSystemWatcher
{
Path = #"C:\Users\XXX\Google Drive",
NotifyFilter = NotifyFilters.LastAccess
| NotifyFilters.LastWrite
| NotifyFilters.FileName
| NotifyFilters.DirectoryName,
Filter = "*.*",
};
watcher.Created += OnChanged;
watcher.EnableRaisingEvents = true;
}
private static void OnChanged(object source, FileSystemEventArgs e)
{
FooPrintClass.SendToPrinter(e.FullPath);
}
As you see I'm watching a Google Drive folder. That folder is also synced on my server. From time to time a system on my server will create 2 pair of files with the same name but with diffrent type:
(Foo.pdf, Foo.txt)
Sometimes the system will create over 50 of those pairs and they all will be synced to my Google Drive folder.
So far so good, now to my problem:
My FileSystemWatcher service do work as expected, but it dosen't treat them in any sorting matter at all.
I need my service to actually process each pair at a time.
Expected Result:
Foo.pdf, Foo.txt
Bar.pdf, Foo.txt
Actual Result:
Bar.txt, Foo.pdf
Foo.txt, Bar.pdf
As the expected result show, I need to print the pairs in order first.
There are many ways to implement a "queue" solution, but in my case I don't know how many files there will be. So I don't know the total of the files and therefor it'll be harder to build a queue and sorting algorithm.
Any tips?

As you use 3d party system for syncing files you have no control how it is done. You may have problems - no control in which order or they are synced, no guaranty that when you get a notification from your watched a file is not locked.
To easy the problem with sync order you may sync files in bundles.
If you could modify the system that creates these files, you can ZIP both files in one zip file. Having Foo.zip you can print both files in order you want.
It doesn't solve the problem with possible locking. If you could notify your service somehow about a new pair of files, you can just download these files directly from Google Drive using the API. In this case you will have full control over files and the order you get them.

You could use Reactive Extensions to buffer a number of events and sort them before continuing.
An example would be something like this:
Observable
.FromEventPattern<FileSystemEventArgs>(watcher, "Created")
.Buffer(TimeSpan.FromSeconds(10))
.Subscribe(onNext);
public void onNext(IList<string>) { ... }
The example buffers all changes happening in 10 seconds and passes them to onNext as a list. This allows you to sort the files before doing anything else.
This ignores some edge cases like files being created right at the time when the buffer window ends. But there are multiple ways to solve those issues.

File system filter driver for specific file types

I need to detect when either of two file types are accessed in any way across an entire windows file system.
As I understand it the only way to do this without causing serious slow downs for the operating system is to create a file system filter driver?
Essentially all I need to do is take a copy of any doc(x) files and pdf's that are opened. I decided on this approach as it was either that or use file monitors in C# which wouldn't be effective for an entire drive.
My question is two fold, is there an easier way and secondly how would I go about simply taking a copy of each doc(x)/pdf file as it's accessed?
The solution needs to be deployable with the package we're currently producing.
UPDATE
I'm going to benchmark the file system watcher, after discussing it with people here I think it's possible that it may be acceptable, my concern is the fact that I need to monitor the common user directories where downloads will occur( so "C:\Users\SomeUser*" as well as the outlook temporary folder.

You will need to create a file system watcher. Here is a code example that will watch for changes to docx files.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Security.Permissions;
namespace filewatchtest
{
class Program
{
static void Main(string[] args)
{
Run();
}
[PermissionSet(SecurityAction.Demand, Name="FullTrust")]
public static void Run()
{
string[] args = System.Environment.GetCommandLineArgs();
// if directory not specified then end program
if (args.Length != 2)
{
Console.WriteLine("Usage: filewatchtest.exe directory");
return;
}
// create a new fileSystemWatcher and set its properties
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = args[1];
// set the notify filters
watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite | NotifyFilters.FileName | NotifyFilters.DirectoryName;
// set the file extension filter
watcher.Filter = "*.docx";
// add event handlers
watcher.Changed += new FileSystemEventHandler(OnChanged);
watcher.Created += new FileSystemEventHandler(OnChanged);
watcher.Deleted += new FileSystemEventHandler(OnChanged);
watcher.Renamed += new RenamedEventHandler(OnRenamed);
// bengin watching
watcher.EnableRaisingEvents = true;
// wait for the user to quit the program
Console.WriteLine("Plress q to quit the program");
while (Console.Read()!='q');
}
static void OnRenamed(object sender, RenamedEventArgs e)
{
Console.WriteLine("File: {0} renamed to {1}", e.OldFullPath, e.FullPath);
}
static void OnChanged(object sender, FileSystemEventArgs e)
{
Console.WriteLine("File:" + e.FullPath + " " + e.ChangeType);
}
}
}

I think that creating a copy on read will cause a lot of problems. For instance: virus scanners. Consider the following:
I open file "test.pdf"
Your program creates "test_copy.pdf"
Virus scanner detects new file and checks (reads) "test_copy.pdf"
Your program detects read access, and creates "test_copy_copy.pdf"
Virus scanner...
Now you ofcourse you could create copies with a different extension to prevent this, but still there will be a lot of READ actions on files. I sometimes open a file like 10 times, just because I closed it accidentally or I want to recheck something I just read. Now you'll have 10 copies?
I would definitly go with Hans Passant's suggestion of creating a copy on change/create. That happens a lot less by definition, because you always need to open it to alter it, but don't have to alter it when you open it.
The second problem would be to detect a read to a file. Now with docx you could check for the creation of hidden files like '~$_____.docx', but that doesn't work for PDF. Also like you mentioned, you will have to check an entire disk. There is no way around it. If a file can be in any folder, you'll have to check all the folders. Creating an internal list of docx and PDF files in a service could be faster, but as you'll have to loop trough each file again at set intervals it depends on how many files are on the system.
So if you really need to check read access, a file system driver is all you got. But since it will be called on every file access, causing problems or slow systems would be a mayor concern.
If you still want to, check out this File System Filter Driver Tutorial to learn how to do it. Personally, I wouldn't go there.

From what I read in the comments, a File System Watcher would probably work well. I am not exactly sure whether Search Everything uses one, but if it does, I cannot notice any impact.
Another option might be ETW - Windows Event Tracing as used by Process Monitor. Even with millions of changes, I can also hardly notice the impact.
I you want to go for Volume Shadow Copies as proposed by Hans Passant, Alpha Volume Shadow Copies might be a suitable library offering support for it.
Conclusion: a filter driver is probably not needed and keeps you away from other problems, although I admit that the description of hierarchical storage management systems might match your approach, thinking of the upload store as the next hierarchy after hard disk.

FileSystemWatcher getting too many event

I'm trying to implement a file watcher that will raise an event if the file content was changed.
The problem that once the file I am watching was modified, I am getting 2 events.
(I want to get it only once)
_automationStatusWatcher = new FileSystemWatcher(fileInfo.Directory.FullName,
fileInfo.Name);
_automationStatusWatcher.NotifyFilter = NotifyFilters.LastWrite;
_automationStatusWatcher.Changed += OnAutomationStatusChanged;
_automationStatusWatcher.EnableRaisingEvents = true;
The file that i'm watching is not intended to be recreated/moved/deleted/whatever. its purpose is to be an xml database file that i need to monitor it once it changes. i want to get only 1 event when the file is modified and to ignore the other events.
How can I do that?

Manual:
Note
Common file system operations might raise more than one event. For example, when a file is moved from one directory to another, several OnChanged and some OnCreated and OnDeleted events might be raised. Moving a file is a complex operation that consists of multiple simple operations, therefore raising multiple events. Likewise, some applications (for example, antivirus software) might cause additional file system events that are detected by FileSystemWatcher.

I had to cope with this problem some time ago. I wanted to be notified when a file was created. The problem was that the event is triggered when the file is still empty or still being written to, not when the new file is closed.
The solution was to create the file using a temporary name and when finished renaming the file to its final name. I then watched for the rename-event.
You could name the file to myfile.tmp while writing to it and when finished rename it to myfile.xml and watch the rename-event.

You could attempt to unhook your event handler once you catch a change until you are ready to accept more changes:
_automationStatusWatcher.Changed -= OnAutomationStatusChanged;
But that is not necessarily thread safe and you could get multiple firings anyway.
Another option is to use a semaphore variable to see if you are handling the change:
private bool _inStatusChange = false;
private void OnAutomationStatusChanged(object sender, args...)
{
if (_inStatusChange)
{
return;
}
else
{
_inStatusChange = true;
//do work
_inStatusChange = false;
}
}
You should use appropriate locking to keep access to the semaphore variable thread safe.

i meet this problem too, now i found out this way :
new Thread(() => {
while (true) {
var r = watch.WaitForChanged(WatcherChangeTypes.All);
this.Invoke(new ThreadStart(() => {
listBox1.Items.Add(string.Format("{0} {1} {2}", DateTime.Now, r.Name, r.ChangeType));
}));
}
}) { IsBackground = true }.Start();
its very similar to nio in java

I had a similar problem for checking updates in logging configurations.
I read about the multiple events problem of FileSystemWatcher. So I decided to implement another solution.
I check for every access to my configuration file if it has changed by comparing modified date. Later I added a latence (2 seconds) to avoid too many accesses to filesystem. Maybe you can also use that way.

If you look at the documentation for FileSystemWatcher it says that it will fire multiple times.
Moving a file is a complex operation that consists of multiple simple operations, therefore raising multiple events. Likewise, some applications (for example, antivirus software) might cause additional file system events that are detected by FileSystemWatcher.
I would recommend doing some kind of queuing mechanism so that you don't have multiple actions on the same file.

FileSystemWatcher - event not firing the second time

I have an application that launches other applications, and then waits for them to create a specific data file (it watches one application at a time). Each time an application is launch it watches a specific directory for a specific file to be created. I am using the FileSystemWatcher to do this (set it to the directory, then filter for the correct file name). This works great the first time (always), but the second application launched never fires the event. The only way it seems to fire the event is if I place a break-point in the event handler, or if I have a Thread.Sleep command in the event handler. This seems very strange to me...is there some race condition that I'm not aware of? Here is the code. Notice I have a Thread.Sleep(500). With this line the code works every time. Without it will fail. I'm really not comfortable relying on a Sleep command. I'm not sure what condition will cause that not to work as well.
public static void watchFiles(string path)
{
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = path;
watcher.Created += new FileSystemEventHandler(watcher_Handler);
watcher.EnableRaisingEvents = true;
}
public static void watcher_Handler(object sender, FileSystemEventArgs e)
{
//Hack - the sleep allows the second and third application to be caught by this event
Thread.Sleep(500);
switch (e.ChangeType.ToString())
{
case "Changed":
break;
case "Deleted":
break;
case "Created":
if (e.Name == "log.dat")
{
parseDataFile();
moveHTMLtoLMS();
}
break;
default:
break;
}
}
Anyone know why I need to have that Sleep (or break-point) to get the code to work a second time?

According to the documentation of the System.IO.FileSystemWatcher class:
The Windows operating system notifies your component of file changes in a buffer created by the FileSystemWatcher. If there are many changes in a short time, the buffer can overflow. This causes the component to lose track of changes in the directory, and it will only provide blanket notification. Increasing the size of the buffer with the InternalBufferSize property is expensive, as it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small yet large enough to not miss any file change events. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties so you can filter out unwanted change notifications.
It might be that the event isn't being consumed fast enough and the internal buffer isn't large enough to handle all the notifications. By default, the watcher handles FileName, DirectoryName, LastWrite notifications yet you only consume creation events (both file and directory). Are your applications running in quick succession? I'd try putting a delay between the invocations of your applications (instead of the event handler), use more specific filters (just the FileName notification or watch only for log files using the Filter property), increase the internal buffer size or any combination of the above. I think that should fix your problem.

public static void watchFiles(string path)
{
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = path;
watcher.Created += new FileSystemEventHandler(watcher_Handler);
watcher.EnableRaisingEvents = true;
}
The watcher variable is eligible for garbage collection at the end of this method. Instead of being a local variable, make it a class-level member as such:
private static FileSystemWatcher watcher;
public static void watchFiles(string path)
{
if (watcher != null)
{
watcher.EnableRaisingEvents = false;
watcher.Created -= new FileSystemEventHandler(watcher_Handler);
}
watcher = new FileSystemWatcher();
watcher.Path = path;
watcher.Created += new FileSystemEventHandler(watcher_Handler);
watcher.EnableRaisingEvents = true;
}

You are listenting to only one "Created" event. You need to listen to all other ones too - OnChanged, OnDeleted - http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx
EDIT: Most programs will not "Create" file when one already exists. You can use FileMon (now Process Monitor - http://technet.microsoft.com/en-us/sysinternals/bb896645 ) to see what operations each program perform with your file.

I'm facing the exact same problem here (running Windows XP). Your hack solves the problem. I would like to add some notes that might be relevant.
In my case the filename is always the same: C:\blah.txt is created, deleted, created and so forth. Also, I'm using a trick to hide my application:
Integrator.StartMonitor(); // Start the file monitor!
Form f = new Form();
f.ShowInTaskbar = false;
f.ShowIcon = false;
f.StartPosition = FormStartPosition.Manual;
f.Location = new Point(-32000, -32000);
f.Show();
f.Hide();
Application.Run();
My file watcher works in debug mode or when I add the sleep-hack of yours. It certainly looks like a bug in the FileSystemWatcher.