Develop Reference

How to watch around 100 folders in parallel? FileSystemWatcher or other more efficient options?

I have a base folder under a drive Data and under this I have around 100 folders.
In each folder Folder1.....100, one of the 3rd part application pushing zip file (zip contains 1 or more files).
I have to write a window service which will watch all 100 folders for file arrival.
Once file is available I need to extract the zip file and placing all the extracted files into a second folder and this I need to do for each folder (Folder 1 .. 100) as soon as files available.
Below code suggest me that through C# FileSystemWatcher, I can watch one folder at a time and act on that.
Question is, how to do watch for 100 folders in parallel?
class ExampleAttributesChangedFiringTwice
{
public ExampleAttributesChangedFiringTwice(string demoFolderPath)
{
var watcher = new FileSystemWatcher()
{
Path = demoFolderPath,
NotifyFilter = NotifyFilters.LastWrite,
Filter = "*.txt"
};
watcher.Changed += OnChanged;
watcher.EnableRaisingEvents = true;
}
private static void OnChanged(object source, FileSystemEventArgs e)
{
// extract zip file, do the validation, copy file into other destination
}
}
The target folder, is it the same folder regardless of the source folder of the zip? That is, it doesn't matter if it's from Folder1 or Folder2, both will be extracted to FolderX?
Target folder is common for all "C:\ExtractedData".
So every folder under Data will be watched? No "blacklisted" folder? What about if a zip appears in Data itself instead of its subfolder? What if a new subfolder is created, should it be watched too?
"zip" always comes inside "subfolders", it will never create inside Data folder.
Yes, there is a chance in future, more subfolders will come and need watch.
And does the extracted files goes into a separate subfolder inside the target folder based on their zip filename, or do they just get extracted on the target folder, eg, if it's A.zip, does the content goes to Target\A or just Target.
For example, if A.zip contains 2 files, "1.txt" and "2.txt", then both files goes to "C:\ExtractedData". This will be common for each zip files arrives at different subfolders.

The "100 folders in parallel" part turn out to be a red herring. Since all the new zip files are treated the same regardless of where they show up, just adding IncludeSubdirectories=true is enough. Note the following codes are prone to exceptions, read the comments
class WatchAndExtract
{
string inputPath, targetPath;
public WatchAndExtract(string inputPath, string targetPath)
{
this.inputPath = inputPath;
this.targetPath = targetPath;
var watcher = new FileSystemWatcher()
{
Path = inputPath,
NotifyFilter = NotifyFilters.FileName,
//add other filters if your 3rd party app don't immediately copy a new file, but instead create and write
Filter = "*.zip",
IncludeSubdirectories = true
};
watcher.Created += OnCreated; //use Changed if the file isn't immediately copied
watcher.EnableRaisingEvents = true;
}
private void OnCreated(object source, FileSystemEventArgs e)
{
//add filters if you're using Changed instead
//https://stackoverflow.com/questions/1764809/filesystemwatcher-changed-event-is-raised-twice
ZipFile.OpenRead(e.FullPath).ExtractToDirectory(targetPath);
//this will throw exception if the zip file is being written.
//Catch and add delay before retry, or watch for LastWrite event that already passed for a few seconds
}
}
If it skipped some files, you either have too many files created at once and/or too big zip to process. Either increase the buffer size or start them in new thread. On HDD with busy IO or extremely large zip files, the events might exceed the storage capability and skipped files after a prolonged busy period, you'll have to consider writing to a different physical (not just a different partition in the same device) drive instead. Always verify with your predicted usage pattern.

is there any way for me to know if a file has been modified?

I am making a program that detects if a file has been modified. but the file is not from me. I wanna know if it is possible to know if the files I'm getting were already modified or not.
Is there a way for me to know that? I've tried the creation date and date modified but sometimes when I modify a file their values will be the same.
P.S. I don't have the original file. I wanna know if it is possible to know if the I'm getting were unchanged before i get it.

The following example creates a FileSystemWatcher to watch the directory specified at run time. The component is set to watch for changes in LastWrite and LastAccess time, the creation, deletion, or renaming of text files in the directory. If a file is changed, created, or deleted, the path to the file prints to the console. When a file is renamed, the old and new paths print to the console.
Use the System.Diagnostics and System.IO namespaces for this example.
using System;
using System.IO;
using System.Security.Permissions;
public class Watcher
{
public static void Main()
{
Run();
}
[PermissionSet(SecurityAction.Demand, Name="FullTrust")]
public static void Run()
{
string[] args = System.Environment.GetCommandLineArgs();
// If a directory is not specified, exit program.
if(args.Length != 2)
{
// Display the proper way to call the program.
Console.WriteLine("Usage: Watcher.exe (directory)");
return;
}
// Create a new FileSystemWatcher and set its properties.
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = args[1];
/* Watch for changes in LastAccess and LastWrite times, and
the renaming of files or directories. */
watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
| NotifyFilters.FileName | NotifyFilters.DirectoryName;
// Only watch text files.
watcher.Filter = "*.txt";
// Add event handlers.
watcher.Changed += new FileSystemEventHandler(OnChanged);
watcher.Created += new FileSystemEventHandler(OnChanged);
watcher.Deleted += new FileSystemEventHandler(OnChanged);
watcher.Renamed += new RenamedEventHandler(OnRenamed);
// Begin watching.
watcher.EnableRaisingEvents = true;
// Wait for the user to quit the program.
Console.WriteLine("Press \'q\' to quit the sample.");
while(Console.Read()!='q');
}
// Define the event handlers.
private static void OnChanged(object source, FileSystemEventArgs e)
{
// Specify what is done when a file is changed, created, or deleted.
Console.WriteLine("File: " + e.FullPath + " " + e.ChangeType);
}
private static void OnRenamed(object source, RenamedEventArgs e)
{
// Specify what is done when a file is renamed.
Console.WriteLine("File: {0} renamed to {1}", e.OldFullPath, e.FullPath);
}
}
Reference: Microsoft MSDN

If you want to know if the content of a file changed, compute its hash:
using System.IO;
using System.Security.Cryptography;
using (HashAlgorithmalgorithm = new .SHA512Managed())
{
using (Stream fileStream = new FileStream(#"path", FileMode.Open))
{
byte[] hash = algorithm.ComputeHash(fileStream);
}
}
Persist the hash as you like on the first run, then recompute it and match it upon the saved value. If they are different, the file changed.

I think this question has a few issues, as there are multiple ways to answer this.
1) If the question is, "I have received a file (that previously was not on the computer/network/location), and want to know if it has been modified...", then the answer is - no, because you're looking at a just a chunk of bits, then all of those bits are subject to being modified and adjusted in any way the sender wants - timestamps, file-status indicators (For instance, the 'archive' tag on Windows systems, or structured metadata on newer file systems) or anything else about that file can be set to what they want.
2) If the question is, "I have a file (on the local network or file-system that I can 'poke' every so often, or locally on the drive that I can 'poke' every so often) and I want to know if it has been modified between the times my application has run", then probably the easiest way to do this is to store a computed hash of that file (As answered by #Albireo).
3) If the question is, "I have a file (on the local network or file-system) and I want to know if it was modified prior to any running of my application", then you're looking at something that may or may not be possible, and will not be possible in a completely reliable way - having some understanding of the underlying structure and what you're expecting in that file will be necessary, and you'd be better off updating your question with more details of what the file consists of and why you're looking for changes programatically so that we can determine the best way of doing this.

A new file in the folder event

Can someone please help me to understand how to build a software that run 24/7 that listen to a specific folder (ex. C:\Actions) and each time I place a new file in that folder, the software needs to read and processing it.
If there isn't files in the folder the software shouldn't do nothing only to wait to the next file to come.
Example of the file (action1.txt) content (1+1)
The software is processing (1+1), saving the answer(2) to anther folder and delete the file(action1.txt) from the "C:\Actions\" folder.
I know how to read the file and process it..
I'm having difficulty to understand how to trigger the software only when there is new file in the folder and how to run the software for 24/7 without using so much memory or causing to memory leakage…
Till now I've used it in the primitive way of looping endless and each 60 sec (Sleep) I'm checking the folder for new files. That's so useless and not so effective.
I'll be happy if someone can help me to understand how to make it more effective..
thank you very much

Use the FileSystemWatcher
An example from that page:
// Create a new FileSystemWatcher and set its properties.
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = "C:\\Actions";
/* Watch for changes in LastAccess and LastWrite times, and
the renaming of files or directories. */
watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
| NotifyFilters.FileName | NotifyFilters.DirectoryName;
// Only watch text files.
watcher.Filter = "*.txt";
// Add event handlers.
watcher.Changed += new FileSystemEventHandler(OnChanged);
watcher.Created += new FileSystemEventHandler(OnChanged);
watcher.Deleted += new FileSystemEventHandler(OnChanged);
// Begin watching.
watcher.EnableRaisingEvents = true;
And the changed event:
// Define the event handlers.
private static void OnChanged(object source, FileSystemEventArgs e)
{
// Specify what is done when a file is changed, created, or deleted.
Console.WriteLine("File: " + e.FullPath + " " + e.ChangeType);
}
There are some things you need to watch out for when using this class though. It does not work well on network drives/UNC paths. Also if you paste a lot of files to the directory, it will overfill the buffer and you might not get events for every file that is added to the folder.

Understand this class http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher(v=vs.110).aspx and you are ready to go. It has the needed events.

Have a look at the FileSystemWatcher Class:
Listens to the file system change notifications and raises events when a directory, or file in a directory, changes.
Changed Occurs when a file or directory in the specified Path is changed.
Created Occurs when a file or directory in the specified Path is created.

FileSystemWatcher and Monitoring Config File Changes

I have about 5-6 Server Manager programs that write their own configuration file out to a particualr folder, such as C:\ACME. The config files all end with a *ServerConfig.cfg" where * = Program name that created it.
I have a Windows service that has a FileSystemWatcher setup that I want to FTP the configuration files each time the program updates. I've gotten everything to work, but I'm noticing that the different Server Manager programs are behaving differently.
When saving a configuration file, the FileSystemWatcher is picking up two "change" events. This is causing my program to FTP the configuration file twice where I only need it once.
In other instances I'm seeing where it may create 4, 5, or 6 "change" events when saving a configuration file.
What is the best way to handle processing/FTPing these files when they are really done saving only one time.
I really dont want o set something up to poll the directory for filechanges every so often... and like the idea that each time a configuration is saved, I get a duplicate copy along with a date/timestamp appended to the filename copied elsewhere.
I have seen lots of suggestions Googling around and even here on Stackoverflow, but nothing that seems to be all-in-one for me.
I suppose I could put the filename in a queue when a "change" event occurred if it didn't already exist in the queue. Not sure if this is the best approx.
Here is my sample code:
Startup-code:
private DateTime _lastTimeFileWatcherEventRaised = DateTime.Now;
_watcherCFGFiles = new FileSystemWatcher();
_watcherCFGFiles.Path = #"C:\ACME";
_watcherCFGFiles.IncludeSubdirectories = true;
_watcherCFGFiles.Filter = "*ServerConfig.cfg";
_watcherCFGFiles.NotifyFilter = NotifyFilters.Size;
//_watcherCFGFiles.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.FileName;
_watcherCFGFiles.Changed += new FileSystemEventHandler(LogFileSystemChanges);
_watcherCFGFiles.Created += new FileSystemEventHandler(LogFileSystemChanges);
_watcherCFGFiles.Deleted += new FileSystemEventHandler(LogFileSystemChanges);
_watcherCFGFiles.Renamed += new RenamedEventHandler(LogFileSystemRenaming);
_watcherCFGFiles.Error += new ErrorEventHandler(LogBufferError);
_watcherCFGFiles.EnableRaisingEvents = true;
Here is that actual handler for the "change" event. I'm skipping the first "change" event if the second is within 700ms. But this doesn't account for the files that make 3-4 change events...
void LogFileSystemChanges(object sender, FileSystemEventArgs e)
{
string log = string.Format("{0} | {1}", e.FullPath, e.ChangeType);
if( e.ChangeType == WatcherChangeTypes.Changed )
{
if(DateTime.Now.Subtract(_lastTimeFileWatcherEventRaised).TotalMilliseconds < 700)
{
return;
}
_lastTimeFileWatcherEventRaised = DateTime.Now;
LogEvent(log);
// Process file
FTPConfigFileUpdate(e.FullPath);
}
}

I had the exact same issue. I used a HashMap that mapped filenames to times of writes, I then used this as a lookup table for files to check and see if the changed event had been applied very quickly. I defined some epsilon (for me it was about 2 seconds to make sure events were flushed). If the time found in the map was older than that I would put it on a queue to be processed. Essentially all I had to do was keep the HashMap up to date with events and changes and this worked out (although you may want to change your epsilon value depending on your application).

Its normal this behavior because the antivirus system or other programs make more writes when a file change the content. I usually create a (global) HashTable and check if the filename exists, if don't, put the filename in it and start and an asynchronous operation to remove the filename after 3-5 seconds.

This is expected behavior - so you need to figure out how to handle it in your particular case.
The file system does not have a concept of "program done working with this file". I.e. one can write editor that updates (open/write/close) file on every keystroke. File system will report a lot of updates, but from the user point of view there is only one update when the editor is closed.

Windows: How to determine if a file has been modified since a given date

I have a utility which goes through a processes a set of files in a directory - the process is relatively slow (and there are a lot of files) and so I've tried to optimise the process by only processes files that have a "last modified" later than the last processing date.
Usually this works well however I've found that as copying a file doesn't change the last modified date, and so there are various scenarios involving copying files in which certain files that have changed are skipped by the process, for example:
The user processes the directory at 9:00.
A file is then copied from this directory and modified so that it has a last modified date of 9:30
The directory is then processed again at 10:00
The modified file is then copied back into the directory at 10:30
Finally the directory is processed again at 11:00
As the modified date of the given file is 9:30, and the directory was last processed at 10:00 the file is skipped when it shouldn't be.
Unfortunately the above tends to happen far too often in certain situations (such as in a collaborative environment with source control etc...). Clearly my logic is flawed - what I really need is a "last modified or copied" date. does such a thing exist?
Failing that, is there another way to quickly determine with reasonable reliability if a given file has changed?

You might want to look at using the FileSystemWatcher class. This class lets you monitor a directory for changes and will fire an event when something is modified. Your code can then handle the event and process the file.
From MSDN:
// Create a new FileSystemWatcher and set its properties.
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = args[1];
/* Watch for changes in LastAccess and LastWrite times, and
the renaming of files or directories. */
watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
| NotifyFilters.FileName | NotifyFilters.DirectoryName;
// Only watch text files.
watcher.Filter = "*.txt";
// Add event handlers.
watcher.Changed += new FileSystemEventHandler(OnChanged);
watcher.Created += new FileSystemEventHandler(OnChanged);
watcher.Deleted += new FileSystemEventHandler(OnChanged);
watcher.Renamed += new RenamedEventHandler(OnRenamed);

Have you thought of running MD5 checksums on the files and storing them later for comparison? If your always processing a certain directory, this might be feasible.

You can use the FileInfo class to get the required change information (which you might be already using). You need to check two properties of a file, which are LastWriteTime and CreationTime. If either of them is higher than your last processing date, you need to copy the file. It is a common misconception that CreationTime is always less than LastWriteTime. It's not. If a file is copied to another file, the new file retains the LastWriteTime of the source but the CreationTime will be the time of the copy.

Have you considered adding a process to watch your directory instead? Using a FileSystemWatcher? Then you move from using a batch process and a real time system for monitoring your files.

As you've observed, copying a file to an existing destination file keeps the existing file's CreationTime, and sets LastWriteTime to the source file's LastWriteTime, rather than current system time when doing the copy. Two possible solutions:
Do a delete-and-copy, ensuring a destination CreationTime will be system's current time.
Check for file's Archived attribute as well, and clear it while processing. When copying source->dest, dest +A attribute will be set.