I have a situation here. I want to read files based on their creation of last modified time. Initially i used FileSystemWatcher so that i was notified when a new file was coming, but later i realized that if the system on which my software is running goes down or restarts the location where files were being dropped will still continue.
To make it easier for understanding i will give an example:
System A - File Server (Files are created every 2 min in a directory on this server)
System B - My Software will run and Monitor files from the Path of System A
If System B goes restarts and is up again after 10 min the FileSystemWatcher will skip all these files which were generated in those 10 min.
How Can I ensure that those files generated in those 10 min of time are also captured?
Let me know if my question is still not understandable.
If you don't want to split it up in two systems, you have to persist a little bit of information.
You could store the current timestamp in a file, every time a new event was fired on the filesystem watcher. Every time your service starts, you can read all files from the filesystem that are newer than the last timestamp. This way you shouldn't miss a file.
I would split this application into two parts and running a filesystemwatcher-wcf service that buffers the files created in this 10 minutes and will send it to system b when it is restarted. I can't see a other way, sorry.
I think the FileSystemWatcher must write info about file system into DB (or other type of storage). When System B starts, watcher compares current file system with this info and will raise events about changes.
Copy the entire files from the Source Machine and paste into the destination based on condition..
string dirPath = #"C:\A";
string DestinPath = #"C:\B";
if (Directory.Exists(dirPath) && Directory.Exists(DestinPath))
{
DirectoryInfo di = new DirectoryInfo(dirPath);
foreach (var file in di.GetFiles())
{
string destinFile = DestinPath + "\\" + file.Name;
if (File.Exists(destinFile))
{
continue;
}
else
file.CopyTo(destinFile);
}
}
Not sure if I understood your question correctly, but based on what I get and assuming both systems are in sync in terms of time, if for example you want to get files that have been modified within ten minutes ago:
DateTime tenMinutesAgo = DateTime.Now.AddMinutes(-10);
string[] systemAFiles = System.IO.Directory.GetFiles(systemAPath);
foreach (string files in systemAFiles)
{
DateTime lastWriteTime = System.IO.File.GetLastWriteTime(files);
if (lastWriteTime > tenMinutesAgo) //produced after ten minutes ago
{
//read file
}
}
I understood that these files are "generated" so they have been created or modified. If they have simply been moved from one folder to another this will not work. In that case the best way is to write a snapshot of the files in that list (and writing it to some sort of a save file) and compare it when it is running again.
Related
here is a scenario:
I copy a file to a folder
and I expect it to be taken by other program.
is there a way to monitor the time that a file exists in a folder
I want to raise an event in case the file is in the folder over X time
Please advise what can I use to accomplish that
I had in mind using File system watcher and save the time of it created
and compare if the file exists after X time
If you only care about timeout, then simply check for it after copying. E.g. using a Timer:
var timer = Timer(5000);
timer.Elapsed += (s, a) =>
{
// checking if file is still there
if(File.Exists(...)
{
// do something
}
}
timer.Start();
Depending on how precise you need to be, you could just have a process that checks the contents of the folder every so often and lets you know about any files that have been in the folder for too long.
To list files in a directory you could use Directory.GetFiles as below.
Then iterate thru all the files checking for file age.
string[] files = System.IO.Directory.GetFiles("c:\temp", "*.txt", System.IO.SearchOption.TopDirectoryOnly);
foreach(string file in files)
{
if (System.DateTime.UtcNow.Subtract(System.IO.File.GetCreationTimeUtc(file)).TotalMinutes > 5)
System.Diagnostics.Debug.WriteLine("TODO: Alert, file older than 5 minutes...");
}
I have a function that checks every file in a directory and writes a list to the console. The problem is, I don't want it to include files that are currently being copied to the directory, I only want it to show the files that are complete. How do I do that? Here is my code:
foreach (string file in Directory.EnumerateFiles("C:\folder"))
{
Console.WriteLine(file);
}
There's really no way to tell "being copied" vs "locked for writing by something". Relevant: How to check for file lock? and Can I simply 'read' a file that is in use?
If you want to simply display a list of files that are not open for writing, you can do that by attempting to open them:
foreach (string file in Directory.EnumerateFiles("C:\folder"))
{
try {
using (var file = file.Open(file, FileMode.Open, FileAccess.Read, FileShare.ReadWrite) {
Console.WriteLine(file);
}
} catch {
// file is in use
continue;
}
}
However -- lots of caveats.
Immediately after displaying the filename (end of the using block) the file could be opened by something else
The process writing the file may have used FileShare.Read which means the call will succeed, despite it being written to.
I'm not sure what exactly you're up to here, but it sounds like two processes sharing a queue directory: one writing, one reading/processing. The biggest challenge is that writing a file takes time, and so your "reading" process ends up picking it up and trying to read it before the whole file is there, which will fail in some way depending on the sharing mode, how your apps are written, etc.
A common pattern to deal with this situation is to use an atomic file operation like Move:
Do the (slow) write/copy operation to a temporary directory that's on the same file system (very important) as the queue directory
Once complete, do a Move from the temporary directory to the queue directory.
Since move is atomic, the file will either not be there, or it will be 100% there -- there is no opportunity for the "reading" process to ever see the file while it's partially there.
Note that if you do the move across file systems, it will act the same as a copy.
There's no "current files being copied" list stored anywhere in Windows/.NET/whatever. Probably the most you could do is attempt to open each file for append and see if you get an exception. Depending on the size and location of your directory, and on the security setup, that may not be an option.
There isn't a clean way to do this, but this... works...
foreach (var file in new DirectoryInfo(#"C:\Folder").GetFiles())
{
try
{
file.OpenRead();
}
catch
{
continue;
}
Console.WriteLine(file.Name);
}
I am currently using this code:
if (!Directory.Exists(command2)) Directory.CreateDirectory(command2);
if (Directory.Exists(vmdaydir)) Directory.Delete(vmdaydir,true);
if (!Directory.Exists(vmdaydir)) Directory.CreateDirectory(vmdaydir);
var dir = Path.GetDirectoryName(args[0]);
sb.AppendLine("Backing Up VM: " + DateTime.Now.ToString(CultureInfo.InvariantCulture));
Microsoft.VisualBasic.FileIO.FileSystem.CopyDirectory(dir, vmdaydir);
sb.AppendLine("VM Backed Up: " + DateTime.Now.ToString(CultureInfo.InvariantCulture));
As you can see, I am deleting the directory, then I am copying the folder back. This is taking way to long since the directory is ~80gb in size. I realized that I do not need to copy all the files, only the ones that have changed.
How would I copy the files from one folder to another but only copying the files that are newer? Anyone have any suggestions?
==== edit ====
I assume I can just do a file compare of each file and then copy it to the new directory, iterating through each folder/file? Is there a simpler way to do this?
Use the FileInfo class, and use the LastWriteTime property to get the last modified time of the file. Compare it to the time you're checking against and take only files that are later.
Loop through the files in the directory, checking the last modified time (FileInfo.LastWriteTime) - any files that are newer are copied over.
See FileInfo Class for more information.
You need to be careful when trying to do this that you can get a lock on the file otherwise another application may not be finished with it and you may try to copy it before you are ready.
So follow these steps...
1) attempt to lock file
2) if(got lock) copy file
3) else wait a short time
4) goto 1
:)
I have about 5-6 Server Manager programs that write their own configuration file out to a particualr folder, such as C:\ACME. The config files all end with a *ServerConfig.cfg" where * = Program name that created it.
I have a Windows service that has a FileSystemWatcher setup that I want to FTP the configuration files each time the program updates. I've gotten everything to work, but I'm noticing that the different Server Manager programs are behaving differently.
When saving a configuration file, the FileSystemWatcher is picking up two "change" events. This is causing my program to FTP the configuration file twice where I only need it once.
In other instances I'm seeing where it may create 4, 5, or 6 "change" events when saving a configuration file.
What is the best way to handle processing/FTPing these files when they are really done saving only one time.
I really dont want o set something up to poll the directory for filechanges every so often... and like the idea that each time a configuration is saved, I get a duplicate copy along with a date/timestamp appended to the filename copied elsewhere.
I have seen lots of suggestions Googling around and even here on Stackoverflow, but nothing that seems to be all-in-one for me.
I suppose I could put the filename in a queue when a "change" event occurred if it didn't already exist in the queue. Not sure if this is the best approx.
Here is my sample code:
Startup-code:
private DateTime _lastTimeFileWatcherEventRaised = DateTime.Now;
_watcherCFGFiles = new FileSystemWatcher();
_watcherCFGFiles.Path = #"C:\ACME";
_watcherCFGFiles.IncludeSubdirectories = true;
_watcherCFGFiles.Filter = "*ServerConfig.cfg";
_watcherCFGFiles.NotifyFilter = NotifyFilters.Size;
//_watcherCFGFiles.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.FileName;
_watcherCFGFiles.Changed += new FileSystemEventHandler(LogFileSystemChanges);
_watcherCFGFiles.Created += new FileSystemEventHandler(LogFileSystemChanges);
_watcherCFGFiles.Deleted += new FileSystemEventHandler(LogFileSystemChanges);
_watcherCFGFiles.Renamed += new RenamedEventHandler(LogFileSystemRenaming);
_watcherCFGFiles.Error += new ErrorEventHandler(LogBufferError);
_watcherCFGFiles.EnableRaisingEvents = true;
Here is that actual handler for the "change" event. I'm skipping the first "change" event if the second is within 700ms. But this doesn't account for the files that make 3-4 change events...
void LogFileSystemChanges(object sender, FileSystemEventArgs e)
{
string log = string.Format("{0} | {1}", e.FullPath, e.ChangeType);
if( e.ChangeType == WatcherChangeTypes.Changed )
{
if(DateTime.Now.Subtract(_lastTimeFileWatcherEventRaised).TotalMilliseconds < 700)
{
return;
}
_lastTimeFileWatcherEventRaised = DateTime.Now;
LogEvent(log);
// Process file
FTPConfigFileUpdate(e.FullPath);
}
}
I had the exact same issue. I used a HashMap that mapped filenames to times of writes, I then used this as a lookup table for files to check and see if the changed event had been applied very quickly. I defined some epsilon (for me it was about 2 seconds to make sure events were flushed). If the time found in the map was older than that I would put it on a queue to be processed. Essentially all I had to do was keep the HashMap up to date with events and changes and this worked out (although you may want to change your epsilon value depending on your application).
Its normal this behavior because the antivirus system or other programs make more writes when a file change the content. I usually create a (global) HashTable and check if the filename exists, if don't, put the filename in it and start and an asynchronous operation to remove the filename after 3-5 seconds.
This is expected behavior - so you need to figure out how to handle it in your particular case.
The file system does not have a concept of "program done working with this file". I.e. one can write editor that updates (open/write/close) file on every keystroke. File system will report a lot of updates, but from the user point of view there is only one update when the editor is closed.
IT has been tasked with reducing the file-server usage rate so I'd like to do my part my compressing old files(ie Excel/Access/Txt).
We have some folders that contain thousands of files so I don't want to just zip the whole directory into one large file - it would be preferrable to have a number of smaller files so that it would be easier for a user to find the data 'bucket' they are looking for.
Is there a way using C# to read through a directory and zip the files into year-month groups (all files from year-month placed together in one zip)?
Or would it be better to use a script like AutoIT?
Or are there programs already existing to do this so I don't have to code anything?
Im not sure if your question is about zipping, selecting files from particular year/month or both.
About zipping Peter already mentioned 7-zip and SharpZipLib. I have personally only experience with the latter but its all positive, easy to work with.
About grouping your files it could be done by iterating all the files in the folder and group them by either there created date or last modified date.
pseudo:
var files = new Dictionary<DateTime, IList<string>>();
foreach (var file in Directory.GetFiles(...)) {
var fi = new FileInfo(file);
var date = fi.CreatedDate();
var groupDate = new DateTime(date.Year, date.Month);
if (!files.ContainsKey(groupDate)) files.Add(groupDate, new Collection<string>());
files[groupDate].Add(file);
}
now your should have a dictionary containing distinct year/month keys and foreach key a list of files belonging to that group. So for zipping
pseudo:
foreach (var entry in files) {
var date = entry.Key;
var list = entry.Value;
// create zip-file named date.ToString();
foreach (var file in list) {
// add file to zip
}
}
Surely you can do this with a bit of C# and libraries like 7-zip or SharpZipLib.
You could use System.IO.Directory.GetFiles() to loop through the files on each directory, parsing out by file name and adding them to a 7-zip or SharpZipLib object. If it's thousands of files it might be best to throw it in a service or some kind of scheduled task to run overnight so as not to tax the fileshare.
Good luck to you !
EDIT: As an addendum you could use a System.IO.FileInfo object for each file if you need to parse by created date or other file attirbutes.