I have an issue with a file system watcher application I am working on. It seems to run fine with a single file but not with multiple files or a folder with files inside.
in the code below, I call IsFileReady to determine if the file has completed writing before trying to copy it to the other folder. However, when a new folder is created with files inside, for some reason it's hanging and not continuing.
I think it's because multiple files are being written at the same time and my code is referencing just a single file, but I am unsure of how to correct this.
Any assistance is appreciated and thank you.
static void Init()
{
string directory = watch_path;
Program._watcher = new FileSystemWatcher(directory);
Program._watcher.Created +=
new FileSystemEventHandler(Program._watcher_Changed);
Program._watcher.EnableRaisingEvents = true;
Program._watcher.IncludeSubdirectories = true;
}
static void _watcher_Changed(object sender, FileSystemEventArgs e)
{
Console.WriteLine("CHANGED, NAME: " + e.Name);
Console.WriteLine("CHANGED, FULLPATH: " + e.FullPath);
try
{
Console.WriteLine("Checking If File Exists at copy location");
if (!File.Exists(copy_path + "\\" + e.Name))
{
Console.WriteLine("Waiting for File to complete write");
WaitForFile(e.FullPath);
Console.WriteLine("Copying file to remote folder");
File.Copy(e.FullPath, copy_path + "\\" + e.Name);
Console.WriteLine("Copy Completed writing to log");
error_handling("File Copy Completed : ", copy_path + "\\" + e.Name, e.FullPath);
}
else
{
Console.WriteLine("Copy Failed Writing to Log");
error_handling("File Copy Skipped, File exists at destination : ", copy_path + "\\" + e.Name, e.FullPath);
}
}
catch (Exception error)
{
Console.WriteLine("Copy process totally failed" + error.Message);
error_handling("File Copy Failed ", "Exception: " + error.Message, "00");
}
}
public static void WaitForFile(string filename)
{
while (!IsFileReady(filename)) { }
}
public static bool IsFileReady(string filename)
{
try
{
using (FileStream inputStream = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.None))
return inputStream.Length > 0;
}
catch (Exception)
{
return false;
}
*Edit
When it hangs, it seems that the WaitForFile never actually finishes up. When copying a single file, it finishes no problem, but I think when there are multiple files it get stuck somehow and it will not go past the
WaitForFile(e.FullPath);
I am intentionally hooking the Created event. the use of this application is to monitor a folder that needs to be identical between multiple web servers. the sync process needs to be as fast as possible as the web servers are load balanced.
the Call to error_handling is just a write to log function as seen below:
public static void error_handling(string message, string fileident, string source)
{
using (System.IO.StreamWriter myFile = new System.IO.StreamWriter(log_folder + "\\" + log_file, true))
{
string finalMessage = string.Format("{0}: {1} SOURCE: {3} - DEST: {2}", DateTime.Now, message, fileident, source, Environment.NewLine);
myFile.WriteLine(finalMessage);
myFile.Close();
}
}
Related
I have around 1.5 million files to process
I process these files by taking the top 300 files in my folder, then process and insert them into the database, and then move these files to the archive folder.
the bottleneck in this process is moving files to the archive folder
I am using this code to move the files
foreach (string file in Files)
{
string OutputFileName = Path.GetFileName(file);
string OutputBackupFile = OutputBackupFolder + OutputFileName;
MoveWithReplace(file, OutputBackupFile);
}
private void MoveWithReplace(string sourceFileName, string destFileName)
{
Log("In MoveWithReplace : " + sourceFileName + " to " + destFileName);
try
{
//first, delete target file if exists, as File.Move() does not support overwrite
if (File.Exists(destFileName))
{
File.Delete(destFileName);
}
File.Move(sourceFileName, destFileName);
}
catch (Exception ex)
{
Log("ERROR (MoveWithReplace): " + ex.ToString());
}
}
I wonder if there is a faster way to move these files in one batch instead of one by one
or a faster way to do the move?
P.S. file size is 100KB on average.
You can try move the files in parallel way with following code
Parallel.ForEach(Files, file => {
string OutputFileName = Path.GetFileName(file);
string OutputBackupFile = OutputBackupFolder + OutputFileName;
MoveWithReplace(file, OutputBackupFile);
});
private static void MoveWithReplace(string sourceFileName, string destFileName)
{
Log("In MoveWithReplace : " + sourceFileName + " to " + destFileName);
try
{
//first, delete target file if exists, as File.Move() does not support overwrite
if (File.Exists(destFileName))
{
File.Delete(destFileName);
}
File.Move(sourceFileName, destFileName);
}
catch (Exception ex)
{
Log("ERROR (MoveWithReplace): " + ex.ToString());
}
}
I had not tried compile it
When I started catching and displaying my logging exceptions on my console application, I realized that quite a number of my log messages are absent due to the following error
The process cannot access the file '[MY_LOG_FILE_PATH]' because it is
being used by another process
This is how I log my messages on a log file (I create a new file which every run of the application). The variables callingClass and callingMethod are respectively the class and the method which are calling the Log function.
try
{
var stackTrace = new StackTrace();
string callingMethod = stackTrace.GetFrame(1).GetMethod().Name;
string callingClass = stackTrace.GetFrame(1).GetMethod().ReflectedType.Name;
string logText = string.Format(format, DateTime.Now, "INFO", callingClass, callingMethod, message);
if (!Directory.Exists("log"))
Directory.CreateDirectory("log");
if (!File.Exists(logFilePath))
File.Create(logFilePath);
using (FileStream f = new FileStream(logFilePath, FileMode.Append, FileAccess.Write, FileShare.ReadWrite))
using (StreamWriter sw = new StreamWriter(f))
sw.WriteLine(logText);
}
catch(IOException)
{
Console.WriteLine("Processes locking the file:");
var lockingProcesses = FileHelper.WhoIsLocking(logFilePath);
foreach (Process p in lockingProcesses)
Console.WriteLine("Process: " + p.ProcessName + " Machine:" + p.MachineName);
}
catch (Exception ex)
{
Console.ForegroundColor = ConsoleColor.DarkRed;
Console.WriteLine("The following log could not be written to file: " + message);
Console.WriteLine("Error: " + ex.Message);
Console.ResetColor();
}
When I caught the IOException causing the above situation that, I found out that the process that possesses the log file is "dotnet" residing on machine ".", which I assume is the current machine.
How do I not lose my log messages?
IDE: Visual Studio Community 2019 Version 16.0.4
.NET Framework Version: 4.7.03190
OS: Windows 10 Pro 64x
File.Create creates the file and also opens it. So in the next line when you try to create a new FileStream the file is already opened and throws an error.
One option is to immediately call Close on the file stream when creating it:
File.Create(logFilePath).Close();
Another is to just use StreamWriter to open/create the file and skip using the FileStream altogether. Passing true means to append to the file if it exists.
try
{
var stackTrace = new StackTrace();
string callingMethod = stackTrace.GetFrame(1).GetMethod().Name;
string callingClass = stackTrace.GetFrame(1).GetMethod().ReflectedType.Name;
string logText = string.Format(format, DateTime.Now, "INFO", callingClass, callingMethod, message);
if (!Directory.Exists("log"))
Directory.CreateDirectory("log");
using (var sw = new StreamWriter(logFilePath, true))
{
sw.WriteLine(logText);
}
}
catch(IOException)
{
Console.WriteLine("Processes locking the file:");
var lockingProcesses = FileHelper.WhoIsLocking(logFilePath);
foreach (Process p in lockingProcesses)
Console.WriteLine("Process: " + p.ProcessName + " Machine:" + p.MachineName);
}
catch (Exception ex)
{
Console.ForegroundColor = ConsoleColor.DarkRed;
Console.WriteLine("The following log could not be written to file: " + message);
Console.WriteLine("Error: " + ex.Message);
Console.ResetColor();
}
Try pushing your messages into a shared BlockingCollection, and have a thread which reads from this and writes to your log file.
In the code below, I check if file uploaded by only checking on single folder but I noticed that the code stops if the file /new/name.ext is not exist.
Do we have to check for each to continue with code? Is there a way to continue with code without having to check if file exist for every file - or even easier way?
if (System.IO.File.Exists(HttpContext.Current.Server.MapPath("/products/cats/thumb/" + strGuid + strExt))) {
System.IO.File.Delete(HttpContext.Current.Server.MapPath("/products/cats/icons/" + strGuid + strExt));
System.IO.File.Delete(HttpContext.Current.Server.MapPath("/products/cats/thumb/" + strGuid + strExt));
System.IO.File.Delete(HttpContext.Current.Server.MapPath("/products/cats/new/" + strGuid + strExt));
System.IO.File.Delete(HttpContext.Current.Server.MapPath("/products/cats/large/" + strGuid + strExt));
System.IO.File.Delete(HttpContext.Current.Server.MapPath("/products/cats/full/" + strGuid + strExt));
}
Although I normally wouldn't recommend "eating" exceptions, this might be an exception to that exception rule.
I'd write a single method:
void DeleteFile(string filePath)
{
try
{
if(File.Exists(filePath)
{
File.Delete(filePath);
}
}
catch(DirectoryNotFoundException ex)
{
// depending on your environment you might
// be prompted for some comment to indicate
// that you meant to do this because
// it's usually bad.
}
}
Only catch DirectoryNotFoundException. If the directory doesn't exist then the file doesn't exist.
The reason for placing this in a separate method is because if you're concerned about this scenario, then you don't want one Delete to throw an exception that keeps the subsequent deletes from executing.
It doesn't entirely sit well with me to use exception handling for something when there's another way to check. It's better to verify that the file exists. This is only for extreme paranoia.
Here's a question - how likely are these files and directories to get deleted before you delete them? And if one file throws an exception that prevents the others from getting deleted, how serious are the consequences? It might be better to just allow the delete to fail then to get extra paranoid and overdo the checking an exception handling. But it's a short method so if you're worried about it it can't hurt. But this sort of thing can become habit-forming.
Based on UweKeim advice and the source of the ZetaLongPaths
library he mentioned in his comment (Thankful for you) and since I don't need all the Long File Name/Path issue so I have picked few pieces to handle my little application. I didn't fully tested it but it works fine for the code in my question. Hope someone can check and provide better tested code.
using System;
using System.ComponentModel;
using System.Diagnostics;
using System.IO;
using System.Web.UI;
using System.Web.UI.HtmlControls;
//------------------------------------------------------------------------------
// ZetaLongPaths/Source/Runtime/ZlpSafeFileOperations.cs
// https://github.com/UweKeim/ZetaLongPaths/blob/master/Source/Runtime/ZlpSafeFileOperations.cs#L30
//------------------------------------------------------------------------------
namespace MyNameSpace {
public class FileHandling {
//------------------------------------------------------------------------------
///<summary>
/// Simple File Operations Handling
///</summary>
///<remarks>
///
///</remarks>
//------------------------------------------------------------------------------
public static bool SafeFileExists(FileInfo filePath) {
return filePath != null && SafeFileExists(filePath.FullName);
}
public static bool SafeFileExists(string filePath) {
return !string.IsNullOrEmpty(filePath) && System.IO.File.Exists(filePath);
}
public static void SafeMoveFile(FileInfo sourcePath, FileInfo dstFilePath) {
SafeMoveFile(
sourcePath?.FullName.ToString(),
dstFilePath?.FullName);
}
public static void SafeDeleteFile(FileInfo filePath) {
if (filePath != null) {
SafeDeleteFile(filePath.FullName);
}
}
public static void SafeDeleteFile(
string filePath) {
Trace.TraceInformation(#"About to safe-delete file '{0}'.", filePath);
if (!string.IsNullOrEmpty(filePath) && SafeFileExists(filePath)) {
try {
var attributes = System.IO.File.GetAttributes(filePath);
// Remove read-only attributes.
if ((attributes & FileAttributes.ReadOnly) != 0) {
System.IO.File.SetAttributes(
filePath,
attributes & (~(FileAttributes.ReadOnly)));
}
System.IO.File.Delete(filePath);
} catch (UnauthorizedAccessException x) {
var newFilePath =
$#"{filePath}.{Guid.NewGuid():N}.deleted";
Trace.TraceWarning(#"Caught UnauthorizedAccessException while deleting file '{0}'. " +
#"Renaming now to '{1}'. {2}", filePath, newFilePath, x.Message);
try {
System.IO.File.Move(
filePath,
newFilePath);
} catch (Win32Exception x2) {
Trace.TraceWarning(#"Caught IOException while renaming upon failed deleting file '{0}'. " +
#"Renaming now to '{1}'. {2}", filePath, newFilePath, x2.Message);
}
} catch (Win32Exception x) {
var newFilePath =
$#"{filePath}.{Guid.NewGuid():N}.deleted";
Trace.TraceWarning(#"Caught IOException while deleting file '{0}'. " +
#"Renaming now to '{1}'. {2}", filePath, newFilePath, x.Message);
try {
System.IO.File.Move(
filePath,
newFilePath);
} catch (Win32Exception x2) {
Trace.TraceWarning(#"Caught IOException while renaming upon failed deleting file '{0}'. " +
#"Renaming now to '{1}'. {2}", filePath, newFilePath, x2.Message);
}
}
} else {
Trace.TraceInformation(#"Not safe-deleting file '{0}', " +
#"because the file does not exist.", filePath);
}
}
public static void SafeMoveFile(string sourcePath, string dstFilePath) {
Trace.TraceInformation(#"About to safe-move file from '{0}' to '{1}'.", sourcePath, dstFilePath);
if (sourcePath == null || dstFilePath == null) {
Trace.TraceInformation(
string.Format(
#"Source file path or destination file path does not exist. " +
#"Not moving."
));
} else {
if (SafeFileExists(sourcePath)) {
SafeDeleteFile(dstFilePath);
var d = Path.GetDirectoryName(dstFilePath);
if (!System.IO.Directory.Exists(d)) {
Trace.TraceInformation(#"Creating non-existing folder '{0}'.", d);
System.IO.Directory.CreateDirectory(d);
}
System.IO.File.Move(sourcePath, dstFilePath);
} else {
Trace.TraceInformation(#"Source file path to move does not exist: '{0}'.", sourcePath);
}
}
}
}
}
Then tested with
if (MyNameSpace.FileHandling.SafeFileExists(HttpContext.Current.Server.MapPath("/products/cats/thumb/" + strGuid + strExt))) {
MyNameSpace.FileHandling.SafeDeleteFile(HttpContext.Current.Server.MapPath("/products/cats/icons/" + strGuid + strExt).ToString());
MyNameSpace.FileHandling.SafeDeleteFile(HttpContext.Current.Server.MapPath("/products/cats/thumb/" + strGuid + strExt).ToString());
MyNameSpace.FileHandling.SafeDeleteFile(HttpContext.Current.Server.MapPath("/products/cats/new/" + strGuid + strExt).ToString());
MyNameSpace.FileHandling.SafeDeleteFile(HttpContext.Current.Server.MapPath("/products/cats/large/" + strGuid + strExt).ToString());
MyNameSpace.FileHandling.SafeDeleteFile(HttpContext.Current.Server.MapPath("/products/cats/full/" + strGuid + strExt).ToString());
}
Couldn't you just iterate through all the files in folder ..cats/ and its sub folders with right attributes(name / extension)? Then you could just use System.IO.File.Exists().
if (System.IO.File.Exists(Path/to/file.ext)){
System.IO.File.Delete(Path/to/file.ext);
}
The iteration could be done finding all files with
string[] files = Directory.GetFiles(txtFolderPath.Text, "*ProfileHandler.cs", SearchOption.AllDirectories);
and then just doing something like:
foreach(file in files){
if (System.IO.File.Exists(Path/to/file.ext)){
System.IO.File.Delete(Path/to/file.ext);
}
}
E: Sorry for not being able to give you the exact syntax, currently using a computer without a proper IDE to write and test the code.
I have this function to delete files older than X amount of months, but it doesn't seem to delete them when I run the code: any guesses to why? When I check folder they are still there.
public static void deleteFiles()
{
try
{
DirectoryInfo ddi = new DirectoryInfo(destination_path);
if (ddi.Exists)
{
logFile.WriteLine("Log Entry: {0}", String.Format("{0:f}", dt) + System.Environment.NewLine);
foreach (var filename in ddi.EnumerateFiles())
{
FileInfo fi = new FileInfo(filename.ToString());//(destination_path + "\\" + filename);
if (fi.CreationTime < dt.AddMonths(-1) )
{
try
{
fi.Delete();
logFile.WriteLine("{0} was deleted successfully.", destination_path + "\\" + filename);
}
catch (Exception ex)
{
logFile.WriteLine("The deletion process failed: {0}", ex.Message);
}
}
}
logFile.WriteLine(String.Concat(Enumerable.Repeat("-------", 25)));
logFile.WriteLine();
}
}
catch (DirectoryNotFoundException ex)
{
logFile.WriteLine("Log Entry: {0}", String.Format("{0:f}", dt) + System.Environment.NewLine);
logFile.WriteLine("Could not delete files from specified directory: {0}", ex.Message);
logFile.WriteLine(String.Concat(Enumerable.Repeat("-------", 25)));
logFile.WriteLine();
}
}
As "Alessandro D'Andria" points out the documentation says:
https://msdn.microsoft.com/en-us/library/system.io.fileinfo.delete(v=vs.110).aspx
"If the file does not exist, this method does nothing."
This makes you think that the file delete has worked and is masking the issue.
If you look at the line:
foreach (var filename in ddi.EnumerateFiles())
this is returning a FileInfo object which is basically a link to the file you want to delete.
But the next line, creates a new FileInfo object based on the filename.
FileInfo fi = new FileInfo(filename.ToString());
But the filename.ToString() is just returning the filename, there is no path information in here.
If you run through the debugger you will see that the fi object will have a pathname (fi.Directory) which is not your destination_path, but is in fact the path of your running executable.
Therefore, fi.Delete() does not actually find the file, so it does nothing (as per the documentation) but you still write a 'success' message to your log file and no exception is thrown, so you wrongly think that everything has worked.
So you actually want something more like this;
if (filename.CreationTime < dt.AddMonths(-1))
{
try
{
if (!filename.Exists)
throw new Exception("File does not exist");
filename.Delete();
WriteLine("{0} was deleted successfully.", destination_path + "\\" + filename);
}
catch (Exception ex)
{
WriteLine("The deletion process failed: {0}", ex.Message);
}
}
Try with the full path:
string path = Path.Combine(destination_path,filename.ToString());
FileInfo fi = new FileInfo(path);
Try something like.... (Ive left out some of your logic but the principle is the same....)
You don't need to use fileinfo as far as I can see.
{
string[] files = Directory.GetFiles(destination_path);
foreach (string filename in files)
{
if (File.Exists(destination_path + "\\" + filename))
{
try
{
File.Delete(destination_path + "\\" + filename);
}
catch (Exception ex)
{
logFile.WriteLine("The deletion process failed: {0}", ex.Message);
}
}
}
}
I'm programming a video player in C# (the video works fine) and what I need now is to get the libvlc logs as well as my custom logs to print them in a file.
I use NLog which handles the libvlc logs (with nVLC) and I raise an event for my custom logs, and in buth cases this function is called :
private static void tracerlogs(string erreur, VLCControl.ControleUtilisateurVLC.LogLevels LvLog)
{
string path = "logs.txt";//Sera redéfini dans l'appli
if (!File.Exists(path))
{
// Create a file to write to.
using (StreamWriter sw = File.CreateText(path))
{
sw.WriteLine(erreur + " " + LvLog.ToString());
sw.Close();
}
}
else
{
using (StreamWriter sw = File.AppendText(path))
{
sw.WriteLine(erreur + " " + LvLog.ToString());
sw.Close();
}
}
Console.WriteLine(erreur + " " + LvLog.ToString());
}
The problem is that I'm getting at random times a System.IO.IOException telling that "the process cannot access the file because it is being used by another process". Although I do close my StreamWriter (which should normally not be useful in a using block)... This makes my app crash. Does anyone have any idea why it does this ?
I finally solved it by adding a resource : as there was a conflict between different threads trying to access this function, I wrapped this :
private static void tracerlogs(string erreur, VLCControl.ControleUtilisateurVLC.LogLevels LvLog)
{
lock (LockLog) {
string path = "logs.txt";//Sera redéfini dans l'appli
if (!File.Exists(path))
{
// Create a file to write to.
using (StreamWriter sw = File.CreateText(path))
{
sw.WriteLine(erreur + " " + LvLog.ToString());
sw.Close();
}
}
else
{
using (StreamWriter sw = File.AppendText(path))
{
sw.WriteLine(erreur + " " + LvLog.ToString());
sw.Close();
}
}
Console.WriteLine(erreur + " " + LvLog.ToString());
}
}
And I declare a public static readonly object LockLog = new Object(); in my class. This works just fine ! Thanks to those who told me that this had to see with threading.