Develop Reference

FileSystemWatcher on separate thread

I have to create script which will push some files into provided location and grab the output generated by other script.
Copy files to location.
Other script will grab the files and process them.
Wait for all results.
Get data from files, remove files.
Repeat steps for next stack of files.
On my PC everything works fine but when I run my script on the machine where Windows Server 2016 is running some error pop-up:
So I created a SystemWatcher which is monitoring a provided location
[PermissionSet(SecurityAction.Demand, Name = "FullTrust")]
private void RunWatcher(string location)
{
// Create a new FileSystemWatcher and set its properties.
using (FileSystemWatcher watcher = new FileSystemWatcher())
{
watcher.Path = location;
// Watch for changes in LastAccess and LastWrite times, and
// the renaming of files or directories.
watcher.NotifyFilter = NotifyFilters.LastAccess
| NotifyFilters.LastWrite
| NotifyFilters.FileName
| NotifyFilters.DirectoryName;
watcher.Filter = "*";
// Add event handlers.
watcher.Created += OnCreate;
// Begin watching.
watcher.EnableRaisingEvents = true;
// quit the program.
while (!endWatcher);
}
}
private void OnCreate(object source, FileSystemEventArgs e)
{
string fileExtension = Path.GetExtension(e.FullPath);
if (extensionWatch.Contains(fileExtension))
{
SaveFileData(fileExtension, e.FullPath);
}
}
The watcher is created on separate thread. Then a "while" loop inside the "for" is checking if all files has been created or if the "Abandon time" has been exceeded.
private void PushFiles(string destinationPath, int timeAbandon, BackgroundWorker worker)
{
Console.WriteLine("Before start watcher");
Task.Run(() =>
{
RunWatcher(destinationPath);
});
Console.WriteLine("After start watcher");
timeAbandon *= 60; // to sec
for (int i = 0; i < fileCollect.Count; i++)
{
double timeDiff;
start = DateTime.UtcNow;
currentListIdx = i;
CopyPtx(fileCollect[i], destinationPath);
while (!CheckIfDone(i))
{
timeDiff = (DateTime.UtcNow - start).TotalSeconds;
if (timeDiff > timeAbandon)
{
break;
}
Thread.Sleep(1000);
Console.WriteLine("Waiting for results...");
}
CopyAndRemoveFiles(fileCollect[i], destinationPath);
worker.ReportProgress(1);
}
endWatcher = true;
}
private bool CheckIfDone(int stackId)
{
bool isDone = true;
foreach (var file in fileCollect[stackId])
{
if (file["finished"] == "false")
{
isDone = false;
}
}
if (isDone)
{
return true;
}
return false;
}
I did have a look on Just-In-Time settings and "Managed, Native, Script" are selected. Also I use .net Core as I can't install any dependencies on the server machine.
To run the script on the server I do publish the script first with a following settings:
I tried to change the Configuration setting to "Debug|AnyCPU" but it didn't help.
Can anyone help me to get more information about the error? Or possible solution to my problem?

There was en error in function SaveFileData(fileExtension, e.FullPath);. The error appeard because I was trying to read file before saving into file was finished. So The title of this post is not correct.
To solve the problem with access to file I used a following code.
protected virtual bool IgnoreErrorMsg(FileInfo file, string ignoreMsg)
{
bool isAccess = false;
while (!isAccess)
{
try
{
using (FileStream stream = file.Open(FileMode.Open, FileAccess.Read, FileShare.None))
{
//if file can be read, then read it
if (stream.CanRead)
{
byte[] buffer = new byte[stream.Length];
int bytesRead = stream.Read(buffer, 0, buffer.Length);
string errorString = Encoding.ASCII.GetString(buffer, 0, bytesRead);
if (errorString.Contains(ignoreMsg))
{
return true;
}
// access has been granted so loop can be exit
isAccess = true;
}
stream.Close();
}
}
catch (IOException e)
{
//the loop will continue as long as "cannot access" error appear
isAccess = false;
}
Thread.Sleep(500);
}
//if return false, then script will wait for optimisation unitil AbandonTime
return false;
}
In that case the title of the post is not correct because it's not related to multithreading.

Read text file line by line using timer

StreamReader sr = new StreamReader("C:/CR EZ Test/Log.txt"); //use with IF
private void timer2_Tick(object sender, EventArgs e)
{
if ((line = sr.ReadLine()) != null)
{
//FileStream fs = File.Open("C:/CR EZ Test/Log.txt", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
//StreamReader sr = new StreamReader(fs); //use with While can't use with }else{
//while ((line = sr.ReadLine()) != null)
//{
string[] dataLog = line.Split(new[] { ',' }, StringSplitOptions.None);
mpa = (dataLog[1]);
ml = (dataLog[2]);
lph = (dataLog[3]);
elapsedTime = float.Parse(dataLog[4]) / 1000;
if (testStatus > 0) time = elapsedTime.ToString("0.0");
tb2.Value = int.Parse(dataLog[6]);
if (chart1.Series[0].Points.Count > tb1.Value && tb1.Value > 0)
{
chart1.Series[0].Points.RemoveAt(0);
chart1.Series[1].Points.RemoveAt(0);
}
chart1.Series[0].Points.AddXY(dataLog[5], int.Parse(dataLog[1]));
chart1.Series[1].Points.AddXY(dataLog[5], int.Parse(dataLog[6]));
//}
}
else
{
sr.DiscardBufferedData();
sr.BaseStream.Seek(0, SeekOrigin.Begin);
sr.BaseStream.Position = 0;
//sr.Close();
//alertTB.Text = "";
timer2.Enabled = false;
}
alertTB.ForeColor = Color.Red;
alertTB.Text = "Data Log Viewing In Progress";
}
The issue is I am reading a text file full of variables back through a GUI, like replaying a video. As the code is shown, it works and I can control the timer tick to change the replay speed. The issue is the file is in use, so I can't write to or delete the text while the file is in use, without closing it first. I would like to either be able to find a workaround of the Streamreader, or use the Filestream to Streamreader code that will allow me to edit the file while it is in use. The issue there is, I can't figure out how to make it work with the timer, it just reads the entire file very quickly. Any help or ideas are greatly appreciated.
The issue here is how to have the commented out code to:
read a line of the text file,
have the timer to tick
then read the next line of the text file, and so on. Obviously handling the data as it arrives.

Opening a file while it is in use
I think what you are looking for is FileStream with FileShare.ReadWrite for the instance of your StreamReader (not the instance you have commented out),
var fs = new FileStream("C:\foo.txt", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
var sr = new StreamReader(fs);
Setting the position of the stream
It also seems like based on your comments, you are having trouble with positioning the stream, this is how you could do that...
fs.Position = 0; // note this is the FileStream not the StreamReader!
// alternatively, you could use Seek
Difference between sequential and random access
Lastly, you might want to take a look below to see the difference between sequential and random access
A Potential Solution
Here is a class called FileMonitor that will check the file and update the list whenever the file is changed / updated.
I understand that you want a timer to poll the data in the text file, but in case the timer is very fast, I have optimized the FileMonitor to watch the file for changes and only extract when there is a change.
Please note that this only continues to read where it was left off, based on the position of the stream. So, it will not work if lines are deleted or modified prior to getting "extracted". This means it only functions based on your requirements and is not improved to handle a lot of other scenarios, but it should adequately cover your requirements.
public class FileMonitor : IDisposable
{
private readonly FileStream _file;
private readonly StreamReader _reader;
private long _position;
private List<string> _lines;
public FileMonitor(string file)
{
if (String.IsNullOrEmpty(nameof(file))) throw new ArgumentNullException(nameof(file));
_lines = new List<string>();
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = Path.GetDirectoryName(file);
watcher.Filter = Path.GetFileName(file);
watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite | NotifyFilters.FileName | NotifyFilters.DirectoryName;
watcher.Changed += new FileSystemEventHandler(OnChanged);
//watcher.Created += new FileSystemEventHandler(OnCreated);
//watcher.Deleted += new FileSystemEventHandler(OnDeleted);
//watcher.Renamed += new RenamedEventHandler(OnRenamed);
// begin watching.
watcher.EnableRaisingEvents = true;
// begin reading
_file = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
_reader = new StreamReader(_file);
_lines = ReadLines(_reader).ToList();
_position = _file.Position;
}
private void OnChanged(object source, FileSystemEventArgs e)
{
List<string> update = ReadLines(_reader).ToList();
// fix to remove the immidate newline
if (update.Count() > 0 && String.IsNullOrEmpty(update[0])) update.RemoveAt(0);
_lines.AddRange(update);
_position = _file.Position;
// just for debugging, you should remove this
Console.WriteLine($"File: {e.FullPath} [{e.ChangeType}]");
}
public IEnumerable<string> Lines { get { return _lines; } }
public void Reset()
{
_file.Position = 0;
_position = _file.Position;
_lines.Clear();
}
private static IEnumerable<string> ReadLines(StreamReader reader)
{
string line;
while ((line = reader.ReadLine()) != null)
{
yield return line;
}
}
public void Dispose()
{
_reader.Dispose();
_file.Dispose();
}
}
Here is how you could use it with your timer
private IEnumerable<string> _lines; // holds all the lines "extracted"
void Main()
{
string file = #"C:\Data\foo.txt";
using (var timer = new System.Timers.Timer())
{
timer.Interval = 2000; // 2 second interval
timer.Elapsed += OnTimedEvent; // attach delegate
timer.Enabled = true; // start the timer
// open the file
using (var monitor = new FileMonitor(file))
{
_lines = monitor.Lines;
// loop forever, remove this
while (true) { }
}
}
}
public void OnTimedEvent(object sender, EventArgs e)
{
// just for debugging, you should remove this
Console.WriteLine($"current count: {_lines.Count()}");
}
If it isn't clear, the data extracted is held in a list of strings. Above, you can grab the "extracted" data from the monitor using the monitor.Line property.

A Proven Working Solution
string line;
if (!File.Exists(logFile))
{
viewLog.Text = "Play";
alertTB.ForeColor = Color.Red;
alertTB.Text = "File Does Not Exist | Log Data To Create File";
chart.Text = "Scope On";
}
if (File.Exists(logFile))
{
var lineCount = File.ReadLines(logFile).Count();//read text file line count to establish length for array
if (lineCount < 2)
{
viewLog.Text = "Play";
alertTB.ForeColor = Color.Red;
alertTB.Text = "File Exists | No Data Has Been Recorded";
chart.Text = "Scope On";
}
if (counter < lineCount && lineCount > 0)//if counter is less than lineCount keep reading lines
{
line = File.ReadAllLines(logFile).Skip(counter).Take(lineCount).First();
string[] dataLog = line.Split(new[] { ',' }, StringSplitOptions.None);
//-----------------------------------------Handling my data
counter++;
}
else
{
counter = 0;
timer2.Enabled = false;
}
}
This is the fix I arrived at, it allows editing the file or deleting the contents of the file. I get the line count before trying to load the file. I then use the counter to iterate through the lines. I can change the delay between the next line read based upon the timer tick interval, pause it, or stop it.

Delete file using file watcher not allowing me to delete second time

My task is to delete file once the processing is completed . I am using FileWatcher to complete this task. It is watching specific folder . Suppose If i copy one file and put that in filewatcher folder it is deleting. Second time when i copy the same file and paste that in the same watching folder. This time it says that Another process is using that file . and exception is throwing . I think i am missing something. Here is my code
private static void Main(string[] args)
{
var fw = new FileSystemWatcher(EmailSetting.DataFolder)
{
IncludeSubdirectories = false
,
EnableRaisingEvents = true
};
fw.Created += (sender, e) =>
{
File.Delete(e.FullPath);
};
Console.ReadLine();
}

You receive the Created event when the file was created (hence the name). But at this point in time the other process that is actually creating it, didn't finish writing content into that file. So the file might be already there, but the other is still working on it (imagine you would copy a 8 GB file).
It would be wiser to simply write the path of the file into a list within the event and let another thread regularly check this concurrent bag (e.g. once a second). First it checks if the file exists and if yes, try to delete it. If succeeded, remove it from the bag, otherwise try again next time.
Code example
private static readonly ConcurrentQueue<FileInfo> _FileCandidates = new ConcurrentQueue<FileInfo>();
private static void Main(string[] args)
{
var watcher = new FileSystemWatcher
{
Path = #"R:\TestFolder",
IncludeSubdirectories = false,
Filter = "*.*",
};
Console.WriteLine("Start watching folder... " + watcher.Path);
watcher.Created += OnFileCreated;
watcher.EnableRaisingEvents = true;
var timer = new Timer
{
AutoReset = true,
Interval = 1000,
};
timer.Elapsed += OnTimerElapsed;
timer.Enabled = true;
Console.ReadKey();
}
static void OnTimerElapsed(object sender, ElapsedEventArgs e)
{
FileInfo file;
var stillInUseFiles = new List<FileInfo>();
Console.WriteLine("Check for file candidates...");
while (_FileCandidates.TryDequeue(out file))
{
try
{
Console.WriteLine("Delete " + file.FullName);
if (file.Exists)
file.Delete();
}
catch (IOException)
{
Console.WriteLine("Could not delete file, try again next time.");
stillInUseFiles.Add(file);
}
}
foreach (var unhappyFile in stillInUseFiles)
{
_FileCandidates.Enqueue(unhappyFile);
}
}
static void OnFileCreated(object sender, FileSystemEventArgs e)
{
Console.WriteLine("Found new file candidate " + e.FullPath);
_FileCandidates.Enqueue(new FileInfo(e.FullPath));
}

FileSystemWatcher or Directory Read?

Conext
I have a scenario in which i have to check the specific directory after every 30 seconds for new files. If there is any new file then i have to process the files but can be multiple in a batch.
Questions
Should i use FileSystemWatcher or Read the directory and process the files in parallel ?
I am using Windows service which will process CSV files and output will be shown to windows form application.
Can we schedule the FileSystemWatcher on Timer ?
What will be the best approach in this case ?
If i choose the Directory read instead of FileSystemWatcher how to process the batch of 100 files in parallel and send to other application ?
Thanks

I was also trying and found this article interesting.
https://connectvishal.wordpress.com/2015/11/05/filesystemwatcher-and-queues-with-parallel-execution/
protected override void OnStart(string[] args)
{
current_directory = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
try
{
strDir = ConfigurationManager.AppSettings["Directory"];
fileMask = ConfigurationManager.AppSettings["FileMask"];
strBatfile = ConfigurationManager.AppSettings["Batch"];
strlog = ConfigurationManager.AppSettings["Log"];
Task.Factory.StartNew(QueueHandler);
var fsw = new FileSystemWatcher();
fsw.Created += (o, e) =>
{
// add a file to the queue
filenames.Enqueue(e.FullPath);
};
fsw.Path = strDir + "\\";
fsw.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
| NotifyFilters.FileName | NotifyFilters.DirectoryName;
fsw.Filter = fileMask;
fsw.EnableRaisingEvents = true;
fsw.Deleted += new FileSystemEventHandler(OnDeleated);
fsw.Renamed += new RenamedEventHandler(OnRenamed);
fsw.EnableRaisingEvents = true;
}
catch (Exception exception)
{
CustomException.Write(CustomException.CreateExceptionString(exception.ToString()));
}
}
and a queue handler :
static void QueueHandler()
{
bool run = true;
AppDomain.CurrentDomain.DomainUnload += (s, e) =>
{
run = false;
filenames.Enqueue("stop");
};
try
{
while (run)
{
string filename;
if (filenames.TryDequeue(out filename) && run)
{
var proc = new Process();
proc.StartInfo.FileName = Service1.strBatfile; //here .exe can be added
proc.Start();
;
Log.getLogger("File Processed after executing batch\.exe: Filename - :" + filename + " " + "Batch File Executed- > " + Service1.strBatfile + " at timestamp : " + DateTime.Now.ToString(), Service1.strlog);
proc.WaitForExit(); // this blocks until the process ends....
}
}
}
catch (Exception exception)
{
CustomException.Write(CustomException.CreateExceptionString(exception.ToString()));
}

Cannot read from txt file although the file close

In this class i am watching a txt file and every new line is handle:
If the first word is Start (the second is the file name) i am opening Wiresahrk process and start capturing.
If it start with Stop i am kill the process who running (all running processes stored in list and associate to the file name)
string _file = #"D:\file.txt";
public void startWatch()
{
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = Path.GetDirectoryName(_file);
watcher.Filter = Path.GetFileName(_file);
watcher.NotifyFilter = NotifyFilters.LastWrite;
watcher.Changed += watcher_Changed;
watcher.EnableRaisingEvents = true;
}
public void watcher_Changed(object sender, FileSystemEventArgs e)
{
readLine();
}
private void readLastLine()
{
string lastLine = string.Empty;
using (StreamReader sr = new StreamReader(_file))
{
string str = sr.ReadToEnd();
int x = str.LastIndexOf('\n');
lastLine = str.Substring(x + 1);
}
validateString(lastLine);
}
private void validateString(string str)
{
string[] arr = str.Split(' ');
if (arr.Length != 2 && arr[0] != "start" && arr[0] != "stop" && arr[0] != "finish")
return;
Tshark tshark = new Tshark(arr[1]);
tshark.startCapturing(); // Start wireshark process and start capturing
}
After i read the last line from my file everything works fine, after the second time i try to read an error occurs: The process cannot access the file because it is being used by another process.

Try explicitly setting, how file will be opened/shared (assuming file exists, but anyway good idea will by to wrap this region in try/catch block).
using (var stream = new StreamReader( File.Open(_file, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)))
{
string str = stream.ReadToEnd();
int x = str.LastIndexOf('\n');
string lastline = str.Substring(x + 1);
}