Develop Reference

Logging best approaches

I'm preparing to start developing a new application which has to connect to different API's of different machines which will make the application fairly complicated.
What approaches are there to implement logging into my application, and what are their respective advantages and disadvantages? To be more precise, I'm looking into the best way to export the logs to an external logfile.
Details:
Currently I'm using a system where I add a logentry to a list of logs which I then process line by line through a backgroundworker.
My idea behind this is that by putting the logsprocessing on a different thread I will not disturb the main UI-thread or any other threads for that matter.
However, I think that I'm still using unnecessary resources in my UI-thread because I'm compiling and adding my logentry on the UI-thread. Should I put the actual entry as well on a different thread?
Here is the code that I currently use:
List<LoggingClass> Logs;
BackgroundWorker bgLogs;
public void startLogging()
{
// INITIALIZE THE BACKGROUND WORKER
bgLogs = new BackgroundWorker();
// SET UP THE BACKGROUND WORKER
bgLogs.WorkerSupportsCancellation = true;
bgLogs.WorkerReportsProgress = false;
bgLogs.DoWork += bgLogs_DoWork;
// START THE BACKGROUND WORKER
bgLogs.RunWorkerAsync();
}
private void bgLogs_DoWork(object sender, DoWorkEventArgs e)
{
BackgroundWorker worker = sender as BackgroundWorker;
while (worker.IsBusy == true)
{
if (worker.CancellationPending == true)
{
e.Cancel = true;
break;
}
else
{
if (Logs != null && Logs.Count > 0)
{
processLogs();
}
Thread.Sleep(1000);
}
}
}
public void log(DateTime logTimeStamp, LogType logType, string logContent)
{
// CHECK IF THE LIST OF LOGS EXISTS
if (Logs == null)
{
Logs = new List<LoggingClass>();
}
// CREATE A TEMPORARY LOG ENTRY
LoggingClass tmpLog = new LoggingClass();
tmpLog.TimeStamp = logTimeStamp;
tmpLog.Type = logType;
tmpLog.Content = logContent;
// ADD THE LOG TO THE LIST OF LOGS
Logs.Add(tmpLog);
// CLEAR THE TEMPRARY LOG
tmpLog = null;
}
private void processLogs()
{
string logOutput = string.Empty;
do
{
// CREATE A TEMPORARY LOG ENTRY
LoggingClass tmpLog = Logs[0];
// APPEND THE LOG DATA INTO THE TEMPORARY STRING
logOutput += string.Format("[{0}]\t{1}\t{2}", tmpLog.TimeStamp.ToString("HH:mm:ss.fff"), tmpLog.Type.ToString(), tmpLog.Content) + Environment.NewLine;
// REMOVE THE FIRST LOG ENTRY
Logs.RemoveAt(0);
} while (Logs.Count > 0);
// GET THE FILENAME FOR THE LOGGING FILE
DateTime currentDate = DateTime.Now;
string logFile = Path.GetFullPath(Path.Combine(Application.ExecutablePath, #"..\LOGS\" + currentDate.ToString("yyyyMMdd") + "_log.log"));
// CHECK AND CREATE IF NEEDED THE LOGS DIRECTORY
Directory.CreateDirectory(Path.GetDirectoryName(logFile));
// ADD THE LOGGINGCONTENT TO THE LOG FILE
File.AppendAllText(logFile, logOutput, Encoding.UTF8);
}
class LoggingClass
{
public DateTime TimeStamp { get; set; }
public LogType Type { get; set; }
public string Content { get; set; }
}
Are there any issues with using this method?

C# Waiting for a file to be closed by an external application

Is there an Event I can capture for when a known file has been closed by an external application?
For example, a user is editing a workbook in Excel and I want to read that file as soon as the user finishes working on it and closes the file.
My current solution is to use a combination of FileSystemWatcher and Timer. The FileSystemWatcher will detect when changes have been made to a file, and start a new thread running a Timer to check when the file has closed (via try-catch) However I don't feel as though this is a good solution. If the user forgot to close the file and heads home for the weekend, it feels wasteful for my Timer to be running the whole time. If I increase the interval on my Timer, then my program won't be as responsive. Is there a solution that doesn't involve polling?
EDIT: updated with code example of what I have
private System.Windows.Forms.Timer processTimer;
private string blockedFile;
// Starts here. File changes were detected.
private void OnFileSystemWatcher_Changed(object source, FileSystemEventArgs e)
{
FileSystemWatcher fsw = (FileSystemWatcher)source;
string fullpath = Path.Combine(fsw.Path, fsw.Filter);
StartFileProcessing(fullpath);
}
private void StartFileProcessing(string filePath)
{
if (isFileOpen(new FileInfo(filePath)))
{
blockedFile = filePath;
processTimer = new System.Windows.Forms.Timer();
processTimer.Interval = 1000; // 1 sec
processTimer.Tick += new EventHandler(processTimer_Elapsed);
processTimer.Enabled = true;
processTimer.Start();
}
else
ProcessFile(filePath);
}
private void ProcessFile(string filePath)
{
// Do stuff, read + writes to the file.
}
// GOAL: Without polling, how can I get rid of this step just know right away when the file has been closed?
private void processTimer_Elapsed(object sender, EventArgs e)
{
if (isFileOpen(new FileInfo(blockedFile)) == false)
{
// The file has been freed up
processTimer.Enabled = false;
processTimer.Stop();
processTimer.Dispose();
ProcessFile(blockedFile);
}
}
// Returns true if the file is opened
public bool isFileOpen(FileInfo file)
{
FileStream str = null;
try
{
str = file.Open(FileMode.Open, FileAccess.Read, FileShare.None);
}
catch (IOException)
{
return true;
}
finally
{
if (str != null)
str.Close();
}
return false;
}

Need help making a File Export function remember the path that the user selects

I have a function in my program that exports data. The exporting works great, but I'm having trouble with something: When the user clicks Export, they are presented with a folder browser to choose where on their hard drive they would like to export to. The browser launches initially with all the hard drives/folders collapsed like they should be, but I can't figure out how to make it so the program remembers which location the user chose, so that the next time they want to export, it automatically opens to that location instead of once again opening with everything collapsed. I'm just having trouble with the logic, I suppose. Anyone have any tips?
Also, just for clarification, I'm trying to get it to remember the location only for the duration of the session, not like permanently on the register.
Here's my export function so far, if you think that would be helpful:
private void Export(int formatVersion, bool pureXmlDriver)
{
if (Device != null)
{
Utilities.StripShortNameFromLongNames(Device);
using (var folderBrowser = new FolderBrowserDialog())
{
folderBrowser.Description = Resources.SelectExportFolder;
if (folderBrowser.ShowDialog() == DialogResult.OK)
{
string selectedFolder = folderBrowser.SelectedPath;
try
{
Cursor = Cursors.WaitCursor;
HandleExport(formatVersion, pureXmlDriver, selectedFolder);
}
finally
{
Cursor = Cursors.Default;
}
}
}
}
}

This should do it. You just need a class field to keep the last value in.
public class MyClass
{
private string selectedPath = "";
public void Export(int formatVersion, bool pureXmlDriver)
{
if (Device != null)
{
Utilities.StripShortNameFromLongNames(Device);
using (var folderBrowser = new FolderBrowserDialog())
{
folderBrowser.Description = Resources.SelectExportFolder;
folderBrowser.SelectedPath = selectedPath;
if (folderBrowser.ShowDialog() == DialogResult.OK)
{
selectedFolder = folderBrowser.SelectedPath;
try
{
Cursor = Cursors.WaitCursor;
HandleExport(formatVersion, pureXmlDriver, selectedFolder);
}
finally
{
Cursor = Cursors.Default;
}
}
}
}
}
}

The code works but it lags when it works. It works with foreach and loop it

When the code is working so laggy it would be very good so that it is not laggy when it works.
How the code works:
It searches the computer for a file is then when it find it to change the file but if the file is running line will loop until it manages to do its job.
Main class
public Form1(string[] Args)
{
InitializeComponent();
backgroundWorker1.RunWorkerAsync();
}
private void backgroundWorker1_DoWork(object sender, DoWorkEventArgs e)
{
Thread.Sleep(1000); // One second.Thread.Sleep(1000); // One second.
MessageBox.Show("Testing");
Fille mc = new Fille();
mc.Search();
}
Fille clss
private static ArrayList list2 = new ArrayList();
private static ArrayList listRemove = new ArrayList();
public void Search()
{
try
{
foreach (string file in Directory.EnumerateFiles(#"C:\Users\user\Downloads\MCFILE\trrtrt\", "*.exe", SearchOption.AllDirectories))
{
// Display file path.
if (SHA1Hash.GetSHA1Hash(file) == "1233456") // fake SHA1Hash
{
try
{
COPYWithReplace(#"C:\Users\user\Downloads\MCFILE\Fake2\Test.exe", file);
}
catch (IOException)
{
// log errors
if (list2.Count == 0)
{
list2.Add(file);
Thread thread = new Thread(new ThreadStart(Test2));
thread.Start();
}
else
{
Thread thread = new Thread(new ThreadStart(Test2));
thread.Abort();
list2.Add(file);
thread.Join();
}
}
}
}
}
catch (Exception ex)
{
// log errors
}
}
private void Test2()
{
if (list2.Count == 0)
{
}
else
{
foreach (string _item in list2)
{
try
{
//Run
COPYWithReplace(#"C:\Users\user\Downloads\MCFILE\Fake2\Test.exe", _item);
listRemove.Add(_item);
}
catch (IOException)
{
//error
}
}
foreach (var Remove in listRemove)
{
list2.Remove(Remove);
}
listRemove.Clear();
if (list2.Count == 0)
{
}
else
{
Thread thread = new Thread(new ThreadStart(Test2));
thread.Start();
}
}
}
I made a new thread because I found the problem. But now it's just that it lags.

I suspect the reason it's "lagging" is because you have the system in a very convoluted but rather processor intensive and I/O intensive loop. If a file fails the first test, your code starts a thread that tries it again. And if that fails then you start another thread to try it again, lather, rinse, repeat.
That's going to absolutely kill performance. You're basically doing this:
while (forever)
{
if I can overwrite the file
{
break;
}
}
Except if you have multiple files that you're trying to write, then you're doing that loop for every file. Concurrently. And you're not just using a loop. Instead, you're starting and stopping threads like nobody's business.
Yeah, that's going to slow down your computer.
A more reasonable way to do this would be with a thread to do the first check, and a timer that will limit how often you do the other checks. Communication is with a simple queue, because only one thread will be accessing it at a time.
Here's what I would recommend:
private static Queue<string> filesToCheck = new Queue<string>();
private System.Timers.Timer copyTimer;
public void Search()
{
try
{
foreach (string file in Directory.EnumerateFiles(#"C:\Users\user\Downloads\MCFILE\trrtrt\", "*.exe", SearchOption.AllDirectories))
{
// Display file path.
if (SHA1Hash.GetSHA1Hash(file) == "1233456") // fake SHA1Hash
{
if (!TryToCopy(file)) // See function below
{
filesToCheck.Enqueue(file);
}
}
}
// Checked all the files once.
// If there are any in the queue, start the timer.
if (filesToCheck.Count > 0)
{
copyTimer = new System.Timers.Timer(CopyTimerProc, null, 1000, Timeout.Infinite);
}
}
catch (Exception)
{
// do your error handling
}
}
private void CopyTimerProc(object state)
{
string filename = filesToCheck.Dequeue();
if (TryToCopy(filename))
{
// success. If the queue is empty, kill the timer.
if (filesToCheck.Count == 0)
{
copyTimer.Dispose();
}
}
else
{
// File still locked.
// Put it back on the queue and reset the timer.
filesToCheck.Enqueue(filename);
copyTimer.Change(1000, 0);
}
}
private bool TryToCopy(string filename)
{
try
{
COPYWithReplace(#"C:\Users\user\Downloads\MCFILE\Fake2\Test.exe", filename);
return true;
}
catch (IOException)
{
// log error
return false;
}
}
The timer is a one-shot that is reset after each time it ticks. The reason I did it this way is to prevent another tick coming along while the previous tick is still processing. After all, it takes time to copy a file.
There's no reason to do this with a bunch of threads. The file system can only do one thing at a time, anyway, and it's not like an extra second or two while you wait for a file to become available is going to hurt anything.

How to implement both the FileSystemWatcher and Timer in Multithreading in C# console application?

I need to create a C# Console Application that will parse the file from SFTP directory when the new file created. For that I implemented FileSystemWatcher with FileCreated event which enqueue the new file path and create a new thread and parse the file.
I read in the blogs that some times FileSystemWatcher may fail to detect new files, for that I implemented Timer which will fire every 1 hr and if the FileSystemWatcher thread is in waitsleep state then will read the IMCOMING SFTP folder and parse the file.
Below is the code i written for FileSystemWatcher and Timer, but its not working properly and I think filesystemwatcher is not in Multithreading. Please help me to get right solution.
MAIN
static void Main(string[] args)
{
try
{
string path = incomingFilePath;
if (Directory.Exists(path))
{
#region Initiate Timer
Thread t = new Thread(new ParameterizedThreadStart(ThreadLoop));
t.Start((Action)fileProcessor.StartTimer);
#endregion
#region FileSystemWatcher
watcher = new FileSystemWatcher { Path = path, Filter = "*.CUST", IncludeSubdirectories = true };
watcher.Created += new
FileSystemEventHandler(watcher_FileCreated);
watcher.Error += new
ErrorEventHandler(watcher_OnError);
watcher.EnableRaisingEvents = true;
#endregion
}
}
catch (Exception Err)
{
}
}
FILESYSTEMWATCHER CODE:
private static void watcher_FileCreated(object sender, FileSystemEventArgs e)
{
if (e.FullPath.ToUpper().Contains("INCOMING"].ToString()))
{
fileProcessor.EnqueueFile(e.FullPath);
lock (lockObject)
{
files.Enqueue(path);
}
if (FileWacherThread == null || shouldStop)
{
FileWacherThread = new Thread(new ThreadStart(Work));
FileWacherThread.Start();
}
// If the thread is waiting then start it
else if (FileWacherThread.ThreadState == ThreadState.WaitSleepJoin)
{
waitHandle.Set();
}
}
}
private void Work()
{
while (!shouldStop)
{
string path = String.Empty;
lock (lockObject)
{
if (files.Count > 0)
{
path = files.Dequeue();
}
}
if (!String.IsNullOrEmpty(path))
{
// Process the file
ParseFile(path);
}
else
{
// If no files are left to process then wait
waitHandle.WaitOne();
}
}
}
TIMER CODE
public void StartTimer()
{
lock (lockObject)
{
if (FileWacherThread == null || FileWacherThread.ThreadState == ThreadState.WaitSleepJoin)
{
if (files.Count == 0)
{
IEnumerable<string> result = new List<string>(Directory.GetFiles(incomingFilePath, "*.CUST", SearchOption.AllDirectories)).Where(s => s.Contains(incomingFilePrefix));
foreach (string path in result)
{
ParseFile(path);
}
}
}
}
}

Things to check...
is waitHandle an AutoResetEvent or a ManualResetEvent? (from the way that you are using it, it should be an AutoResetEvent
If shouldStop is true, is FileWacherThread(sic) set to null when Work() exits...
How are you protecting access to FileWacherThread? if it is accessed from multiple threads (to check its state, assign etc, then it too should be protected with a lock).
You shouldn't worry about the state of the FileWacherThread when you set the event. If you want to signal to that thread, just set it, (i.e. build your multithreaded code such that the publisher doesn't know/care about the current state of the subscriber).
Currently there are states that your FileWacherThread can be in where it isn't waiting but it might still need to be signaled. If you always set the event, the worst that can happen is that it loops one time unnecessarily.