I have a program which uses SLF for logging. This program runs 24/7 and I would like to upload the log files to a remote server every night for later review.
My question is, how do I release the file lock for the log without closing the program?
I was hoping to suspend logging, upload the logs, either delete the log file or erase the contents, and then resume logging.
ILogger logger = LoggerService.GetLogger(typeof(TaskScheduler).FullName);
// Other initialization here
foreach (var task in managedTasks.OrderBy(t => t.Priority))
{
if (task.NextRunTime <= DateTime.Now)
{
dataManager.CurrentStatus = AppStatus.Running;
if (task.Name == "Log Sender")
{
logger = null;
}
// Run the task
if (task.Name == "Log Sender")
{
logger = LoggerService.GetLogger(typeof(TaskScheduler).FullName);
}
dataManager.CurrentStatus = AppStatus.Idle;
}
}
Currently, when I do this, I'm still getting an IOException because the file is still locked by the Task Scheduler.
EDIT: I'm using SLF over log4net if that helps.
I would consider this as a design issue rather than technical. Log events for each day in an individual file and upon next day you can do anything you want with yesterday's log file. You can narrow log period according to your need (i.g. every 12 hours).
After posting this question and getting the recommendations to reconsider my design. I started to look into creating a rolling log file for each date. I happened across this, which solved my problem completely:
<appender name="InfoRollingLogFileAppender" type="log4net.Appender.RollingFileAppender,log4net">
<param name="File" value="log.txt" />
<param name="StaticLogFileName" value="true"/>
<maximumFileSize value="1024KB" />
<appendToFile value="true" />
<lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%logger: %date{dd MMM yyyy HH:mm:ss} [%thread] %-5level - %message %newline" />
</layout>
</appender>
I configured the log4net appender in my app.config to follow the lockingModel to include a "MinimalLock" this enabled me to pull the contents of the logs, upload them to the server and then erase the local logs without having to create separate log files for a specified time/date span.
Again, thanks to everyone for your help, I wouldn't have stumbled across this handy change without your suggestions.
Related
I'm looking at using log4net as my logging framework of choice for a new project starting shortly. One issue that I've run into during prototyping that I can't find a definitive answer for is how you can clean or mask message content in a configurable and tidy way.
Hypothetically let's say I want several cleaners to be put in action but I also want to follow the single responsibility principle. Some cleaner examples:
Cardnumber/PAN cleaner
Password cleaner
Private data cleaner
I know that you should never be logging this sort of information in plain text and the code executing the logs will never knowingly be doing this. I want to have a last level of protection however in case data becomes malformed and sensitive data somehow slips into somewhere it shouldn't; logs being the worst case scenario.
Option 1:
I've found this StackOverflow article which details a possible solution however it involves the use of reflection. This is not desirable for performance but it also seems hacky to manipulate internal storage mechanisms.
Editing-log4net-messages-before-they-reach-the-appenders
Option 2:
The suggested answer on the same question suggests the use of a PatternLayoutConverter. This is fine for a single cleaner operation but you are unable to use multiple operations such as the below:
public class CardNumberCleanerLayoutConverter : PatternLayoutConverter
{
protected override void Convert(TextWriter writer, LoggingEvent loggingEvent)
{
string message = loggingEvent.RenderedMessage;
// TODO: Replace with real card number detection and masking.
writer.Write(message.Replace("9", "*"));
}
}
<layout type="log4net.Layout.PatternLayout">
<converter>
<name value="cleanedMessage" />
<type value="Log4NetPrototype.CardNumberCleanerLayoutConverter, Log4NetPrototype" />
</converter>
<converter>
<name value="cleanedMessage" />
<type value="Log4NetPrototype.PasswordCleanerLayoutConverter, Log4NetPrototype" />
</converter>
<conversionPattern value="%cleanedMessage" />
</layout>
In the case of a naming collision as demonstrated above, the converter loaded last will be the one which is actioned. Using the above example, this means that passwords will be cleaned but not card numbers.
Option 3:
A third option which I've tried is the use of chained ForwarderAppender instances but this quickly complicates the configuration and I wouldn't consider it an ideal solution. Because the LoggingEvent class has an immutable RenderedMessage property we are unable to change it without creating a new instance of the LoggingEvent class and passing it through as demonstrated below:
public class CardNumberCleanerForwarder : ForwardingAppender
{
protected override void Append(LoggingEvent loggingEvent)
{
// TODO: Replace this with real card number detection and masking.
string newMessage = loggingEvent.RenderedMessage.Replace("9", "*");
// What context data are we losing by doing this?
LoggingEventData eventData = new LoggingEventData()
{
Domain = loggingEvent.Domain,
Identity = loggingEvent.Identity,
Level = loggingEvent.Level,
LocationInfo = loggingEvent.LocationInformation,
LoggerName = loggingEvent.LoggerName,
ExceptionString = loggingEvent.GetExceptionString(),
TimeStamp = loggingEvent.TimeStamp,
Message = newMessage,
Properties = loggingEvent.Properties,
ThreadName = loggingEvent.ThreadName,
UserName = loggingEvent.UserName
};
base.Append(new LoggingEvent(eventData));
}
}
public class PasswordCleanerForwarder : ForwardingAppender
{
protected override void Append(LoggingEvent loggingEvent)
{
// TODO: Replace this with real password detection and masking.
string newMessage = loggingEvent.RenderedMessage.Replace("4", "*");
// What context data are we losing by doing this?
LoggingEventData eventData = new LoggingEventData()
{
Domain = loggingEvent.Domain,
Identity = loggingEvent.Identity,
Level = loggingEvent.Level,
LocationInfo = loggingEvent.LocationInformation,
LoggerName = loggingEvent.LoggerName,
ExceptionString = loggingEvent.GetExceptionString(),
TimeStamp = loggingEvent.TimeStamp,
Message = newMessage,
Properties = loggingEvent.Properties,
ThreadName = loggingEvent.ThreadName,
UserName = loggingEvent.UserName
};
base.Append(new LoggingEvent(eventData));
}
}
Matching configuration (very hard to follow):
<log4net>
<appender name="LocatedAsyncForwardingAppender" type="Log4NetPrototype.LocatedAsyncForwardingAppender, Log4NetPrototype">
<appender-ref ref="CardNumberCleanerForwarder" />
</appender>
<appender name="CardNumberCleanerForwarder" type="Log4NetPrototype.CardNumberCleanerForwarder, Log4NetPrototype">
<appender-ref ref="PasswordCleanerForwarder" />
</appender>
<appender name="PasswordCleanerForwarder" type="Log4NetPrototype.PasswordCleanerForwarder, Log4NetPrototype">
<appender-ref ref="LogFileAppender" />
</appender>
<appender name="LogFileAppender" type="Log4NetPrototype.LogFileAppender, Log4NetPrototype">
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%m" />
</layout>
</appender>
<root>
<level value="DEBUG" />
<appender-ref ref="LocatedAsyncForwardingAppender" />
</root>
</log4net>
Does anyone have another suggestion for how this could be implemented where theoretically n number of cleaners could be configured at the cost of performance?
In your question you are already saying that you should go to the cause and not logging any sensitive data. This can be enforced by a fourth option of using code reviews, and look at the data being logged. Your logging statements should never log any sensitive data, because the cause a security risk. Trusting on any code with filters the sensitive data will probably fail if you make changes to your project. Your QA process has to be really good to catch this kind of mistakes (I've never seen a tester going through all logs). So I would go for option 4 which is making sure you do not log this kind of information in the first place.
I'm curious about timestamping in NLog when using async targets.
I know that according to What's the meaning of the time stamp in nlog when async is on? the timestamp is generated when the log entry is queued, as you would expect.
However, I noticed something in one of my log files, so I decided to whip up a quick test.
static void Main(string[] args)
{
for (int i = 0; i < 10000; i++)
{
_logger.Info("Timestamp: {0}, LogNumber: {1}",DateTime.Now.ToString("HH:mm:ss.fff"), i);
}
_logger.Factory.Flush();
}
My NLog.config looks like:
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<targets async="true">
<target xsi:type="File" name="f" fileName="${basedir}/logs/${shortdate}.log"
layout="${longdate} ${uppercase:${level}} ${message}" />
</targets>
<rules>
<logger name="*" minlevel="Trace" writeTo="f" />
</rules>
</nlog>
Now if I look at the output, we see that all entries between 754, and 9962 have the same NLog timestamp, however, DateTime.Now shows the milliseconds progressing:
2015-02-12 08:19:23.3814 INFO Timestamp: 08:19:23.376, LogNumber: 0
...
2015-02-12 08:19:23.3853 INFO Timestamp: 08:19:23.384, LogNumber: 754
...
2015-02-12 08:19:23.4033 INFO Timestamp: 08:19:23.399, LogNumber: 9963
...
I can understand that with the overheads, a DateTime.Now stamp of .384 could be logged an .385, however, it doesn't make sense to me that .399 comes out as .385.
The way that the NLog timestamp progresses, almost looks like the timestamp is generated during a logging cycle, rather than the log call. Which would be contrary to the above article.
So, is this something to do with the time source NLog uses, or rather, when the timestamp is generated?
Behavior you see is defined by time source system of NLog.
NLog 4.x provides 4 different time sources, and you can plug in your own. The default time source of NLog is optimized for performance and does some caching of the values that are being returned. The caching mechanism uses Environment.TickCount intervals to return time stamps for new log entries. Resolution of TickCount property is limited to resolution of system timer which is not a fixed value but typically varies in the 10-16 ms range.
So, the effect you see is defined by default time source of NLog. And, btw, it's not specific to async targets - remove async property from your example and you will see exactly same behavior.
If you need a more precise resolution for time stamps in your log events, you can configure NLog to use another time source:
<nlog>
<time type="AccurateUTC" />
</nlog>
However, this will not guarantee that all your events will have different time stamps, you might get 2-3 events with same time. But it will use straight value of DateTime.UtcNow to fill in the date/time field of log event, without any caching. And without any date time zone conversions too...
You can find more info about time sources in NLog wiki
In our C# app, we write files to Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments). Our log4net logfile should go there too, so we've defined application.conf as follows:
<appender name="LogFile" type="log4net.Appender.RollingFileAppender">
<appendToFile value="true"/>
<file value="%USERPROFILE%\My Documents\MyApp\log.txt"/>
...snip...
</appender>
This works, until we run in on a PC which has a non-English Windows. Because then, SpecialFolder.MyDocuments points to the folder Mijn Documenten, while the log still goes to My Documents. Confusion ensues, because now our files are in two places.
I want to write my log to the "real" My Documents folder. How do I do this?
I tried to find an environment variable like %USERPROFILE%, but there doesn't seem to exist one for My Documents.
There's a registry key that defines the true location of My Documents but it's not accessible from application.conf.
I tried to override the File parameter of my appender programmatically, like this:
public static void ConfigureLogger()
{
XmlConfigurator.Configure();
Hierarchy hierarchy = (Hierarchy)log4net.LogManager.GetRepository();
foreach (var appender in hierarchy.Root.Appenders)
{
if (appender is FileAppender)
{
var fileAppender = appender as FileAppender;
var logDirectory = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments), "MyApp");
var pathElements = fileAppender.File.Split(Path.DirectorySeparatorChar);
var logFileName = pathElements.Last();
var fullLogFilePath = Path.Combine(logDirectory, logFileName);
fileAppender.File = fullLogFilePath;
}
}
}
This doesn't work either: when I inspect the internals of my logger, the File property happily reports Mijn Documenten, but in the mean time the logs still go to My Documents.
I'm running out of ideas!
Change the line
<file value="%USERPROFILE%\My Documents\MyApp\log.txt"/>
to
<file type="log4net.Util.PatternString" value="%envFolderPath{MyDocuments}\MyApp\log.txt" />
The accepted answer is out of date.
You should now use:
<file value="${UserProfile}\Documents\log-messages.log" />
(This should work even on Windows 7, where "Documents" is referred to by the alias "My Documents".)
${UserProfile} will map to C:\Users[UserName], even if you don't see this variable explicitly defined in your environment variable list.
There seem to be at least 2 approaches. The simplest is kind of a hack:
Specify a custom environment variable to indicate the root path in you log4net config:
<file value="%MYAPP_USER_ROOTFOLDER%\MyApp\log.txt"/>
At startup, before initializing logging, set this environment variable value:
Environment.SetEnvironmentVariable("MYAPP_USER_ROOTFOLDER", Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments));
The more complicated but recommended approach is explained here: http://marc.info/?l=log4net-user&m=110142086820117&w=2 and here http://ziqbalbh.com/articles/log4net-another-way-to-change-log-file-location-on-runtime/
Does anyone know how to dump environment name in log4net?
e.g.
<conversionPattern value="%date{ISO8601} Server:%property{log4net:HostName} [%-5level] %m" />
This will give computer name.
Not sure what do you mean by "environment name", but with log4net contexts you can log anything: logged on user, ip address if dynamic etc.
Not sure how to add it to the e-mail subject, but in the body pattern would be:
<conversionPattern value="%date{ISO8601} Server:%property{environment} [%-5level] %m" />
after you init log4net, call this:
log4net.GlobalContext.Properties["environment"] =
Environment.GetEnvironmentVariables("EnvironmentName")+"AppName";
It seems you can just add $(COMPUTERNAME) to your conversionPattern. If it is the computername you were looking for?
I currently use log4net with a RollingFileAppender.
As each Log call is made I'd like to store this in memory. At the end of my console application run I'd like to (if an app.config setting is true) take only the Warns and Fatals and send all these messages in an Email. I notice MemoryAppender but not quite sure how to use it. Also see SMTPAppender but not sure it is the right tool, else I'll use MemoryAppender and somehow filter out only events of Levels Warn/Fatal and then email using the SmtpClient class.
How to achieve this?
Thanks
Update
My last part of log4net config now looks like.
<appender name="MemoryAppender" type="log4net.Appender.MemoryAppender" >
<onlyFixPartialEventData value="true" />
<threshold value="WARN" />
</appender>
<root>
<level value="DEBUG" />
<appender-ref ref="Console" />
<appender-ref ref="RollingFile" />
<appender-ref ref="MemoryAppender" />
</root>
In code I do:
private static MemoryAppender MemoryAppender
{
get
{
if (memoryAppender == null)
{
Hierarchy h = LogManager.GetRepository() as Hierarchy;
memoryAppender = h.Root.GetAppender("MemoryAppender") as MemoryAppender;
}
return memoryAppender;
}
}
Then when I want the events I call:
MemoryAppender.GetEvents();
I've tried MemoryAppender.GetEvents()[0].RenderedMessage but that is not the correct output, how do I get the message string as it was written to the File/Console logs with the correct pattern and time etc and build myself a StringBuilder? I'll then put this in the body of my Email and send it using the SmtpClient. RenderMessage is just giving me the string that was provided to the Log.Warn() call not what was written to the log. Is this due to not setting a layout pattern on the MemoryAppender?
Thanks
MemoryAppender will only "append" to memory and is thus mostly useful only for development and testing purposes. And there is currently no appender that will only append on application shutdown.
The SMTPAppender is something in between, since it inherits the BufferingAppenderSkeleton. These appenders have a BufferSize property which controls how many messages are kept in memory before they are flushed.
Which messages to pass to the appenders are controlled with the level settings either on the root element or on individual logger elements. In your case use a level of WARN which will let through WARN, ERROR and FATAL. If you don't want the ERROR messages you will have to put a level filter on your appender.
Update: MemoryAppender is not using any layout to "render" message objects. What you get from MemoryAppender is just the raw message objects as they are produced by log4net. You will have to convert those to meaningfull text yourself.
Alternatively, if you require both layout functionality and in-memory appending you could look into subclassing AppenderSkeleton. That way you get the basic Layout support. When implementing the Append method you can do what MemoryAppender does, that is just appending to an internal list of messages.
Update 2: to implement the MemoryAppender alternative I suggest taking the MemoryAppender as a starting point. MemoryAppender is a subclass of AppenderSkeleton and have thus access to the RenderLoggingEvent method. So, we subclass MemoryAppender and add a method that renders the current batch of logging events:
public class RenderingMemoryAppender : MemoryAppender
{
public IEnumerable<string> GetRenderedEvents()
{
foreach(var loggingEvent in GetEvents())
{
yield return RenderLoggingEvent(loggingEvent);
}
}
}
You can use SMTPAppender and look at how the flush functionality works. Log4net keeps all messages in memory until flush is called (if it's setup this way), so the email will be sent when you flush it.
Another thing you can do is create a separate appender (Rolling or FileAppender) with filters WARN and FATAL, then attach this appender to the same logger, and at the end of your run email this file if it's non-empty (and you can choose to send it as an attachment or right in the body of the email). Let me know if you want more details, this is almost the same I'm doing know.
Good luck!
Ricardo.