Develop Reference

NLog Async target timestamps

I'm curious about timestamping in NLog when using async targets.
I know that according to What's the meaning of the time stamp in nlog when async is on? the timestamp is generated when the log entry is queued, as you would expect.
However, I noticed something in one of my log files, so I decided to whip up a quick test.
static void Main(string[] args)
{
for (int i = 0; i < 10000; i++)
{
_logger.Info("Timestamp: {0}, LogNumber: {1}",DateTime.Now.ToString("HH:mm:ss.fff"), i);
}
_logger.Factory.Flush();
}
My NLog.config looks like:
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<targets async="true">
<target xsi:type="File" name="f" fileName="${basedir}/logs/${shortdate}.log"
layout="${longdate} ${uppercase:${level}} ${message}" />
</targets>
<rules>
<logger name="*" minlevel="Trace" writeTo="f" />
</rules>
</nlog>
Now if I look at the output, we see that all entries between 754, and 9962 have the same NLog timestamp, however, DateTime.Now shows the milliseconds progressing:
2015-02-12 08:19:23.3814 INFO Timestamp: 08:19:23.376, LogNumber: 0
...
2015-02-12 08:19:23.3853 INFO Timestamp: 08:19:23.384, LogNumber: 754
...
2015-02-12 08:19:23.4033 INFO Timestamp: 08:19:23.399, LogNumber: 9963
...
I can understand that with the overheads, a DateTime.Now stamp of .384 could be logged an .385, however, it doesn't make sense to me that .399 comes out as .385.
The way that the NLog timestamp progresses, almost looks like the timestamp is generated during a logging cycle, rather than the log call. Which would be contrary to the above article.
So, is this something to do with the time source NLog uses, or rather, when the timestamp is generated?

Behavior you see is defined by time source system of NLog.
NLog 4.x provides 4 different time sources, and you can plug in your own. The default time source of NLog is optimized for performance and does some caching of the values that are being returned. The caching mechanism uses Environment.TickCount intervals to return time stamps for new log entries. Resolution of TickCount property is limited to resolution of system timer which is not a fixed value but typically varies in the 10-16 ms range.
So, the effect you see is defined by default time source of NLog. And, btw, it's not specific to async targets - remove async property from your example and you will see exactly same behavior.
If you need a more precise resolution for time stamps in your log events, you can configure NLog to use another time source:
<nlog>
<time type="AccurateUTC" />
</nlog>
However, this will not guarantee that all your events will have different time stamps, you might get 2-3 events with same time. But it will use straight value of DateTime.UtcNow to fill in the date/time field of log event, without any caching. And without any date time zone conversions too...
You can find more info about time sources in NLog wiki

How to access existing NLog MemoryTarget

I'm reworking an existing logging system to use NLog Instead. I've manually added a reference to NLog in my configuration file:
<targets>
<target xsi:type="File" name="fileLogger" fileName="${basedir}\TRACE\${date:format=yyyy-MM-dd}.log" layout="TimeStamp:[${date}]|${message}" />
<target xsi:type="Memory" name="MemLogger" layout="TimeStamp:[${date}]|${message}|${Type}" />
</targets>
<rules>
<logger name="ApplicationLogger" levels="Info,Warn,Error" writeTo="fileLogger" />
<logger name="ApplicationLogger" levels="Info,Warn,Error" writeTo="fileLogger" />
</rules>
What I want to do is in code pull out the MemLogger Logs and access the different parts (Type, Message, Timestamp). How would I accomplish this? I've seen how to create a new log from scratch, but I don't want to create a new memorytarget, I want to access the existing one in my config, pull the log data out of it, and then clear the memory (so that I don't have a memory leak).
How to I access the MemLogger MemoryTarget in C#?

Unfortunately you cannot access different parts of log. Logs stored as rendered strings, with all layout renderers already replaced with their values. All you can do is manually parse each log string. E.g.
var target =(MemoryTarget)LogManager.Configuration.FindTargetByName("MemLogger");
foreach (string log in target.Logs)
{
var parts = log.Split('|');
var date = parts[0].Replace("TimeStamp:[", "").TrimEnd(']');
var message = parts[1];
var type = parts[2];
//...
}

Something like this:
var target = LogManager.Configuration.FindTargetByName("MemLogger");
Optionally, you can cast received target to MemoryTarget type.

I've just had to do this myself:
Natively:
IList<string> logs = LogManager.Configuration.FindTargetByName<MemoryTarget>("MyTarget").Logs;
or using a Cast:
IList<string> logs = ((MemoryTarget)LogManager.Configuration.FindTargetByName("MyTarget")).Logs;

How to log complex synchronization process?

I'm developing a complicated distributed service that makes iterative synchronization process. It synchronise every 10 seconds business entities in different information systems. One iteration is consist of bunch of 3d party service calls to retrieve current state of business objects (count of customers, goods, certain customer and goods details etc.), queries to local DB and then get differences between them and smooth out, synchronize this differences.
There are different types of iterations. They are fast (only changes in set of objects) and slow iterations (full revieweing of data). Fast are every 10 seconds and slow are once a day.
So, how can I log this processes using NLog? I'm using SQLite for storing data. But I'm stuck in DB design for logs.
So I want to log flow of every iteration:
1. Request for current state of objects to 3d party service
2. Query the local database for current state of objects
3. Get differences list
4. Invoke external service to commit insufficient data
5. Update local database for insufficient data
But there is so many kinds of info to log so I can't just put it into one TEXT field.
At the moment I'm using such structure for logs:
CREATE TABLE [Log] (
[id] INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
[ts] TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
[iteration_id] varchar,
[request_response_pair] varchar,
[type] VARCHAR NOT NULL,
[level] TEXT NOT NULL,
[server_id] VARCHAR,
[server_alias] VARCHAR,
[description] TEXT,
[error] Text);
So every service request and response puts to description and request_response_pair is a key to link every response to every request.
Here is my NLog config:
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" internalLogFile="D:\nlog.txt" internalLogLevel="Trace">
<targets>
<target name="Database" xsi:type="Database" keepConnection="false"
useTransactions="false"
dbProvider="System.Data.SQLite.SQLiteConnection, System.Data.SQLite, Version=1.0.82.0, Culture=neutral, PublicKeyToken=db937bc2d44ff139"
connectionString="Data Source=${basedir}\SyncLog.db;Version=3;"
commandText="INSERT into Log(iteration_id, request_response_pair, type, level, server_id, server_alias, description, error) values(#Iteration_id, #Request_response_pair, #Type, #Loglevel, #server_id, #server_alias, #Description, #Error)">
<parameter name="#Type" layout="${message}"/>
<parameter name="#Loglevel" layout="${level:uppercase=true}"/>
<parameter name="#Request_response_pair" layout="${event-context:item=request_response_pair}"/>
<parameter name="#Iteration_id" layout="${event-context:item=iteration_id}"/>
<parameter name="#server_id" layout="${event-context:item=server_id}"/>
<parameter name="#server_alias" layout="${event-context:item=server_alias}"/>
<parameter name="#Description" layout="${event-context:item=description}"/>
<parameter name="#Error" layout="${event-context:item=error}"/>
</target>
</targets>
<rules>
<logger name="*" minlevel="Trace" writeTo="Database" />
</rules>
</nlog>
Here is how I log:
namespace NLog
{
public static class LoggerExtensions
{
public static void InfoEx(this Logger l, string message, Dictionary<string, object> contextParams)
{
LogEventInfo eventInfo = new LogEventInfo(LogLevel.Info, "", message);
foreach (KeyValuePair<string, object> kvp in contextParams)
{
eventInfo.Properties.Add(kvp.Key, kvp.Value);
}
l.Log(eventInfo);
}
public static void InfoEx(this Logger l, string message, string server_id, string server_alias, Dictionary<string, object> contextParams = null)
{
Dictionary<string, object> p = new Dictionary<string, object>();
p.Add("server_id", server_id);
p.Add("server_alias", server_alias);
if (contextParams != null)
{
foreach (KeyValuePair<string, object> kvp in contextParams)
{
p.Add(kvp.Key, kvp.Value);
}
}
l.InfoEx(message, p);
}
}
}
I know about logging levels but I need all this verbose logs, so I log it as info. I can't find any tutorial how to log these complicated, structured logs. Only plain dumb log messges.

I am assuming, you are talking about "Logs" for the typical "log things so we have something to look at if we need to inspect our workflow (errors/performance)". I assume that you do NOT mean logs as e.g. in "We need accounting information and the log is part of our domain data and is included in the workflow."
And from what I got from your posting, you are worrying about the backend storage format on the logs, so that you can later process them and use for said diagnostics?
Then I'd recommend that you make the logging code independend of the domain specifics.
Question: How will the logs you create be processed? Do you really need to access them all over the place so you need the database to provide you a structured view? Is it in any kind relevant how fast you can filter your logs? Or will they end up in one big log-analyzer application anyway, that is ran only ever second week when something bad happened?
In my opinion, the biggest reasons you want to avoid any domain specifics in the log are that "logs should work if things go wrong" and "logs should work after things changed".
Logs should work if things go wrong
If you have columns in your log table for domain specific values like "Request_response_pair", and there is no pair, then to write the log itself might fail (e.g. if its an index field). Of course, you can make sure to not have non-null columns and no restrictions in your DB-design, but take a step back and ask: Why do you want the structure in your log database anyway? Logs are meant to work as reliable as possible, so any kind of template you press them on may restrict the use cases or may make you not be able to log crucial information.
Logs should work after things changed
Especially if you need logs for detecting and fixing bugs or improve performance, that means that you will regulary compare logs from "before the change" to logs from "after the change". If you need to change the structure of your log database, because you changed your domain data, this is going to hurt you when you need to compare the logs.
True, if you make a data structure change, you probably still need to update some tools like log analyzer and such, but there is usually a large part of logging/analyzing code that is completely agnostic to the actual structure of the domain.
Many systems (including complex ones) can live with "just log one simple string" and later write tools to take the string apart again, if they need to filter or process the logs.
Other systems write logs in simple string key/value pairs. The log function itself is not domain specific but just accepts a string dictionary and writes it off (or even easier, a params string[] which should have an even number of parameters and you use every second parameter as key - if you aren't scared by that proposition :-D).
Of course, you will probably start writing another tooling layer on top of the base log functions that knows about domain specific data structures and then composes the string dictionary and pass it on. You certainly don't want to copy the decomposing code all place around. But make the base functions available at all places where you might want to log something. Its really helpfull if you indeed run into "strange" situations (exception handler) where some information is missing.

Best way to store associative array in app.config

I've been working on my software lately and I have been wondering what the best way is to store an associative array.
The only thing I could come up with out of the blue is to do something like this:
<add key="disks" value="C|1|10,D|2|20,E|1|5,Z|1|3"/>
But this doesn't offer a lot of readability in my config file and I want my config file to be readable as it is a console application.
The reason for this because I've written a program that checks the diskspace of the disks specified in the app.config file but I want different thresholds for different disks.
How would you solve it?
Here's a part of my current config file.
<!-- DISK FEATURE SETTINGS -->
<!-- Type 1 is threshold by percentage and type 2 is threshold by a certain limit -->
<add key="threshold_type" value="1" />
<add key="threshold_limit" value="0,1" />
<!-- Space_type defines if you want to limit using kilobytes (1), megabytes (2) or gigabytes (3) if using threshold_type 2 -->
<add key="space_type" value="3" />
<!-- Put the disks here delimited by a comma like this: C,D,E -->
<add key="disks" value="C,D,E,Z"/>
<!-- SERVICE FEATURE SETTINGS -->
<!-- Put the services here delimited by a comma like this: C,D,E -->
<add key="services" value="spooler,ekrn,RadeonPro Support Service,TeamViewer6"/>
<!-- Put this on 1 if you want to log your output to a text file -->
<add key="logging" value="1"/>
I want to use the same principle for my performancecounter program that uses the perfmon counters to get some data and store it in a text file.
I hope people can help me for a bit here :)

I suggesst you to create your own configuration section. Custom configuration gives more readability and type safety. Here are links to create custom configuration
http://msdn.microsoft.com/en-us/library/2tw134k3.aspx and
http://haacked.com/archive/2007/03/11/custom-configuration-sections-in-3-easy-steps.aspx (old one but easy to follow).

As far as standard configuration mechanism works with XML serialization, the best (and, IMHO, the wise) way to store dictionaries in App.config is a List<KeyValuePair<K,V>>.

you may want to use Hashtable from system.collection or List<>
below are few pointers for hashtable,
http://www.dotnetperls.com/hashtable
http://www.tutorialspoint.com/csharp/csharp_hashtable.htm
i hope this helps!! thanks :)

c# log4net store entries in memory and Email under certain conditions

I currently use log4net with a RollingFileAppender.
As each Log call is made I'd like to store this in memory. At the end of my console application run I'd like to (if an app.config setting is true) take only the Warns and Fatals and send all these messages in an Email. I notice MemoryAppender but not quite sure how to use it. Also see SMTPAppender but not sure it is the right tool, else I'll use MemoryAppender and somehow filter out only events of Levels Warn/Fatal and then email using the SmtpClient class.
How to achieve this?
Thanks
Update
My last part of log4net config now looks like.
<appender name="MemoryAppender" type="log4net.Appender.MemoryAppender" >
<onlyFixPartialEventData value="true" />
<threshold value="WARN" />
</appender>
<root>
<level value="DEBUG" />
<appender-ref ref="Console" />
<appender-ref ref="RollingFile" />
<appender-ref ref="MemoryAppender" />
</root>
In code I do:
private static MemoryAppender MemoryAppender
{
get
{
if (memoryAppender == null)
{
Hierarchy h = LogManager.GetRepository() as Hierarchy;
memoryAppender = h.Root.GetAppender("MemoryAppender") as MemoryAppender;
}
return memoryAppender;
}
}
Then when I want the events I call:
MemoryAppender.GetEvents();
I've tried MemoryAppender.GetEvents()[0].RenderedMessage but that is not the correct output, how do I get the message string as it was written to the File/Console logs with the correct pattern and time etc and build myself a StringBuilder? I'll then put this in the body of my Email and send it using the SmtpClient. RenderMessage is just giving me the string that was provided to the Log.Warn() call not what was written to the log. Is this due to not setting a layout pattern on the MemoryAppender?
Thanks

MemoryAppender will only "append" to memory and is thus mostly useful only for development and testing purposes. And there is currently no appender that will only append on application shutdown.
The SMTPAppender is something in between, since it inherits the BufferingAppenderSkeleton. These appenders have a BufferSize property which controls how many messages are kept in memory before they are flushed.
Which messages to pass to the appenders are controlled with the level settings either on the root element or on individual logger elements. In your case use a level of WARN which will let through WARN, ERROR and FATAL. If you don't want the ERROR messages you will have to put a level filter on your appender.
Update: MemoryAppender is not using any layout to "render" message objects. What you get from MemoryAppender is just the raw message objects as they are produced by log4net. You will have to convert those to meaningfull text yourself.
Alternatively, if you require both layout functionality and in-memory appending you could look into subclassing AppenderSkeleton. That way you get the basic Layout support. When implementing the Append method you can do what MemoryAppender does, that is just appending to an internal list of messages.
Update 2: to implement the MemoryAppender alternative I suggest taking the MemoryAppender as a starting point. MemoryAppender is a subclass of AppenderSkeleton and have thus access to the RenderLoggingEvent method. So, we subclass MemoryAppender and add a method that renders the current batch of logging events:
public class RenderingMemoryAppender : MemoryAppender
{
public IEnumerable<string> GetRenderedEvents()
{
foreach(var loggingEvent in GetEvents())
{
yield return RenderLoggingEvent(loggingEvent);
}
}
}

You can use SMTPAppender and look at how the flush functionality works. Log4net keeps all messages in memory until flush is called (if it's setup this way), so the email will be sent when you flush it.
Another thing you can do is create a separate appender (Rolling or FileAppender) with filters WARN and FATAL, then attach this appender to the same logger, and at the end of your run email this file if it's non-empty (and you can choose to send it as an attachment or right in the body of the email). Let me know if you want more details, this is almost the same I'm doing know.
Good luck!
Ricardo.