I'm developing a complicated distributed service that makes iterative synchronization process. It synchronise every 10 seconds business entities in different information systems. One iteration is consist of bunch of 3d party service calls to retrieve current state of business objects (count of customers, goods, certain customer and goods details etc.), queries to local DB and then get differences between them and smooth out, synchronize this differences.
There are different types of iterations. They are fast (only changes in set of objects) and slow iterations (full revieweing of data). Fast are every 10 seconds and slow are once a day.
So, how can I log this processes using NLog? I'm using SQLite for storing data. But I'm stuck in DB design for logs.
So I want to log flow of every iteration:
1. Request for current state of objects to 3d party service
2. Query the local database for current state of objects
3. Get differences list
4. Invoke external service to commit insufficient data
5. Update local database for insufficient data
But there is so many kinds of info to log so I can't just put it into one TEXT field.
At the moment I'm using such structure for logs:
CREATE TABLE [Log] (
[id] INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
[ts] TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
[iteration_id] varchar,
[request_response_pair] varchar,
[type] VARCHAR NOT NULL,
[level] TEXT NOT NULL,
[server_id] VARCHAR,
[server_alias] VARCHAR,
[description] TEXT,
[error] Text);
So every service request and response puts to description and request_response_pair is a key to link every response to every request.
Here is my NLog config:
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" internalLogFile="D:\nlog.txt" internalLogLevel="Trace">
<targets>
<target name="Database" xsi:type="Database" keepConnection="false"
useTransactions="false"
dbProvider="System.Data.SQLite.SQLiteConnection, System.Data.SQLite, Version=1.0.82.0, Culture=neutral, PublicKeyToken=db937bc2d44ff139"
connectionString="Data Source=${basedir}\SyncLog.db;Version=3;"
commandText="INSERT into Log(iteration_id, request_response_pair, type, level, server_id, server_alias, description, error) values(#Iteration_id, #Request_response_pair, #Type, #Loglevel, #server_id, #server_alias, #Description, #Error)">
<parameter name="#Type" layout="${message}"/>
<parameter name="#Loglevel" layout="${level:uppercase=true}"/>
<parameter name="#Request_response_pair" layout="${event-context:item=request_response_pair}"/>
<parameter name="#Iteration_id" layout="${event-context:item=iteration_id}"/>
<parameter name="#server_id" layout="${event-context:item=server_id}"/>
<parameter name="#server_alias" layout="${event-context:item=server_alias}"/>
<parameter name="#Description" layout="${event-context:item=description}"/>
<parameter name="#Error" layout="${event-context:item=error}"/>
</target>
</targets>
<rules>
<logger name="*" minlevel="Trace" writeTo="Database" />
</rules>
</nlog>
Here is how I log:
namespace NLog
{
public static class LoggerExtensions
{
public static void InfoEx(this Logger l, string message, Dictionary<string, object> contextParams)
{
LogEventInfo eventInfo = new LogEventInfo(LogLevel.Info, "", message);
foreach (KeyValuePair<string, object> kvp in contextParams)
{
eventInfo.Properties.Add(kvp.Key, kvp.Value);
}
l.Log(eventInfo);
}
public static void InfoEx(this Logger l, string message, string server_id, string server_alias, Dictionary<string, object> contextParams = null)
{
Dictionary<string, object> p = new Dictionary<string, object>();
p.Add("server_id", server_id);
p.Add("server_alias", server_alias);
if (contextParams != null)
{
foreach (KeyValuePair<string, object> kvp in contextParams)
{
p.Add(kvp.Key, kvp.Value);
}
}
l.InfoEx(message, p);
}
}
}
I know about logging levels but I need all this verbose logs, so I log it as info. I can't find any tutorial how to log these complicated, structured logs. Only plain dumb log messges.
I am assuming, you are talking about "Logs" for the typical "log things so we have something to look at if we need to inspect our workflow (errors/performance)". I assume that you do NOT mean logs as e.g. in "We need accounting information and the log is part of our domain data and is included in the workflow."
And from what I got from your posting, you are worrying about the backend storage format on the logs, so that you can later process them and use for said diagnostics?
Then I'd recommend that you make the logging code independend of the domain specifics.
Question: How will the logs you create be processed? Do you really need to access them all over the place so you need the database to provide you a structured view? Is it in any kind relevant how fast you can filter your logs? Or will they end up in one big log-analyzer application anyway, that is ran only ever second week when something bad happened?
In my opinion, the biggest reasons you want to avoid any domain specifics in the log are that "logs should work if things go wrong" and "logs should work after things changed".
Logs should work if things go wrong
If you have columns in your log table for domain specific values like "Request_response_pair", and there is no pair, then to write the log itself might fail (e.g. if its an index field). Of course, you can make sure to not have non-null columns and no restrictions in your DB-design, but take a step back and ask: Why do you want the structure in your log database anyway? Logs are meant to work as reliable as possible, so any kind of template you press them on may restrict the use cases or may make you not be able to log crucial information.
Logs should work after things changed
Especially if you need logs for detecting and fixing bugs or improve performance, that means that you will regulary compare logs from "before the change" to logs from "after the change". If you need to change the structure of your log database, because you changed your domain data, this is going to hurt you when you need to compare the logs.
True, if you make a data structure change, you probably still need to update some tools like log analyzer and such, but there is usually a large part of logging/analyzing code that is completely agnostic to the actual structure of the domain.
Many systems (including complex ones) can live with "just log one simple string" and later write tools to take the string apart again, if they need to filter or process the logs.
Other systems write logs in simple string key/value pairs. The log function itself is not domain specific but just accepts a string dictionary and writes it off (or even easier, a params string[] which should have an even number of parameters and you use every second parameter as key - if you aren't scared by that proposition :-D).
Of course, you will probably start writing another tooling layer on top of the base log functions that knows about domain specific data structures and then composes the string dictionary and pass it on. You certainly don't want to copy the decomposing code all place around. But make the base functions available at all places where you might want to log something. Its really helpfull if you indeed run into "strange" situations (exception handler) where some information is missing.
Related
I've got an application that runs without problem with the File and Console Sinks and now I'm trying to add the MSSqlServer Sink.
Looking at the documentation on Github I've got my application to write to the SQL Database as well as the other sinks.
<add key="serilog:write-to:MSSqlServer.connectionString" value="Server=servername;Database=databasename;User Id=userid;Password=password;"/>
<add key="serilog:write-to:MSSqlServer.tableName" value="Logs"/>
<add key="serilog:write-to:MSSqlServer.autoCreateSqlTable" value="true"/>
One improvement is I'd like to add a custom column to the Logs table that Serilog uses to store a number indicating a unique RunId for my application. The idea being that a simple query would allow grouping by RunId to see all messages for that one run.
So I added the following, based on the documentation (and I haven't been able to find any other examples) as it seemed logical:
<add key="serilog:write-to:MSSqlServer.columnOptions.ColumnName" value="RunId"/>
<add key="serilog:write-to:MSSqlServer.columnOptions.PropertyName" value="RunId"/>
<add key="serilog:write-to:MSSqlServer.columnOptions.DataType" value="SqlDbType.Int"/>
<add key="serilog:write-to:MSSqlServer.columnOptions.DataLength" value="32"/>
and then in my code all I need to do is:
Log.Information("{RunId}{Message}", RunId, Message);
to see a new entry with {RunId} in the RunId column and {Message} in the Message column... however everytime I do this nothing is written to the RunId column, it remains as NULL whereas every log message the console/file has is also duplicated in the Table.
So it seems logging is working, it must be the keys wrong and I'm really not sure what should be used.
Would anyone be able to point me in the direction I need to be going or where I've gone wrong?
Thank you.
Logging definitely was working but some digging and finally I found out I had two issues:
Permissions on database where insuffident for the user serilog was using, and
AppSettings settings I was using were wrong
#1 was easy enough to fix, I created a dedicated Serilog account for this database and fixed the permissions as per documentation.
#2 however was very frustrating but eventually I was able to get the following to work:
<configSections>
<section name="MSSqlServerSettingsSection" type="Serilog.Configuration.MSSqlServerConfigurationSection, Serilog.Sinks.MSSqlServer"/>
</configSections>
<MSSqlServerSettingsSection DisableTriggers="false" ClusteredColumnstoreIndex="false" PrimaryKeyColumnName="Id">
<!-- SinkOptions parameters -->
<TableName Value="Logs"/>
<Columns>
<add ColumnName="RunId" DataType="int"/>
</Columns>
</MSSqlServerSettingsSection>
On compile and execution my program will now create the Logs table in the database and then create a RunId column which I have been able to populate with a {RunId} expression in my Log.Debug() calls.
FWIW: I hope this will be helpful and if I can work out how I'll see if I can add this to documentation as an example of using AppSettings for this use case. Searching this question most people seem to be using JSON and not AppSettings.
I'm pretty new to NLog so please forgive my basic question.
I've inherited a Winforms application written by some contractors, I'm mainly a database developer but I've managed to do some development of the application but I sometimes struggle with tracking down error messages encountered by users, therefore I'm attempting to retrofit logging into the application via NLog, I first encountered the issue in NLog 4.4.5 and I've since updated to 4.4.12 and that hasn't solved the problem. I've verified that NLog is catching the errors correctly as it will output to a text file but when I try to direct it to a database output I can't get it to work.
This is my database table:
My problem is that I can only get errors written to the database only if I don't pass any parameters to the insert statement (which is pretty useless). That is to say that the following in my NLog.config file works:
<target name="database" xsi:type="Database">
<commandText>INSERT INTO [tblException] (DbVersionID, ExceptionDateTime) SELECT MAX(DbVersionID), GETDATE() FROM tblDbVersion</commandText>
<dbProvider>System.Data.SqlServerCe.4.0</dbProvider>
<connectionString>Data Source=${basedir}\Database.sdf</connectionString>
</target>
But this doesnt':
<target name="database" xsi:type="Database">
<commandText>INSERT INTO [tblException] (DbVersionID, ExceptionDateTime, Message) SELECT MAX(DbVersionID), GETDATE(), #message FROM tblDbVersion</commandText>
<parameter name="#message" layout="${message}" />
<dbProvider>System.Data.SqlServerCe.4.0</dbProvider>
<connectionString>Data Source=${basedir}\Database.sdf</connectionString>
</target>
I've enabled internal logging and the following is what I get:
2017-11-28 11:26:45.8063 Trace Executing Text: INSERT INTO [tblException] (DbVersionID, ExceptionDateTime, Message) SELECT MAX(DbVersionID), GETDATE(), #Message FROM tblDbVersion
2017-11-28 11:26:45.8063 Trace Parameter: '#message' = 'Test Error Message' (String)
2017-11-28 11:26:45.8063 Error Error when writing to database. Exception: System.Data.SqlServerCe.SqlCeException (0x80004005): A parameter is not allowed in this location. Ensure that the '#' sign is in a valid location or that parameters are valid at all in this SQL statement.
at System.Data.SqlServerCe.SqlCeCommand.ProcessResults(Int32 hr)
at System.Data.SqlServerCe.SqlCeCommand.CompileQueryPlan()
at System.Data.SqlServerCe.SqlCeCommand.ExecuteCommand(CommandBehavior behavior, String method, ResultSetOptions options)
at System.Data.SqlServerCe.SqlCeCommand.ExecuteNonQuery()
at NLog.Targets.DatabaseTarget.WriteEventToDatabase(LogEventInfo logEvent)
at NLog.Targets.DatabaseTarget.Write(LogEventInfo logEvent)
It appears that the parameter isn't getting replaced in the query before it's being run. I tried adding single quotes in the command text just in case that would help but it just resulted in the literal string '#message' being inserted into the database field.
I can't see anything that I've done differently to the examples, so any help would be appreciated.
Regards,
Alex
As per the comment from #pmcilreavy
Have you tried temporarily removing the MAX and replacing with a literal and changing to INSERT INTO .. (blah) VALUES (blah, #Message); SqlCe has quite a few limitations and quirks compared to Sql Server so might be worth simplifying things where possible.
The issue appears to be that SQLCE doesn't support parameters in the select part of a select statement.
I need to:
get data from users (from a UI), and put that data into Properties
import the Properties into some C# code using a Custom Action
do some stuff to the Properties (encrypt the values),
export the values back to WiX, where I will
create a registry key and put the encrypted values into them
I can accomplish everything on that list except for #4. That is, I can't seem to import and export values into the C# code. I think the problem is in the timing.
Here's an example of a Custom Action that is used to import some properties into some C# code:
<Property Id="VALUE" Value="value"/>
<SetProperty Id="CustomAction_PassProperty"
Value="VALUE=[VALUE]"
Sequence="execute"
Before="CustomAction_PassProperty"/>
<Binary Id="Binary_PassProps"
SourceFile="$(var.CreateRegistryKey.TargetDir)CreateRegistryKey.CA.dll"/>
<!-- Note that 'Impersonate="no"' elevates the privilege of the C# code, needed to create keys -->
<CustomAction Id="CustomAction_PassProperty"
BinaryKey="Binary_PassProps"
DllEntry="CreateKeys"
Execute="deferred"
Impersonate="no"
Return="check"
HideTarget="yes"/>
<InstallExecuteSequence>
<Custom Action="CustomAction_PassProperty"
After="InstallInitialize"/>
</InstallExecuteSequence>
Notice that the action is done after InstallInitialize.
Next, how to take the imported properties and convert them into variables in the C# code:
[CustomAction]
public static ActionResult CreateKeys(Session session)
{
string value = session.CustomActionData["VALUE"];
return ActionResult.Success;
}
Next, here's an example of how to export variables in C# code back into WiX, as properties:
<Binary Id="Binary_CustomActionTemplate"
SourceFile="$(var.CustomAction.TargetDir)CustomAction.CA.dll"/>
<CustomAction Id="CustomAction_CustomActionTemplate"
BinaryKey="Binary_CustomActionTemplate"
DllEntry="CustomActionTemplate"
Execute="immediate"
Return="check"/>
<InstallUISequence>
<Custom Action="CustomAction_CustomActionTemplate" After="LaunchConditions"/>
</InstallUISequence>
And this time the action is done after LaunchConditions.
Finally, how to create a Property, give it a value, and send it back to WiX in C#:
[CustomAction]
public static ActionResult CreateKeys(Session session)
{
session["VALUE"] = "Hello, world.";
return ActionResult.Success;
}
I think the problem lies in when -- that is, when during the installation sequence -- that I do both things (import and export), but I'm not sure. That is, I need to import the data into the C# code, do stuff in the C# code, then export data from C# code. But how?!? (waves hands dramatically at sky)
To sum up: how to import and export data into C# Custom Action in WiX (using the same C# code)?
AFAIK you cannot alter property values once in the execute sequence. Someone can correct me if I'm wrong and that would be an alternative answer to your question.
You should be able to encrypt the properties in the UI phase of the install. In your immediate custom action you can access the msi properties with session["PROPERTYNAME"] as opposed to session.CustomActionData["PROPERTYNAME"]. You can also set the properties: session["PROPERTYNAME"] = "new value string";.
Then on your Next/Install control on the page where the information is added you can put
<Control Id="Install" ...>
<Publish Event="DoAction" Value="EncodeAndSetValuesAction">1</Publish>
....
</Control>
If they are marked Secure='yes' and public (all caps name) then they should be usable in the execute phase of your installation with the encoded values.
Do note that if you launch your custom action with a DoAction control event (clicking next on the page where the user enters the values) you won't be able to get logging information in the msi log. See here for more info.
Small edit: I was writing this answer when you edited the question which shows you already know how to get the properties and set them in an immediate action.
Alternatively, you may be able to just schedule an immediate custom action to encrypt and set the values in the InstallExecuteSequence before InstallInitialize (or before WriteRegistryValues or anywhere?) and it will set the properties before the elevated Server portion of the install starts. I think the InstallExecuteSequence happens in two parts. One where it does a pass figuring out values and which files to install making an install script for the elevated Server portion of the install where it actually copies the files around and writes registry keys ect. This part is still the client context where you can run immediate custom actions and read/set properties. Secondly it will start the Server portion of the install with set in stone properties and follow the execution script that was just created.
I decided to put a short simplified explanation about the differences between immediate and deferred custom actions here as well.
There are basically two main differences between deferred and immediate custom action. Immediate custom actions can access the msi DB directly to read and/or modify properties but can't do anything that requires elevation unless the installer was run "as administrator" since it runs in the user's context.
Deferred actions can only run in the execute sequence and can only read property values set explicitly through CustomActionData with the same name as the action but they always have elevated privileges.
I use nlog logger in my project.
My program generates xml files based on data which i getting from sql server. I'm doing this with PLINQ. But also i have to log tracing info to be able make some investigations on exceptional cases on prod environment.
Result logs looks awful, when it came from multiple threads. For example:
Operation 1 started
Deserializing XXX
Operation 2 started
Deserializing XXX finished with status X
Filling XXX with data from Z
Deserializing YYY....
And its just for degree of parallelism 2.
I'd like to see result like this:
Operation 1 started
Deserializing XXX
Deserializing XXX finished with status X
Filling XXX with data from Z
Operation 1 finished
Operation 2 started
Deserializing YYY....
I see some solutions, but they're not looking good enough:
Save logging data to some buffer and flush it when parallel task ends - I will be ought to pass context to all inner methods (looks terrible!).
Add some kind of prefix to logging message to help getting context for some messages - i have to pass prefix to every inner message (also looks terrible).
Is there some clean solutions for this problem?
In NLog config files, there is the ${threadid} syntax. Use it like this:
<target name="file" xsi:type="File"
layout="${longdate} [${threadid}] ${level:uppercase=true} ${message} ${exception:format=tostring}"
fileName="${basedir}/logs/log.txt"
archiveFileName="${basedir}/logs/log.{#####}.txt"
archiveAboveSize="10485760"
archiveNumbering="Sequence"
concurrentWrites="true"
keepFileOpen="false" />
More info:
https://github.com/NLog/NLog/wiki/ThreadId-Layout-Renderer
I used it in production and generally it works. It is not perfect, but all operations (which I logged) is in sequence and this threadid describe which operation is in which ThreadId.
I'm curious about timestamping in NLog when using async targets.
I know that according to What's the meaning of the time stamp in nlog when async is on? the timestamp is generated when the log entry is queued, as you would expect.
However, I noticed something in one of my log files, so I decided to whip up a quick test.
static void Main(string[] args)
{
for (int i = 0; i < 10000; i++)
{
_logger.Info("Timestamp: {0}, LogNumber: {1}",DateTime.Now.ToString("HH:mm:ss.fff"), i);
}
_logger.Factory.Flush();
}
My NLog.config looks like:
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<targets async="true">
<target xsi:type="File" name="f" fileName="${basedir}/logs/${shortdate}.log"
layout="${longdate} ${uppercase:${level}} ${message}" />
</targets>
<rules>
<logger name="*" minlevel="Trace" writeTo="f" />
</rules>
</nlog>
Now if I look at the output, we see that all entries between 754, and 9962 have the same NLog timestamp, however, DateTime.Now shows the milliseconds progressing:
2015-02-12 08:19:23.3814 INFO Timestamp: 08:19:23.376, LogNumber: 0
...
2015-02-12 08:19:23.3853 INFO Timestamp: 08:19:23.384, LogNumber: 754
...
2015-02-12 08:19:23.4033 INFO Timestamp: 08:19:23.399, LogNumber: 9963
...
I can understand that with the overheads, a DateTime.Now stamp of .384 could be logged an .385, however, it doesn't make sense to me that .399 comes out as .385.
The way that the NLog timestamp progresses, almost looks like the timestamp is generated during a logging cycle, rather than the log call. Which would be contrary to the above article.
So, is this something to do with the time source NLog uses, or rather, when the timestamp is generated?
Behavior you see is defined by time source system of NLog.
NLog 4.x provides 4 different time sources, and you can plug in your own. The default time source of NLog is optimized for performance and does some caching of the values that are being returned. The caching mechanism uses Environment.TickCount intervals to return time stamps for new log entries. Resolution of TickCount property is limited to resolution of system timer which is not a fixed value but typically varies in the 10-16 ms range.
So, the effect you see is defined by default time source of NLog. And, btw, it's not specific to async targets - remove async property from your example and you will see exactly same behavior.
If you need a more precise resolution for time stamps in your log events, you can configure NLog to use another time source:
<nlog>
<time type="AccurateUTC" />
</nlog>
However, this will not guarantee that all your events will have different time stamps, you might get 2-3 events with same time. But it will use straight value of DateTime.UtcNow to fill in the date/time field of log event, without any caching. And without any date time zone conversions too...
You can find more info about time sources in NLog wiki