I'd like to save the logged message ${message} in several columns in a database, as shown on the following example:
My logger messages will follow the path User|Action, e.g:
logger.Info("John Doe|Logged in application"}
logger.Info("Mike Doe|Deleted a file"}
Now I'd like to save User in a column in my database, e.g logsTable.user, and Action in another column, e.g logsTable.action.
Is there any way to parse the ${message} with a regex or some other rules (to separate messages according to a specific character, in my example it's a "|") to save into parameters (in my case, i'd like the first part of the message to go in #user parameter and the second part in #action parameter)?
According to NLog documentation it shouldn't be too complicated to add your own properties to a log event. Then you could do an extension method on the correct NLog interface and write something like this (uncompiled):
public void LogSomething(this ILog logger, string username, string message)
{
LogEventInfo myEvent = new LogEventInfo(LogLevel.Debug, "", message);
myEvent.LoggerName = logger.Name;
myEvent.Properties.Add("User", username);
logger.Log(myEvent);
}
Now you should be able to refer to ${event-context:item=User}
Related
Recently I focused in Serilog to point out a templated path based on the current Date of every LogEvent.
After figuring how to implement this, I finally resolve the path on the fly by using the Date field into LogEvent by using Serilog.Sinks.Map, such as shown below:
return new LoggerConfiguration().WriteTo
.Map(
// Log key
(LogEvent le) => le.Timestamp.Date,
// Log Action
(DateTime date, LoggerSinkConfiguration lc) =>
{
string path = GetFilesPath(date, logName);
lc.File(path);
}
);
public string GetFilePath(DateTime date, string logName) =>
Path.Combine("./Logs", $"{date:yyyy-MM-dd}", $"{logName}.log");
With this, I achieved my goal: writing logs with in a sub folder based on the Date.
The issue is, since Serilog does not know that the pointing path changed, it does not close or dispose the file stream as expected. So, my application leaves files opened day to day, ad infinitum.
It'd be great if someone has faced this approach, to manually close the stream, or if Serilog API exposes somehow automatically close those streams.
Btw, I am using
Serilog 2.9.0
Serilog.Sinks.File 4.1.0
Serilog.Sinks.Map 1.0.1
Edit 05/06/2020 for those reading this afterwards.
Keying every single log event by the Timestamp is a bad idea. By doing so, we are in fact adding an entry per log event (supposing that no events are being emitted at the same time, for simplicity).
Even if we specify the sinkMapCountLimit to 0, which in theory won't keep any event in our map, if that event is configured to write to file (specially with the RollingFile sink), those sinks won't be disposed nor erased from memory.
So, the chunk of code above is leaking memory (and pretty fast).
The Map.Sink documentation warns about this, indeed.
...but isn't suitable when the set of possible key values is open-ended.
Serilog.Sinks.Map accepts a parameter sinkMapCountLimit to control this:
return new LoggerConfiguration().WriteTo
.Map(
// Log key
(LogEvent le) => le.Timestamp.Date,
// Log Action
(DateTime date, LoggerSinkConfiguration lc) =>
{
string path = GetFilesPath(date, logName);
lc.File(path);
},
sinkMapCountLimit: 5
);
For a certain portion of my code, I need to log all messages to a second file, specific to each instance of the action completed in two classes.
These classes (Parser and Importer) both have the following logger definition:
private static ILog _logger = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
Both also have this method:
public void AddAppender(IAppender appender)
{
((Logger)_logger.Logger).AddAppender(appender);
}
The whole project (ASP.NET WebForms) logging is initialized in the AssemblyInfo.cs:
[assembly: XmlConfigurator]
Now, when the user triggers a certain action, this happens:
var logFile = new FileInfo(ConfigurationManager.AppSettings["uploadDir"] + "/" + importId + "_log.txt");
var appender = new FileAppender() {
Threshold = Level.Info,
Layout = new PatternLayout("%message%newline"),
File = logFile.FullName
};
var parser = new Parser(...);
parser.AddAppender(appender);
var candidates = parser.Parse(File.ReadAllLines(uploadFile.FullName));
var importer = new Importer(...);
importer.AddAppender(appender);
importer.Import(candidates, false, ignoreWarnings.Checked);
My expectation would be, that a file will be created at the logFile location, that would contain all INFO and above message from either of the two classes.
This, however, is not the case - no file is created at all.
What am I doing wrong and how can I make this work?
Per the documentation of FileAppender:
This appender will first try to open the file for writing when
ActivateOptions is called. This will typically be during
configuration. If the file cannot be opened for writing the appender
will attempt to open the file again each time a message is logged to
the appender. If the file cannot be opened for writing when a message
is logged then the message will be discarded by this appender.
This documentation is slightly ambiguous, since it seems to imply the file will be opened when you attempt to log a message in all cases, but this isn't true -- unless you call ActivateOptions, the appender is never considered ready for appending.
How to dynamically change the FileName using a variable from C#? My idea is to create a log file like Log_<UserId_From_DB>_${date:format=yyyy-MM-dd}.log.
Any ideas?
Another option is to use the Global Diagnostic Context - $(GDC):
Set the value in C#
GlobalDiagnosticsContext.Set("UserId_From_DB","42");
In the config (nlog.config):
<target type="file" filename="Log_${gdc:item=UserId_From_DB}_${date:format=yyyy-MM-dd}.log" ..>
Please avoid modifying NLog Variables at runtime (See previous answer below). They should be seen as readonly, because they are not threadsafe. NLog Variables will also be affected if LoggingConfiguration is reloaded.
Previous answer with NLog Variables:
Set the value in C#
LogManager.Configuration.Variables["UserId_From_DB"] = "42";
In the config (nlog.config):
<target type="file" filename="Log_${var:UserId_From_DB}_${date:format=yyyy-MM-dd}.log" ..>
If the value is set again, the filename will automatically changed.
While the posted answer works, it suffers from concurrency issues. That variable is a global variable and you may end up with conflicts.
There is a better solution available. There is a way to pass event properties to NLog.
Link to the relevant NLog documentation.
Let's assume you want to log an error message:
Logger myLog = LogManager.GetLogger(name);
LogLevel level = LogLevel.Error;
string message = "This is an error message!";
You turn this information into a LogEventInfo object:
LogEventInfo logEvent = new LogEventInfo(level , myLog.Name, message);
You can then add properties to this event (the string indexes are free to choose):
logEvent.Properties["MySpecialValue"] = "SPECIAL";
And then you write to the log:
myLog.Log(logEvent);
The interesting thing here is that in your NLog configuration, you can use this custom property in any field that the Nlog documentation refers to as a "Layout" value.
You use ${event-properties:item=MySpecialValue} in the layout to access the property. For example:
<target xsi:type="File"
name="file"
fileName="${basedir}/logs/${event-properties:item=MySpecialValue}/my_${event-properties:item=MySpecialValue}_file.log"
layout="${event-properties:item=MySpecialValue} ${message}" />
Following the posted example, you will get a folder named SPECIAL, inside of which is a log file named my_SPECIAL_file.log in which you find the message SPECIAL This is an error message!. Just to prove the point that you can use this custom value in many different ways and shapes.
I commonly use this to make entity-specific logging (where the filename of the log equals the entity's ID value), which is essentially the same as you want to do here.
As a quick tip, I tend to wrap the NLog Logger in a class of my own:
public class UserLogger
{
private readonly Logger _log;
private readonly User _user;
public UserLogger(User u)
{
_user = u;
_log = LogManager.GetCurrentClassLogger();
}
public void Error(string message)
{
LogEventInfo logEvent =
new LogEventInfo(LogLevel.Error, _log.Name, message);
logEvent.Properties["UserId"] = _user.Id;
_log.Log(logEvent);
}
}
This is just a simple example to get you started. The cool feature I'm using here is that I've defined the log's filename (in the Nlog.Config target) using the UserId value and thus can ensure that each user gets logged to their own unique log file.
This way, you can enforce that the user ID is known when you want to log to the "user log" target. As an additional bonus, it also neatly decouples NLog dependencies from your calling code.
Assuming you have a log file called mylogfile.log in your nlog.config file
FileTarget target = LogManager.Configuration.FindTargetByName("mylogfile.log") as FileTarget;
String customlog = "Log_" + GetUserId(UserId_From_DB) + "_" + DateTime.Now.ToString("yyyy-MM-dd") + ".log";
target.FileName = customlog;
I am working on a program that will read in a text file and then insert areas of the text file into different columns on a database. The text file is generally set up like this:
"Intro information"
"more Intro information"
srvrmgr> "information about system"
srbrmgr> list parameters for component *ADMBatchProc*
"Headers"
*Name of record* *alias of record* *value of record*
The columns create a table containing all of the setting information for this component. One all of the settings are listed, the file moves to another component and returns all the information for that component in a new table. I need to read in the component and the information on the tables without the headers or the other information. I will then need to be able to transfer that data into a database. The columns are fixed width on each table within the file.
Any recommendations about how to approach this are welcome. I have never read in a file this complex so I dont really know how to approach ignoring alot of information while trying to get other information ready for a database. Also the component value I am trying to gather always follows the word component on a line that starts with "srvrmgr".
The '*' represents areas that will be put into datbase.
Siebel Enterprise Applications Siebel Server Manager, Version 8.1.1.11 [23030] LANG_INDEPENDENT
Copyright (c) 1994-2012, Oracle. All rights reserved.
The Programs (which include both the software and documentation) contain
proprietary information; they are provided under a license agreement containing
restrictions on use and disclosure and are also protected by copyright, patent,
and other intellectual and industrial property laws. Reverse engineering,
disassembly, or decompilation of the Programs, except to the extent required to
obtain interoperability with other independently created software or as specified
by law, is prohibited.
Oracle, JD Edwards, PeopleSoft, and Siebel are registered trademarks of
Oracle Corporation and/or its affiliates. Other names may be trademarks
of their respective owners.
If you have received this software in error, please notify Oracle Corporation
immediately at 1.800.ORACLE1.
Type "help" for list of commands, "help <topic>" for detailed help
Connected to 1 server(s) out of a total of 1 server(s) in the enterprise
srvrmgr> configure list parameters show PA_NAME,PA_ALIAS,PA_VALUE
srvrmgr>
srvrmgr> list parameters for component ADMBatchProc
PA_NAME PA_ALIAS PA_VALUE
---------------------------------------------------------------------- ------------------------------------- --------------------------------------------------------------------------------------------------------------------
ADM Data Type Name ADMDataType
ADM EAI Method Name ADMEAIMethod Upsert
ADM Deployment Filter ADMFilter
213 rows returned.
srvrmgr> list parameters for component ADMObjMgr_enu
PA_NAME PA_ALIAS PA_VALUE
---------------------------------------------------------------------- ------------------------------------- --------------------------------------------------------------------------------------------------------------------
AccessibleEnhanced AccessibleEnhanced False
This is the beginning of the text file. It a produced in a system called Siebel to show all of the settings for this environment. I need to pull the component name (there are multiple on the actual file but the ones shown here are 'ADMBatchProc' and 'ADMObjMgr_enu'), and then the data shown on the table below it that was created by Siebel. The rest of the information is irrelevant for the purpose of the task I need.
I would recommend using Test-Driven Development techniques in this case. I'm guessing that your possible variations of input format are near infinite.
Try this:
1) Create an interface that will represent the data operations or parsing logic you expect the application to perform. For example:
public interface IParserBehaviors {
void StartNextComponent();
void SetTableName(string tableName);
void DefineColumns(IEnumerable<string> columnNames);
void LoadNewDataRow(IEnumerable<object> rowValues);
DataTable ProduceTableForCurrentComponent();
// etc.
}
2) Gather as many small examples of discrete inputs that have well-defined behaviors as possible.
3) Inject a behaviors handler into your parser. For example:
public class Parser {
private const string COMPONENT_MARKER = "srvrmgr";
private readonly IParserBehaviors _behaviors;
public Parser(IParserBehaviors behaviors) {
_behaviors = behaviors;
}
public void ReadFile(string filename) {
// bla bla
foreach (string line in linesOfFile) {
// maintain some state
if (line.StartsWith(COMPONENT_MARKER)) {
DataTable table = _behaviors.ProduceTableForCurrentComponent();
// save table to the database
_behaviors.StartNextComponent();
}
else if (/* condition */) {
// parse some text
_behaviors.LoadNewDataRow(values);
}
}
}
}
4) Create tests around the expected behaviors, using your preferred mocking framework. For example:
public void FileWithTwoComponents_StartsTwoNewComponents() {
string filename = "twocomponents.log";
Mock<IParserBehaviors> mockBehaviors = new Mock<IParserBehaviors>();
Parser parser = new Parser(mockBehaviors.Object);
parser.ReadFile(filename);
mockBehaviors.Verify(mock => mock.StartNextComponent(), Times.Exactly(2));
}
This way, you will be able to integrate under controlled tests. When (not if) someone runs into a problem, you can distill what case wasn't covered, and add a test surrounding that behavior, after extracting the case from the log being used. Separating concerns this way also allows your parsing logic to be independent from your data operation logic. The needs of parsing specific behaviors seems to be central to your application, so it seems like a perfect fit for creating some domain-specific interfaces.
You'll want to read the text file using StreamReader:
using (FileStream fileStream = File.OpenRead(path))
{
byte[] data = new byte[fileStream.Length];
for (int index = 0; index < fileStream.Length; index++)
{
data[index] = (byte)fileStream.ReadByte();
}
Console.WriteLine(Encoding.UTF8.GetString(data)); // Displays: your file - now you can decide how to manipulate it.
}
Perhaps then you'll use Regex to capture the date you'd like to insert:
You might insert into the db like this:
using (TransactionScope transactionScope = new TransactionScope())
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
SqlCommand command1 = new SqlCommand(
“INSERT INTO People ([FirstName], [LastName], [MiddleInitial])
VALUES(‘John’, ‘Doe’, null)”,
connection);
SqlCommand command2 = new SqlCommand(
“INSERT INTO People ([FirstName], [LastName], [MiddleInitial])
VALUES(‘Jane’, ‘Doe’, null)”,
connection);
command1.ExecuteNonQuery();
command2.ExecuteNonQuery();
}
transactionScope.Complete();
}
Examples adapted from Wouter de Kort's C# 70-483.
How do i read the last entered event log or is there any other way to limit to write only once the same event? This piece of code will help, but is there any other approach available?
EventLog eventLog;
eventLog = new EventLog();
eventLog.Log = eventLogName;
eventLog.Source = sourceName;
foreach (EventLogEntry log in eventLog.Entries)
{
//log.Source - do your stuff
}
There is an excellent article on Reading and Writing to the Event Log.
Use the Entries member when reading from the event log.
Because the property is read-only, you cannot modify an entry or write
to the log using Entries. Instead, specify a Source and call
WriteEntry to write a new log entry. You can use Entries to count the
number of entries in the event log, and view each EventLogEntry in the
collection. Use the indexed Item member to retrieve information about
a specific entry, such as Message, Category, TimeWritten, or
EntryType.
It is not necessary to specify a Source when only reading from a log.
You can specify only the Log name and MachineName (server computer
name) properties for the EventLog instance. In either case, the
Entries member is automatically populated with the event log's list of
entries. You can select the appropriate index for an item in this list
to read individual entries.
An important distinction between reading and writing log entries is
that it is not necessary to explicitly call a read method. After the
Log and MachineName are specified, the Entries property is
automatically populated. If you change the value of the Log or
MachineName property, the Entries property is repopulated the next
time you read it.
An example would be:
using System;
using System.Diagnostics;
class MySample{
public static void Main(){
EventLog myLog = new EventLog();
myLog.Log = "MyNewLog";
foreach(EventLogEntry entry in myLog.Entries){
Console.WriteLine("\tEntry: " + entry.Message);
}
}
}
You'll obviously want to create a method that will be able to filter or sort out your request based on certain criteria to help limit your choices. Hopefully this helps.
You may encounter certain permission issues; as they are tied into an Administrator Category with the EventLogPermissionAccess.Administer.
The information provided is directly from Microsoft here.