I am currently trying to understand serilogs for structured Logging.
Is there a way to enforce common Property to be used in serilog . Like if I have a log written in code like below
log.Info("Disk Quota {DiskQuota} exceeded by user {Username}", 100, "User1")
How can I use message template to ensure that any future log written in 2 to 3 classes where disk quota exceeding warning could be thrown and needs to be logged, uses always {Username} only and not {user} or {userid} etc.
log.Info("Disk Quota {DiskQuota} exceeded by user {User}", 100, "User2") // Disallow , Possibly ??
There is nothing out-of-the-box in Serilog that does that for you.
A good solution to this would be to implement your own source code analyzer, that can perform these checks during the build and emit warnings and/or errors when messages are similar but have different property names.
You'd have to define a String metric to decide what "similar" means to you. There are a number of String metrics you can use, and each of them are similar but have some different characteristics, with the popular ones being:
Levenshtein Distance: The minimum number of single-character edits required to change one word into the other. Strings do not have to be the same length;
Hamming Distance: The number of characters that are different in two equal length strings;
Smith–Waterman: A family of algorithms for computing variable sub-sequence similarities;
Sørensen–Dice Coefficient: A similarity algorithm that computes difference coefficients of adjacent character pairs.
You also leverage parts of the source code of the SerilogAnalyzer project to find message templates in code, and perform the checks.
A much more simple solution (but not as effective because relies on good developer behavior) would be to declare these standard message templates in a class, inside of a project that is shared across your solution, so that you can reference them instead everywhere instead of typing.
e.g.
public static class LogMessageTemplates
{
public const string DiskQuotaExceededByUsername = "Disk Quota {DiskQuota} exceeded by user {Username}";
}
And then everywhere you'd rely on developers to always use these pre-defined message templates, instead of typing the strings directly. e.g.:
log.Info(LogMessageTemplates.DiskQuotaExceededByUsername, 100, "User1");
If you use this approach you'll probably want to install SerilogAnalyzer in the project to help you identify places where the message template has X parameters but you are using Y given that it's now more difficult to spot how many properties a message template has just by looking at the name.
Related
I have been following Logging in ASP.NET Core Which is working just fine.
I have a question about this line
_logger.LogWarning(LoggingEvents.GetItemNotFound, "GetById({ID}) NOT FOUND", id);
I am wondering why they are not using $ - string interpolation?
_logger.LogWarning(LoggingEvents.GetItemNotFound, $"GetById({ID}) NOT FOUND");
Why would the LogWarning extension have a params object[] args paramater?
Whats the point when you can just send everything in string message.
I assume there is a reason for this but i haven't been able to find an explanation anywhere. I am wondering which method i should be using in order to log properly in .net core.
I suspect the question can be rephrased to :
Why didn't they provide overloads that accept a FormattableString to pass message templates and parameters using string interpolation syntax, like EF Core does for parameterized queries?
I'd say they got it right. At this point in time using FormattableString offers minimal benefits but creates a lot of confusion.
I just found that Serilog's author explains why this isn't such a good idea even though a semantic logging library looks like a natural fit for this scenario
Semantic Logging
One can argue that FormattableString would be a great addition to semantic logging libraries like Serilog. In this case an interpolated string does lose important information.
The call
Log.Information("Logged in {UserId}", loggedInUserId);
Won't just log a string based on the template, it will keep the names and values of the parameters and provide them to filters and targets. Wouldn't it be great to get the same result with :
Log.Information($"Logged in {loggedInUserId}");
Serilog's author doesn't think so and explains that :
A good variable name is not necessarily a good property name
Holes don’t always have obvious names, eg Log.Information($"Enabling categories {new[]{1, 2, 3}}");
and concludes that
String interpolation is a great feature, one I’ve looked forward to in C# for a long time. The idea of providing direct support for Serilog is a very interesting one and worth exploring, but I’m increasingly convinced it’s unnecessary.
Interpolation is nice when it keeps code DRY, cutting out the redundant clutter of {0} and {1} and preventing parameter mismatches.
In the case of Serilog, I think it’s incorrect to consider the property names like {UserId} as redundant; in a well-implemented logging strategy they’re an incredibly important part of the picture that deserve their own consideration. You wouldn’t let variable names determine the table and column names in a relational database – it’s exactly the same trade-off being considered here.
Original explanation
That's one of the most controversial features of EF Core actually, and can easily result in the very SQL injection and conversion problems one wants to avoid by using parameters.
This call :
string city = "London";
var londonCustomers = context.Customers
.FromSql($"SELECT * FROM Customers WHERE City = {city}");
calls FromSql(FormattableString) and will create a parameterized query :
SELECT * FROM Customers WHERE City = #p0
And pass London as a parameter value.
On the other hand this :
string city = "London";
var query=$"SELECT * FROM Customers WHERE City = {city}";
var londonCustomers = context.Customers.FromSql(query);
calls FromSql(string) and will generate :
SELECT * FROM Customers WHERE City = London
Which is invalid. It's far too common to fall in this trap even when you do know about the risk.
It doesn't help at all when you already have predefined queries or messages. This usage is far more common in logging, where you (should) use specific message templates defined in well known locations, instead of sprinkling similar looking strings in every log location.
One could argue that this addition made EF Core 2.0 somewhat safer because people had already started using string interpolation in EF Core 1.0, resulting in invalid queries. Adding the FormattableString overload the EF Core team was able to mitigate that scenario while making it easier to accidentally cause a different problem.
At this point in time the logging designers decided to avoid this confusion. Logging a raw string doesn't have such catastrophic consequences.
At least two reasons.
First, logging pre-dates string interpolation, and Microsoft have not yet invented a time machine. String interpolation was only introduced in C# 6 in July 2015, but the logging methods follow the same pattern used in Microsoft.Build.Utilities since dotnet framework 2.0.
Second, Performance. If string interpolation is used and a string is passed as a parameter, then the interpolation is done before the call to Log. However not every call to Log results in something being logged - it depends on the configuration.
If you log something at DEBUG level and your current configuration is for INFORMATION level, the it is a waste of time to do string interpolation, you can just say "Thanks, but no thanks", and return immediately after doing nothing with the arguments.
Expanding on Second, Performance
Internally, the most logger look basically like this:
void LogDebug(string Message, params object[] args){
if(this.DebugEnabled){
Log.Write(string.Format(Message,args));
}
}
// 1 with parameters
LogDebug("GetById({ID}) NOT FOUND", id);
// 2 interpolated
LogDebug($"GetById({id}) NOT FOUND");
So if Debug is not enabled, one less interpolation operation is done.
I know this is 4 years old but I don't feel any of the answers provided are really correct.
The reason is structured logging, if you use string interpolation you are just logging a single string, with structured logging you log the variables seperatly so you can access them more easily.
Imagine you have a website and you want to alert or provide a report on how long a page takes to load. You log it like this logger.LogDebug($"{PageName} took {ms}ms to load."); All your log can contain is "Index took 53ms to load."
If you log it like this instead logger.LogDebug("{PageName} took {ms}ms to laod.", PageName, sw.ElapsedMilliseconds); Then depending on your logging provider you can access all the properties seperatly and group by the PageName easily and filter all load times over 50ms. Without having to revert to Regular Expressions. E.g. SELECT PageName, Count(*) FROM Logs WHERE LogText = '{PageName} took {ms}ms to laod.' and ms > 50 GROUP BY PageName
The reason in my opinion is that .NET is using multiple levels of logging.
Trace = 0, Debug = 1, Information = 2, Warning = 3, Error = 4, Critical = 5 and None = 6.
There are differences between below codes although they look similar.
Structural logging
Logger.LogInformation("Hello {Name}", myName);
and
String Interpolation
Logger.LogInformation($"Hello {myName}");
String interpolation under the hood is identical to the use of string.Format(). Which means string.Format will always be executed
disregarding the log level and pass the formatted string to the logger.
On the other hand, when structural logging is used, the string will only be formatted as per the log level is met.
I am trying to create a fluent interface that has two wrinkles;
It has multiple terminal points
It has possible invalid intermediate states
I can handle either one of these, but am having trouble with both. Let me give a (slightly simplified) concrete example, assuming that I am extending Stream and the stream is seekable.
I want to support the two flows as follows;
stream.GoTo.Offset(x); // Seek from the beginning
stream.GoTo.Offset(x).From(origin); // Seek from the stated origin, which could be the end
I therefore could end on an Offset or a From (two terminals). Also, seeking to a point before the start is not allowed. This means if the position was at 10 and the length was 1000, that stream.GoTo.Offset(-100) should fail but stream.GoTo.Offset(-100).From(SeekOrigin.End) should succeed.
Normally if I have two terminals, I would apply the parts sequentially and all is good; add the Offset to the position first, and at the From, add 0, subtract the original position, or add the length depending on the case. This doesn't work for the problem of the invalid interim state though.
It could well be that there is no solution. I feel like I've simplified it to the point where I can see the options and have eliminated them all. However, I don't write many fluent interfaces and maybe there is at least a work-around I'm not seeing. Any help would be appreciated, even if just to confirm it can't be done.
The methods in the .NET platform's DirectorySecurity namespace (e.g. GetAccessRules()) are far too slow for my purposes. Instead, I wish to directly query the NTFS $Secure metafile (or, alternatively, the $SDS stream) in order to retrieve a list of local accounts and their associated permissions for each file system object.
My plan is to first read the $MFT metafile (which I've already figured out how to do) - and then, for each entry therein, look up the appropriate security descriptor in the metafile (or stream).
The ideal code block would look something like this:
//I've already successfully written code for MFTReader:
var mftReader = new MFTReader(driveToAnalyze, RetrieveMode.All);
IEnumerable<INode> nodes = mftReader.GetNodes(driveToAnalyze.Name);
foreach (NodeWrapper node in nodes)
{
//Now I wish to return security information for each file system object
//WITHOUT needing to traverse the directory tree.
//This is where I need help:
var securityInfo = GetSecurityInfoFromMetafile(node.FullName, node.SecurityID);
yield return Tuple.Create(node.FullName, securityInfo.PrincipalName, DecodeAccessMask(securityInfo.AccessMask));
}
And I would like my output to look like this:
c:\Folder1\File1.txt jane_smith Read, Write, Execute
c:\Folder1\File1.txt bill_jones Read, Execute
c:\Folder1\File2.txt john_brown Full Control
etc.
I am running .NET version 4.7.1 on the Windows 10.
There's no API to read directly from $Secure, just like there is no API to read directly from $MFT. (There's FSCTL_QUERY_FILE_LAYOUT but that just gives you an abstracted interpretation of the MFT contents.)
Since you said you can read $MFT, it sounds like you must be using a volume handle to read directly from the volume, just like chkdsk and similar tools. That allows you to read whatever you want provided you know how to interpret the on-disk structures. So your question reduces to how to correctly interpret the $Secure file.
I will not give you code snippets or exact data structures, but I will give you some very good hints. There are actually two approaches possible.
The first approach is you could scan forward in $SDS. All of the security descriptors are there, in SecurityId order. You'll find there's at various 16-byte aligned offsets, there will be a 20-byte header that includes the SecurityId among other information, and following that there's the security descriptor in serialized form. The SecurityId values will appear in ascending order in $SDS. Also every alternate 256K region in $SDS is a mirror of the previous 256K region. To cut the work in half only consider the regions 0..256K-1, 512K..768K-1, etc.
The second approach is to make use of the $SII index, also part of the $Secure file. The structure of this is a B-tree very similar to how directories are structured in NTFS. The index entries in $SII have SecurityId as the index for lookups, and also contain the byte offset you can go to in $SDS to find the corresponding header and security descriptor. This approach will be more performant than scanning $SDS, but requires you to know how to interpret a lot more structures.
Craig pretty much covered everything. I would like to clear some of them. Like Craig, no code here.
Navigate to the node number 9 which corresponds to $Secure.
Get all the streams and get all the fragments of the $SDS stream.
Read the content and extract each security descriptor.
Use IsValidSecurityDescriptor to make sure the SD is valid and stop when you reach an invalid SD.
Remember that the $Secure store the security descriptors in self-relative format.
Are you using FSCTL_QUERY_FILE_LAYOUT? The only real source of how to use this function I have found is here:
https://wimlib.net/git/?p=wimlib;a=blob;f=src/win32_capture.c;h=d62f7d07ef20c08c9bec93f261131033e39b159b;hb=HEAD
It looks like he solves the problem with security descriptors like this:
He gets basically all information about files from the MFT, but not security descriptors. For those he gets the field SecurityId from the MFT and looks in a hash table whether he already has a mapping from this ID to the ACL. If he has, he just returns it, otherwise he uses NtQuerySecurityObject and caches it in the hash table. This should drastically reduce the amount of calls. It assumes that there are few security descriptors and that the SecurityID field correctly represents the single instancing of the descriptors
I'm currently working on an application which allows the user to read and a configuration for a device. The configuration is stored as XML.
The issue I'm faced with is how to define validation for the application. For example, most of the values I'm storing in the XML file have to be within different ranges, e.g. 0 - 2, 1 - 50, 10 characters or 20 characters, etc.
There are a lot of these constraints that I have to validate against, and I don't want to hard-code the ranges because when version 2 of the device comes out the configuration file will have different set of ranges. E.g. instead of 0 - 2, it will be 0 - 4 and instead of 20 characters, 40 is now allowed.
How should I approach this? Should I store the validation rules in separate XML files? Should I define a class with hard-coded configuration ranges for this device, and create a new class for the version 2 of the device with its configuration ranges?
It could be done inside a XML, so kind of declarative programming, where in XML you define a behaviour. But it's not flexible and you easily can jump into pretty conplicated scenarious.
What I personally would prefer to do is maintain the logic inside the code, but the parameters range that the data in source XML has to be checked against store in some MatchData.xml.
Hope this helps.
I almost always prefer an external configuration file in cases like this. You could define an object that performs the validation (validater). When the validater object is instantiated it instantiates a validation rules object that contains all the ranges for the various validation items. I would serialize/de-serialize this object using an XML file and that file would be included in your app distribution.
For the project that I'm currently on, I have to deliver specially formatted strings to a 3rd party service for processing. And so I'm building up the strings like so:
string someString = string.Format("{0}{1}{2}: Some message. Some percentage: {3}%", token1, token2, token3, number);
Rather then hardcode the string, I was thinking of moving it into the project resources:
string someString = string.Format(Properties.Resources.SomeString, token1, token2, token3, number);
The second option is in my opinion, not as readable as the first one i.e. the person reading the code would have to pull up the string resources to work out what the final result should look like.
How do I get around this? Is the hardcoded format string a necessary evil in this case?
I do think this is a necessary evil, one I've used frequently. Something smelly that I do, is:
// "{0}{1}{2}: Some message. Some percentage: {3}%"
string someString = string.Format(Properties.Resources.SomeString
,token1, token2, token3, number);
..at least until the code is stable enough that I might be embarrassed having that seen by others.
There are several reasons that you would want to do this, but the only great reason is if you are going to localize your application into another language.
If you are using resource strings there are a couple of things to keep in mind.
Include format strings whenever possible in the set of resource strings you want localized. This will allow the translator to reorder the position of the formatted items to make them fit better in the context of the translated text.
Avoid having strings in your format tokens that are in your language. It is better to use
these for numbers. For instance, the message:
"The value you specified must be between {0} and {1}"
is great if {0} and {1} are numbers like 5 and 10. If you are formatting in strings like "five" and "ten" this is going to make localization difficult.
You can get arround the readability problem you are talking about by simply naming your resources well.
string someString = string.Format(Properties.Resources.IntegerRangeError, minValue, maxValue );
Evaluate if you are generating user visible strings at the right abstraction level in your code. In general I tend to group all the user visible strings in the code closest to the user interface as possible. If some low level file I/O code needs to provide errors, it should be doing this with exceptions which you handle in you application and consistent error messages for. This will also consolidate all of your strings that require localization instead of having them peppered throughout your code.
One thing you can do to help add hard coded strings or even speed up adding strings to a resource file is to use CodeRush Xpress which you can download for free here: http://www.devexpress.com/Products/Visual_Studio_Add-in/CodeRushX/
Once you write your string you can access the CodeRush menu and extract to a resource file in a single step. Very nice.
Resharper has similar functionality.
I don't see why including the format string in the program is a bad thing. Unlike traditional undocumented magic numbers, it is quite obvious what it does at first glance. Of course, if you are using the format string in multiple places it should definitely be stored in an appropriate read-only variable to avoid redundancy.
I agree that keeping it in the resources is unnecessary indirection here. A possible exception would be if your program needs to be localized, and you are localizing through resource files.
yes you can
new lets see how
String.Format(Resource_en.PhoneNumberForEmployeeAlreadyExist,letterForm.EmployeeName[i])
this will gave me dynamic message every time
by the way I'm useing ResXManager