Structural logging on the example of Serilog and Seq

Structural logging takes one small step forward compared to regular recordings.

Idea

Regular log entries consist of strings and in order to search for entries among an array of text, we have to use regular expressions.

_logger.Warning("   .    ID"+postId); 

Structural logging stores records in the form of objects (structures). For example, using JSON

 _logger.Warning("   .    {ID}", postId); 

Two entries will be saved:

1. The template for the final output line "Write to the log file. Post on habre {id} "
2. The object that we will substitute in the template: {"ID": 1, type: Number}

The main advantage of this approach is that you can now store an integer ID object as a separate parameter, for example, in a NoSql database. And perform a convenient and fast search using typed comparison operations instead of writing regular expressions.

Serilog

One of the convenient .NET libraries that support structural logging is Serilog .
The library supports all the basic logging functions that log4net, Nlog, and other well-known libraries have:

• Several common record types:
  Verbose - the lowest level and detailed logging (for example, the arguments to the method)
  Debug - data for debugging code, one level higher than Verbose (for example, which method was run, and the result of the execution)
  Warning - a warning for a business process, it should not contain Debug data (for example, they start a payroll calculation)
  Error - an error in the application that was not expected
  Fatal is an exceptional error stopping the business processes of the application (for example, redirecting the user to PayPal and the buyer’s payment is not equal to the expected amount).
  
• Different types of storage, called Serilog in stock: text file, relational databases, NoSql databases, Windows Events, http / s requests, etc.
• Convenient configuration through code, and through .config files

')
So, after installing the Serilog Nuget package, configure the configuration. We do this in code instead of the .config file.

  var logger = new LoggerConfiguration() .MinimumLevel.Verbose() //     Verbose  ,    Information .WriteTo.ColoredConsole() //     .WriteTo.RollingFile(@"C:\Logs\Log-{Date}.txt") //     ,     //    Verbose    ,  , Error  Windows Event Logs .CreateLogger(); 

Now you can pass a logger object through Dipendency Injection, or use it immediately.
Suppose we create a program to diagnose a car and we need information about the manufacturer, the series and the intake date. And of course, to keep everything in a structural form.

 logger.Debug("Request for vehicles {Manufature}, {Series}, {ProductionYear}", "BMW", "F02", new DateTime(2015, 1,1)); // search returns 10 vehicles logger.Verbose("Found {FoundVehiclesCount}", 10); 

Since we used the output in the console and the file, the result will be in the form of a familiar line.


One of the convenient features is the addition of frequently used properties. For a web application, this may be a user session, a visited URL, browser data. In a more general case, you can add a version of the application. You can do this using the Enrich property and already built-in classes, or write your own.

  var logger = new LoggerConfiguration() .Enrich.With<HttpRequestIdEnricher>() .Enrich.With<HttpRequestUrlEnricher>() .Enrich.With<HttpRequestUserAgentEnricher>() .Enrich.With<UserNameEnricher>(); .Enrich.WithProperty("ApplicationVersion", config.ApplicationVersion) .CreateLogger(); 

For convenient output of complex objects to the console or text file, it is better to prompt the Serilog template that we are using a complex object, and not a primitive type, by adding the @ symbol. Otherwise, typeof (MyClass) .ToSting () will be displayed.

 logger.Debug("We are using anonymous object instead of simple string/integer {@Car}", new {Manufacture="Toyota", Production = new DateTime(2015, 1,1)}); 

Seq

The Seq application comes to the rescue for convenient storage and retrieval of structural logs.
Seq works as a windows service that accepts REST requests, and internally stores data in a NoSql database.

After installing Seq, you must also add a package to the Serilog.Sinks.Seq Nuget package. And then make friends with our Serilog Seq.

 var logger = new LoggerConfiguration() .MinimumLevel.Verbose() .WriteTo.Seq("http://localhost:5341") .CreateLogger(); 

Now you can conveniently do a search on our fields, including comparing numbers and dates


Search parameters can be saved and used in other parts of the application. Add Environment to Dashboard

And you can add a real-time Dashboard display of errors that came exactly from the application "MyApp", version "1.2.1" and occurred in the method "Repository.GetUserByIdAndPassword ()".

Business requirements

Like any new, fashionable, glowing program you should check by asking questions:
“What benefits does this program have for business?”, “What problem does it solve and for what price?”.
Often, in terms of logging, goals are reduced to

• Reduce the effort required to find errors in the program.
• Have the ability to restore the history of user actions. For example, to find out whether the goods are paid.
• Be aware of errors, as they lead to user losses and profits.

If the first two points are solved by detailed records and a convenient search, then error reporting is often overlooked. I call it "peace in ignorance." Open the Windows event logs and it turns out that almost every server has programs that crash with critical errors.

With the help of Seq extensions you can inform about the events. For example, immediately send a message to the programmer / administrator if a critical error has occurred. And send a list of all errors that occurred in a day or week. You can configure extensions in the Seq -> Settings -> Apps section

Cons of structural logging

The disadvantage of structural logging is the need to determine the parameters by which the search will be performed. They will then be used to create NoSql indexes. If you forgot to define a parameter, the search is reduced to the old scanning of strings through regular expressions.
A huge step in this direction makes the application Splunk . The idea is to collect string data, absolutely any and in any format (application logs, OS events, etc.). And then a terrific split of the string depending on the queries and the dynamic construction of the result through Map / Reduce. Splunk is a great infrastructure for collecting and analyzing data, and goes beyond the topic of logging and this article.