Filling text templates with model-based data. Implementing on .NET using dynamic functions in bytecode (IL)

Prologue

Recently, there was the task of mass mailing of letters, the text of which is formed on the basis of a template in which, in addition to static content, there is information about the recipient and fragments of text. In my case, this is a template for automatically notifying subscribers about the publication of new articles; accordingly, there is an appeal to the addressee and a beautifully executed link to the publication.

Immediately the question arose - how to implement it? Various solutions came to mind, starting from setting certain constant values in the template, which would be replaced by these models, and ending with full-fledged Razor views (the site is built on MVC 5).

After a brief battle with myself, I came to the conclusion that it’s time to solve this fairly common task once and for all, and that its solution should not be very difficult (i.e. should not depend on libraries that are not part of the .NET Framework 4), but at the same time functional enough to solve the set task and to have a reserve for expandability.
')
In this article I will talk about a solution based on a byte-code generator that meets these requirements, and also comment on the most interesting code fragments.

If you are only interested in the template engine, the links below are:

Source codes of the template engine (Genesis.Patternizer) and the test console in the SourceForge project: https://sourceforge.net/projects/open-genesis/?source=navbar
Or in the archive in one file: Patternizer.zip

Formulation of the problem

To begin with we will be defined with syntax. Personally, I like the function string.Format, which is widely used for formatting simple values. We use its syntax to denote the places where values are inserted into the template:

'{' <> [ ':' < > ] [ '|' <  > ] '}'

Examples: {User.GetFIO ()}, {User.Name | user interface}, {User.Birthdate: dd.MM.yyyy}, {User.Score: 0.00 | nothing}.

The default value will be substituted if the desired value is null (null) or absent altogether, i.e. if the specified property / field / method is not found in the model. For shielding braces we will use double braces (as in the function string.Format), for escaping characters in the format string and the default value - a slash.

Here is an example of a ready-made template that will be used in the test example:

 , {User.GetFIO()|}!   ,   : function PrintMyName() {{ Console.WriteLine("My name is {{0}}. I'm {{1}}.", "{UserName|}", {User.Age:0}); }}     {Now:dd MMMM yyyy}  {Now:HH:mm:ss}

Initially, I assumed that the template would only support the public properties of the model, but during the development process we added support for the fields, as well as methods (with the possibility of passing arguments such as string, number, boolean type and null) and calls to an array of any dimension. Those. The following expression will also be a valid pattern:

  : {User.Account[0].GetSomeArrayMethod("a", true, 8.5, null)[5,8].Length:0000|NULL}

Decision

Parser

First you need to understand what to do with the text template. Of course, you can analyze the pattern data, look up and substitute values for each model substitution call. But this is a very slow way. It will be much more effective to break the pattern into separate logical fragments (pattern elements) once and later to operate with these elements. There are three obvious element types: a string constant (that part of the template that directly goes to the result unchanged), a substitution (what's inside the curly brackets) and a comment (this element is not implemented, but I suppose you understand what this is about ).

Based on these arguments, we describe the base class for the template element:

 /// <summary> ///   /// </summary> public abstract class PatternElement { /// <summary> ///     /// </summary> public virtual int EstimatedLength { get { return 0; } } /// <summary> ///      /// </summary> public abstract string GetNullValue(); }

The meaning of the EstimatedLength property and the GetNullValue () method will be explained below.

Next, we will describe specific implementations - a string constant and a substitution (let's call it “expression”):

 public class StringConstantElement : PatternElement { public string Value { get; set; } public override int EstimatedLength { get { return Value == null ? 0 : Value.Length; } } public override string GetNullValue() { return Value; } } public class ExpressionElement : PatternElement { public string Path { get; set; } public string FormatString { get; set; } public string DefaultValue { get; set; } public override int EstimatedLength { get { return Math.Max(20, DefaultValue == null ? 0 : DefaultValue.Length); } } public override string GetNullValue() { return DefaultValue; } }

We also describe the interface of the IPatternParser parser, which accepts a text pattern as input, and outputs a sequence of elements:

 public interface IPatternParser { IEnumerable<PatternElement> Parse(string pattern); }

A parser based on curly brackets and let's call it BracePatternParser . Having no great experience in writing parsers (and this is what parser actually does), I will not delve into its implementation.

BracePatternParser.cs

 using System; using System.Collections.Generic; using System.Linq; using System.Text; namespace Genesis.Patternizer { /// <summary> ///      /// </summary> public class BracePatternParser : IPatternParser { private object _lock = new object(); private HashSet<char> PATH_TERMINATOR_CHARS; private HashSet<char> FORMAT_TERMINATOR_CHARS; private HashSet<char> PATTERN_TERMINATOR_CHARS; private string pattern; //  private int length; //   private int length_1; //     private int index; //     private StringBuilder constantBuilder; private StringBuilder expressionBuilder; /// <summary> ///  /// </summary> public BracePatternParser() { PATH_TERMINATOR_CHARS = new HashSet<char>(":|}".ToCharArray()); FORMAT_TERMINATOR_CHARS = new HashSet<char>("|}".ToCharArray()); PATTERN_TERMINATOR_CHARS = new HashSet<char>("}".ToCharArray()); } /// <summary> ///    /// </summary> /// <param name="chars"> - </param> /// <returns></returns> private string ParsePatternPath(HashSet<char> chars) { //    expressionBuilder.Clear(); Stack<char> brackets = new Stack<char>(); bool ignoreBrackets = false; for (index++; index < length; index++) { char c = pattern[index]; if (c == '(') { brackets.Push(c); expressionBuilder.Append(c); } else if (c == ')') { if (brackets.Peek() == '(') { brackets.Pop(); } else { //   ignoreBrackets = true; } expressionBuilder.Append(c); } else if (c == '[') { brackets.Push(c); expressionBuilder.Append(c); } else if (c == ']') { if (brackets.Peek() == '[') { brackets.Pop(); } else { //   ignoreBrackets = true; } expressionBuilder.Append(c); } else if (chars.Contains(c) && (ignoreBrackets || brackets.Count == 0)) { //   break; } else { expressionBuilder.Append(c); } } return expressionBuilder.Length == 0 ? null : expressionBuilder.ToString(); } /// <summary> ///     /// </summary> /// <param name="chars"> - </param> /// <returns></returns> private string ParsePatternPart(HashSet<char> chars) { //    expressionBuilder.Clear(); for (index++; index < length; index++) { char c = pattern[index]; if (c == '\\') { //     if (index < length_1) { expressionBuilder.Append(pattern[++index]); } } else if (chars.Contains(c)) { //   break; } else { expressionBuilder.Append(c); } } return expressionBuilder.Length == 0 ? null : expressionBuilder.ToString(); } /// <summary> ///    /// </summary> /// <returns></returns> private ExpressionElement ParsePattern() { string path = ParsePatternPath(PATH_TERMINATOR_CHARS); if (path == null) { //   //      (}) for (; index < length; index++) { char c = pattern[index]; if (c == '\\') { index++; } else if (c == '}') { break; } } return null; } else { ExpressionElement element = new ExpressionElement(path); //    if (index < length && pattern[index] == ':') { //   element.FormatString = ParsePatternPart(FORMAT_TERMINATOR_CHARS); } if (index < length && pattern[index] == '|') { //    element.DefaultValue = ParsePatternPart(PATTERN_TERMINATOR_CHARS); } return element; } } /// <summary> ///   /// </summary> /// <param name="pattern">  </param> /// <returns></returns> public IEnumerable<PatternElement> Parse(string pattern) { lock (_lock) { if (pattern == null) { //   yield break; } else if (string.IsNullOrWhiteSpace(pattern)) { yield return new StringConstantElement(pattern); yield break; } //   this.pattern = pattern; //   length = pattern.Length; length_1 = length - 1; index = 0; //  constantBuilder = new StringBuilder(); expressionBuilder = new StringBuilder(); //    for (; index < length; index++) { char c = pattern[index]; if (c == '{') { if (index < length_1 && pattern[index + 1] == c) { //   '{' constantBuilder.Append(c); index++; } else { //    if (constantBuilder.Length != 0) { yield return new StringConstantElement(constantBuilder.ToString()); //   constantBuilder.Clear(); } var patternElement = ParsePattern(); if (patternElement != null) { yield return patternElement; } } } else if (c == '}') { if (index < length_1 && pattern[index + 1] == c) { //   '}' constantBuilder.Append(c); index++; } else { //      ,     constantBuilder.Append(c); } } else { constantBuilder.Append(c); } } //    if (constantBuilder.Length != 0) { yield return new StringConstantElement(constantBuilder.ToString()); } //   this.pattern = null; constantBuilder = null; expressionBuilder = null; index = length = length_1 = 0; } } } }

Builder generator

The parser described above performs only part of the overall task. It is not enough to get a set of template elements, we still need to process them. To do this, we describe another interface that represents the main component of the system, the IBuilderGenerator :

 public interface IBuilderGenerator { Func<object, string> GenerateBuilder(List<PatternElement> pattern, Type modelType); }

To achieve the highest performance, for each new type of model ( modelType ) we will create a new builder and write it into a hash. The builder itself is a normal function, which takes an object (model) as input and the return string is a filled pattern. The specific implementation of this interface will be given below, and before that we consider the last component of the system, which links everything together.

Template engine

The template engine itself is a class that binds the template, the parser and the builder. Its code also does not represent anything super interesting.

Patternizator.cs

 using System; using System.Collections.Generic; using System.Linq; using System.Text; using BUILDER = System.Func<object, string>; namespace Genesis.Patternizer { /// <summary> /// - /// </summary> public class Patternizator { #region Declarations private PatternizatorOptions _options; //   private string _pattern; //  private List<PatternElement> _elements; //   private Dictionary<Type, BUILDER> _builders; //   #endregion #region Properties /// <summary> ///  /// </summary> public string Pattern { get { return _pattern; } set { _pattern = value; PreparePattern(); } } #endregion #region Constructors /// <summary> ///  /// </summary> public Patternizator() { _options = PatternizatorOptions.Default; _builders = new Dictionary<Type, BUILDER>(); } /// <summary> ///  /// </summary> /// <param name="pattern">  </param> public Patternizator(string pattern) { _options = PatternizatorOptions.Default; Pattern = pattern; } /// <summary> ///  /// </summary> /// <param name="options">   </param> public Patternizator(PatternizatorOptions options) { _options = options; _builders = new Dictionary<Type, BUILDER>(); } /// <summary> ///  /// </summary> /// <param name="pattern">  </param> /// <param name="options">   </param> public Patternizator(string pattern, PatternizatorOptions options) { _options = options; Pattern = pattern; } #endregion #region Private methods /// <summary> ///   /// </summary> private void PreparePattern() { //   _elements = _options.Parser.Parse(_pattern).ToList(); //    _builders = new Dictionary<Type, BUILDER>(); //             //string template = string.Join(Environment.NewLine, _elements.Select(e => System.Text.RegularExpressions.Regex.Replace(e.ToString(), @"\s+", " ").Trim()).ToArray()); } #endregion #region Public methods /// <summary> ///   /// </summary> /// <param name="model">  </param> /// <returns></returns> public string Generate(object model) { //       Type modelType = model == null ? null : model.GetType(); Type modelTypeKey = modelType ?? typeof(DBNull); //      BUILDER builder; if (!_builders.TryGetValue(modelTypeKey, out builder)) { //     builder = _options.BuilderGenerator.GenerateBuilder(_elements, modelType); _builders.Add(modelTypeKey, builder); } //   return builder(model); } #endregion } }

PatternizatorOptions.cs

 using System; using System.Collections.Generic; using System.Linq; using System.Text; namespace Genesis.Patternizer { /// <summary> ///   /// </summary> public class PatternizatorOptions { /// <summary> ///   /// </summary> public IPatternParser Parser { get; set; } /// <summary> ///   /// </summary> public IBuilderGenerator BuilderGenerator { get; set; } #region Default private static PatternizatorOptions _default; /// <summary> ///    /// </summary> public static PatternizatorOptions Default { get { if (_default == null) { _default = new PatternizatorOptions { Parser = new BracePatternParser(), BuilderGenerator = new ReflectionBuilderGenerator(), }; } return _default; } } #endregion } }

Options ( PatternizatorOptions ) is an optional argument in which you can instruct the template engine to use a specific implementation of the parser or generator of the builder, for example, if you use a template syntax that is different from the standard one.

An example of using a template engine in the standard version:

 //   string pattern = GetPattern(); //   Patternizator patternizator = new Patternizator(pattern); //   User user = new User { Surname = RandomElement(rnd, SURNAMES), Name = RandomElement(rnd, NAMES), Patronymic = RandomElement(rnd, PATRONYMICS), //   1950 - 1990  Birthdate = new DateTime(1950, 1, 1).AddDays(rnd.NextDouble() * 40.0 * 365.25) }; var model = new { User = user, UserName = user.Name, Now = DateTime.Now, }; //     string text = patternizator.Generate(model);

In this example, the model is an anonymous type, but you should not be confused. Even when generating elements of this type in a loop, the builder will be created only once, when you first call the Generate method. But let's return to the issue of performance at the end of the article, but now we will consider the most interesting, so to speak, the highlight of this publication.

Bytecode builder generator

To begin, we will make a small analysis. How can this problem be solved in theory?
Let me remind you that we have a list of template elements (constants and expressions) and model type. And we need to get the function Func <object, string> , which substitutes the model of the specified type in the template, receiving the output string.

If there are no questions with constants (we just throw them in StringBuilder), then with expressions everything is more complicated.
I see three possible options for how to get the value of an expression from a model:

Through reflection
Generate C # code, compile and link to build
Write a dynamic function ( System.Reflection.Emit.DynamicMethod ) with a body bytecode

The first option clearly suffers from speed, because reflection always works slowly. My advice to you is to never use reflection for operations that are performed very often. The best way to use it is to prepare at the start of the program, i.e. something like “we ran through the classes, found the necessary information in the attributes, built some kind of connections (delegates, events) and then used them without resorting to reflection again”. In general, reflection is clearly not suitable for this task.

The second option is very good, initially I wanted to use it. The code of the builder function would look something like this (for the template given at the beginning):

 public string Generate(object input) { if (input == null) { //    return @", !   ,   : function PrintMyName() { Console.WriteLine("My name is {0}. I'm {1}.", "", 0); }      "; } else { Model model = input as Model; StringBuilder sb = new StringBuilder(); sb.Append(", "); //   if (model.User != null) { var m_GetFIO = model.User.GetFIO(); if (m_GetFIO != null) { sb.Append(m_GetFIO); } else { sb.Append(""); //    } } else { sb.Append(""); //    } sb.Append("!\r\n  ,   :\r\n\r\n ..."); //   \\  .. return sb.ToString(); } }

Of course, the code will be long enough, but who will see it? In general, this option would be optimal if it were not for anonymous types. In the code above, we would not be able to declare a model variable: Model model = input as <?>; if the type of the model did not have a name.

So, there is a third option. Generate the same code directly in bytecode. When writing a template engine, I myself used dynamic functions and a byte-code generator for the first time, and that was what prompted me to write this article so that you, dear readers, would have fewer problems when you decide to master this technology.

Dynamic assemblies, dynamic functions, and a bytecode generator are described in the System.Reflection.Emit namespace, and you do not need to include any additional libraries to use them.

The simplest dynamic function is created as follows:

 //    var genMethod = new DynamicMethod("< >", typeof(< >), new Type[] { typeof(<  1>), typeof(<  2>), ..., typeof(<  N>) }, true); //   - (  IL-  CIL-) var cs = genMethod.GetILGenerator(); //    // ... //   cs.Emit(OpCodes.Ret); //     return genMethod.CreateDelegate(typeof(< >)) as < >;

cs.Emit (OpCodes.Ret); - this is a write operation command in bytecode. Who does not know, bytecode is something like an assembler for the languages of the .NET family.

If you have gathered strength and read the article before this paragraph, then you should have a question, how will I generate the byte code if I don’t know its commands? The answer is quite simple. To do this, we will need an additional program (the project is in the archive), the code of which is given under the spoiler.

ILDasm

 using System; using System.Collections.Generic; using System.Diagnostics; using System.IO; using System.Linq; using System.Text; namespace ILDasm { class Program { #region Static static void Main(string[] args) { new Program().Run(); } #endregion public void Run() { string basePath = AppDomain.CurrentDomain.BaseDirectory; string exeName = Path.Combine(basePath, AppDomain.CurrentDomain.FriendlyName.Replace(".vshost", "")); Process.Start(@"C:\Program Files (x86)\Microsoft SDKs\Windows\v8.1A\bin\NETFX 4.5.1 Tools\x64\ildasm.exe", string.Format(@"/item:ILDasm.TestClass::DoIt ""{0}"" /text /output:code.il", exeName)); } } public class TestClass { public string DoIt(object value) { StringBuilder sb = new StringBuilder(); return sb.ToString(); } } }

The meaning of the program is that it launches the ildasm disassembler built into the studio and sets it to the DoIt function of the TestClass class. The byte code of the body of this function is placed in the code.il file, which can then be opened and analyzed. I quote the byI-code of the DoIt function (too much removed):

 IL_0000: newobj instance void [mscorlib]System.Text.StringBuilder::.ctor() IL_0005: stloc.0 IL_0006: ldloc.0 IL_0007: callvirt instance string [mscorlib]System.Object::ToString() IL_000c: ret

The substance in the skull box, combined with trial and error, will help generate the code by analogy, i.e. Write the body of the DoIt function like what we want in our generated function, run the utility, look at the code and implement it in the generator.

General information about bytecode

Everything is built on the stack.
If we want to perform addition operations a and b , we need to push the value of the variable a to the stack, then push the value of the variable b to the stack, then call the add command. In this case, the stack is cleared of a and b , and the result of addition is placed on its top. If after this we want to multiply the sum by c , put its value on the stack (remember, now there is already the sum a + b ) and call the multiplication operation (mul).

Final bytecode:

 IL_0000: ldarg.1 IL_0001: ldarg.2 IL_0002: add IL_0003: ldarg.3 IL_0004: mul

And this is what it looks like in C #:

 cs.Emit(OpCodes.Ldarg_1); cs.Emit(OpCodes.Ldarg_2); cs.Emit(OpCodes.Add); cs.Emit(OpCodes.Ldarg_3); cs.Emit(OpCodes.Mul);

Similarly, methods and constructors are called (put the arguments on the stack and call the method / constructor). In this case, for non-static methods, the first thing to do is to put an instance of the class, the method of which we call.

This article is not intended to complete the training byte-code generation, so we continue the reasoning about the generator.

The generator core is enclosed in the function that implements its interface ( IBuilderGenerator ):

Generatebuilder

 /// <summary> ///    /// </summary> /// <param name="pattern">   </param> /// <param name="modelType">   </param> /// <returns></returns> public virtual BUILDER GenerateBuilder(List<PatternElement> pattern, Type modelType) { if (modelType == null) { //   ,     StringBuilder sb = new StringBuilder(); foreach (PatternElement item in pattern) { string nullValue = item.GetNullValue(); if (nullValue != null) { sb.Append(nullValue); } } string value = sb.ToString(); return (m) => value; } else { //      string methodName = "Generate_" + Guid.NewGuid().ToString().Replace("-", ""); //    var genMethod = new DynamicMethod(methodName, typeof(string), new Type[] { typeof(object) }, true); //    var cs = genMethod.GetILGenerator(); var sb = cs.DeclareLocal(typeof(StringBuilder)); var m = cs.DeclareLocal(modelType); ReflectionBuilderGeneratorContext context = new ReflectionBuilderGeneratorContext { Generator = cs, ModelType = modelType, VarSB = sb, VarModel = m, }; //   cs.Emit(OpCodes.Ldarg_0); cs.Emit(OpCodes.Isinst, modelType); cs.Emit(OpCodes.Stloc, m); //  StringBuilder      cs.Emit(OpCodes.Ldc_I4, pattern.Sum(e => e.EstimatedLength)); cs.Emit(OpCodes.Newobj, typeof(StringBuilder).GetConstructor(new Type[] { typeof(int) })); cs.Emit(OpCodes.Stloc, sb); foreach (PatternElement item in pattern) { MethodInfo processor; if (_dicProcessors.TryGetValue(item.GetType(), out processor)) { //   processor.Invoke(processor.IsStatic ? null : this, new object[] { context, item }); } } cs.Emit(OpCodes.Ldloc, sb); cs.Emit(OpCodes.Callvirt, typeof(object).GetMethod("ToString", Type.EmptyTypes)); cs.Emit(OpCodes.Ret); return genMethod.CreateDelegate(typeof(BUILDER)) as BUILDER; } }

This is where the GetNullValue () method and the EstimatedLength property of the template element came in handy.

A feature of the generator is its extensibility, because it is not tied to the template element types described at the beginning - a string constant and expression. If you wish, you can come up with your own elements and, by inheriting this generator, add functions responsible for generating bytecode for the element types you created. To do this, in the code, you must describe a function with the PatternElementAttribute attribute, for example, the code generation for a string constant included in the standard generator implementation is described as follows:

 [PatternElement(typeof(StringConstantElement))] protected virtual void GenerateStringConstantIL(ReflectionBuilderGeneratorContext context, StringConstantElement element) { if (element.Value != null) { WriteSB_Constant(context, element.Value); } } /// <summary> ///   StringBuilder   /// </summary> /// <param name="context">   </param> /// <param name="value">  </param> protected virtual void WriteSB_Constant(ReflectionBuilderGeneratorContext context, string value) { if (value != null) { var cs = context.Generator; cs.Emit(OpCodes.Ldloc, context.VarSB); cs.Emit(OpCodes.Ldstr, value); cs.Emit(OpCodes.Callvirt, _dicStringBuilderAppend[typeof(string)]); cs.Emit(OpCodes.Pop); } }

I will not give the code of other methods, since it is very bulky, but if you have any questions, I will try to answer them separately.

Performance test

Since I don't have the opportunity to compare my template with any other, I will make a comparison with a hard-coded template generator based on string.Replace ().

Test function code

 /// <summary> ///   /// </summary> private void Run() { //   string outputPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "Result"); if (!Directory.Exists(outputPath)) Directory.CreateDirectory(outputPath); Random rnd = new Random(0); Stopwatch sw = new Stopwatch(); //   string pattern = GetPattern(); //   string text; double patternTotal = 0; //      () double patternInitialization; //  () double patternFirst = 0; //   () double manualTotal = 0; //      () //   sw.Restart(); Patternizator patternizator = new Patternizator(pattern); sw.Stop(); patternInitialization = sw.Elapsed.TotalMilliseconds; Console.WriteLine(" {0} (v. {1})", patternizator.GetType().Assembly.GetName().Name, patternizator.GetType().Assembly.GetName().Version); //     for (int i = 0; i < COUNT_PATTERNIZATOR; i++) { //   User user = new User { Surname = RandomElement(rnd, SURNAMES), Name = RandomElement(rnd, NAMES), Patronymic = RandomElement(rnd, PATRONYMICS), //   1950 - 1990  Birthdate = new DateTime(1950, 1, 1).AddDays(rnd.NextDouble() * 40.0 * 365.25) }; var model = new { User = user, UserName = user.Name, Now = DateTime.Now, }; //     sw.Restart(); text = patternizator.Generate(model); sw.Stop(); patternTotal += sw.Elapsed.TotalMilliseconds; if (i == 0) { patternFirst = sw.Elapsed.TotalMilliseconds; } //     if (i < COUNT_MANUAL) { // !             //   -      Replace    sw.Restart(); { StringBuilder sb = new StringBuilder(pattern); DateTime now = DateTime.Now; sb.Replace("{User.GetFIO()|}", model.User.GetFIO() ?? ""); sb.Replace("{UserName|}", model.UserName ?? ""); sb.Replace("{User.Age:0}", model.User.Age.ToString("0")); sb.Replace("{Now:dd MMMM yyyy}", now.ToString("dd MMMM yyyy")); sb.Replace("{Now:HH:mm:ss}", now.ToString("HH:mm:ss")); text = sb.ToString(); } sw.Stop(); manualTotal += sw.Elapsed.TotalMilliseconds; } } WriteHeader(""); WriteElapsedTime(" ", patternInitialization); WriteElapsedTime("  ", patternFirst); Console.WriteLine(); WriteElapsedTime(string.Format("    {0} ", COUNT_PATTERNIZATOR), patternTotal); WriteElapsedTime("   ", patternTotal / COUNT_PATTERNIZATOR); WriteHeader(" ()"); WriteElapsedTime(string.Format("    {0} ", COUNT_MANUAL), manualTotal); WriteElapsedTime("   ", manualTotal / COUNT_MANUAL); Console.WriteLine(); Console.WriteLine("    ..."); Console.ReadKey(); }