Metaprogramming

The purpose of this article is to draw the attention of the widest IT public to metaprogramming and all of its many forms and techniques. I tried to collect a classification of everything I know about this topic and show its applicability not only in speculative tasks, but also in the development of ordinary applications. But this is rather a plan for further study and, I hope, a push for more active discussion.

So, I will try to give an alternative definition: metaprogramming is a paradigm of building an information system code with a dynamic change in behavior or structure depending on the data, user actions, or interaction with other systems. The objectives of metaprogramming: increase code abstraction and its flexibility, reuse, accelerate development, simplify intersystem integration. In fact, we all use metaprogramming to some extent, I can even now remember how I used it 15 years ago, without even knowing what it is called, then I still could not make any classification.

What we know metaprogramming techniques (I hope that they will complement me):

1. Compiled technology
1.1. Patterns, macros and parametric polymorphism . For one block of code, at the stage of precompilation, several variants are generated for different types of parameters; this is the oldest type of mapping.
1.2. Optimizing precompilation . Calculation and optimization of all expressions and entire functions that do not contain variables, at the stage of compilation or precompilation.
1.3. Source code generators for compiled languages. Before compiling, at the stage of building the project, in a batch manner, with or without metadata, files, functions, classes, forms, etc. are generated.
1.4. Writing a domain-specific compiled language (for a specific range of tasks) - DSL (Domain-specific Programming Language).
')
2. Interpreted techniques
2.1. Evalue code (eval) from string variables in interpreted languages ​​or languages ​​that support late compilation to bytecode during execution.
2.2. Writing your specialized interpretable imperative language for a task (LOP - Language Oriented Programming).
2.3. Create or use a declarative language , a serialization format, a special syntax, or a subset of such syntaxes as XML, JSON, XAML, etc.
2.4. Dynamic generation and execution of code in query languages , for example: SQL, XQuery, LINQ, etc.

3. Hybrid technology
3.1. Introspection - providing access to internal language structures, data types, classes, functions, etc. Receiving metadata about structures and traversing them in a loop, or obtaining function parameters as an array with the ability to analyze the class of each.
3.2. Dynamic interpretation of metamodels . Special terminology for this type has not yet been developed, more precisely, it has not been settled, so I will describe it in more detail below. And here I will give the main feature: a metamodel (a domain model in a meta-language) contains both imperative and declarative components affecting each other, and the application becomes a “virtual machine” for launching the metamodel.
3.3. Distributed information system with dynamic binding based on the interpretation of metamodels . This is the application of clause (3.2) to client-server or intersystem interaction.

Dynamic application environment for business objects

Now more about the dynamic interpretation of meta-models. How do imperative and declarative constructions affect each other? Obviously, we analyze declarative descriptors — some kind of formalized data format, regular grammar, which you can easily parse with regular expressions, string operations, or take a ready-made parser (for example, JSON or XML). Thus, the imperative algorithm can change the structure and sequence of code execution. But moreover, the meta-model also contains interpretable scripts for expressing business logic, the scripts get the declarative part of the meta-model already deployed in memory at their complete disposal and can access the libraries. Here everyone thought about JavaScript of course. I note that the scripts from the meta-model do not have access to the introspection of the “virtual machine” , but only to the data and metadata, i.e. to domain model. Such a script may not even know where it is running, on the client or on the server. The application environment is a layer between the application model and the operating system, which allows you to virtualize the application application, untie it from the system and platform.

Cloud technologies and virtualization have now risen only to the level of infrastructure and platform, the next, the application layer is practically not affected . The need for a new application architecture is becoming increasingly clear. Service architecture (SOA) undoubtedly allows to satisfy most of the needs of corporate systems, however, due to its heavy weight, static nature and development complexity, only large companies can afford to develop software in this technology. This is due to the complex technological cycle of software creation, including design, programming, debugging, testing, building, delivery to the user, installation, configuration and support. Each new version of an application system requires a multitude of steps in order to become available at the user's workplace. And the transition between versions is fraught with the difficulty of data conversion and the backward compatibility of data exchange formats and file formats. All this is the result of the following features of the architecture of software systems:
1. Mixing in the program code abstractions of different levels within a single class or module. For example, the implementation of reading / writing from the database, business logic and visualization in the same class.
2. High connectivity code of two adjacent abstract layers of the application. With one-sided or two-way dependency layers. For example, a separate windowed AWP or web interface is rigidly tied to the set of server API functions, their parameters, data types and classes, and the server application is tied rigidly to the structure of tables in the database. Often, high connectivity is expressed in the presence of class and function identifiers wired into the code, interfaces and parameters, tables and fields.
3. Assembly-oriented life cycle and compilation of business models into machine code or byte-code with subsequent manual deployment on the server.
4. Rigid fixing of interfaces between system modules and network interfaces (by structure, call parameters and data types).

The use of a dynamic application environment using interpretable metadata in preparing business models for launching in a “virtual machine” makes it possible to drastically reduce the time for system modification, to the extent that it is possible to make a flexible modification with the usual standard operating mode of the application system, rather than stitching the model in compiled code. Then someone may argue that system performance will drop significantly, but not, and that's why: most of the executable code during the operation of such a system is algorithms written in a compiled language and providing API for meta-modems. It is a library for parsing, a library for processing vector graphics, network libraries and libraries of visual components. All this is compiled, but it is completely abstracted from the task and constitutes a “virtual machine”, and the interpreted code only binds, “stitches” all this set of tools in such a way as to solve an applied problem.

Now we invert 4 reasons for the inflexibility of systems using various metaprogramming techniques and get a recipe for systems with dynamic interpretation of the metamodel :
1. Separation of abstractions of different levels in the program code. For example, the level of visual components, the level of network transport, the level of the library of applied algorithms and the level of the business model may not be related to each other at the compilation stage of the environment, however, links will be dynamically built based on metadata at the time of the request to the corresponding functionality and cached until metamodel changes.
2. The absence of direct and inverse dependencies in the abstract layers of the application , the use of metaprogramming techniques, introspection, declarative and active languages ​​to describe business objects. At the same time, the internal connectedness of classes within the business model can be enhanced.
3. Use of compilation for the level of the application environment and the principle of interpretation for the level of business models. For the environment, the life cycle remains assembly, but the business model can change at least every minute without redeploying.
4. Introduction of dynamic interfaces between the modules (described by declarative languages) and network interfaces supporting intra-inspection at the network protocol layer for the interaction of applications with dynamic structure and parameters.

So, the application environment itself will be a classic application and should go through all the stages of development and deployment, however, the business model running in the environment should not be compiled and installed in a computer to increase flexibility, it is dynamically deployed during runtime. Data structures and scripts prepared for execution by the lexer or even converted to bytecode can be cached in the application environment until the metamodel is modified or the individual class or parameter of the metamodel changes. The first launch of the necessary function will be accompanied by the parsing of structures, and the preparation of a new byte code or data structures, however, subsequent operations should not yield to speed as compared to machine code.

Distributed Dynamic Binding Information System

When two or more systems interact through network application interfaces, dynamic binding based on the interpretation of the meta-model will allow application information systems to interact, which were not even supposed to be connected before. When transmitting a meta-model over the network, the parties do not know in advance the structure and parameters of information objects, and are not tied rigidly to the names of functions and parameter sets when making intersystem calls. Instead, the parties know the meta description language, which allows them to dynamically interpret the data and make calls, forming parameters and interpreting the answers of the remote party.

Dessert

And finally, in order to entertain the aesthetic side of a respected audience, I will show a small example of the use of metaprogramming for animation in JavaScript. Recently there was an article about impress.js and somehow so coincided with the fact that I thought about such a decision in the context of my insanity in metaprogramming and that's what happened, see a live example . And here is the code from it:

The imperative part:

$(function() { var Images = $("#sky div"), Timers = []; Images.each(function() { fly = function(Id,Range,Duration,Direction) { var Way1 = {}, Way2 = {}; Way1[Direction] = '+='+Range; Way2[Direction] = '-='+Range; $('#'+Id).animate(Way1,Duration).animate(Way2,Duration); } var Image = $(this), Id = Image.attr('id'), Range = Image.attr('data-fly-range'), Duration = parseInt(Image.attr('data-fly-duration')), Direction = Image.attr('data-fly-direction'); Timers[Id] = setInterval('fly("'+Id+'","'+Range+'",'+Duration+',"'+Direction+'")',Duration*2); fly(Id,Range,Duration,Direction); }); }); 

Declarative part:

 <div id="sky"> <div id="castle-white" class="castle" style="background:url(images/castle-white.png) no-repeat; left:350px; top:-10px; width: 287px; height: 698px;" data-fly-range="10" data-fly-duration="3000" data-fly-direction="top"></div> <div id="castle-indigo" class="castle" style="background:url(images/castle-indigo.png) no-repeat; left:560px; top:235px; width: 471px; height: 397px;" data-fly-range="15" data-fly-duration="2500" data-fly-direction="top"></div> <div id="fog-03" class="fog" style="background:url(images/fog-03.png) no-repeat; left:500px; top:325px; width: 599px; height: 310px;" data-fly-range="15" data-fly-duration="4000" data-fly-direction="left"></div> <div id="castle-blue" class="castle" style="background:url(images/castle-blue.png) no-repeat; left:690px; top:350px; width: 521px; height: 288px;" data-fly-range="10" data-fly-duration="2000" data-fly-direction="top"></div> <div id="fog-02" class="fog" style="background:url(images/fog-02.png) no-repeat; left:50px; top:280px; width: 1279px; height: 375px;" data-fly-range="20" data-fly-duration="3500" data-fly-direction="left"></div> <div id="castle-green" class="castle" style="background:url(images/castle-green.png) no-repeat; left:470px; top:60px; width: 353px; height: 621px;" data-fly-range="15" data-fly-duration="1500" data-fly-direction="top"></div> <div id="castle-yellow" class="castle" style="background:url(images/castle-yellow.png) no-repeat; left:210px; top:50px; width: 267px; height: 630px;" data-fly-range="5" data-fly-duration="500" data-fly-direction="top"></div> <div id="fog-01" class="fog" style="background:url(images/fog-01.png) no-repeat; left:100px; top:435px; width: 580px; height: 194px;" data-fly-range="25" data-fly-duration="3000" data-fly-direction="left"></div> <div id="castle-orange" class="castle" style="background:url(images/castle-orange.png) no-repeat; left:-110px; top:-30px; width: 524px; height: 736px;" data-fly-range="20" data-fly-duration="1000" data-fly-direction="top"></div> </div>