Metaprogramming (with JavaScript examples)

This article is another attempt to rethink metaprogramming , which I periodically make. The idea is clarified each time, but this time it was possible to pick up fairly simple and understandable examples that are both very compact and illustrative, have real useful applications and do not pull libraries and dependencies. At the time of publication I will report this topic on OdessaJS , therefore, the article can be used as a place for questions and comments on the report. The format of the article makes it possible to more fully present the material than in the report, whose listeners are not exempt from reading .

A popular understanding of metaprogramming is usually very vague, and most often ends with such options:

Compilation templates and macros
Program that changes itself
A program that generates another program.

I propose the following definition:
Metaprogramming is a programming paradigm built on a programmatic change in the structure and behavior of programs.
And then we will analyze how it works, why it is needed and what advantages and disadvantages we get in the end.

What is modeling?

The concept of metaprogramming is closely related to modeling , because the method itself involves increasing the level of model abstraction by removing metadata from the model. As a result, we get the metamodel and metadata . During early or late binding (when compiling, translating, or running the program), we again obtain the model from the metamodel and metadata using an automatic program method. The created model can change many times, without changing the program code of the metamodel, and often, even without stopping the program.
')
Surprisingly, a person is able to successfully solve problems, the complexity of which exceeds the capabilities of his memory and thinking, with the help of building models and abstractions . The accuracy of these models determines their usefulness for decision-making and the development of control actions. The model is always not accurate and displays only a small part of reality, one or several of its sides or aspects. However, in limited conditions of use, the model may be indistinguishable from the real object of the domain. There are physical, mathematical, imitation and other types of models, but we will be interested, first of all, in information data models and program logic models. Programming paradigms, these are the models of program logic, for example, imperative, declarative, functional, event. As programmers, we can designate not writing code, but, first of all, building models and abstractions. Thus, metaprogramming allows us to raise the level of abstraction in models, which makes them more universal, and our work is much more interesting.

What is metaprogramming?

Metaprogramming is not something new; you have always used it, if you have experience in practical programming in any language and in any applied field of application. All programming paradigms, at least for the computer of the Vonnemann architecture , in one way or another inherit the basic principles of modeling from this architecture. The most important principle of the von Neumann architecture is the mixing of data and commands that define the logic of data processing in one universal memory. That is, the lack of a fundamental difference between the program and the data . This gives a lot of consequences, first, the machine needs to distinguish where the command is, and where is the number and what is its width and type, where is the address, and where is the array, where is the string, and where is the length of this string, in what encoding is it represented ., up to complex structures, as objects and scopes. All this is determined by metadata; without metadata, nothing happens at all in the programming languages for the von Neumann architecture. Secondly, the program gains access to the memory in which it is stored, other programs, their source code, and can process the code as data, which allows translation and interpretation, automated testing and optimization, introspection and debugging, dynamic linking, and much more. another.

Definitions:
Metadata is data about data. For example, the type of the variable is the metadata of the variable, and the names and types of the parameters of the function are the metadata of this function.
Introspection is a mechanism that allows a program to receive metadata about memory structures during work, including metadata about variables and functions, types and objects, classes and prototypes.
Dynamic binding (or late binding) is a function call through an identifier, which is converted to the address of a call only at the execution stage.
A metamodel is a high-level abstract model, from which metadata is taken out, and which dynamically generates a specific model when receiving metadata.
Metaprogramming is a programming paradigm built on a programmatic change in the structure and behavior of programs.

So, one cannot begin to apply metaprogramming from today, but one can realize, analyze and apply the tool consciously. This is paradoxical, but many seek to separate data and logic using the Vonleyman architecture. Meanwhile, they should not be separated, but combined in the right way. There are other architectures, such as analog solvers, digital signal processors (DSP), programmable logic integrated circuits (FPGAs), and others. In these architectures, the calculations are not made imperatively, that is, not by the sequence of processing operations specified by the algorithm, but by parallel digital or analog elements that implement many mathematical and logical operations in real time and have an answer ready at any moment. These are analogues of reactive and functional programming. In FPGA circuit switching occurs during reprogramming, and in DSP, the imperative logic controls the small re-switching of circuits in real time. Metaprogramming is also possible for systems with non-imperative or hybrid logic, for example, I see no reason that one FPGA could not reprogram the other.

Now we will consider the generalized model shown in the diagram of the program module. Each module necessarily has an external interface and software logic. And such components as configuration, state and permanent memory can either be absent or play the main role. A module receives requests from other modules through the interface and responds to them, exchanging data in certain protocols. The module sends requests to the interfaces of other modules from any place of its program logic, therefore incoming communications are integrated by the interface, and outgoing communications are scattered throughout the module body. Modules are part of larger modules and are themselves built from several or many sub-modules. The generic model is suitable for modules of any scale, ranging from functions and objects, to processes, servers, clusters and large information systems. When modules interact, requests and responses are data, but they necessarily contain metadata that affect how the module will process the data or how it tells the other module to process the data.

Typically, a set of metadata is limited by the fact that the protocol necessarily requires the structure of the transmitted data to be read. In binary formats, metadata is less than in syntax formats used to serialize data (such as JSON and MIME). Information about the structure of binary formats, for the most part, is located at the receiving module in the form of a struct (structures for C, C ++, C #, and other languages) or “wired” into the logic of the interpreting module in another way. It is rather difficult to divide where the data processing with the use of metadata ends and the metaprogramming begins. Conventionally, you can define the following criterion: when metadata not only describes structures, but increases the abstraction of program code in a module that interprets data and metadata, this is where metaprogramming begins. In other words, when the transition from the model to the metamodel occurs. The main feature of such a transition is the expansion of the universality of the module, and not the expansion of the universality of the protocol or data format. The diagram on the right shows how metadata is extracted from the data and enters the module, changing its behavior during data processing. Thus, the abstract metamodel contained in the module at the execution stage is transformed into a concrete model .

Before proceeding to the consideration of the techniques and techniques of metaprogramming, I would like to quote one, which I always quote when it comes to metaprogramming. It suggests the idea that metaprogramming is a reflection of such a fundamental law on which all cybernetic systems are based. That is, the systems are “living”, in which control, correction of behavior and activity parameters takes place with the help of closed-loop control. This allows systems to reproduce their state and structure in different conditions and with different modifications, while maintaining significant and varying behavior, including generating derivative systems for this.

“This is what I mean by producing work, or, as I called it last time,“ opera operans. ” In philosophy, there is a distinction between "natura naturata" and "natura naturans" - the generated nature and the generating nature. By analogy one could form - “cultura culturata” and “cultura culturans”. For example, the novel “In Search of Lost Time” is not built as a work, but as “cultura culturans” or “opera operans”. This is what the Greeks called Logos. "
// Merab Mamardashvili "Lectures on ancient philosophy"

How does metaprogramming work?

Based on the definition, you need to disassemble the following three questions:

When do changes occur?
What exactly is changing?
What are the changes taking place?

When changes occur : development time metaprogramming, for example, when the IDE analyzes your code as data, helps to modify it, prompts the names of objects and functions, their types and even generates templates or automatically builds blocks of code from schemes or visual modeling tools, for example , in the visual editors of user interfaces, databases and other CAD / CAM tools for automated development. Examples of compile-time changes: translators, including for creating typed algorithms from untyped and for generating code from a language with a higher level of abstraction into a language that is executed in a particular environment, up to the OS and the hardware platform. But we are more interested in changing the behavior of programs during their work , which we will consider in more detail below.

So, I propose the following classification of metaprogramming by time of changes in behavior and structure:

during development (Design time)
at compile time
while the application is running (Run time)
- during the execution of the task (Just-in-Time)
- between tasks (lazy)
  - on time (Timer)
  - external call (pull)
  - event (Push)

Just-in-Time interpreting and linking is not the best way, but sometimes it is the only possible way if the metadata comes along with the data. But metadata, however, changes less frequently than requests occur, so the model can be built in advance and cached while waiting for requests and data. You can update the model with specific calls, to minimize requests, or you can periodically poll the source of the storage of metadata for changes. It is best, of course, to have a channel of notifications from the source, so that it initiates an update on the push principle.

What exactly is changing?

data types and data structures ;
identifiers (the names of classes, types, variables, both inside the module, and the names by which the module refers to other modules);
calls (names of functions and methods, dynamic linking, including using the stub / skeleton pattern, in which, in one module, a stub is built representing the object, class or function located in the address space of the remote module, so that all calls to the stub were identical to remote calls);
parameters of data processing algorithms or parameters of models, which may vary;
substitution of expressions, formulas and logical expressions, regular expressions, etc .;
the code itself is dynamically interpreted (metadata is usually declarative, but this is not necessary, it can be imperative or it can contain imperative code fragments);
serialization / deserialization of data, objects and classes, as well as marshaling with code correction during transmission from one module to another (usually address correction, but there may be other corrections).

What are the changes taking place?

Parsing and translating syntax structures (with string operations or regular expressions).
Access to identifiers by name or index (including object parameters, associative arrays, etc.).
Complete introspection (see definition above, if the concept is unusual).
Individuation of first-class objects (main way):
- functions, through closures;
- objects, through dynamic creation and impurities;
- by all conceivable and inconceivable means provided by a programming language.

Why do we need metaprogramming?

Now we can identify the main tasks and cases where metaprogramming significantly simplifies the implementation or even makes the solution possible:

Expansion of functionality, increasing software versatility.
Dynamic subject areas when changes are regular mode.
Simplifying intersystem integration is a separate topic, but it helps a lot.

Example 1

Consider the simplest example of extracting metadata from a model and building a metamodel (see the example on github ). First, we define the problem of the example: there is an array of strings, you need to filter them according to certain rules: the length of suitable strings should be from 10 to 200 characters, inclusive, but excluding strings from 50 to 65 characters long; the line should begin with "Mich" and not begin with "Abu"; the string must contain “V” and not contain “Lev”; the string must end in “ov” and must not end in “iov”. We define the data for an example:

let names = [ 'Marcus Aurelius Antoninus Augustus', 'Darth Vader', 'Victor Michailovich Glushkov', 'Gottfried Wilhelm von Leibniz', 'Mao Zedong', 'Vladimir Sergeevich Soloviov', 'Ibn Arabi', 'Lev Nikolayevich Tolstoy', 'Muammar Muhammad Abu Minyar al-Gaddafi', 'Rene Descartes', 'Fyodor Mikhailovich Dostoyevsky', 'Benedito de Espinosa' ];

We implement logic without metaprogramming:

 function filter(names) { let result = [], name; for (let i=0; i<names.length; i++) { name = names[i]; if ( name.length >= 10 && name.length <= 200 && name.indexOf('Mich') > -1 && name.indexOf('V') === 0 && name.slice(-2) === 'ov' && !( name.length >= 50 && name.length <= 65 && name.indexOf('Abu') > -1 && name.indexOf('Lev') === 0 && name.slice(-3) === 'iov' ) ) result.push(name); } return result; }

Select the metadata from the model of the solution of the problem and form them into a separate structure:

 { length: [10, 200], contains: 'Mich', starts: 'V', ends: 'ov', not: { length: [50, 65], contains: 'Abu', starts: 'Lev', ends: 'iov' } }

Building a meta model:

 function filter(names, conditions) { let operations = { length: (s, v) => s.length >= v[0] && s.length <= v[1], contains: (s, v) => s.indexOf(v) > -1, starts: (s, v) => s.indexOf(v) === 0, ends: (s, v) => s.slice(-v.length) === v, not: (s, v) => !check(s,v) }; function check(s, conditions) { let valid = true; for (let key in conditions) valid &= operations[key](s, conditions[key]); return valid; } return names.filter(s => check(s, conditions)); }

The advantage of solving the problem using metaprogramming is obvious; we have obtained a universal string filter with configurable logic. If you need to filter more than once, but several, on the same metadata configuration, then the metamodel can be wrapped into a closure and get cached for an individual function to speed up the work.

Example 2

We will write the second example right away with the help of metaprogramming (see the example on github ), because if I imagine its size in size in a govnokode, then it becomes scary to me. Task description: you need to make HTTP GET / POST requests from specific URLs or load data from files and transfer received or read data via HTTP PUT / POST to other URLs and / or save them to files. There will be several such operations and they need to be performed at various time intervals. The task can be described as metadata as follows:

 [ { interval: 5000, get: 'http://127.0.0.1/api/method1.json', save: 'file1.json' }, { interval: '8s', get: 'http://127.0.0.1/api/method2.json', put: 'http://127.0.0.1/api/method4.json', save: 'file2.json' }, { interval: '7s', get: 'http://127.0.0.1/api/method3.json', post: 'http://127.0.0.1/api/method5.json' }, { interval: '4s', load: 'file1.json', put: 'http://127.0.0.1/api/method6.json' }, { interval: '9s', load: 'file2.json', post: 'http://127.0.0.1/api/method7.json', save: 'file1.json' }, { interval: '3s', load: 'file1.json', save: 'file3.json' }, ]

We solve the problem using metaprogramming:

 function iterate(tasks) { function closureTask(task) { return () => { console.dir(task); let source; if (task.get) source = request.get(task.get); if (task.load) source = fs.createReadStream(task.load); if (task.save) source.pipe(fs.createWriteStream(task.save)); if (task.post) source.pipe(request.post(task.post)); if (task.put) source.pipe(request.put(task.put)); } }; for (let i = 0; i < tasks.length; i++) { setInterval(closureTask(tasks[i]), duration(tasks[i].interval)); } }

We see that we have written "beautiful columns" and we can make another convolution, having rendered the metadata already inside the metamodel. How metamodel configured by metadata will look like:

 function iterate(tasks) { // Metamodel configuration metadata // let sources = { get: request.get, load: fs.createReadStream }; let destinations = { save: fs.createWriteStream, post: request.post, put: request.put }; // Metamodel logic // function closureTask(task) { return () => { console.dir(task); let verb, source, destination; for (key in sources) if (task[key]) source = sources[key](task[key]); for (key in destinations) if (task[key]) source.pipe(destinations[key](task[key])); } } for (let i = 0; i < tasks.length; i++) { setInterval(closureTask(tasks[i]), duration(tasks[i].interval)); } }

I note that the example uses closures for the individualization of tasks.

Example 3

The second example uses the duration function, which returns a value in milliseconds, which we did not consider. This function interprets the interval value specified as a string in the format: “Dd Hh Mm Ss”, for example “1d 10h 7m 13s”, each component of which is optional, for example “1d 25s”, if the function receives a number, then it returns it, it is necessary for the convenience of setting metadata if we specify the interval directly in milliseconds.

 // Parse duration to seconds, example: duration('1d 10h 7m 13s') // Parse duration to seconds // Example: duration('1d 10h 7m 13s') // function duration(s) { let result = 0; if (typeof(s) === 'string') { let days = s.match(/(\d+)\s*d/), hours = s.match(/(\d+)\s*h/), minutes = s.match(/(\d+)\s*m/), seconds = s.match(/(\d+)\s*s/); if (days) result += parseInt(days[1]) * 86400; if (hours) result += parseInt(hours[1]) * 3600; if (minutes) result += parseInt(minutes[1]) * 60; if (seconds) result += parseInt(seconds[1]); result = result * 1000; } if (typeof(s) === 'number') result = s; return result; }

Now we implement the interpretation configured by the metadata:

 function duration(s) { if (typeof(s) === 'number') return s; let units = { days: { rx: /(\d+)\s*d/, mul: 86400 }, hours: { rx: /(\d+)\s*h/, mul: 3600 }, minutes: { rx: /(\d+)\s*m/, mul: 60 }, seconds: { rx: /(\d+)\s*s/, mul: 1 } }; let result = 0, unit, match; if (typeof(s) == ='string') for (let key in units) { unit = units[key]; match = s.match(unit.rx); if (match) result += parseInt(match[1]) * unit.mul; } return result * 1000; }

Example 4

Now let's look at metaprogramming with introspection applied to integrate modules. First, we define the remote methods on the client using this structure and show how to use these calls when writing application logic:

 let ds = wcl.AjaxDataSource({ read: { get: 'examples/person/read.json' }, insert: { post: 'examples/person/insert.json' }, update: { post: 'examples/person/update.json' }, delete: { post: 'examples/person/delete.json' }, find: { post: 'examples/person/find.json' }, metadata: { post: 'examples/person/metadata.json' } }); ds.read({ id: 5 }, (err, data) => { data.phone = '+0123456789'; ds.update(data, () => console.log('Data saved')); });

Now let's initialize from the metadata received from another module and show that the applied logic has not changed:

 let ds = wcl.AjaxDataSource({ introspect: { post: "examples/person/introspect.json" } }); ds.read({ id:3 }, (err, data) => { data.phone ="+0123456789"; ds.update(data, () => console.log('Data saved')); });

In the next example, we will create a local data source with the same interface as the remote one and show that the application logic has not changed either:

 let ds = wcl.MemoryDataSource({ data: [ { id: 1, name: 'Person 1', phone: '+380501002011', emails: [ 'person1@domain.com' ], age: 25 }, { id: 2, name: 'Person 2', phone: '+380501002022', emails: [ 'person2@domain.com', 'person2@domain2.com' ], address: { city: 'Kiev', street: 'Khreschatit', building: '26' } }, { id: 3, name: 'Person 3', phone: '+380501002033', emails: [ 'person3@domain.com' ], tags: [ { tag: 'tag1', color: 'red' }, { tag: 'tag2', color: 'green' } ] }, ]}); ds.read({ id: 3 }, (err, data) => { data.phone ="+0123456789"; ds.update(data, () => console.log('Data saved')); });

findings

Metaprogramming Techniques

Task description style: declarative (metadata), use of imperative and functional inserts
Hashes (associative arrays) do not know the key in advance: let a = {}; a [key] = value;
Interpret strings, invent your own syntax or take common (json, js, regexp ...)
Impurities (mixins): do not know in advance where to add function mixin (a) {a.fn = () => {...}}
Closures: we personalize the functions fn = (a => () => a * 2) (value)

The implications of metaprogramming

Code size: often decreases dramatically, but sometimes it may increase slightly
Speed: slightly reduced, but with proper implementation remains about the same
Flexibility: the software code becomes more versatile, the scope of software is expanding
Integration: usually much easier and requires less code changes
Pleasure from work: it is more interesting to metaprogramme therefore pleasure and motivation is more
Development speed: develop longer, and maintain much easier, save a lot of time