Introduction

A few days ago, I discovered a wonderful Clojure PL, one of the modern Lisp dialects, which features a good implementation of multi-threading tools, compilation into jvm bytecode, respectively, the ability to use java libraries, jit compilation, etc. About Clojure can be read for example here . But this article is about metaprogramming. Lisp is designed in such a way that the data and the code in it are the same. Function declarations, macros, function calls, macro deployment - in Lisp, these are all just lists, possibly nested into each other.

(defn square [foo] (* foo foo)) (defmacro show-it [foo] `(println ~foo)) 

')
This combination of code and data provides powerful metaprogramming capabilities — code that writes code, writes code, writes code, etc. - This is the most common thing for programming in Lisp. In compile-time, all the functionality of the language is fully accessible to us, we can call functions, deploy macros, possibly recursively. For example, if we define this macro:

 (defmacro recurs [foo bar] (println "hello from compiler" foo) (case (<= bar 0) true `(defn foo [] ~foo) false `(recurs ~(- foo 1) ~(- bar 1)))) 

And in the code we insert the expression

 (recurs 2 1) 

That at compile time we will see

 hello from compiler 2 hello from compiler 1 

And the macro in this case will unfold into the definition of the function foo with arity 1 and the return value 1, i.e.

 (def user/foo (clojure.core/fn ([] 1))) 

If we write (recurs 3 1) then the function foo will return the value 2, etc. Lisp macros are an excellent tool for hiding any complex logic or control structures behind a simple syntax and accordingly for extending the expressive means of the language itself. Many constructions of a language like "defn", "- >>", "->" are in fact also simply macros.

Lisp macros are ordinary functions, with the exception that they are executed at compile time, and to master the technique of writing them, in principle, it is enough to know how the following 4 “special forms” work

 `(expr) '(expr) ~(expr) ~@(expr) 

About this you can read in detail here . In a nutshell, the quote ('and `) construct is just a notation to the compiler:“ this expression should not be tried, but returned as it is, i.e. in the form of a code, and the unquote (~ and ~ @) construction roughly means “execute this expression and insert the result in this place of the code”. Naturally, unqoute constructs only make sense within quote constructs. Thus, macros are functions that take as arguments of a quote construct, returning as values ​​of a quote construct and are executed in compile-time.

Why do I need pipe matching?

To demonstrate the power of Lisp macros, we will write a pipe matching implementation for Lisp. What is pipe matching? As the name implies, this is a composition of two control structures: pipe and pattern matching.

Pipes in Clojure's PL are just macros that take the n-th number of expressions as arguments and expand in a certain way to the composition of these expressions, to put it in quite simple terms:
(-> expr1 expr2 expr3 ...) here the pipe "->" inserts expr1 as the first argument into expr2 (all other arguments move 1 position to the right), the resulting expr12 inserts as the first argument into expr3, etc. Example

 (-> (func1 "foo") (func2 "bar") (func3 123)) 

 (func3 (func2 (func1 "foo") "bar") 123) 

These two expressions are the same. The question of the readability of the code is a matter of taste, but personally it’s obvious to me that the pipes are needed. There are other pipes, for example - >>, which, if not difficult to guess, works in a similar way ->, it only inserts not the first but the last argument. In general, you can write any kind of pipe, but as a rule these two are most often used.

Pattern Matching is an obvious thing, for example, for erlang / elixir, haskell, ml is a developer, but in a few words it’s not so easy to explain here, you should rather feel it. Very distant pm can be defined as a composition of assignment and comparison. The uninitiated may refer to Wikipedia or see my previous article , where I briefly described the use of pm in PL elixir / erlang. Clojure does not have native pattern matching, but as you might guess there are libraries that implement the pm macros almost identical pm in erlang. We will not reinvent the wheel and use the library macro match in this project.

Now back to the question - why do you need to connect pm and pipe in one entity pipe matching? When we write programs of the “hello world” level, where all functions are pure, there are no side - effects and everything is extremely deterministic, perhaps this does not make much sense. But in real production you cannot do without dirty functions. For example, we need to upload a photo to an album in the social network VK. According to the documentation for this, we need the album id, access key and the actual data to be downloaded. The general loading algorithm is as follows

    url  (get-http ,  json)   (post-http ,  json)    (get-http ,  json) 

Reading a file, parsing json, http requests are all dirty functions. During their call, anything can happen - there is no file at the given address, invalid json, valid json without the required fields, disconnection, the server answered not 200, but something else, etc. and etc. In each of these functions, everything is very bad. And everything would be fine, but each next function requires correct results from the previous one. In each individual function, we can write clauses for bad cases and avoid exceptions, but you need to have it all work together, and if something goes wrong - we want to know exactly what and where exactly went wrong. Perhaps many readers will want to write something like this.



But this is a shame) Which is difficult not only to maintain and debug, but simply to read / write / understand. It is in such situations (which will agree pretty often) pipe matching will make the code very simple by hiding all the complex logic. An example of using the macro pipe_matching and pipe_not_matching

 (defn nested_process foo bar baz (pipe_matching {:ok some_data} (simple_func1 foo) (simple_func2) (simple_func3 bar) (simple_func4 baz))) 

What happens here: the nested_process function accepts foo, bar, and baz arguments. And then follow-up calls of possibly dirty functions simple_func1 with arity 1, simple_func2 with arity 1, simple_func3 with arity 2 and simple_func4 with arity 2 begin, while as in ordinary pipes the result of the previous expression is the first argument of the next one. And now the most important thing is that we set the pattern {: ok some_data} with the first argument of the pipe_matching macro. Under this pattern is suitable map, where there is a key: ok with any value. And while simple_func functions return values ​​suitable for this pattern, the next function is called (as in a normal pipe). But as soon as any of the simple_func functions returns a value not matching this pattern, it will return as the value of the nested_process function and will not be passed further along the chain. For example, if simple_func3 returns {: error "on simple_func3 server ans 500"}, the function simple_func4 will not be called at all, and nested_process will return {: error "on simple_func3 server ans 500"}. You can also make a similar macro pipe_not_matching, which can be used like this

 (defn nested_process foo bar baz (pipe_not_matching {:error some_error} (simple_func1 foo) (simple_func2) (simple_func3 bar) (simple_func4 baz))) 

I think how it works - it is clear from the context. If simple_func3 returns here {: error “on simple_func3 server ans 500”}, and the 1st and 2nd functions return the values ​​that do not match the pattern before, the result will be the same as in the previous example. In practice, I prefer pipe_not_matching. As a result, we have almost literal serialization of the logic of the problem into the code without any if / else / elseif / case / switch etc. In general, everything for what we love FP - the code is the task itself without unnecessary abstract entities. Cool? Now let's see how to write such macros on Lisp literally in a couple of lines.

We write pipe matching for Lisp

First of all, let's write the dependencies for the namespace of our library - we need only one macro “match”

 (ns pmclj.core (:use [clojure.core.match :only (match)])) 

We define in general 2 macros, which are our goal

 (defmacro pipe_matching [pattern init_expression & other_expressions ] (pipe_matching_inner {:pattern pattern, :result init_expression, :expressions other_expressions, :continue_on_match true})) (defmacro pipe_not_matching [pattern init_expression & other_expressions ] (pipe_matching_inner {:pattern pattern, :result init_expression, :expressions other_expressions, :continue_on_match false})) 

The macro will take pattern as the first argument, the remaining arguments will be expressions for pm itself. It is logical that we need at least one expression, so we will call it init_expression and set it as the obligatory second argument. Pay attention to the & sign - here it means that other_expressions is a list of any length (possibly empty) consisting respectively of possible arguments (third, fourth, etc.). Thus, a macro can be expanded with any number of arguments greater than 1.

Further, for convenience, we simply serialize the arguments to the map with the keys: pattern,: result,: expressions,: continue_on_match. result here - the result obtained in the previous step, expressions - the remaining expressions not expanded into the result code of the macro, continue_on_match - true / false: means what to do if result matched the pattern - continue the chain of calls or return the value.

The resulting map is passed to the recursive function pipe_matching_inner which returns the code we need. Yes, this is the beauty of Lisp, you can call functions in compile-time, so long as they were already compiled at the time of the call. The function is as follows.

 (defn pipe_matching_inner [{pattern :pattern, result :result, expressions :expressions, continue_on_match :continue_on_match}] (case (or (= expressions nil) (= expressions ())) true result false (let [to_pipe (first expressions) rest_expr (rest expressions)] `(let [~'res ~result] (case (= ~continue_on_match (check_match ~'res ~pattern)) true ~(pipe_matching_inner {:pattern pattern, :result `(-> ~'res ~to_pipe), :expressions rest_expr, :continue_on_match continue_on_match}) false ~'res))))) 

It takes a map, which we formatted in the body of the macro. Here, by the way, you can see that native pm in some form and with a rather strange syntax in Clojure is still there: [{pattern: pattern, result: result, expressions: expressions, continue_on_match: continue_on_match}].
Case - expression here says the following: if we have already processed all the expressions - nothing remains how to return the result. Otherwise - we process. Next comes the very important Lisp special form - let. There are no global variables and assignments here and there cannot be, but we can do local binding in the spirit of the expression => symbol. The first and rest functions actually return the first one and everything except the first element of the list, everything is transparent here - we take the following expression to process.
Further, the most interesting is the actual code, which we dynamically generate in compile-time. There will already have to make a little mental effort to understand what is happening. We locally bind the result from the previous expression to the res symbol (in order not to execute this expression more than once at runtime). Pay attention to ~ '- this is a small hack, connected with the fact that when compiling the characters are attached to the corresponding namespace, the combination ~' unhooks them as it were, we will not delve into these issues in the context of this narration. This is followed by a case - an expression that will be honestly executed at runtime - we will check if our res pattern matches the pattern. check_match - the macro itself, very simple, based on the match macro library

 (defmacro check_match [obj pattern] `(match ~obj ~pattern true :else false)) 

And further, depending on whether the expression check_match is expected true - for pipe_matching and false - for pipe_not_matching, either the call chain will continue or res will return. The subsequent call chain is also beautifully drawn by the recursive calls to the pipe_matching_inner function.

Let's see how it will look like now.
Here is such a beautiful pipe matching

 (pipe_matching {:ok some} (func1 "foo") (func2 "bar") (func3 123)) 

In fact, it unfolds in such a local hell

 (clojure.core/let [res (func1 "foo")] (clojure.core/case (clojure.core/= true (pmclj.core/check_match res {:ok some})) true (clojure.core/let [res (clojure.core/-> res (func2 "bar"))] (clojure.core/case (clojure.core/= true (pmclj.core/check_match res {:ok some})) true (clojure.core/-> res (func3 123)) false res)) false res)) 

I want to draw your attention that to build the same logic without pipe matching, you would have to write all this hell by yourself)

Fully code can be found here .

In conclusion, I want to say that at first glance, the Lisp syntax is certainly not very user-friendly, but IMHO it fits well for building large / complex systems. Of course, after erlang / elixir you feel without otp as without hands, but I am sure that it is a matter of time. Anyway, this is my first jvm experience, and I think it will be pretty successful)

UPD:

Made some refactoring after which the macros themselves do not look so scary. He listened to the advice and added 4 more macros:

pred_matching / pred_not_matching - the same, only accepts the first argument lambda with arity 1.
key_matching / key_not_matching - the same, only accepts a key with the first argument, and at runtime searches for the results of the calls with the specified key value, if the value is nil or missing, it is false otherwise - true.

usage example

 (defn func1 [arg] {:ok arg}) (defn func2 [arg1 arg2] {:fail (+ (get arg1 :ok) arg2)}) (defn func3 [arg1 arg2] {:ok (+ (get arg1 :ok) arg2)}) (defn example_pred [] (pred_matching #(contains? % :ok) (func1 1) (func2 2) (func3 3))) (defn example_pm [] (pipe_matching {:ok some} (func1 1) (func2 2) (func3 3))) (defn example_key [] (key_matching :ok (func1 1) (func2 2) (func3 3))) 

By the way, you will get an example of code where a correct return from the previous function is needed: the place (+ (get arg1: ok) arg2) potentially contains exceptions, for example, if the get function returns nil.
As expected, all three functions of example will return the same value in this case.

 (example_pred) {:fail 3} (example_pm) {:fail 3} (example_key) {:fail 3} 

Also added rkey_matching / rkey_not_matching macros, work in the same way, just look for key entries in the structure recursively, and if they find, terminate the call chain and return the error found. As practice has shown - this is the most convenient option in terms of ease of use / versatility.

The universality of control structures here is naturally such pred_matching> pipe_matching> rkey_matching> key_matching. Which of these to use depends on personal taste and the complexity of the data we work with.

The resulting code is here.