Python code generation with Hy

1. What is Hy

Hy is a Lisp dialect that is embedded in a python.

Due to the fact that Hy transforms its Lisp-like code into an Abstract Syntax Tree (AST) of a python, with the help of Hy, the whole wonderful world of python is at your fingertips and in the shape of a Lisp.

2. About Hy syntax, very briefly

Hy is a peculiar language, similar to each of its parents (more, of course, to Lisp). For those who are not familiar with the Lisp syntax, it can be summed up in this case.

1. Indent does not matter. Instead, there are levels of nesting in expressions from round brackets.
2. In all function calls, the function name is in brackets with the argument list in the first place; commas in the argument list are not used.
3. All operators are written as if they were functions.
4. Colons are not used.
5. Literals for strings and dictionaries work as before; strings are written in double quotes, tuples look like function calls ",".

Although it may seem unusual at first, in practice, due to the simplicity of this syntax (which is achieved by reducing the number of special characters used), you can get used quickly.

3. Terminology notes

It is necessary to separately discuss the terminology used. The basic terms in English are quoting, unquoting, quaziquoting, splicing, macro expansion. In the Russian translation of the book Practical Common Lisp, the words “quoting”, “quoting”, “quasi-quotation” are used for them - and for the last of them - “the disclosure of macros”. I do not think this translation option is convenient.

This material will be used as translations "hiding" for quoting, "opening" for unquoting, "quasi-covering" for quaziquoting, "structural opening" for splicing, "expanding macro" for macro expansion.

In the following code examples, you can see the syntax of these operations:

• ' :: hide; applies to the following form of Hy; instead, it will be processed as data.
• ` :: quasi-hiding; more complex form of hiding, allowing you to build more complex syntactic structures.
• ~ :: disclosure; since it is used in python for the constructor of tuples, the character used is different from the traditional Lisp comma. It is used in a quasi-hidden form and places into it the result of the execution of the form following it.
• ~@ :: structural disclosure; works similarly to the previous operation with the following difference: the result of the form evaluation must be a list, and its elements are placed in the ambient quasi-open form.

Execution indicates a function call if the form is a list, and access to the value of the symbol otherwise; literals when executed remain by themselves.

4. The essence of the method

To obtain a construction from hy as an object with which it is possible to carry out manipulations, it is possible with the help of concealment. Extending macros alone will not help - because the macro-extended code is executed immediately. In order to even simply inspect its extension without concealment is indispensable, for example:

 (macroexpand '(my-macro param1 param2 (do (print "hello!")))) 

Therefore, you can immediately focus on obtaining structures - for example, by generating their functions on some input data.

Here we have several difficulties that we should not forget.

1. The hidden construct itself does not have to be syntactically correct for hy itself. In our case, correctness is necessary.
2. Not all correct hy constructs can be translated to the correct python code. In particular, this refers to variable names - the rules for the names of characters in hy much more relaxed.

If there is a well-generated code structure in any variable (for example: the result of a call to the generating function), you can get the code on python, for example, like this:

 (with [fd (open "some/python/file.py" "a")] (.write fd "\n") (.write fd (disassemble code True))) 

5. Generation of names

When generating code on python, in contrast, for example, to writing macros, it is important for us which names are new characters, i.e. in the case of python, the names of the newly generated functions, classes, variables. In other words, the standard method in Lisp ( (gensym) ) does not suit us. Also in hy there is no standard for many Lisp (intern) , used to turn an arbitrary string (adjusted for grammar constraints) into a character.

Fortunately, the entire hy code base is available, and a quick search HySymbol sure that (gensym) works by creating HySymbol objects. We can do the same.

The following example, despite what was said earlier, is a macro.

 (defmacro make-vars [data] (setv res '()) (for [element data] (setv varname (HySymbol (+ "var" (str element)))) (setv res (cons `(setv ~varname 0) res))) `(do ~@res)) 

In addition to generating the name of a variable, there is one more useful detail. This is the collection of the resulting AST from fragments by compiling a list of these fragments, and then disclosing this list in the right frame.

6. Example and remarks

When using hy for code generation (as opposed to just working on it), some aspects emerge that are hidden when sending code for execution.

First of all, it concerns the fact that in the context of AST and the execution context the same expressions mean different things.

• [ ] not just a python list, but a HyList ;
• { } does not open the Python dictionary, but HyDict , and in the internal model, hy is presented as a list;
• "" not just a string variable, but HyString.

and so on. The main conclusion that can be drawn from this is that the listed (and other) constructions, while hidden, will be correctly converted into corresponding python literals when disassembling.

In order to statically populate lists or dictionaries in python code, you will need to use a structured expansion operation.

 (setv class-def [`(defclass ~class-name [~(HySymbol (. meta-base __name__))] [army_name ~army-name faction_base ~(HyString faction) alternate_factions [~@(map HyString alternate-fac-list)] army_id ~army-id army-factions [~@(map HyString army-factions)]] (defn --init-- [self &optional [parent None]] (apply .--init-- [(super ~class-name self)] {~@(interleave (map HyString class-grouping) (repeat 'True)) "parent" parent}) ~@(map (fn [key] `(.add-classes (. self ~(HySymbol key)) [~@(genexpr (HySymbol (. ut __name__)) [ut (get class-grouping key)])])) class-grouping)))])))) 

In the given example, lists are filled in the fields alternate_factions and army-factions class being declared. Note that in the Python code, both of these fields will be through the underscore. The filling is based on the lists of strings, so the structural disclosure of the result of the conversion of the python to the HyString in alternating strings is HyString .

From the above code fragment in hy you can generate the following code fragment in python:

 class DetachPatrol_adeptus_ministorum(DetachPatrol): army_name = u'Adeptus Ministorum (Patrol detachment)' faction_base = u'ADEPTUS MINISTORUM' alternate_factions = [] army_id = u'patrol_adeptus_ministorum' army_factions = [u'IMPERIUM', u'ADEPTA SORORITAS', u'<ORDER>', u'ADEPTUS MINISTORUM'] def __init__(self, parent=None): super(DetachPatrol_adeptus_ministorum, self).__init__(*[], **{u'heavy': True, u'troops': True, u'transports': True, u'hq': True, u'fast': True, u'elite': True, u'parent': parent, }) self.heavy.add_classes([Exorcist, Retributors, PenitentEngines]) self.troops.add_classes([BattleSisters]) self.transports.add_classes([ASRhino, Immolator]) self.hq.add_classes([Celestine, Canoness, Jacobus]) self.fast.add_classes([Dominions, Seraphims]) self.elite.add_classes([ArcoFlagellants, Assassins, Celestians, Dialogus, Hospitaller, Imagifier, Mistress, Priest, Repentia, PriestV2, Crusaders]) return None 

Separately, I would like to note how the call to the parent class constructor is described.

• For functions from a class (which begin with . ), apply treats the first positional argument provided to it (the first element of the list, which is its second parameter) as an object whose method is invoked;
• You can populate a dictionary of named arguments with a structural expansion;
• To match each key (the string converted to HyString ) of a value, interleave is used, which iterates over two lists, interspersing their elements;
• The True symbol is hidden, in the python code it will be converted to itself;
• In a hidden construct, you can use anywhere else (free) characters that will be converted to variables with the same name. Note; at least in the hidden construct there is a declaration of the parent symbol as a parameter of the class method; during the execution of the function that returns the hidden code construct, such a symbol does not exist;
• You can generate a series of operations of the same type from lists, producing a structural disclosure of the list of hidden hy constructs (obtained by conversion from the original list).

7. Materials used

When writing this article, materials from the Hy documentation and the Russian translation of Practical Common Lisp were used.