Using parse_transform
Disclaimer: The described tool has a controversial reputation. I do not urge to use it wherever I get, I only introduce the concepts used in order to reduce some of the thrill of technology.
The source code as well as a text copy of the article can be found on the githab .
parse_transform is the mechanism for changing AST before compiling. It is intended to change the meaning of constructs (semantics), without going beyond the Erlang syntax.
')
Unfortunately, there is little information on the Web about this, which makes the entry threshold very high for a non-guru erlang.
In this article, I will talk a little about AST Erlang, give an example of simple transformations, and show the process of writing a parse_transform to create a stateless gen_server (the task doesn’t have much sense, but as an example of use it will do), and in the end I’ll give you a link on a set of novice transformer.
Just in case: AST definition
It is better to see AST once, than read its description a hundred times. Therefore, we will write a small module to see how each line is transformed.
So, the source text astdemo.erl :
To see the AST, you need to set a parse_file function from the epp module to this file:
It can be seen that each expression is converted to a length of at least 3, with the first two elements being always a type and a string, followed by a specific description. If it is not clear what is in a particular place, the documentation is at your service.
Let's now do a dummy-parse_transform to see what we have to deal with next. To do this, we will create a module that will deal with the transformation, and instead of manipulating the AST, we simply print it.
So demo_pt.erl :
Paste the appropriate directive into astdemo.erl :
Compile:
As you can see, the AST is the same (up to offset lines), but this time it is printed at compile time.
It should be noted that compiler directives have already been removed on the AST arriving for transformation.
What is passed in the options, an inquisitive reader is likely to learn on their own. This article is about AST.
Let's do an exercise that is useless in practice - rename the “hello / 0” function to “hi / 0”. This will be easy to do, since hello / 0 is not called from inside the module, but only has the ability to be called from outside. Therefore, it suffices to change the list of exports and the function header.
Since AST (Forms Binding) is a list, each element of which is a form of a very short list of types, it is logical to skip all Forms through the mutator function. Since the task is simple and the transformation of each expression does not depend on the rest of the content, lists: map will suit us.
The function that will change the export and function headers will look something like this:
We enable this function by changing the code of the parse_transform function:
We compile demo_pt , make sure that we are not messing up.
We are trying to compile astdemo with a new transformer:
Perfectly! It worked, as they wanted. Time to do something a little more useful.
Sometimes when writing a module with the behavior of gen_server, there is no need to drag along the State, since there is nothing to store in it, and dragging the State from the handle_anything to the final expression clogs the code. Let's make parse_transform, which will allow us to define handle_call / 2, handle_cast / 1, handle_info / 1 . Or not. To make the article a bit shorter, I will show only the transformation handle_call / 2 -> handle_call / 3 , and those who are interested will have the rest defined.
The behavior of gen_server requires the definition of handle_call (for simplicity) as follows ( documentation ):
As we get rid of the need to consider State, let our syntax be:
We will train on it. Determined by the handle_call in our syntax and its analogue in canonical form for comparing and writing a transformer.
Everything was written the same as last time - looking at the output of epp: parse_file and adjusting what is, under what is needed.
Success! It remains to finish the owl and put it on the githab.
An interested reader, I hope, met AST in Erlang, and also got a rough idea about the methods of its transformation. Perhaps someone first learned about the parse_transform.
The article contains information that should be enough to start writing your own transform. Slightly below will be criticism and a link to a library useful for transformations.
First, the use of parse_transform (in the event that it is in a separate project) is added dependent on your project. In the case of rebar, this is fatal.
Second, people who read (and, especially, edit) such code may not immediately understand the concept. Therefore, we need not only good documentation, but also a noticeable link to it at the beginning of the source code.
Thirdly, the possibilities for writing your own dialects are very limited. Before the AST gets under your scalpel, a regular parser works. Therefore, the introduction of tricky keywords and own operators can break the parser, greatly complicating the task.
parse_trans is a useful thing for writing parse_transforms. It allows you to make a recursive map to a tree, which is extremely useful when modifying expressions at irregular depths. In the examples there is a very concise way of rewriting the operator “!” To the gproc: send call.
The source code as well as a text copy of the article can be found on the githab .
What is parse_transform
parse_transform is the mechanism for changing AST before compiling. It is intended to change the meaning of constructs (semantics), without going beyond the Erlang syntax.
')
Unfortunately, there is little information on the Web about this, which makes the entry threshold very high for a non-guru erlang.
What do we do
In this article, I will talk a little about AST Erlang, give an example of simple transformations, and show the process of writing a parse_transform to create a stateless gen_server (the task doesn’t have much sense, but as an example of use it will do), and in the end I’ll give you a link on a set of novice transformer.
AST in Erlang
Just in case: AST definition
It is better to see AST once, than read its description a hundred times. Therefore, we will write a small module to see how each line is transformed.
So, the source text astdemo.erl :
-module(astdemo). -export([hello/0, hello/2]). hello() -> hello("world", 1). hello(_What, 0) -> ok; hello(What, Count) -> io:format("Hello, ~s~n", [What]), hello(What, Count - 1). To see the AST, you need to set a parse_file function from the epp module to this file:
Eshell V5.8.5 (abort with ^ G)
1> {ok, Forms} = epp: parse_file ("astdemo.erl", [], []), io: format ("~ p ~ n", [Forms]).
[{attribute, 1, file, {"astdemo.erl", 1}},
{attribute, 1, module, astdemo},
{attribute, 2, export, [{hello, 0}, {hello, 2}]},
{function, 4, hello, 0,
[{clause, 4, [], [],
[{call, 5,
{atom, 5, hello},
[{string, 5, "world"}, {integer, 5,1}]}]}]],
{function, 7, hello, 2,
[{clause, 7, [{var, 7, '_ What'}, {integer, 7.0}], [], [{atom, 8, ok}]},
{clause, 9,
[{var, 9, 'What'}, {var, 9, 'Count'}],
[],
[{call, 10,
{remote, 10, {atom, 10, io}, {atom, 10, format}},
[{string, 10, "Hello, ~ s ~ n"},
{cons, 10, {var, 10, 'What'}, {nil, 10}}]},
{call, 11,
{atom, 11, hello},
[{var, 11, 'What'},
{op, 11, '-', {var, 11, 'Count'}, {integer, 11,1}}]}]}]},
{eof, 12}]
ok
It can be seen that each expression is converted to a length of at least 3, with the first two elements being always a type and a string, followed by a specific description. If it is not clear what is in a particular place, the documentation is at your service.
Parse_transform / 2 function
Let's now do a dummy-parse_transform to see what we have to deal with next. To do this, we will create a module that will deal with the transformation, and instead of manipulating the AST, we simply print it.
So demo_pt.erl :
-module(demo_pt). -export([parse_transform/2]). parse_transform(Forms, _Options) -> io:format("~p~n", [Forms]), Forms. Paste the appropriate directive into astdemo.erl :
-module(astdemo). -compile({parse_transform, demo_pt}). -export([hello/0, hello/2]). ........... Compile:
Eshell V5.8.5 (abort with ^ G)
1> c (astdemo).
[{attribute, 1, file, {"./ astdemo.erl", 1}},
{attribute, 1, module, astdemo},
{attribute, 3, export, [{hello, 0}, {hello, 2}]},
{function, 5, hello, 0,
[{clause, 5, [], [],
[{call, 6,
{atom, 6, hello},
[{string, 6, "world"}, {integer, 6,1}]}]}]},
{function, 8, hello, 2,
[{clause, 8, [{var, 8, '_ What'}, {integer, 8.0}], [], [{atom, 9, ok}]},
{clause, 10,
[{var, 10, 'What'}, {var, 10, 'Count'}],
[],
[{call, 11,
{remote, 11, {atom, 11, io}, {atom, 11, format}},
[{string, 11, "Hello, ~ s ~ n"},
{cons, 11, {var, 11, 'What'}, {nil, 11}}]},
{call, 12,
{atom, 12, hello},
[{var, 12, 'What'},
{op, 12, '-', {var, 12, 'Count'}, {integer, 12,1}}]}]}]},
{eof, 13}]
{ok, astdemo} As you can see, the AST is the same (up to offset lines), but this time it is printed at compile time.
It should be noted that compiler directives have already been removed on the AST arriving for transformation.
What is passed in the options, an inquisitive reader is likely to learn on their own. This article is about AST.
First transformations
Let's do an exercise that is useless in practice - rename the “hello / 0” function to “hi / 0”. This will be easy to do, since hello / 0 is not called from inside the module, but only has the ability to be called from outside. Therefore, it suffices to change the list of exports and the function header.
Single form transformer
Since AST (Forms Binding) is a list, each element of which is a form of a very short list of types, it is logical to skip all Forms through the mutator function. Since the task is simple and the transformation of each expression does not depend on the rest of the content, lists: map will suit us.
The function that will change the export and function headers will look something like this:
% hello_to_hi replaces occurences of hello/0 with hi/0 hello_to_hi({attribute, Line, export, Exports}) -> % export attribute. Replace {hello, 0} with {hi, 0} HiExports = lists:map( fun ({hello, 0}) -> {hi, 0}; (E) -> E end, Exports), {attribute, Line, export, HiExports}; hello_to_hi({function, Line, hello, 0, Clauses}) -> % Header of hello/0. Just replace hello with hi {function, Line, hi, 0, Clauses}; hello_to_hi(Form) -> % Default: do not modify form Form. Now all together
We enable this function by changing the code of the parse_transform function:
parse_transform(Forms, _Options) -> HiForms = lists:map(fun hello_to_hi/1, Forms), io:format("~p~n", [HiForms]), HiForms. We compile demo_pt , make sure that we are not messing up.
Check
We are trying to compile astdemo with a new transformer:
Eshell V5.8.5 (abort with ^ G)
1> c (astdemo).
[{attribute, 1, file, {"./ astdemo.erl", 1}},
{attribute, 1, module, astdemo},
{attribute, 3, export, [{hi, 0}, {hello, 2}]},
{function, 5, hi, 0,
[{clause, 5, [], [],
[{call, 6,
{atom, 6, hello},
[{string, 6, "world"}, {integer, 6,1}]}]}]},
{function, 8, hello, 2,
[{clause, 8, [{var, 8, '_ What'}, {integer, 8.0}], [], [{atom, 9, ok}]},
{clause, 10,
[{var, 10, 'What'}, {var, 10, 'Count'}],
[],
[{call, 11,
{remote, 11, {atom, 11, io}, {atom, 11, format}},
[{string, 11, "Hello, ~ s ~ n"},
{cons, 11, {var, 11, 'What'}, {nil, 11}}]},
{call, 12,
{atom, 12, hello},
[{var, 12, 'What'},
{op, 12, '-', {var, 12, 'Count'}, {integer, 12,1}}]}]}]},
{eof, 13}]
{ok, astdemo}
2> astdemo: hi ().
Hello, world
ok Perfectly! It worked, as they wanted. Time to do something a little more useful.
Stateless gen_server parse_transform
Sometimes when writing a module with the behavior of gen_server, there is no need to drag along the State, since there is nothing to store in it, and dragging the State from the handle_anything to the final expression clogs the code. Let's make parse_transform, which will allow us to define handle_call / 2, handle_cast / 1, handle_info / 1 . Or not. To make the article a bit shorter, I will show only the transformation handle_call / 2 -> handle_call / 3 , and those who are interested will have the rest defined.
Concept
The behavior of gen_server requires the definition of handle_call (for simplicity) as follows ( documentation ):
handle_call(Request, From, State) -> ..... {reply,Reply,NewState}. As we get rid of the need to consider State, let our syntax be:
handle_call(Request, From) -> ..... Reply. Transformation plan
- Find and change in exports handle_call / 2 to handle_call / 3
- Among the definitions of functions for handle_call / 2, add the parameter State and frame the final expression in each slander in {reply, ..., State}
Cat
We will train on it. Determined by the handle_call in our syntax and its analogue in canonical form for comparing and writing a transformer.
-module(sl_gs_demo). -behavior(gen_server). -compile({parse_transform, sl_gs}). -export([handle_call/2, ref_handle_call/3]). -export([handle_cast/2, handle_info/2]). -export([init/1, terminate/2, code_change/3]). % This will be transformed handle_call(Req, From) -> {Req, From}. % That's what handle_call should finally look like ref_handle_call(Req, From, State) -> {reply, {Req, From}, State}. % Dummy functions to make gen_server happy % Exercise: Try to insert them automatically during transformations :) handle_cast(_, State) -> {noreply, State}. handle_info(_, State) -> {noreply, State}. init(_) -> {ok, none}. terminate(_, _) -> ok. code_change(_, State, _) -> {ok, State}. Code
Everything was written the same as last time - looking at the output of epp: parse_file and adjusting what is, under what is needed.
-module(sl_gs). -export([parse_transform/2]). parse_transform(Forms, _Options) -> lists:map(fun add_missing_state/1, Forms). add_missing_state({attribute, Line, export, Exports}) -> % export attribute. Replace {handle_call, 2} with {handle_call, 3} NewExports = lists:map( fun ({handle_call, 2}) -> {handle_call, 3}; % You can add more clauses here for other function mutations (E) -> E end, Exports), {attribute, Line, export, NewExports}; add_missing_state({function, Line, handle_call, 2, Clauses}) -> % Mutate clauses NewClauses = lists:map(fun change_call_clause/1, Clauses), % Finally, change arity in header {function, Line, handle_call, 3, NewClauses}; add_missing_state(Form) -> % Default Form. change_call_clause({clause, Line, Arguments, Guards, Body}) -> % Change arity in clauses. NewArgs = Arguments ++ [{var, Line, 'State'}], % Add State argument % Then replace last statement of each clause with corresponding tuple NewBody = change_call_body(Body), {clause, Line, NewArgs, Guards, NewBody}. change_call_body([Statement | Rest=[_|_] ]) -> % Rest has to be non-empty list for this % Recurse to change only last statement [Statement|change_call_body(Rest)]; change_call_body([LastStatement]) -> % Put it into tuple. Lines are zero to omit parsing LastStatement [{tuple,0, [{atom,0,reply}, LastStatement, {var,0,'State'}] }]. Health check
Eshell V5.8.5 (abort with ^ G)
1> c (sl_gs_demo).
{ok, sl_gs_demo}
2> {ok, D} = gen_server: start_link (sl_gs_demo, [], []).
{ok, <0.39.0>}
3> gen_server: call (D, hello).
{hello, {<0.32.0>, # Ref <0.0.0.83>}}
Success! It remains to finish the owl and put it on the githab.
Results
An interested reader, I hope, met AST in Erlang, and also got a rough idea about the methods of its transformation. Perhaps someone first learned about the parse_transform.
The article contains information that should be enough to start writing your own transform. Slightly below will be criticism and a link to a library useful for transformations.
Criticism of the method
First, the use of parse_transform (in the event that it is in a separate project) is added dependent on your project. In the case of rebar, this is fatal.
Second, people who read (and, especially, edit) such code may not immediately understand the concept. Therefore, we need not only good documentation, but also a noticeable link to it at the beginning of the source code.
Thirdly, the possibilities for writing your own dialects are very limited. Before the AST gets under your scalpel, a regular parser works. Therefore, the introduction of tricky keywords and own operators can break the parser, greatly complicating the task.
Library parse_trans
parse_trans is a useful thing for writing parse_transforms. It allows you to make a recursive map to a tree, which is extremely useful when modifying expressions at irregular depths. In the examples there is a very concise way of rewriting the operator “!” To the gproc: send call.
Source: https://habr.com/ru/post/140374/
All Articles