Babel and handlebars, it's time to make friends
I think many people know such a package as Babel, or PyBabel.
An excellent package for localization, which is based on gettext, like everything else (at least known to me) in the modern world.
The task was to prepare the site for future localization, all templates were made in Handlebars, this template engine was the first and foremost reference point that you need to make friends with it. The question is who.
I should make a reservation in advance that I had no restrictions on the choice of technologies.
We ultimately use the full set for the static build - ruby ​​(compass), node (coffee, grunt, requirejs), python (backend and the basis of everything and everything), shell scripts, in general, there is no limit.
By the way, if anyone is interested, I can describe in detail about the build of a separate post, there requirejs + scss + all of the above, at the moment there are about 1000+ files included in the build + deployment on heroku and non-heroku with one button. In my opinion, the whole process is interesting, but I don’t know what to focus on
')
That is, in fact, all that was needed was to find who is capable of:
a) Go through the handlebars patterns
b) To be able to work on the same principle as Babel — that is, the string is the key, not the constant, when everything is stored in separate files.
c) Prepare from the found lines .po file
Then the hands are already untied and there are enough opportunities.
Having spent several hours searching for a finding was disappointing, no one did this for some reason.
I categorically did not want to believe this, but everything that existed was “not that” and “not so.”
What did existing solutions do and know how?
They with varying success pulled out regeksami lines. They didn’t know how to work with ngettext, or didn’t know how to work according to the principle “string = key”, but only constants.
All this was necessary to eliminate.
The task was immediately broken into points, just take a moment and describe the entire chain that I wanted to receive.
1) Define the text of the translation in the templates, the principle of "line = key"
So much more comfortable in the work than when the key = the name of a constant. The cost of the error is also less, the maximum anyone will receive the text in English, instead of translation
2) Send text to babel, as well as line number
This is necessary, because during updates, babel will not be able to determine what exactly has changed if he does not have line numbers.
3) Build files from .po files, this is actually a babel task
4) From the .po files after the translation, make .json files and transfer them to someone else on the client side. Conversion - po2json task
5) Make helper s for handlebars, both single-line and block with ngettext support
Again, diversity for the convenience of everyday
6) Both options should be able to work with parameters
7) Both options should transfer the text for translation "to the top", to someone who holds those .json files
8) The same someone should substitute the parameters obtained from the template into the final line.
At the 8th point, Jed was chosen, an excellent library with embedded sprintf, which immediately solved the problem of convenient parameter transfer.
The first point was the key, it was impossible to spy it anywhere.
For a long time he looked at regexps and tried his own - he didn’t like everything. Crooked, not enough, unreliable.
I considered template code compiled into javascript even longer. From there, getting the strings was pretty easy.
But the very idea of ​​this approach was terrifying.
In addition, it did not solve the problem that you need to know where the line was met in the original template.
I had to look at the source files of handlebars, and here came an insight - his lexer was written with the help of jison and the parser jisona, using the vocabulary directly of handlebars, sewn inside.
And by calling the Handlebars.parse (template) method, you can get a JSON template structure.
It would seem that everything is great - but there are no line numbers.
But happiness was close and it was necessary to dig in the same direction - but already in the assembled handlebars, in the code of the generated parser, it remained to find the right place where to enter two lines and voila, the output is the template structure, where for each block the initial line and the final line are indicated. All this is in the parser itself, it just did not pass out at the right time in the right place.
Then there was the matter of technology, put it all together, screw it to the babel as an extractor
Those:
A python function (extractor) that calls the script that loads the patched handlebars.js node.js script.
Further, this script recursively goes through the structure, collects the string and returns to the extractor in the desired format. The whole logic of the babel remains unchanged. The string extraction logic is also native to handlebars.js. All this can not but rejoice.
By itself, this post would not make sense if there were no subsequent lines. I think that at least the time spent searching for the right direction for exporting rows will save someone.
- pip install pybabel-hbs
- github.com/tigrawap/pybabel-hbs
The package is installed along with the patched handlebars, and in the source code there are examples of the implementation of helper (true on coffee)
The very use of the templates turned out to be perfect for me personally, 4 helper on githubs, the thickest here, for clarity
In the config babel, you need to add
For work it is necessary that node.js be in the environment.
The plans include optimization, so that node.js does not start for each file separately each time, but once for the entire lifetime of the main babel process.
This is implemented in version 0.2.0, while the post was moderated.
By itself, the acceleration turned out to be quite good, a pack of 150 templates began to be processed in ~ a minute.
Before that, it took a couple of minutes, which was completely unacceptable.
But a minute for such a cola too much.
Essentially (another 4 times), it turned out to be optimized due to the fact that initially transferred the data to node.js from python via pexpect, now (version 0.2.1) I transfer only the file name and already node.js reads it. (here of course the minus is that both open the file in turn)
Now processing 150 files takes less than 15 seconds. There is still room for speeding up the process, but for the time being it suits me.
Given the fact that you often do not need to run - good enough result.
“The moral of this story is that, before spending hours writing and testing regular expressions, looking for devious ways to parse any language — it’s worth seeing how it parses itself. Maybe it's easier.
Ultimately, the whole problem was solved by several lines inscribed in handlebars.js itself, and the fact that today is offered on the Internet is ... at least not convenient.
An excellent package for localization, which is based on gettext, like everything else (at least known to me) in the modern world.
The task was to prepare the site for future localization, all templates were made in Handlebars, this template engine was the first and foremost reference point that you need to make friends with it. The question is who.
I should make a reservation in advance that I had no restrictions on the choice of technologies.
We ultimately use the full set for the static build - ruby ​​(compass), node (coffee, grunt, requirejs), python (backend and the basis of everything and everything), shell scripts, in general, there is no limit.
By the way, if anyone is interested, I can describe in detail about the build of a separate post, there requirejs + scss + all of the above, at the moment there are about 1000+ files included in the build + deployment on heroku and non-heroku with one button. In my opinion, the whole process is interesting, but I don’t know what to focus on
')
That is, in fact, all that was needed was to find who is capable of:
a) Go through the handlebars patterns
b) To be able to work on the same principle as Babel — that is, the string is the key, not the constant, when everything is stored in separate files.
c) Prepare from the found lines .po file
Then the hands are already untied and there are enough opportunities.
Having spent several hours searching for a finding was disappointing, no one did this for some reason.
I categorically did not want to believe this, but everything that existed was “not that” and “not so.”
What did existing solutions do and know how?
They with varying success pulled out regeksami lines. They didn’t know how to work with ngettext, or didn’t know how to work according to the principle “string = key”, but only constants.
All this was necessary to eliminate.
The task was immediately broken into points, just take a moment and describe the entire chain that I wanted to receive.
1) Define the text of the translation in the templates, the principle of "line = key"
So much more comfortable in the work than when the key = the name of a constant. The cost of the error is also less, the maximum anyone will receive the text in English, instead of translation
2) Send text to babel, as well as line number
This is necessary, because during updates, babel will not be able to determine what exactly has changed if he does not have line numbers.
3) Build files from .po files, this is actually a babel task
4) From the .po files after the translation, make .json files and transfer them to someone else on the client side. Conversion - po2json task
5) Make helper s for handlebars, both single-line and block with ngettext support
Again, diversity for the convenience of everyday
6) Both options should be able to work with parameters
7) Both options should transfer the text for translation "to the top", to someone who holds those .json files
8) The same someone should substitute the parameters obtained from the template into the final line.
At the 8th point, Jed was chosen, an excellent library with embedded sprintf, which immediately solved the problem of convenient parameter transfer.
The first point was the key, it was impossible to spy it anywhere.
For a long time he looked at regexps and tried his own - he didn’t like everything. Crooked, not enough, unreliable.
I considered template code compiled into javascript even longer. From there, getting the strings was pretty easy.
But the very idea of ​​this approach was terrifying.
In addition, it did not solve the problem that you need to know where the line was met in the original template.
I had to look at the source files of handlebars, and here came an insight - his lexer was written with the help of jison and the parser jisona, using the vocabulary directly of handlebars, sewn inside.
And by calling the Handlebars.parse (template) method, you can get a JSON template structure.
It would seem that everything is great - but there are no line numbers.
But happiness was close and it was necessary to dig in the same direction - but already in the assembled handlebars, in the code of the generated parser, it remained to find the right place where to enter two lines and voila, the output is the template structure, where for each block the initial line and the final line are indicated. All this is in the parser itself, it just did not pass out at the right time in the right place.
Then there was the matter of technology, put it all together, screw it to the babel as an extractor
Those:
A python function (extractor) that calls the script that loads the patched handlebars.js node.js script.
Further, this script recursively goes through the structure, collects the string and returns to the extractor in the desired format. The whole logic of the babel remains unchanged. The string extraction logic is also native to handlebars.js. All this can not but rejoice.
By itself, this post would not make sense if there were no subsequent lines. I think that at least the time spent searching for the right direction for exporting rows will save someone.
- pip install pybabel-hbs
- github.com/tigrawap/pybabel-hbs
The package is installed along with the patched handlebars, and in the source code there are examples of the implementation of helper (true on coffee)
The very use of the templates turned out to be perfect for me personally, 4 helper on githubs, the thickest here, for clarity
{{#ntrans num_to_check_aganst param_1="something" num=num_to_check_against}} Some text to be translated with %(param_1)s and %(num)s {{else}} Some plural text to be translated with %(param_1)s and %(num)s {{/ntrans}} In the config babel, you need to add
[hbs: path/to/project/**.hbs] For work it is necessary that node.js be in the environment.
This is implemented in version 0.2.0, while the post was moderated.
By itself, the acceleration turned out to be quite good, a pack of 150 templates began to be processed in ~ a minute.
Before that, it took a couple of minutes, which was completely unacceptable.
But a minute for such a cola too much.
Essentially (another 4 times), it turned out to be optimized due to the fact that initially transferred the data to node.js from python via pexpect, now (version 0.2.1) I transfer only the file name and already node.js reads it. (here of course the minus is that both open the file in turn)
Now processing 150 files takes less than 15 seconds. There is still room for speeding up the process, but for the time being it suits me.
Given the fact that you often do not need to run - good enough result.
“The moral of this story is that, before spending hours writing and testing regular expressions, looking for devious ways to parse any language — it’s worth seeing how it parses itself. Maybe it's easier.
Ultimately, the whole problem was solved by several lines inscribed in handlebars.js itself, and the fact that today is offered on the Internet is ... at least not convenient.
Source: https://habr.com/ru/post/198280/
All Articles