Geekly Articles each Day
📜 ⬆️ ⬇️

20 projects, 20 languages, deadline yesterday. Part 2

I continue a series of articles on setting up continuous localizations. The first you can find here . In this article, I will explain how to integrate the Serge-Smartcat-GitLab bundle and configure Serge configuration files using the example of a test project. Instructions for Ubuntu.


Installation via vagrantfile


If you don’t have a Ubuntu server at hand, you can use Vagrant. How it works:


  1. Install Vagrant and VirtualBox following the instructions .
  2. Download the vagrantfile and bash script.
  3. Run the vagrantfile following the instructions .

The result of the Vagrant build will be a virtual machine with Ubuntu, on which Serge is installed with the Smartcat plugin. All necessary files will be copied, ssh-key is created. You can immediately proceed to setting up the project and launching the application.


Manual installation


Create user


Create a user group serge:


sudo groupadd serge

Create a user with the home directory / usr / local / serge:


 sudo useradd -g serge serge -m -d /usr/local/serge

Install Serge


Serge's complete documentation is here .
Update the local database package metadata:


 sudo apt-get -qq update

Install the build tools as well as the necessary packages. Before starting the installation, make sure that Perl 5.10 or higher is installed.


 sudo apt-get -qq -y install build-essential libssl-dev libexpat-dev unzip wget

Go to the serge user's home directory and install Serge with all necessary dependencies.


 cd /usr/local/serge sudo -u serge wget https://github.com/evernote/serge/archive/master.zip -O serge-master.zip sudo -u serge unzip serge-master.zip sudo -u serge unlink serge-master.zip cd serge-master sudo cpan App::cpanminus sudo cpanm --installdeps . sudo -u serge sudo ln -s /usr/local/serge/serge-master/bin/serge /usr/local/bin/serge

Installing the Smartcat plugin


 sudo cpanm HTTP::Daemon@6.01 sudo cpanm Serge::Sync::Plugin::TranslationService::Smartcat sudo cpan install LWP::Protocol::https

Creating keys


Create a ssh key for the serge user that the serge user will use to go to GitLab:


 sudo -u serge ssh-keygen -t rsa -N "" -f .id_rsa_serge

Integration with GitLab


Create a user serge in GitLab and add it to all repositories with developer rights.
Copy the ssh key from the id_rsa_serge.pub file to the serge user profile settings in GitLab. This is necessary so that Serge can receive and send strings to GitLab.


Installation completed!


Integration with Smartcat


To get started you need to:


  1. Sign up for Smartcat .
  2. Create a project specifying the languages ​​into which you intend to translate. When setting up the plugin, you will need the language codes used in Smartcat.
  3. Get project_id , token and token_id .
  4. Write to support@smartcat.ai and ask to enable support for Serge file parsing algorithm

Setting configuration Serge-Smartcat


Before proceeding to configure, I will describe the general principle of operation. The standard localization cycle includes 5 steps:
image



Serge and git repositories


How does Serge find out which lines need to be sent for translation? To define new lines, the standard git tools are used: the difference between the master branch and the current one is taken. Diff sent to transfer.


image


Important : As a rule, the master branch is protected, so an additional translation branch is used to define diffs, we call it base-translate. That is, after the merge release in master, master must be additionally added to base-translate.


There is an important point. It is necessary to build the processes in such a way that, on the one hand, the developers understand what will go to the translation at what point, and on the other hand, so that the techist does not run around in circles, holding his head, because he has a diff of 1000 lines. And it seems only the title was corrected.


Principle of operation


Serge enters the repository, takes all the branches, and starts comparing them with the master branch. For each branch in which diff is found, it creates a set of documents for translation in the Smartcat project. In order not to get bogged down in the chaos of irrelevant branches and not stand near each developer with the question: “did you remember to delete the branch?”, It is convenient to use prefixes. For example, Serge works with us only with branches that have the translate- prefix.


It is also important that already read and approved texts, ready for translation, fall into the branch with the translate prefix. No one wants to sort out the history of commits in vain attempts to understand what these texts are and why does diff look like that?


Content preparation


Our content preparation process is as follows:


image


Formulation of the problem. When the task is at the stage of production, sometimes there are already texts that should be unchanged into the application. If they are, a copywriter and technical writer help them finalize them.


Design layouts. When the designer has prepared the layouts, the technical writer and copywriter read all the texts. Thus, the subtracted layouts in English fall into development. The developer does not have to think about the texts, he just takes the finished lines from the layouts.


Development. Unfortunately, the previous steps cannot cover all possible types of texts. For example, layouts do not include validators, API errors, some modal windows. If during the development it was necessary to add lines, the technical writer and copywriter provide the developer with all the necessary texts.


Thus, at some point in time, we have a state where the branch contains all the new texts needed for the part of the code that is being developed in it. At this moment a new branch is created from it with the prefix translate- and Serge can begin the translation process. When the translations are ready, the developer takes them from the translate branch, and then acts on the git flow adopted in the team.


This process minimizes developer involvement in the localization process, and also reduces the amount of manual labor to an absolute minimum: you just need to create a branch.


The further description of the customization process is largely based on the fact that there are similar arrangements and processes in the team.


Project Setup


Log in as serge:


 sudo -Hu serge -i

Create the groups directory:


 mkdir groups

In this directory will be placed projects corresponding to the repositories in GitLab. It makes sense to repeat the structure of groups of repositories and their location to facilitate navigation.


Create a directory for the first project:


 cd groups mkdir myproject cd myproject

Copy the Serge configuration files. You can install the midnight commander and manage files using the file manager. Before installing the MC, you must exit the serge account.


 exit sudo apt install mc sudo -Hu serge -i

Files that need to be copied from / usr / local / serge / serge-master / bin / tools / feature-branch-config-generator:



 cp /usr/local/serge/serge-master/bin/tools/feature-branch-config-generator/{myproject.cfg,myproject.inc,myproject.serge.tmpl,myproject_branches.txt} /usr/local/serge/serge-master/groups/myproject cp /usr/local/serge/serge-master/bin/tools/feature-branch-config-generator/fbcgen.pl /usr/local/serge/serge-master/

Create a file to record the plugin's logs:


 mkdir log cd log touch smartcat.log

Configuring myproject.inc



Configure myproject.serge.tmpl



Important! In source_path_prefix, the end point should be% FBCGEN_BRANCH.


A call to the Smartcat plugin must be added to the appropriate section of myproject.serge.tmpl


 sync { ts { plugin Smartcat data { project_id 12345678-1234-1234-1234-1234567890123 token_id 12345678-1234-1234-1234-1234567890123 token 1_qwertyuiopasdfghjklzxcvbn push { disassemble_algorithm_name Serge.io PO } pull { complete_projects NO complete_documents YES } log_file ./log/smartcat.log } }

Description of some parameters:



Sample customized project


Below are the configuration files for a project with 6 languages. The project stores source lines in js-format. A 5-character file naming format is used. Path to resource files: ./branches/base-translate/client/src/translations.


myproject.cfg


 # This is a configuration file for fbcgen.pl # Usage: fbcgen.pl myproject.cfg # Root directory where the master branch checkout is located. # (path is relative to the location of the configuration file itself). # The local checkout should be initialized *before* fbcgen.pl is run. # You can run `serge --initialize onboarding-frontend.serge.tmpl` # to do an initial checkout of the project data. $data_dir = './branches/base-translate'; $branch_list_file = 'myproject_branches.txt'; # Where to load Serge config template from. # (path is relative to the location of the configuration file itself). $template_file = "myproject.serge.tmpl"; # Where to save the localized Serge config file. # (path is relative to the location of the configuration file itself). $output_file = "myproject.local.serge"; our $skip_branch_mask = '^(master)$'; # skip these branches unconditionally our $unmerged_branch_mask = '^(translate-)'; # process unmerged branches matching this mask our $any_branch_mask = '^(translate-)'; # additionally, process these branches even if they were merged # Filter out commits that match this mask when determining if branch is inactive. $skip_commit_mask = '^l10n@example.com'; # This sub returns a hash map of additional parameters # that can be referenced in template as `$FBCGEN_<VARIABLE_NAME>`. # For example, `EXTRA_INCLUDE` parameter generated in the function below # is referenced in `myproject.serge.tmpl` file as `$FBCGEN_EXTRA_INCLUDE`. $calculate_params = sub { my ($branch) = @_; return { # for branch names starting with `release/`, return an empty string; # otherwise, return a string that will be used in the `@include` directive EXTRA_INCLUDE => $branch =~ m!^release/! ? '' : "myproject.inc#skip-saving-localized-files\n" } }

myproject.inc


 # Here we define a job template (common parameters) that will be reused # across all jobs in the generated configuration file. # Certain job parameters (like job ID and paths) will be overridden # in each feature branch job. job-template { name JS file processing ('master' branch) id myproject.js.base-translate # master job id db_namespace myproject destination_languages ru ko de ja zh-Hans source_dir ./myproject/base-translate/res/en output_file_path ./myproject/base-translate/res/%LANG%/%FILE% source_match \.js$ parser { plugin parse_js } ts_file_path ./po/myproject/%LOCALE%/%FILE%.po callback_plugins { :feature_branch { plugin feature_branch data { master_job myproject.js.base-translate # this must match your master job id } } } } # This block will be included conditionally # for all branches except the `release/` ones (see myproject.cfg). # This allows one to skip saving localized files in non-release branches # (but still gather from them strings for translation). skip-saving-localized-files { callback_plugins { :skip-saving-localized-files { plugin process_if phase can_generate_localized_file data { if { lang_matches . then { return NO } } } } } }

myproject.serge.tmpl


 sync { ts { plugin Smartcat data { project_id %project_id% token_id %token_id% token %token% push { disassemble_algorithm_name Serge.io PO } pull { complete_projects NO complete_documents YES } } } vcs { plugin git data { local_path ./branches add_unversioned YES name L10N Robot email l10n-robot@example.com remote_path { base-translate git@gitlab.loc:common/myproject.git#base-translate /* FBCGEN_BRANCH_REMOTES $FBCGEN_DIR_PADDED git@gitlab.loc:common/myproject.git#$FBCGEN_BRANCH */ } } } } jobs { :develop { id job.base-translate name myproject source_language en destination_languages ru zh-Hans ko de ja optimizations NO source_dir ./branches/base-translate/client/src/translations source_match `en-US.js` debug NO parser { plugin parse_js } leave_untranslated_blank YES db_source DBI:SQLite:dbname=./myproject.db3 db_namespace myproject ts_file_path ./po/1bd80338-a0b5-48b3-822c-e90affd2cdcc/%LANG%/%FILE%.po output_file_path ./branches/base-translate/client/src/translations/%CULTURE%.%EXT% output_bom NO output_lang_rewrite { zh-Hans zh } callback_plugins { :feature_branch { plugin feature_branch data { master_job job.base-translate } } } } /* FBCGEN_BRANCH_JOBS :$FBCGEN_DIR { @inherit .#jobs/:develop id job.$FBCGEN_DIR $FBCGEN_EXTRA_INCLUDE source_path_prefix $FBCGEN_BRANCH. source_dir ./branches/$FBCGEN_DIR/client/src/translations/ output_file_path ./branches/$FBCGEN_DIR/client/src/translations/%CULTURE%.%EXT% } */ }

Application launch


Cloning repository


 GIT_SSH_COMMAND="ssh -i /usr/local/serge/serge-master/.id_rsa_serge" git clone -b base-translate git@gitlab.loc:groups/myproject.git branches/base-translate/

Localization cycle


A .serge file is generated:


 GIT_SSH_COMMAND="ssh -i /usr/local/serge/serge-master/.id_rsa_serge" /usr/local/serge/serge-master/fbcgen.pl myproject.cfg

Taking changes from all branches of the repository:


 GIT_SSH_COMMAND="ssh -i /usr/local/serge/serge-master/.id_rsa_serge" serge pull --initialize myproject.local.serge

Collection of translation changes for all branches from Smartcat:


 GIT_SSH_COMMAND="ssh -i /usr/local/serge/serge-master/.id_rsa_serge" serge pull-ts myproject.local.serge

Formation of the database and .po:


 GIT_SSH_COMMAND="ssh -i /usr/local/serge/serge-master/.id_rsa_serge" serge localize myproject.local.serge

Sending new data from the repository to Smartcat:


 GIT_SSH_COMMAND="ssh -i /usr/local/serge/serge-master/.id_rsa_serge" serge push-ts myproject.local.serge

Sending received transfers to the repository in the appropriate branches:


 GIT_SSH_COMMAND="ssh -i /usr/local/serge/serge-master/.id_rsa_serge" serge push myproject.local.serge

The following articles will be devoted to troubleshooting and describing special cases of integration.


')

Source: https://habr.com/ru/post/446640/

More articles:


All Articles