Meta development tool: FutoIn CID

Now, few people will be surprised with project dependency management tools like npm , composer , bundler , pip , maven , cargo and others. Their common drawback is the inability to control the runtime directly. Such a task is solved through nvm , php-build , rvm , virtualenv , sdkman , rustup and other global "manager" versions of runtime, usually written under Bash / Zsh.

The next level of “problems” begins when a universal developer deals with projects every day using completely different technologies. The environment variables turn into a mess, and the shell launch can take several seconds. Inevitably begin everyday mistakes in working with this zoo.

Further, confusion and reeling overlaps with Continuous Integration & Delivery, where manual dances with a tambourine of installing tools and activating specific versions are completely discouraged, but ideally, it is required to abstract as much as possible from the technologies used and bring the process to primitive neutral teams: download, prepare, build, pack, upload, check, approve (sign), roll out.

This is where a tool begs itself, unified working on existing technologies,
which is the FutoIn CID - FutoIn Continuous Integration & Delivery tool .

FutoIn CID philosophy

1. Use specialized projects inside, not replace them.
2. Work without configuration and automatically determine the technologies used.
3. Rid the developer user of the details of installing the project dependencies.
4. Work even on bare Python 2.7 and 3.x
5. Leave the ability to customize the CID and override the default commands.
6. Separate FTN16 specification allowing alternative implementations.
7. Support for all common operating systems and an unlimited number of development tools.
8. Complete abstraction from a particular tool and its features. Only the division of tools by type of application: for assembly, for testing, version control system, release control system, runtime, data migration, etc.
9. Forced unification and restriction of too free tools (for example, Subversion in terms of branches and tags).
10. The entire state is stored in a tree structure that fully reflects the configuration options, the deployment environment and the dynamic values ​​of the command line parameters.

Basic operations

• cid tag - prepare the code for the release, zategit.
• cid prepare - get and prepare the code for a clean build, install all dependencies.
• cid build - build a project.
• cid package - prepare a binary artifact.
• cid check - run tests that do not require deployment.
• cid promote - send a binary artifact to RMS or promote it earlier
  loaded into another pool.
• cid deploy - deploy from a binary artifact, from a VCS tag, or from a branch.
• cid run - run a project by configuring "entry points", by a specific entry point or by a named command.
• cid ci_build - prepare , build , package and promote in one bottle for execution on a CI server.
• cid tool (install|uninstall|update|test|env) - an interface for managing tools.
• cid tool (prepare|build|check|package|migrate) - call standard operations for a specific tool.
• cid tool (list|describe) - a list of supported tools and a more detailed description.
• cid init - initialization of the configuration file futoin.json .

Some real action

Everything happens on a debian/jessie64 pristine vagrant box.

1. Install using the example of an APT-based distribution (Debian, Ubuntu and other forks).

 sudo apt-get install -y python-pip sudo pip install futoin-cid #   RMS mkdir -p ~/Releases/Builds ~/Releases/Prod 

2. Configure sudo

In order to find a balance between paranoia and convenience, it is proposed to allow only installation of packages without a password. Read more in the readme .

For some cases, Docker and JRE / JDK from Zulu require the ability to install signature verification keys and additional repositories. If there are concerns in the framework of sudo , then it is not so difficult to do it with your hands, because all executable commands are output.

In the case of Vagrant, we can do everything by default.

3. Let's try to build some HelloWorld.

 cid ci_build master Builds --vcsRepo=git:https://github.com/laravel/quickstart-basic.git --rmsRepo=scp:${HOME}/Releases 

many small letters

And now let's see what happened.

 tar tJf ~/Releases/Builds/laravel-CI-*.txz --exclude="*/*/*" 

 ./ ./composer.json ./storage/ ./vendor/ ./resources/ ./.package.checksums ./LICENSE.txt ./composer.lock ./database/ ./.env.example ./.env ./LICENSE.txt.gz ./node_modules/ ./artisan ./.gitattributes ./phpunit.xml ./tests/ ./gulpfile.js ./app/ ./gulpfile.js.gz ./config/ ./package.json ./public/ ./server.php ./composer.json.gz ./readme.md ./package.json.gz ./bootstrap/ 

We see a full-fledged binary artifact, ready for deployment. By default, the CID will also make the GZIP versions of the .js, .json, .css, .svg and * .txt files found for use with the gzip_static module in nginx.

As an improvised manifest, a .package.checksums file is .package.checksums with SHA-512 hashes.

Note: project root packaging is only a standard action in the absence of another implementation. For example, maven or puppet themselves create artifacts.

4. Deploy with RMS

 cid deploy Builds --rmsRepo=scp:${HOME}/Releases --deployDir=deploy 

Note: By default, the CID itself finds the latest artifact in the selected pool for deployment, unless the name is explicitly indicated. The same applies to tags.

Of course, to deploy a not very serious project on one machine, you can skip such a step and deploy it directly from the VCS tag.

5. Deploy directly from VCS

Due to the absence of tags in the example, we will use the live branch of the project.

 cid deploy vcsref master --vcsRepo=git:https://github.com/laravel/quickstart-basic.git --deployDir=deploy 

many small letters

Repeat without changes branches and get a warning about the already deployed revision. With tags and artifacts will be the same.

 cid deploy vcsref master --vcsRepo=git:https://github.com/laravel/quickstart-basic.git --deployDir=deploy 

Sample code preparation project for release

Prepare a specific version by calling from a local copy.

 cid tag master 1.0.0 

Release the next incremental version without a local copy at the time of the call.

 cid tag master --vcsRepo=git:user@somehost:project.git 

Standard actions: synchronization with a remote repository, updating all possible metadata files, commit, creating a tag. Example release of the CID itself:

The project may use additional plugins. For example, CID updates the version in the source code and in CHANGELOG.

We work with individual tools

 rubyVer=2.3 cid tool exec ruby -- -e 'puts "Hello World!"' 

If the tool is not yet installed, all the necessary operations will be performed. In this case, the installed RVM, and then the current Ruby 2.3. *. At the same time, the RVM will not load when the shell starts and the standard environment variables will not be polluted.

Tool information

 cid tool describe maven 

A bit of technical details

File futoin.json

By analogy with the well-known package.json , composer.json , Cargo.toml and others is the description of the project, if it is located in its root. It is allowed to set general project settings, but the presence of a file is not mandatory.

When launched, CID looks for possible settings for the deployment environment in the deployment root folder - it allows you to set both project settings and environment settings ( .env node), and in the home folder of the active user and in /etc/futoin/futoin.json , only the .env node .

Note: The .env node is not equivalent and does not replace the "dotenv" files.

Valid values ​​are documented in more detail in FTN16 .

The most interesting sites:

• actions - allows you to override or supplement the standard operations tag / prepare / build / package / check.
• plugins - allows you to add additional tools.
• tools - allows you to hard specify the list of tools used and optionally their version.
  
  • Version '*' and true - requires the default version
  • Otherwise, the version falls directly into the {tool}Ver environment variable and is available in plug-ins as a .env.{tool}Ver node of the global state.
• toolTune - the possibility of specific fine-tuning of each tool. Details are available in cid tool describe {tool} .
• entryPoints - entryPoints named entry points.
• persistent - list of read-write paths in the project. Specific to the web.

An example of the futoin.json file in the project root:

 { "name": "example-package", "version": "0.4.2", "actions": { "package": [] }, "plugins": { "examplerelease": "some.project.specific.release", "examplehelper": "some.other.helpertool" }, "vcs": "git", "tools": { "examplerelease": true, "python": "*", "node": "stable", "gradle": "*" }, "toolTune" : { "gradle": { "package": "jar" } }, "rms": "scp", "rmsRepo": "rms@somehost", "rmsPool": "ReleaseBuilds", "entryPoints": { "app": { "tool": "python", "path": "app.py", "tune": {} } } } 

Deployment Nuances

• {.deployDir} is our conditional root folder for deploying a specific project.
• {.deployDir} / persistent - storage area for read-write files. In a cluster, an area of ​​the network file system is implied.
• {.deployDir} / current - a symbolic link to the current version.
• {.deployDir} / {version_dir} - the final location of a specific version:
  
  • {artifact} - package name from RMS without extension,
  • {vcsref} _ {vcsrevision} - in case of deployment from a branch,
  • {vcstag} - in case of deployment from a tag.
• {.deployDir} / {version_dir} .tmp - temporary folder in processing.
• {.deployDir} / {version_dir}. {ext} - unpacked binary artifact in rms mode.
• {.deployDir} / vcs is a local copy of the repository in vcsref and vcstag .
• {.deployDir} /. futoin-deploy.lock is a lock file, as well as a security feature against unintentional cleaning of an incorrectly specified target folder.
• everything else is trimmed.

A little about guts to add tools

All tools are divided into several types. One tool can immediately apply to several. In the future, they can be more.

• SubTool - base, interface auto-definition, dependency, install, update, delete.
• BuildTool - the base for the build tools (prepare, build, package).
• MigrationTool - base for migration tools (migrate).
• RmsTool is the base for RMS (promote) and internal API.
• RunEnvTool is the base for version managers (nvm, rvm, gvm, etc.).
• RuntimeTool - base for launch tools (run). These include: java, python, ruby, etc.
• TestTool - base for tools that support testing (check).
• VcsTool - base for VCS (tag) and internal API

For auxiliary needs, there is a set of mixins, some of which are already included in SubTool.

What is at the moment?

Almost all common Linux distributions are supported: from CentOS and Debian to ArchLinux and Gentoo .

Added experimental macOS support. No suitable equipment - help in testing is extremely welcome.

The quality of the proof-of-concept supports a wide variety of programming languages ​​and related tools: from PHP , Python and Ruby to Go , Rust and Scala . The ability to expand and add custom plug-ins is almost unlimited.

Three different VCSs are also supported: Git , Mercurial and Subversion .

What else is not?

RMS support is implemented in principle. For testing purposes, only scp is working, but the main sight is on Archiva, Artifactory and Nexus.

Support for unit tests relies on project build tools (make, gradle, grunt, etc.), but the collection of results has not yet been defined. Recommendations are also welcome.

The process of cryptographic signing is not yet defined. It is required to analyze and find the common denominator for use cases.

If necessary, support for * BSD will presumably require minimal effort, while support for Windows with a focus on .Net will rather require an alternative, specialized implementation.

The interface for data migration is provided, but does not yet have a standard implementation.

Summarize

Driven by laziness, pathological rejection of monotonous fuss and a passion for automation, a generalizing concept was born for almost all possible technologies. To it was added the implementation in the form of a specific tool FutoIn CID. Certainly requires serious grinding and running in practice. In addition to simplifying the life of a developer, a long-range view is the automation of the DevOps infrastructure.

Of course, this is not the first tool that supports various technologies, but perhaps the first one that can work without tuning at all, improving labor productivity and reducing the number of everyday mistakes.

Links to the project:

→ GitHub
→ Python Package Index
→ FTN16 Draft