Ship oranges barrels. Golang Releases

This article is a continuation of the instrumental topic covered in the last publication . Today we will try to deal with the assembly of releases of Golang applications as a single executable file, including resource dependencies, and the issue of optimizing the size of the final assembly. Also consider the process of building a working environment that meets the following requirements:

1. Portability The environment should be easily reproducible on different machines.
2. Isolation The environment should not affect the versions of installed libraries and programs on the developer's machine.
3. Flexibility. The environment should allow to collect releases for different versions of Golang and Linux (different versions of distributions and glibc).
4. Repeatability. There should be no magic and secret knowledge, that is, all the steps of building the project and dependencies should be described by code.

Introduction

Golang offers a static way to build applications. It is quite convenient and suitable for many occasions. When developing utilities with a web interface or full-fledged web services, dependencies on:

• configuration files
• templates, styles, scripts and images for the user interface
• sensitive information, such as keys for SSL, if your service works directly with https
• and much more

It would be nice to be able to package the necessary resources statically into an executable file so that the final release is the following set:

• service configuration file
• executable release file
• service configuration file for systemd or its equivalent.

This approach allows us to solve several issues at once:

• simplify the delivery of code to end users.
• provide control over resources
• ensure safety (impossibility of change by third parties)

A similar approach is used, for example, by authors of popular open source projects, such as the mailhog, consul, vault.

Work environment

If you are simultaneously running several projects with different versions of compilers and dependencies, you probably already found a way to isolate the working environments of different projects within the same developer machine.

Since in my practice hybrid systems are widely used, consisting of components written in different languages ​​and included in the delivery at the same time, the question of flexibility and speed of switching between them arises.

In the last article, you can find an explanation of the basic ideas when designing a working environment for an Erlang project. A sandbox for Golang projects is built on similar principles.

The source code of the demo application and the environment for developing Golang projects can be found at https://github.com/Vonmo/relgo

Note: The code was tested on debian-like distributions. To work successfully with this article, docker , docker-compose and GNU Make must be installed on your machine. Docker installation does not take much time, you just need to remember that your user should be added to the docker group.

In make, the following targets are predefined:

• build_imgs . Responsible for assembling base containers to run workbench containers.
• up . Creates or updates sandbox containers and then launches them.
• down . Stops and removes sandbox containers.
• test . Run tests
• rel . Prepares the final release of the application.
• run . Runs the application in the sandbox
• deps . Receive and update dependencies
• new_migration . Creates a database migration
• migrate . Applies Migrations
• format_code . Formats source code using gofmt

Demo application

We will develop an atomic counter (atomic counter). To demonstrate, let's complicate the application by entering a database dependency. In this case, the counters will be stored in postgresql.
Define the requirements:

1. Functionality.
   The application must perform the following functions:
   
   • increment
   • decrement
   • reset
   • value
2. Target system: ubuntu 16.04 LTS
3. Simplest HTTP API
4. Using Golang> = 1.9
5. Availability of an initial interface for monitoring internal application processes

Architecture

The architecture of the application is based on the idea of ​​isolating logically complete units in separate packages. The application has the following layers:

• core - the core of the system, provides basic functions: configuration, logging, metrics; provides a correct start and stop services.
• config - is responsible for parsing the configuration
• models - implements the interface for communicating with the database.
• migrations - migrations for database schema.
• log - adds logging levels to standard log.
• metrics - implements the simplest application metrics.
• services - package includes all system services.

Layers and separation of applications into services make it easy to add new functionality to the system and, in addition, flexibly configure the release in a distributed environment.

Core

The main goal of the kernel is to create the correct environment with basic functions for all services connected to it. The demo application implements kernel functions:

• Configuration
• Logger
• Metrics
• Service Registry

Node configuration

For simple utilities, you can use command-line variables and the flag engine. For more complex programs, the introduction of a configuration file is justified.

I am impressed by the yaml format, so in this example it is the one that is used, but you can replace the parser with a simple refinement in the config.go of the Parse / 1 function and the attributes themselves that are responsible for parsing in the declaration of the config structure.

The entire configuration is divided into sections, grouping logically related parameters:

• Node - a group of parameters that determine the name of the node and its position in the cluster (node ​​name, data center, shelf)
• Runtime - allows you to configure runtime go, for example, the degree of parallelization.
• Dirs - sets the list of directories that the kernel will automatically create. In this case, accessing them in the code is quite simple, an example of getting the path to the data folder is System.Config.Dirs.Data
• Log - defines the parameters of the logger. We can write logs to stdout or to a file, while flexibly adjusting the number of events through logging levels.
• Metrics - provides options for application metrics. The metrics file allows you to get an idea of ​​the processes occurring within the application, for example, if there are too many or too few events. Also, metrics are useful when testing, because they allow you to send a request to the system and see how the counters have changed during the processing of the request.
• DataSources - contains parameters for connecting to external data sources, such as, for example, databases.
• Services - allows you to define a specific set of options for each service.

Services

For the correct initialization of the service, it is necessary to fulfill a number of conditions:

1. Implement the description of the service structure, inheriting the standard description interface:
   An example for our service:

   
   

   type ACounter struct { core.Service http *http.Server } 

   
   

2. Define the init() function in which the service instance is created and started:

   
   

    func init() { go (&ACounter{ Service: core.NewService(Acounter), }).start() } 

   
   
3. Implement the logic of starting and stopping the service

 func (srv *ACounter) start() { waitCore() srv.Ready = true srv.ShutdownFun = func(reason string) { log.Debug("acounter: soft shutdown") ... } core.Register(&srv.Service) ... } 

To register services, use the standard init() call. Since we cannot guarantee the order of init() calls in modules, the kernel provides wait mechanisms:

• core.WaitCore() - the call waits for the completion of the full kernel initialization: setting up metrics, logger, connecting to the database and other internal processes
• core.WaitService(srv string) - allows you to wait for the initialization of another service to complete. This mechanism allows to determine the order of launching related services.

If a stop signal is received from the operating system, the kernel reports this to all services by calling them srv.ShutdownFun .

Dependency management

Dep is included in the base image. This utility allows you to produce vendor locking and is currently the standard. To add dependencies in a project, import it in any project file and run a predefined target:

• $ make deps

All new dependencies will be placed in the vendor directory. I would also like to note that Dep automatically removes unused dependencies.

Logger

There are a lot of libraries for logging on go. For example, glog from google is customizable, it has levels and copes with the task perfectly. But I didn’t want additional dependency in the project, and many libraries didn’t fit for one reason or another.

As a solution, we had to extend the standard log so that it meets the following requirements:

1. Implemented the original API log.
   Call log. * Should work.
2. Allowed to easily switch to it in already existing projects.
   imprt “log” is replaced with import "github.com/Vonmo/relgo/log"
3. Display microsecond time stamps.
4. Showed the file name and the line in which the call to the logging function occurred.
5. Had message typing: Debug, Info, Error, Panic, Fatal and setting the logging level.

Migrations

Since our project is database dependent and there is a data schema in postgresql, it is good practice to use migrations. Approaches to migrations are different, you can write them within the framework of the project language, implementing the functions of migration and rollback, and you can use additional utilities.

The choice of tool depends on the preference of the developer and the environment of the project. If you work in several languages ​​at the same time and would like uniformity in migrations, you can use an external utility. For myself, I chose sql-migrate . This utility generates plain text in which you need to put the sql migration and rollback.

Testing

In the world of go offer to test http api without starting the server, by directly calling handlers (see https://golang.org/pkg/net/http/httptest/ ). However, I want to move away from unit tests and conduct a full interface testing to get closer to integration testing.

In the example, you can find the implementation of testing a full kernel and all specific services in a test environment.

Note: I’m probably not looking well enough, but is there a testing framework on Golang that is close in functionality to Erlang's Common Test?

Total file size

The question arises: why are the Golang assemblies so big? For example, if we build a simple main(){ fmt.Println("done.") } get about 1.9 MB, of which the runtime golang takes about 1 MB (the data is relevant for go 1.9.2 amd64).

With real projects, things are even more interesting:

• consul-linux-x86_64 - 42.8 MB
• vault-linux-x86_64 - 75.5 MB
• mailhog-linux-x86_64 - 11.2 MB

Since the release comes with stable code, you can disable DWARF debuggin information and general debug information by specifying the -w and -s keys as build options. This measure will reduce the size by about 30%.

The next step is to use executable file wrappers. One of the popular packers is upx. After applying the packer, the final assembly size was reduced from the initial 1.9 MB to 423 kb, so we reduced the size of the source file by almost 78%. With this you can already live.

Results

We managed to create an environment in which you can easily develop applications. Just as with the Erlang work environment, this environment allows you to get a repetitive result, regardless of the development machine, to get rid of library conflicts and tool versions, to simplify further CI and, most importantly, identify problems in the early stages of development, since with the help of docker you can reproduce quite complex environments.

Dear readers, thank you for your time and interest in the topic.