We are preparing to build Go-applications in production

In June, at the RIT ++ conference, my colleague Igor Dolzhikov and I shared our experience in automating the development of services on Go — from the first commit to the release to the Kubernetes production environment (yes, the video starts at 07:16, and we don’t like it either ). Since the publication of the master class, from time to time I get questions on various topics covered in it. Perhaps the hottest questions deserve separate consideration, and today I would like to talk about the process of building the application. Topics are relevant not only in the preparation of services, but in general for any applications written on Go.

Everything that is described in this article is relevant for the current version of Go - 1.9.

Cherished GOPATH and location code

When preparing an application for production, you need to be as confident as possible that the code that the developer expects will fall into the assembly. By default, Go still does not know how to work with dependency management, which means that when the application is compiled, all “external” dependencies will be searched for Go tools inside the $GOPATH/src directory. How to find out the current GOPATH value if you are not sure about it? This value can be found in the list of variables output by the go env command.

In addition, the code of the project itself should also be inside $GOPATH/src , and I recommend to think in advance about the way in which it will lie. When a project is under the control of a version control system, when it is tightened using, for example, the go get command, it should fall precisely along the path that we defined for the project initially. For example, the service code that is stored in my github account in the rumyantseva / mif repository is deployed inside my $GOPATH/src/github.com/rumyantseva/mif . If the same code were inside the mif repository of some private storage example.com in the namespace services, then the path to it on the developer's machine would look like $GOPATH/src/example.com/services/mif . In order to avoid future problems or ambiguities, the code positioning rule must be observed.
')
Different projects can be stored both within the same GOPATH directory and within several. Accordingly, in the second case, the GOPATH value GOPATH have to be redefined. In order to do this, you will need to reset the corresponding environment variable to the desired value. In case GOPATH not specified at all, Go will consider the go directory in the user's home directory as the working directory. To better understand this, let's conduct several experiments with the console:


However, if you still don’t like the idea of ​​GOPATH, you can turn to tools like gb , which allows assembling regardless of the location of the code.

Work with external dependencies

So, if we are writing an application that uses external (relative to the current repository) dependencies, for a successful build we need all these dependencies to be inside GOPATH . You can pull in dependencies automatically by calling commands such as go get or go install inside the current working project. At the same time, we will download the code of dependencies located in the default branches of the repositories. This is enough for the “here and now” situation, but in the general case, no one guarantees that after 5 minutes, back-incompatible changes will not appear in the same branches of external dependencies. So, the next attempt to deploy the application (for example, on the build machine) can end in failure. What will help us in this situation? Of course, vendoring.

About the vendor directory was repeatedly written on Habré, and in many other places, and I will not repeat. However, I’ll briefly remind you that all the dependencies that we pulled into GOPATH/src can also be added to the vendor directory of the current application. How to do it? Or manually, or with the help of a dependency management manager. As an example of a utility for working with dependencies, we can finally mention dep , the official Go experiment. Despite the status of the "official experiment", dep is still neither absolutely stable nor recommended. Nevertheless, we ventured to try dep in our work projects, and we liked it! If this is your first time with the issue of dependency management, I highly recommend you a ten-minute video from the conference Gophercon 2017, in which the principles of the work of dep are clearly and visually shown.

So, on the basis of using the dependency manager, we got the vendor directory full of packages, and some set of metafiles describing our dependencies. It would seem that metafiles with a description of the used tags and even hashes of commits are enough for us, and the natural desire would be to remove the vendor directory from the control of the version control system. However, in real production projects, you should not do this. Storing code with vendor is the only way to protect against accidents such as dropping a github or completely removing your repository from a third-party developer.

Binary Versioning

Perhaps almost all popular console utilities support a command or version flag, which allows you to display information about the current version of the binary. The same practice can be useful in the case of a running service. In addition, it is sometimes useful to “sew” a binary not only the information about the semantic version, but also the hash of the commit, the build date, and other useful data.

Such information is conveniently stored as variables, for example, in a special package. And by calling the linker with the -X flag, we can automatically replace the values ​​of these variables.

A small example. Create a file hello.go with the following contents:

 package main import "fmt" var hello = "Hello" var world = "World" func main() { fmt.Printf("%s, %s!\n", hello, world) } 

When you start it normally, for example, using the go run hello.go , we get the string “Hello, World!”:

 $ go run hello.go Hello, World! 

And now we add a call to the linker with the -X flags and new variable values:

 $ go run -ldflags="-X main.hello= -X main.world=" hello.go , ! 

It is possible to substitute not only variables from the main package, but also variables from any packages in general. Thus, during the assembly of the application, you can sew into it any necessary meta-information.

upd. Thanks to forsyte for clarifying the -X flag

A set of instructions for building the application

Despite its age, the make utility does not lose its relevance and popularity among those who have to build applications (at least under * nix). Here and among Go-developers this tool is rather widespread.

Consider a specific Makefile example for some service. I prepared a go-zeroservice repository containing a “zero” service, the only functionality of which is to run and display build information.

The make build command builds a binary file, substituting the version specified in the Makefile, the last commit hash, and the current time. In this case, before make build, the clean command is called, which deletes an existing binary (if it was). To update the dependencies, make vendor is provided, which will install dep if it does not already exist, and execute the dep ensure command to update packages inside the vendor. To check the quality of the code, the command make check is proposed, which will install and launch the metalinter .

In our production projects, we put in the Makefile any more or less repetitive actions - running checks on coding standards and running tests, running the package manager, building the application for the OS with the necessary flags, building commands and running the Docker container, and even , allowing you to launch a service release in Kubernetes using Helm.

The presence of such commands on different environments allows you to quickly perform the necessary actions, for example, run tests and collect and run the container on the developer’s local environment, or run tests and build and release as part of the CI / CD processes. In the case of go-zeroservice, you can see the .travis.yml file, which starts the build of the service within the Travis CI and consists of the commands described in the Makefile.

Conclusion

So, in order to deal with the issues of building Go-applications for production, it takes not so much. First, you need to decide on the location of the application code inside GOPATH and make sure that it corresponds to the location of the code in the version control system. Secondly, you need to decide how to interact with the management of external dependencies and select the appropriate tool . Thirdly, it is convenient when you have at hand all the instructions that you have to perform more or less regularly and on different environments, for example, a Makefile helps in storing such instructions.

I hope this article will help the newcomers of the Go world to quickly deal with issues of release preparation. And if you use other practices to build applications, do not hesitate to share them in the comments and thank you in advance for that.

PS By the way, we came up with the continuation of the master class about Go and Kubernetes and plan to present it in September at the DevFest Siberia conference. Join us! ;)