In 2014, I spoke at the opening of the GopherCon conference with a report entitled “Go: Best Practices for Production Environments ”. In SoundCloud, we were one of the first Go users, and by that time already two years had written on it and supported Go in combat in one form or another. During this time we have learned something, and I tried to share a part of this experience.

Since then, I continued to program on Go throughout the entire working day, first in the SoundCloud teams responsible for operations and infrastructure, and now I work at Weaveworks on Weave Scope and Weave Mesh . I also worked hard on the Go kit , an open source microservice toolkit. And all this time I took an active part in the development of the Go-programmers community, met with many developers at meetings and conferences throughout Europe and in the USA, collecting their success stories and failures.

In November 2015, on the sixth anniversary of the release of Go, I recalled my first performance. Which of the best practices have passed the test of time? Which ones are outdated or become ineffective? Have any new techniques appeared? In March, I had the opportunity to speak at QCon London , where I spoke about the best practices of 2014 and the further development of Go until 2016. This post presents a squeeze from my speech.
')
Key points I highlighted in the text as Top Tips - the best tips.

Here is the content:

1. Development environment
2. Repository structure
3. Formatting and style
4. Configuration
5. Program development
6. Logging and metrics
7. Testing
8. Dependency management
9. Build and Deploy
10. Conclusion

Development environment

Go environment agreements are based on the use of GOPATH. In 2014, I defended the view that there should be a single global variable GOPATH. Since then, my position has softened somewhat. I still think that, other things being equal, this is the best option, but much also depends on the specifics of your project, team and other things.

If you or your company create mostly executable binaries, then using a separate GOPATH for each project can provide certain advantages. For such cases, you can use the new gb utility from Dave Cheney and contributors, replacing the standard go tools for this purpose. On the utility there are already many positive reviews.

Some developers use GOPATH with two directories (two-entry), for example $HOME/go/external:$HOME/go/internal . The go-command always knew how to handle such cases: go get downloads the dependency to the directory by the first path, so this solution can be useful if you need to strictly separate the internal code from the third-party.

I noticed that some developers forget to put GOPATH/bin in their PATH . But this makes it easy to run the executable files you get with go get , and also makes it easier to work with the (preferred) go install code mechanism. There is no reason not to do this.

✪ Top Tip - $GOPATH/bin in your $PATH , this will make it easier to access installed programs.

Thanks to all kinds of editors and IDE, the development environment has continuously improved. If you are a vim fan, then everything went perfectly for you: thanks to the tireless and incredibly effective work of Fatih Arslan ( Fatih Arslan ), the vim-go plugin has turned into a real work of art, the best tool in its class. I’m not so familiar with Emacs , but Dominic Honnef is go-mode.el governing this area.

Moving on, many still successfully use the Sublime Text + GoSublime combination . In terms of speed it is difficult to compete. But apparently, recently more and more attention is paid to editors based on Electron . There are a lot of fans of the Atom + go-plus bundle, especially among developers, who often have to switch from some language to JavaScript. A bunch of Visual Studio Code + vscode-go was a dark horse: it runs slower than Sublime Text, but noticeably faster than Atom, and at the same time, by default it perfectly supports important features for me, like click-to-definition (transition to the object definition by click ). I have been using this bundle every day for six months now, since Thomas Adam ( Thomas Adam ) introduced me to her. Great thing.

As for the full IDE, we can mention the specially created LiteIDE , which is regularly updated and has its audience of fans. There is also an interesting plugin for Go Intellij , which is constantly improving.

Repository structure

We had enough time for the projects to become more mature, and as a result a number of clear approaches were developed. What your project is about depends on how you structure your repository. If we are talking about a closed project or an internal project of a company, then you can break away: let it have its own GOPATH, use a custom build tool, do anything if it brings you pleasure and improves your productivity.

But if it is a public project (for example, open source), then the rules become stricter. Your code must be compatible with go get , since this is the way most Go developers will want to take advantage of your work.
The ideal repository structure depends on the types of your entities. If these are exclusively executable binaries or libraries, then you need to be sure that consumers can use go get or import along the base path. So place package main or main code for import into github.com/name/repo , and use subfolders for support packages.

If your repository is a combination of binary files and libraries, then you should define the main entity and put it in the root of the repository. For example, if your repository for the most part consists of executable files, but can also be used as a library, then you probably prefer to structure it like this:

 github.com/peterbourgon/foo/ main.go // package main main_test.go // package main lib/ foo.go // package foo foo_test.go // package foo 

Useful advice: in the lib/ subfolder it is better to name the package according to the name of the library, and not the folder itself; that is, in this example, package foo instead of package lib . This is an exception to fairly strict Go-idioms, but in practice it is very convenient for users. Similarly, the excellent tsenart / vegeta repository is built , a tool for load testing HTTP services.

✪ Top Tip - If your foo repository mainly consists of executable binary files, put the library code in the lib/ subfolder and name the package foo .

If your repository is basically a library, but also includes one or two executable programs, then the structure could be:

 github.com/peterbourgon/foo foo.go // package foo foo_test.go // package foo cmd/ foo/ main.go // package main main_test.go // package main 

It turns out the inverted structure, when the library code is put in the root, and the executable program code is stored in the cmd/foo/ subfolder. The intermediate level cmd/ convenient for two reasons:

• The Go toolkit automatically names the binaries by the folder name in which package main is located, so that we get the best file names without possible conflicts with other packages in the repository.
• If your users use go get on the path that contains /cmd/ , they immediately understand what they got. The repository of the gb build utility is similarly arranged.

✪ Top Tip - If the main purpose of your repository is a library, then place the code of the executable programs in subfolders within cmd/ .

The main point here: take care of users — simplify the use of the core functionality of your project. It seems to me that this abstract idea — focusing on user needs — is in keeping with the very spirit of Go.

Formatting and style

Here, little has changed. This is one of those places where Go has gone the right way, and I really appreciate community agreements and the stability of the language regarding this. Code Review Comments are great and should be the minimum set necessary to meet the criteria during a code revision. And if you have conflicting situations or contradictions in the names, you can use an excellent set of idiomatic naming conventions for Andrew Gerrand.

✪ Top Tip - Use Andrew Gerrand's naming conventions.

As for the toolkit, everything here has only become better. Configure your editor so that when saving gofmt is initiated, and better goimports (I hope no one will have any objections here). Using the go vet utility almost never leads to false positives, so you can easily make it part of your pre-commit hook. And pay attention to the gometalinter code quality monitoring utility. It may have false positives, so it makes sense to somehow label your own agreements .

Configuration

The configuration lies between the runtime environment and the process. It should be explicit and well documented. I still use and recommend using the flag package, but would still prefer the configuration to be more familiar. I would like to get the standard syntax of arguments in getopts-style, so that there are detailed and brief form of the arguments. I also want the usage text to be much smaller.

Applications that follow the Twelve-Factor App are motivated to use environment variables for configuration, and I think this is normal, provided that each variable is also defined as a flag . Here, the obviousness is important: changing the runtime behavior of an application should be performed in an easily detectable and documented way.

I already said in 2014, but I consider it necessary to repeat: define and disassemble the flags inside func main () . Only func main() has the right to decide which flags will be available to the user. If your library allows you to configure your behavior, then configuration parameters should be part of type constructors. Transferring the configuration to the global scope of the packages creates the illusion of benefits, but the savings are false: you break the code modularity, so it will be more difficult for other developers to understand dependencies, and it will also be much more difficult to write independent, parallelized tests.

✪ Top Tip - Only func main() has the right to decide which flags will be available to the user.

I think the community may well create a comprehensive package of flags in which all these properties will be combined. It may already exist. If so, let me know . I would definitely use it.

Program development

In the conversation, I used the configuration as a starting point to discuss a number of other aspects of program development (I did not raise this topic in 2014). First, let's look at the constructors. If we correctly parameterize all our dependencies, then the designers can become quite large.

 foo, err := newFoo( *fooKey, bar, 100 * time.Millisecond, nil, ) if err != nil { log.Fatal(err) } defer foo.close() 

Sometimes it is better to express such a construction with the help of a configuration object: structures that accept optional parameters that determine the behavior of the object being constructed. Suppose that the fooKey parameter fooKey required, and all others either have reasonable default values ​​or are optional.

I often see projects in which configuration objects are constructed in some separate way:

 //    cfg := fooConfig{} cfg.Bar = bar cfg.Period = 100 * time.Millisecond cfg.Output = nil foo, err := newFoo(*fooKey, cfg) if err != nil { log.Fatal(err) } defer foo.close() 

But it is much better to construct an object at a time, using a single expression, using the so-called structure initialization syntax.

 //    cfg := fooConfig{ Bar: bar, Period: 100 * time.Millisecond, Output: nil, } foo, err := newFoo(*fooKey, cfg) if err != nil { log.Fatal(err) } defer foo.close() 

There are no expressions when the object is in an intermediate, wrong state. At the same time, all fields are beautifully delimited and indented, reflecting the definition of fooConfig .

Note that we construct the cfg object and use it immediately. In this case, by directly embedding the structure declaration in the newFoo constructor, we can avoid another stage of the intermediate state and save another line of code.

 //    foo, err := newFoo(*fooKey, fooConfig{ Bar: bar, Period: 100 * time.Millisecond, Output: nil, }) if err != nil { log.Fatal(err) } defer foo.close() 

Fine.

✪ Top Tip - To avoid an incorrect intermediate state, use a literal structure initialization. Wherever possible, embed structure declarations.

Now turn to the topic of reasonable defaults. Note that the Output parameter can be nil .

Suppose it is io.Writer . If we don’t do anything special, then when we want to use it in our foo object, we first have to check for nil.

 func (f *foo) process() { if f.Output != nil { fmt.Fprintf(f.Output, "start\n") } // ... } 

This is not great. It is much better and safer to be able to use the output value without checking for its existence.

 func (f *foo) process() { fmt.Fprintf(f.Output, "start\n") // ... } 

So here we need to provide something useful by default. Thanks to the interface types, we have the ability to transfer something that provides a no-op-implementation (i.e., an implementation that does not perform any operations, a stub. - Note of the translator) of the interface. Therefore, the stdlib ioutil package comes with a no-op io.Writer called ioutil.Discard .

✪ Top Tip - Avoid checking for nil with no-op default implementations.

You could pass this to the fooConfig object, but this is a rather fragile solution. If the calling code forgets to do it in the place of the call, then we again get the parameter nil . Instead, we can protect ourselves inside the constructor.

 func newFoo(..., cfg fooConfig) *foo { if cfg.Output == nil { cfg.Output = ioutil.Discard } // ... } 

This is just the use of Go-idioms "make zero value useful." That is, we allow zero ( nil ) to provide good default behavior (no-op).

✪ Top Tip - Make the null value useful, especially in configuration objects.

Let’s go back to the constructor. The fooKey , bar , period and output parameters are dependencies . The success of starting and running the foo object depends on each of them. What I definitely learned in six years of daily programming on Go and observing large projects is that you need to make dependencies explicit .

✪ Top Tip - Make dependencies explicit!

I believe that ambiguous or implicit dependencies are the cause of an incredible amount of labor for technical support, confusion, bugs and unpaid technical debt. Consider the process () method of type foo :

 func (f *foo) process() { fmt.Fprintf(f.Output, "start\n") result := f.Bar.compute() log.Printf("bar: %v", result) // Whoops! // ... } 

fmt.Printf autonomous, does not affect and does not depend on the global state. In functional terms, it has something like referential transparency . So this is not an addiction. Obviously, it is f.Bar . Curiously, log.Printf affects the global (within the package) log.Printf object, which is simply not obvious due to the free Printf function. So this is also an addiction.

What do we do with all these dependencies? Let's make them explicit . Since the process () method writes to the log during its work, either the method or the foo object itself must accept the logging object as a dependency. For example, log.Printf should become f.Logger.Printf .

 func (f *foo) process() { fmt.Fprintf(f.Output, "start\n") result := f.Bar.compute() f.Logger.Printf("bar: %v", result) // . // ... } 

We are accustomed to counting side specific types of work like logging. Therefore, we are pleased to use supporting libraries like global loggers to ease our burden. But logging, like metrics, often plays a crucial role in the functioning of a service. And hiding dependencies in the global visibility space can — and does it — hit us the same way, either in the form of something seemingly harmless, like logging, or in the form of some other, more important, subject component, which we did not take care of parameterization . Protect yourself from future pain by using strict rules: make all your dependencies explicit .

✪ Top Tip - Loggers are dependencies, as are links to other components, database clients, command line arguments, etc.

Of course, we also need to take care of obtaining a reasonable silence for our logger.

 func newFoo(..., cfg fooConfig) *foo { // ... if cfg.Logger == nil { cfg.Logger = log.New(ioutil.Discard, ...) } // ... } 

Logging and metrics

Speaking of the problem as a whole: with logging I had a lot more experience in combat, which only strengthened my respect for the problem. Logging is expensive, much more expensive than you think, and can quickly become a bottleneck in your system. I covered this topic in detail in a separate post , but if in brief:

• Log only information that provides a basis for action , read by a person or a machine.
• Avoid overly detailed logging, you may need general information and data for debugging.
• Use structured logging. Although I am biased, I recommend go-kit / log .
• Loggers are dependencies!

Where logging is expensive, metrics are cheap. Remove metrics from any significant component of your code base. If this is a resource, like a queue, then measure it using the Brendan Gregg USE method : Utilization, Saturation, Error count (rate). If this is some kind of endpoint, then measure using the RED method of Tom Wilkie : Request count (rate), Error count (rate), Duration.

If you have the opportunity to choose in this matter, then I recommend using Prometheus as a measuring system. And of course, metrics are also dependencies!

Let's digress from loggers and metrics and look directly at the global state. Here are some facts about Go:

• log.Print uses a fixed global log.Logger .
• http.Get uses a fixed global http.Client .
• http.Server by default uses a fixed global log.Logger .
• database/sql uses a fixed global driver registry.
• func init exists only to have a side effect on the global state of the package.

These facts are tolerated separately, but difficult in general. That is, how can we test the output sent to the log by components using a fixed global logger? We'll have to redirect this data, but how to test them in parallel? None? The answer is unsatisfactory. Or, say, there are two independent components that generate HTTP requests with different requirements, how do we manage this? Using standard global http.Client is quite difficult to do. See an example:

 func foo() { resp, err := http.Get("http://zombo.com") // ... } 

http.Get calls global in the http package. It has an implicit global dependency, which we can quite easily get rid of:

 func foo(client *http.Client) { resp, err := client.Get("http://zombo.com") // ... } 

Just pass http.Client as a parameter. But this is a specific type (concrete type), so if we want to test this function, we will have to provide a specific http.Client, which will surely force us to establish the actual connection via HTTP. This is not good. You can do better: pass an interface that can perform ( Do ) HTTP requests.

 type Doer interface { Do(*http.Request) (*http.Response, error) } func foo(d Doer) { req, _ := http.NewRequest("GET", "http://zombo.com", nil) resp, err := d.Do(req) // ... } 

http.Client automatically satisfies the Doer interface, but now we are free to pass on our implementation of Doer to our test. : foo foo , , http.Client , .

…

Testing

2014 , - . (stdlib) . . Go , . , . testing .

TDD/BDD , DSL , , . , . , , , , . When in Go, do as Gophers do ( Go, , ) : — Go, , , .

, . GOPATH, , , DSL . , , . , .

. (Mitchell Hashimoto) ( SpeakerDeck , YouTube ), .

: , Go , . , , , .

✪ Top Tip — .

http.Client , , - , . - -, HTTP-, , , . - - .

✪ Top Tip — , .

. 2014 , (vendor). - : . , Go 1.6 GO15VENDOREXPERIMENT vendor/ . . , , . :

• FiloSottile/gvt . , gb , .
• Masterminds/glide : . .
• kardianos/govendor , , , . - ( , , — vendor.json. — . ).
• constabulary/gb go . , , .

✪ Top Tip — .

. Go . , . , 1.5 , . , : 1 , 2 . , : .

✪ Top Tip — .

, () API. , .

open source , , . , , . GO15VENDOREXPERIMENT , .

, . Etcd , , , Go Windows. , , , . , , , - .

, ( ) go install go build . install $GOPATH/pkg , . $GOPATH/bin , .

✪ Top Tip — go install go build .

, , gb . . , Go 1.5 - « ». GOOS GOARCH, go-. .

, , , , Ruby, Python, JVM. : , — FROM scratch. Go , .

: , , — -. . , AMI EC2, . .

Conclusion

Top Tips:

1. $GOPATH/bin $PATH , .
2. foo , lib/ package foo .
3. — , cmd/.
4. .
5. func main() , .
6. , . , , .
7. nil no-op- .
8. , .
9. !
10. , , , . .
11. .
12. , .
13. .
14. .
15. go install go build .

Go , , - . — — . ( Go Proverbs ), , « » (up the stack) , , Go.

Go.