Article written in February 2018

Go needs to add package versioning.

More precisely, you need to add the concept of versioning to the working dictionary of Go developers and tools so that everyone uses the same version numbers when they mention which program to build, run or analyze. The go command should say exactly which versions of which packages are in a particular build.
')
Version numbering allows you to make reproducible assemblies: if I post the latest version of my program, you will receive not only the latest version of my code, but also the exact same versions of all the packages on which my code depends, so that you and I will create completely equivalent binary files.

Versioning also ensures that tomorrow the program will assemble in the same way as today. Even if new versions of dependencies are released, go will not use them without a special command.

Although you need to add version control, you should not abandon the main advantages of the go command: it is simplicity, speed and clarity. Today, many programmers do not pay attention to the version, and everything works fine. If you make the right model, the programmers will still not pay attention to the version numbers, everything will just work better and become clearer. Existing workflows remain virtually unchanged. The release of new versions is very simple. In general, versioning should go into the background and not take the attention of the developer.

In short, you need to add package version control, but do not break go get . In this article, we offer how to do this, as well as demonstrate a prototype that you can try right now, and which I hope will be the basis for a possible go integration. I hope the article will be the beginning of a productive discussion about what works and what does not. Based on this discussion, I will make adjustments to both my proposal and the prototype, and then I will submit a formal proposal to add an optional function to Go 1.11.

This offer retains all the benefits of go get , but adds reproducible assemblies, supports semantic versioning, eliminates vending, removes GOPATH in favor of a project-based workflow, and provides a smooth departure from dep and its predecessors. However, this proposal is still at an early stage. If the details are not correct, we will correct them before the work gets into the main distribution Go.

General situation

Before examining the proposal, let's look at the current situation and how we got there. Maybe this section is a bit too big, but the story carries important lessons and helps to understand why we want to change something. If you are not interested, you can go directly to the proposal or read the accompanying blog article with an example .

Makefile , goinstall and go get

In November 2009, a compiler, linker and several libraries were released with the original version of Go. To compile and link programs, it was necessary to run 6g and 6l , and we included examples of makefiles in the package. A minimal gobuild shell could build one package and write the appropriate makefile (in most cases). There was no fixed way to share the code with others. We knew that this was not enough - but we released what we had, planning to develop the rest together with the community.

In February 2010, we offered goinstall , a simple command to download packages from version control system repositories, such as Bitbucket and GitHub. Goinstall introduced conventions on import paths, which are now generally accepted. But while no code followed these conventions, goinstall initially only worked with packages that imported nothing but the standard library. But developers quickly moved to a single agreement that we know today, and the set of published Go packages has grown into a coherent ecosystem.

Goinstall also removed the Makefiles, and with them the complexity of the custom build options. Although it is sometimes inconvenient that package authors cannot generate code during each build, this simplification is incredibly important for package users : they do not need to worry about installing the same set of tools that the author used. Such a simplification is also crucial to the operation of the tools. A makefile is an obligatory step-by-step recipe for compiling a package; Using a different tool like go vet or auto-completion can also be quite complex. Even getting dependencies right to re-build packages if necessary and only when necessary is much more difficult with arbitrary Makefiles. Although at that time some objected that they were deprived of flexibility, but looking back, it became clear that the refusal of the Makefile was the right step: the benefits far outweigh the inconvenience.

In December 2011, in preparation for Go 1, we presented the go command , which replaced goinstall with go get .

In general, go get introduced significant changes: it allowed Go developers to share source code and use each other's work. He also isolated the parts of the assembly system within the go command, so that considerable automation was possible with the help of tools. But go get is missing the version control concept. In the very first goinstall discussions, it became clear: you need to do something with version control. Unfortunately, it was not clear what to do. At least, we in the Go team did not clearly understand this. When go get requests a package, it always gets the latest copy, delegating download and update operations to a version control system, such as Git or Mercurial. This “blind work” has led to at least two significant shortcomings.

Version Management and API Stability

The first major drawback of go get is that without the concept of version control, it cannot tell the user what changes to expect in this update.

In November 2013, in the version of Go 1.2, an entry FAQ was added with such advice regarding versioning (the text has not changed to Go 1.10):

Packages for general use must maintain backward compatibility as they evolve. Go 1 compatibility guidelines are relevant here: do not delete exported names, encourage tagging of composite literals, and so on. If new functionality is required, add a new name, but do not change the old one. In case of a major change, create a new package with a new import path.

In March 2014, Gustavo Niemeyer launched gopkg.in under the guise of “stable APIs for the Go language”. This domain is a GitHub redirect with a version, allowing you to import paths like gopkg.in/yaml.v1 and gopkg.in/yaml.v2 for different commits (possibly in different branches) of the same Git repository. According to semantic version control, authors should release a new basic version when making critical changes. Thus, later versions of the v1 import path replace the previous ones, and v2 can render completely different APIs.

In August 2015, Dave Cheney submitted a proposal for semantic versioning . Over the next few months, this caused an interesting discussion: it seemed that everyone agreed that the semantic mark of versions was a great idea, but no one knew how the tools should work with these versions.

Any argument for semantic versioning inevitably meets criticism with reference to the Hyrum Act :

The contract of your API becomes unimportant with a sufficient number of users. Someone depends on any observed behavior of the system.

Although the Hyrum law is empirically true, semantic versioning is still a useful way to form expectations for the relationship between releases. Upgrading from 1.2.3 to 1.2.4 should not break your code, and upgrading from 1.2.3 to 2.0.0 may well. If the code stops working after updating to 1.2.4, then the author will most likely accept the bug report and fix the error in version 1.2.5. If the code stopped working (or even compiling) after upgrading to 2.0.0, then this change was much more likely to be intentional and, accordingly, it is unlikely to fix something in 2.0.1.

I do not want to conclude from the Hyrum law that semantic versioning is impossible. Instead, I believe that builds should be used carefully, using the exact same version of each dependency as the author. That is, the default builds should be as reproducible as possible.

Vending and replicable builds

The second major drawback of go get is that without the concept of version control, the team cannot provide and even express the idea of ​​a replicable build. It’s impossible to make sure that users compile the same code dependency versions as you. In November 2013, the following advice was added to the FAQ for Go 1.2:

If you use an external package and fear that it may change unexpectedly, the simplest solution is to copy it to your local repository (this is the approach used by Google). Save a copy with a new import path that identifies it as a local copy. For example, you can copy original.com/pkg to you.com/external/original.com/pkg . One of the tools for this procedure is goven Keith Rarik.

Keith Rerik began this project in March 2012. The goven utility copies the dependency to the local repository and updates all import paths to reflect the new location. Such source code changes are necessary, but unpleasant. They make it difficult to compare and include new copies, and also require updating other copied code using this dependency.

In September 2013, Keith introduced the godep , “a new tool for fixing package dependencies”. The main achievement of godep was what we now call vendoring, i.e., copying dependencies into a project without changing the source files, without direct support of tools, by means of a certain GOPATH setting.

In October 2014, Keith proposed to add support for “external packages” to Go tools, so that tools would better understand projects using this convention. By that time, several godep -style godep . Matt Farina published the post “Traveling by Sea Go godep Managers”, comparing godep with godep , especially glide .

In April 2015, Dave Cheney introduced gb , “a project-based assembly tool ... with repeatable builds through source vending,” again without rewriting import paths (another motivation for creating gb was to avoid the requirement to store code in certain directories in GOPATH which is not always convenient).

That spring, Jason Buberl studied the situation with Go package management systems, including numerous duplication of effort and wasted work on similar utilities. His survey made it clear to developers that support for vendoring without overwriting import paths should definitely be added to the go command. At the same time, Daniel Theophanes began preparing specifications for a file format that describes the exact origin and version of the code in the vendor catalog. In June 2015, we accepted Keith’s offer as a vendoring experiment in Go 1.5 , which was included by default in Go 1.6. We encouraged the authors of all the tools for vendoring to work with Daniel to adopt a single metadata file format.

The introduction of the concept of vendinging in Go allowed tools like vet more competently analyze programs, and today it is used by about a dozen or two package managers or tools of vending. On the other hand, since everyone has different metadata formats, they do not interact and cannot easily exchange information about dependencies.

More fundamentally, vendoring is an incomplete version control solution. It provides only reproducible builds, but does not help sort out versions of the package and decide which one to use. Package managers like glide and dep implicitly add the concept of version control to Go, in a certain way setting up the vendor directory. As a result, many of the tools in the Go ecosystem are not able to get the correct version information. It is clear that Go needs direct support for package versions.

Official experiment on package management

A group of Go activists gathered at GopherCon 2016 in Hack Day (now Community Day) for a broad discussion of package management issues . One of the results was the formation of a committee and an advisory group to conduct a complex of works in order to create a new package management tool . The idea was for the unified tool to replace the existing ones, although it would still be implemented outside of the direct Go toolkit, using vendor catalogs. The committee included Andrew Gerrand, Ed Muller, Jesse Frasel and Sam Boyer, led by Peter Burgon. They prepared a draft specification , and then Sam and his assistants implemented dep . For an understanding of the overall situation, see Sam’s February 2016 article “So, you want to write a package manager,” his post in December 2016, “The dependency management saga in Go” and a speech in July 2017 on GopherCon “A new era of package management in Go .

Dep performs many tasks: this is an important improvement over current practices. This is an important step towards a future solution, and at the same time an experiment - we call it “official experiment” - which helps us to better understand the needs of developers. But dep not a direct prototype for the possible integration of go commands into package versioning. This is a powerful, flexible, almost universal way to explore design solution space. It is similar to the makefiles we fought at the very beginning. But as soon as we better understand the design decision space and narrow it down to several key functions that need to be supported, it will help the Go ecosystem to remove other functions, reduce expressiveness, adopt mandatory conventions that make Go code bases more uniform and easy to understand.

This article is the beginning of the next step after dep : the first prototype of the final integration with the go command, the package equivalent of goinstall . The prototype is a separate command that we call vgo : a go replacement with support for version control packages. This is a new experiment, and we'll see what comes of it. As well as during the goinstall announcement, some projects and code are already compatible with vgo , while others need changes. We will remove some control and expressiveness, just as the makefiles were removed in due time, in order to simplify the system and eliminate the complexity for users. Most importantly, we are looking for pioneers who will help experiment with vgo to get as many reviews as possible.

The beginning of the experiment with vgo does not mean the termination of support for dep : it will remain available until we achieve full and public integration with go . We will also try to make the final transition from dep to integration with go as smooth as possible, in whatever form this integration takes place. Projects that have not yet been converted to dep can still benefit from this transformation (note that the godep and glide stopped their active development and encourage migration to dep). Some projects may wish to go directly to vgo , if that suits their needs.

Sentence

The proposal to add version control to the go command consists of four steps. First, accept the import compatibility rule , which is indicated by the FAQ and gopkg.in: newer versions of the package with the specified import path must be backward compatible with older versions. Second, adopt a simple new algorithm, known as choosing the minimum version to determine which versions of the package are used in this assembly. Third, introduce the concept of a Go module : groups of packages, versioned as a whole and declaring minimum requirements that must be satisfied by their dependencies. Fourth, determine how to integrate all this into the existing go command so that the main workflows do not change significantly from today. In the rest of the article, we consider each of these steps. They are discussed in more detail in other blog articles .

Import Compatibility Rule

The main problem with package management systems is to try to resolve incompatibilities. For example, most systems allow package B to declare that it needs package D version 6 or later, and then allow package C to declare that it requires D version 2, 3, or 4, but not version 5 or later. Thus, if in your package you want to use B and C, then you are not lucky: it is impossible to choose any version of D that satisfies both conditions, and you can do nothing.

Instead of a system that inevitably blocks the assembly of large programs, our proposal introduces an import compatibility rule for package authors:

If the old and new packages have the same import path, the new package must be backward compatible with the old package.

The rule repeats the FAQ mentioned earlier. That text ended with the words: “In the event of a major change, create a new package with a new import path.” , . , :

 import "github.com/go-yaml/yaml/v2" 

2.0.0 , . , Go . . v1 v2.

, . , , , . . .

, dep cargo , . , . -, « » - , - . , - , , , . -, , , «, X», X .

, . . , . , , . , , , , , .

. , , , . , , , , . , - . , , .

.

— . «, », «, ». : () . .

Go

Go , . , . Git , Git . , .

go.mod , . , go.mod :

 // My hello, world. module "rsc.io/hello" require ( "golang.org/x/text" v0.0.0-20180208041248-4e4a3210bb54 "rsc.io/quote" v1.5.2 ) 

, rsc.io/hello , : golang.org/x/text rsc.io/quote . , go.mod . , - .

, vgo , . rsc.io/quote , github.com/rsc/quote , , 1.5.2. golang.org/x/text . , v0.0.0-yyyymmddhhmmss-commit . v0.0.0 yyyymmddhhmmss-commit . , v0.0.0, . , .

, go.mod , , , . .

Goinstall go get , git hg , , . , bzr Bazaar. , Go HTTP zip-. go get . vgo API .

zip- - . -, . go.mod , , .

go

go . , , go build , go install , go run go test , . golang.org/x/text Go .

— GOPATH . go.mod , , go.mod , , . git clone , cd , . . GOPATH.

?

« Go» , vgo . , vgo . . .

, vgo . . go.mod . , go.mod , dep , glide , glock , godep , godeps , govend , govendor gvt , vgo go.mod .

, Go . , Go, — , go get , GOPATH GOPATH. .

- . , , vgo . , Go 1.11 Go, , Go 1.12 . , go get . , , .

go get . , . , , .