gb - dependency management for Go
The lack of a native dependency and version manager in Go is one of the most frequent points in language criticism. In this article, we will look at the problem in more detail and get acquainted with the new project, with the concise name gb, which is gaining popularity in the Go community and promises to soon become the de facto standard for managing dependencies and versions in Go.
(Credit orig.photo: Nathan Youngman)
First, let's see what all the fuss is about and why Go didn’t initially have an advanced dependency manager.
What is meant by the “advanced” dependency manager and what tasks it should solve. After all, but in Go, simple dependency management is present.
')
In Go, there is a standard “go get” command, which is enough to indicate the name of the project on, say, Github, so that the project code is installed, assembled, and ready for use. “Go get” always downloads the latest version, from the master branch, and the general consensus for open-source libraries is not to break backward compatibility at any cost. This is written in the FAQ, and in the vast majority of cases of third-party libraries, this rule is observed.
Depending on your previous experience, you may reject such an idea in the bud, or foresee that this scheme will not be viable very soon, but so far, after 3 years of Go 1 existence, this scheme works and is even used in production in large companies, although initially and more designed for an open-source ecosystem. I will say this - this is a question of the ratio of complexity and risks that developers are willing to take on.
But moving from the open-source world to the world of commercial development, immediately there is a need to ensure that the code of a specific revision will always be compiled and produce the same result, and not depend on external factors. External factors can be understood as new, breaking compatibility, changes in third-party libraries, and turning off the Internet or the fall of Github.
This problem is solved in two ways, which often go together - versioning dependencies ( versioning ) and vendoring ( vendoring ) - by including a third-party code in your code base. The first allows you to ensure that new commits to a third-party library do not break your build, and the second is that even when the Internet is down, all the code needed for building will be available for building locally.
Actually, this is the answer to the question of what advanced dependency management is - a tool that includes everything needed for versioning and vendoring dependencies.
The short answer is because it's damn hard to make everyone happy . Not that difficult - this is impossible if you want to make the most convenient tool suitable for all cases that happen in real practice.
The authors of Go initially recognized the complexity of the problem, and honestly refused to impose a solution on everyone, suggesting options. This question is well stated in the official FAQ and the answer is as follows:
Fair enough, agree.
Further, the FAQ gives recommendations on how and what to do, and how Google solves this problem in its case (again, emphasizing that “what is suitable for us may not be suitable for you”). And if there is an “ideal solution”, then the community has all the cards on hand to create and offer it as standard for Go.
As practice has shown, there is no universal solution for everyone. But the zoo of solutions for different cases and the constant discussion of the issue led to a fairly good consensus on what would put an end to the controversy about advanced dependency management in Go.
The entire zoo of solutions comes down to utilities that solve the issue of either versioning, or vendoring, or both. The range of solutions is wide - from simple Makefile-c with the replacement of GOPATH and git-self-modules to rewriting of paths in imports and full-fledged all-in-one utilities, the most popular of which is Godep .
But all of them are in essence wrappers over the standard go get and go build commands. And it was this that was the source of the major inconsistencies. Workflow when working with go get / build works great, if you write an open-source project, everything is simple and convenient. But if you write a separate working project that doesn’t even intersect with your open-source activity at all, then all versioning / vendoring solutions become a thorn in the heel and deprive of certain amenities and add complexity. Everything becomes more confusing if you try to mix both scenarios.
The realization of these two main different development scenarios (open-source vs private closed project) led to the understanding that different scenarios require different solutions. And not so long ago, Dave Cheney (one of the Go-contributors and in general, a notable Go-popularizer) proposed to clearly separate these two scenarios, creating a separate tool for the second, which would be similar to the standard go get / build, but originally created for with a project-oriented source tree, with a clear separation between “own code” and “dependencies”.
The main theses underlying gb are:
If you were scared on the phrase “replaces the standard go get / build / test”, then this is a normal reaction :) In fact, the project for gb can be absolutely easy to use with the standard go build / test, but gb build will allow you not to bother on the way to delivered packages.
Now in order.
A separate utility replacing the standard go get / build / test
This is exactly the case, and you will have to put one more command in your working system. This is done simply:
Everything is determined by the directory structure.
By “everything” is meant the fact that gb can work with this project. The rule here is simple - there should be a folder src / in the directory. And this is enough to start working with gb.
Often, when Go is used for only one project, the “GOPATH per project” approach is recommended - in fact, you are asked to change your GOPATH to the path to this project and work in it. gb implements something like this, just not touching your system GOPATH. Slightly more below.
No special description files.
Modern projects already consist of a dozen .dot files with various manifests and descriptions. Another such file needed to build a project would be too, and not in Go-style.
No code changes
The project code always remains the way it was written. There are no import overwrites in the import style of github.com/user/pkg -> import "vendor / github.com / user / pkg" 'is not and will not be.
Clear separation of code into “project” and “dependency”
Returning to the point about the directory structure, gb treats everything that is in src / as the code of your project. All dependent packages are installed in the vendor / directory and it is from there that the code is taken when building with gb.
The best way to understand a tool is to use it.
First, create a new project, ~ / demoproject. The directory does not matter, even in / tmp. When working with gb, you can forget about your standard GOPATH in general.
The project tree here looks like this:
Run the build with the command:
gb build should generate an error stating that the dependency ( github.com/labstack/echo ) was not found:
If you look closely, you can see that gb is looking for a dependent package first in GOROOT (as the stdlib packages are there), then in GOPATH, which is defined for this project (demoproject / src), and, lastly, in demoproject / vendor . Since the package is nowhere to be found, we get an error. You can pull the package with your hands in the vendor (and do not forget to delete the folder. Git), but gb has a functional for this: gb vendor .
Check the project directory structure:
The manifest file contains all the information about the versions of dependencies:
$ cat vendor / manifest
Repeat gb vendor fetch for the remaining dependencies and now try to collect the code:
The binary is placed, as in the usual scenario of working in one GOPATH in the bin / directory.
Now the whole directory can be safely entered into the version control system, and here you, as the project owner, decide - you want to leave only versioning, or vendor dependencies too, or manage everything, and manage the versions manually. In your hands the freedom of choice.
In the case of versioning-only, any developer from your team, after having cloned the repository on his machine, should run:
to get a 1-in-1 dependency tree for building a project.
In general, this description should be enough to understand the principles of gb and the approach to solving the issue of versioning and vending.
The gb project is still in an early stage of development and acceptance by the community, but judging by the reaction of the latter and the stormy support, this is a very good decision.
The project has a lot of open issues in the tracker, but they quickly close and the project is actively developing.
For not yet great personal experience of use - there are still difficulties with cross-platform build. The rest is still normal flight.
Official website: getgb.io
Github repository: github.com/constabulary/gb
Blog post from the author: dave.cheney.net/2015/06/09/gb-a-project-based-build-tool-for-the-go-programming-language
Design rationale: getgb.io/rationale
Theory of operation: getgb.io/theory
(Credit orig.photo: Nathan Youngman)
First, let's see what all the fuss is about and why Go didn’t initially have an advanced dependency manager.
What is dependency management?
What is meant by the “advanced” dependency manager and what tasks it should solve. After all, but in Go, simple dependency management is present.
')
In Go, there is a standard “go get” command, which is enough to indicate the name of the project on, say, Github, so that the project code is installed, assembled, and ready for use. “Go get” always downloads the latest version, from the master branch, and the general consensus for open-source libraries is not to break backward compatibility at any cost. This is written in the FAQ, and in the vast majority of cases of third-party libraries, this rule is observed.
Depending on your previous experience, you may reject such an idea in the bud, or foresee that this scheme will not be viable very soon, but so far, after 3 years of Go 1 existence, this scheme works and is even used in production in large companies, although initially and more designed for an open-source ecosystem. I will say this - this is a question of the ratio of complexity and risks that developers are willing to take on.
But moving from the open-source world to the world of commercial development, immediately there is a need to ensure that the code of a specific revision will always be compiled and produce the same result, and not depend on external factors. External factors can be understood as new, breaking compatibility, changes in third-party libraries, and turning off the Internet or the fall of Github.
This problem is solved in two ways, which often go together - versioning dependencies ( versioning ) and vendoring ( vendoring ) - by including a third-party code in your code base. The first allows you to ensure that new commits to a third-party library do not break your build, and the second is that even when the Internet is down, all the code needed for building will be available for building locally.
Actually, this is the answer to the question of what advanced dependency management is - a tool that includes everything needed for versioning and vendoring dependencies.
Why is this not in Go?
The short answer is because it's damn hard to make everyone happy . Not that difficult - this is impossible if you want to make the most convenient tool suitable for all cases that happen in real practice.
The authors of Go initially recognized the complexity of the problem, and honestly refused to impose a solution on everyone, suggesting options. This question is well stated in the official FAQ and the answer is as follows:
Versioning is a source of considerable complexity, especially in large code bases, and we do not know of such a solution, which, on the scale of the whole variety of possible situations, would work well enough for everyone.
Fair enough, agree.
Further, the FAQ gives recommendations on how and what to do, and how Google solves this problem in its case (again, emphasizing that “what is suitable for us may not be suitable for you”). And if there is an “ideal solution”, then the community has all the cards on hand to create and offer it as standard for Go.
As practice has shown, there is no universal solution for everyone. But the zoo of solutions for different cases and the constant discussion of the issue led to a fairly good consensus on what would put an end to the controversy about advanced dependency management in Go.
A brief history of attempts to solve the problem.
The entire zoo of solutions comes down to utilities that solve the issue of either versioning, or vendoring, or both. The range of solutions is wide - from simple Makefile-c with the replacement of GOPATH and git-self-modules to rewriting of paths in imports and full-fledged all-in-one utilities, the most popular of which is Godep .
But all of them are in essence wrappers over the standard go get and go build commands. And it was this that was the source of the major inconsistencies. Workflow when working with go get / build works great, if you write an open-source project, everything is simple and convenient. But if you write a separate working project that doesn’t even intersect with your open-source activity at all, then all versioning / vendoring solutions become a thorn in the heel and deprive of certain amenities and add complexity. Everything becomes more confusing if you try to mix both scenarios.
The realization of these two main different development scenarios (open-source vs private closed project) led to the understanding that different scenarios require different solutions. And not so long ago, Dave Cheney (one of the Go-contributors and in general, a notable Go-popularizer) proposed to clearly separate these two scenarios, creating a separate tool for the second, which would be similar to the standard go get / build, but originally created for with a project-oriented source tree, with a clear separation between “own code” and “dependencies”.
gb - project-based build tool
The main theses underlying gb are:
- a separate utility replacing the standard go get / build / test
- everything is determined by the directory structure
- no special description files
- no code changes (including import rewrites)
- clear separation of code into “project” and “dependency”
If you were scared on the phrase “replaces the standard go get / build / test”, then this is a normal reaction :) In fact, the project for gb can be absolutely easy to use with the standard go build / test, but gb build will allow you not to bother on the way to delivered packages.
Now in order.
A separate utility replacing the standard go get / build / test
This is exactly the case, and you will have to put one more command in your working system. This is done simply:
go get github.com/constabulary/gb/... Everything is determined by the directory structure.
By “everything” is meant the fact that gb can work with this project. The rule here is simple - there should be a folder src / in the directory. And this is enough to start working with gb.
Often, when Go is used for only one project, the “GOPATH per project” approach is recommended - in fact, you are asked to change your GOPATH to the path to this project and work in it. gb implements something like this, just not touching your system GOPATH. Slightly more below.
No special description files.
Modern projects already consist of a dozen .dot files with various manifests and descriptions. Another such file needed to build a project would be too, and not in Go-style.
No code changes
The project code always remains the way it was written. There are no import overwrites in the import style of github.com/user/pkg -> import "vendor / github.com / user / pkg" 'is not and will not be.
Clear separation of code into “project” and “dependency”
Returning to the point about the directory structure, gb treats everything that is in src / as the code of your project. All dependent packages are installed in the vendor / directory and it is from there that the code is taken when building with gb.
Usage example
The best way to understand a tool is to use it.
First, create a new project, ~ / demoproject. The directory does not matter, even in / tmp. When working with gb, you can forget about your standard GOPATH in general.
mkdir ~/demoproject && cd !$ mkdir -p src/myserver cat > src/myserver/main.go <<END package main import ( "github.com/labstack/echo" "net/http" ) func hello(c *echo.Context) error { return c.String(http.StatusOK, "Hello, World!\n") } func main() { e := echo.New() e.Get("/", hello) e.Run(":1323") } END The project tree here looks like this:
$ tree $(pwd) /Users/user/demoproject └── src └── myserver └── main.go Run the build with the command:
$ gb build gb build should generate an error stating that the dependency ( github.com/labstack/echo ) was not found:
FATAL command "build" failed: failed to resolve import path "myserver": cannot find package "github.com/labstack/echo" in any of: /usr/local/go/src/github.com/labstack/echo (from $GOROOT) /Users/user/demoproject/src/github.com/labstack/echo (from $GOPATH) /Users/user/demoproject/vendor/src/github.com/labstack/echo If you look closely, you can see that gb is looking for a dependent package first in GOROOT (as the stdlib packages are there), then in GOPATH, which is defined for this project (demoproject / src), and, lastly, in demoproject / vendor . Since the package is nowhere to be found, we get an error. You can pull the package with your hands in the vendor (and do not forget to delete the folder. Git), but gb has a functional for this: gb vendor .
$ gb vendor fetch github.com/labstack/echo Cloning into '/var/folders/qp/6bvmky410dn8p1yhn3b19yxr0000gn/T/gb-vendor-097548747'... remote: Counting objects: 1531, done. remote: Compressing objects: 100% (45/45), done. remote: Total 1531 (delta 12), reused 0 (delta 0), pack-reused 1482 Receiving objects: 100% (1531/1531), 317.40 KiB | 191.00 KiB/s, done. Resolving deltas: 100% (911/911), done. Checking connectivity... done. Check the project directory structure:
$ tree -L 6 $(pwd) /Users/user/demoproject ├── src │ └── myserver │ └── main.go └── vendor ├── manifest └── src └── github.com └── labstack └── echo ├── LICENSE ├── README.md ├── context.go ├── context_test.go ├── echo.go ├── echo_test.go ├── examples ├── group.go ├── group_test.go ├── middleware ├── response.go ├── response_test.go ├── router.go ├── router_test.go └── website 10 directories, 14 files The manifest file contains all the information about the versions of dependencies:
$ cat vendor / manifest
{ "version": 0, "dependencies": [ { "importpath": "github.com/labstack/echo", "repository": "https://github.com/labstack/echo", "revision": "1ac5425ec48d1301d35a5b9a520326d8fca7e036", "branch": "master" } ] } Repeat gb vendor fetch for the remaining dependencies and now try to collect the code:
$ gb build github.com/bradfitz/http2/hpack github.com/labstack/gommon/color github.com/mattn/go-colorable golang.org/x/net/websocket github.com/bradfitz/http2 github.com/labstack/echo myserver $ tree bin/ bin/ └── myserver 0 directories, 1 file The binary is placed, as in the usual scenario of working in one GOPATH in the bin / directory.
Now the whole directory can be safely entered into the version control system, and here you, as the project owner, decide - you want to leave only versioning, or vendor dependencies too, or manage everything, and manage the versions manually. In your hands the freedom of choice.
- only versioning: add vendor / src to .gitignore
- vendoring only: add vendor / manifest to .gitignore
- and versioning and vendoring: leave .gitignore unchanged
In the case of versioning-only, any developer from your team, after having cloned the repository on his machine, should run:
$ gb vendor update -all to get a 1-in-1 dependency tree for building a project.
In general, this description should be enough to understand the principles of gb and the approach to solving the issue of versioning and vending.
Afterword
The gb project is still in an early stage of development and acceptance by the community, but judging by the reaction of the latter and the stormy support, this is a very good decision.
The project has a lot of open issues in the tracker, but they quickly close and the project is actively developing.
For not yet great personal experience of use - there are still difficulties with cross-platform build. The rest is still normal flight.
Links
Official website: getgb.io
Github repository: github.com/constabulary/gb
Blog post from the author: dave.cheney.net/2015/06/09/gb-a-project-based-build-tool-for-the-go-programming-language
Design rationale: getgb.io/rationale
Theory of operation: getgb.io/theory
Source: https://habr.com/ru/post/259967/
All Articles