Cross compilation in Go
Despite the fact that cross-platform has become virtually a standard attribute of almost all modern languages and libraries, to create a truly cross-platform product, it was still not easy. Compiled languages and related libraries required complex installation and configuration of the build environment and libraries, and interpretable required to have or deploy the necessary version of the interpreter. There are quite a few projects trying to make this process a bit simpler, but often the only solution left was to install a separate server and compile natively.
In Go, cross-platform has reached the level where, for the first time, you can safely refuse compile farms, specially configured dev-environments, virtual machines for building or chroot / docker-dev solutions. And this is another serious game-changer, about which I want to tell and show examples.
Go.
As you know, Go deliberately abandoned dynamic linking - for a number of reasons, the main one of which is very similar to the usual explanation for the design of almost every aspect of Go - “the advantages [of dynamic linking] are much less than its drawbacks and complexity that it brings to architecture”. Well, the main reason for the emergence of dynamic linking was the desire to save resources — primarily disk space and memory — which are now quite cheap, not only on servers, but also in embedded devices (copters, for example, are carried on board already by 1-2 GB RAM!). In general, to list the pros and cons of a separate linking method - it will pull on a separate post, so for now just accept as is - in Go we always have a static binary on the output.
')
At the moment, for the current version of Go 1.4.1, the following platforms are supported:
1 - kernels 2.6.23 and higher are officially supported, but in reality everything works on earlier kernels of the 2.6 branch - for example, quite a few people use Go on RHEL5 / CentOS5 with 2.6.18.
Go 1.5 is expected to support iOS.
It is also noteworthy that initially there was no Windows support in Go - the team was small, and there was no one to getdirty with implementing code for Windows, but due to the fact that the project was opened for open-source development, the port for Windows was very quickly written by third-party people and integrated in the official code base.
Although the processes described below will be absolutely identical for all platforms (except, perhaps, Android and Native Client (NaCl), which require extra gestures), later in the article it will be assumed by default that you use one of the three most popular desktop platforms - Linux, MacOS X or Windows. In addition, for simplicity, I will imply that we write and use exclusively Go code, without having to link to C libraries (and, thus, without having to use cgo / gcc). There is also a separate case - when you need to use a number of functions from the standard library tied to cgo, but I will write about this in a separate chapter at the end.
The first step that needs to be done is to build a toolchain for the desired platform.
Go to the Go source directory (it’s $ GOROOT / src, you always have it on the machine) and rebuild it to the required platform, say Windows / amd64:
The process takes about 26 seconds on Macbook Air 2012. The make.bash script is a standard Go build script with which you would install Go if you installed from source. He actually builds Go, and the entire standard library, only this time for the windows / amd64 platform.
Also, for the reason mentioned above, we have disabled CGO support.
Table of values of GOOS (if anyone knows how to make a table in Habré in 50% of width - tell me):
AND GOARCH:
Let's write a simple web server that is easier to write in Go than in some languages / libraries to parse the command line.
And build it for Windows 32- and 64-bit:
Checking:
I think it is not necessary to say that both binaries are ready to be copied to the target Windows system and will work.
I’ll say right away that I’m not working tightly with embedded devices, so I don’t know any details - so I’ll try not to go into this topic, but in general I’m following the situation from Go to embedded. Generally speaking, Go was not positioned as a language for embedded platforms, which, however, did not prevent people from actively starting to use it in this area. Perhaps the reason is that the embedded-industry made a leap forward, and now the "embedded" device no longer means a critically small amount of resources, and perhaps the compromises not in favor of saving memory in Go turned out to be much less tangible in practice, but the fact is Go has already created a lot of projects like Gobot (robotics-framework for a whole heap of platforms - from Arduino, Raspberry PI and Beaglebone Back to LeapMotion, Pebble and ArDrone) or EMBD (framework for working with hobby boards), and PayPal has been using for a couple of years Go in your beacon-device for wireless checks and payments .
For example, take the Nokia N9 (or N950, lucky one) - and collect the above example for it:
Just like that, yes.
For ARM platforms, in fact, you may still need to specify the GOARM flag, but here, if the default version is not suitable, the binary on the target platform will produce a clear message, like this:
It would seem that it may be easier to specify one variable before go build. But there are situations when the code needs to be collected and deployed to different platforms 100 times a day. For such tasks there are several projects to automate the process of preparing toolchains and, directly, assembling code for the required platform.
Link: github.com/mitchellh/gox
Installation and preparation of all possible toolchains at once:
Now, instead of “go build”, we write “gox”:
You can specify a specific package or a specific platform:
The remaining command line arguments are identical to go build. Intuitive enough.
GoCX is one of the most famous wrappers around cross-compilation features, but with an emphasis on packaging (it can even do .deb) and various buns for automated builds. I did not use it myself, therefore, for whom it is interesting, see the site and the documentation.
github.com/laher/goxc
If someone watched the video from the GopherCon 2014 conference, which was held last spring in Denver, you may remember Alan Shreve’s “Build Your Developer Tools in Go” speech - and one of the things he says quite categorically: “Do not use cross compile compile natively. " Next comes the explanation - the reason for the Cgo. If you don't need to use cgo, everything is ok. And in fact, a very small part of very specific code in Go needs third-party C-libraries. What is the problem?
The problem is that some functions of the standard library depend on cgo. Ie, if we collect Go with CGO_ENABLED = 0, they will simply not be available and at the compilation stage we will get an error. Despite the fact that there is a very convenient and beautiful workaround, let's see what exactly in the standard library depends on cgo.
Fortunately, this is easy to do:
Briefly go through these files:
Now more about the DNS resolver.
Each file from that list (which is compiled only for its platform thanks to // + build tags) contains the implementation of a single function cgoAddrInfoFlags (), which, in turn, is used in cgoLookupIP (), which is used in dnsclient_unix.go, in which we we find the goLookupIP () function, which serves as a fallback option in the absence of a cgo-enabled code, and immediately find an explanation:
goLookupIP is actually resolvable only by the Hosts file and the DNS protocol, which for most systems is approx. But there can be problems if the system uses non-standard name resolving methods. I believe that in 99% of cases, hosts and dns will be more than enough.
The bottom line is - if your code does not use C / C ++ - libraries through Cgo, and does not use the following two things:
then all the problems with Cgo can be scored .
The first part (from X.509) is actually not so rare. If I understand correctly, this code is needed, if your program uses standard net / http.StartAndListenTLS () - and you use real certificates that you really need to check.
Therefore, in brief about a simple workaround around this topic - it is called gonative , and does one simple thing - download the official golang binary versions from the official website of the required version for the desired platform, which already has compiled binaries of all standard packages and, in fact, completes the process “build toolchain with cgo-code ".
All you need to do is install it (go get github.com/inconshreveable/gonative) and execute one simple command:
And then use the standard cross-compilation procedures, as before, with pens or via gox / gocx.
Read more about gonative here: inconshreveable.com/04-30-2014/cross-compiling-golang-programs-with-native-libraries
Now about the main thing - the application in practice. I used in production so far only three schemes - “build on darwin / amd64 -> deploy on linux / 386”, “linux / amd64 -> linux / 386” and “linux / amd64 -> windows / amd64”. These are products that have been working for more than a year. The third case (deployed on windows) then took me completely by surprise - there was a server running successfully on Linux, and then suddenly it took to run it on Windows. And "that's urgently needed." Remembering the sleepless nights of experience with cross- - yes, there’s cross, just compiling Qt for deploying to Windows - the 60-second process “google it as done → build toolchain → recompile project → deploy to windows” - was just a shock, I even did not believe his eyes.
But this is where the next moment arises - once the cross-compilation and the deployment become so simple and fast, there is an incentive for all the dependencies on the files — be it configs, certificates or anything else — to be embedded in the binary too. However, this is a fairly simple task, even for third-party libraries, thanks to the effective use of the io.Reader interface and the go-bindata package , but this is already a topic for a separate article.
I hope I missed nothing of the main thing.
But overall, this is actually a very significant difference with all previous cross-build experiences. To be honest, I'm still not used to this change. No more virtual machines with configured dev-environment, no need for docker-images to build - yes in general dev-environment disappears as such. This is a too sharp game changer to get used to so quickly.
dave.cheney.net/2012/09/08/an-introduction-to-cross-compilation-with-go
blog.hashbangbash.com/2014/04/linking-golang-statically
www.limitlessfx.com/cross-compile-golang-app-for-windows-from-linux.html
In Go, cross-platform has reached the level where, for the first time, you can safely refuse compile farms, specially configured dev-environments, virtual machines for building or chroot / docker-dev solutions. And this is another serious game-changer, about which I want to tell and show examples.
Go.
As you know, Go deliberately abandoned dynamic linking - for a number of reasons, the main one of which is very similar to the usual explanation for the design of almost every aspect of Go - “the advantages [of dynamic linking] are much less than its drawbacks and complexity that it brings to architecture”. Well, the main reason for the emergence of dynamic linking was the desire to save resources — primarily disk space and memory — which are now quite cheap, not only on servers, but also in embedded devices (copters, for example, are carried on board already by 1-2 GB RAM!). In general, to list the pros and cons of a separate linking method - it will pull on a separate post, so for now just accept as is - in Go we always have a static binary on the output.
')
At the moment, for the current version of Go 1.4.1, the following platforms are supported:
- Linux 2.6 1 and up - amd64, 386, arm
- MacOS X 10.6 and higher - amd64, 386
- Windows XP and above - amd64, 386
- FreeBSD 8 and above - amd64, 386, arm
- NetBSD - amd64, 386, arm
- OpenBSD - amd64, 386
- DragonFly BSD - amd64, 386
- Plan 9 - amd64, 386
- Google Native Client - amd64p32, 386
- Android - arm
1 - kernels 2.6.23 and higher are officially supported, but in reality everything works on earlier kernels of the 2.6 branch - for example, quite a few people use Go on RHEL5 / CentOS5 with 2.6.18.
Go 1.5 is expected to support iOS.
It is also noteworthy that initially there was no Windows support in Go - the team was small, and there was no one to get
Although the processes described below will be absolutely identical for all platforms (except, perhaps, Android and Native Client (NaCl), which require extra gestures), later in the article it will be assumed by default that you use one of the three most popular desktop platforms - Linux, MacOS X or Windows. In addition, for simplicity, I will imply that we write and use exclusively Go code, without having to link to C libraries (and, thus, without having to use cgo / gcc). There is also a separate case - when you need to use a number of functions from the standard library tied to cgo, but I will write about this in a separate chapter at the end.
Toolchain preparation
The first step that needs to be done is to build a toolchain for the desired platform.
Go to the Go source directory (it’s $ GOROOT / src, you always have it on the machine) and rebuild it to the required platform, say Windows / amd64:
cd $(go env GOROOT)/src sudo GOOS=windows GOARCH=amd64 CGO_ENABLED=0 ./make.bash --no-clean The process takes about 26 seconds on Macbook Air 2012. The make.bash script is a standard Go build script with which you would install Go if you installed from source. He actually builds Go, and the entire standard library, only this time for the windows / amd64 platform.
Also, for the reason mentioned above, we have disabled CGO support.
GOOS and GOARCH values
Table of values of GOOS (if anyone knows how to make a table in Habré in 50% of width - tell me):
| OS | $ GOOS |
|---|---|
| Linux | linux |
| MacOS X | darwin |
| Windows | windows |
| Freebsd | freebsd |
| Netbsd | netbsd |
| Openbsd | openbsd |
| DragonFly BSD | dragonfly |
| Plan 9 | plan9 |
| Native client | nacl |
| Android | android |
AND GOARCH:
| Architecture | $ GOARCH |
|---|---|
| x386 | 386 |
| AMD64 | amd64 |
| AMD64 with 32 pointers | amd64p32 |
| ARM | arm |
Example 1. A web server written and compiled in Linux for Windows
Let's write a simple web server that is easier to write in Go than in some languages / libraries to parse the command line.
package main import ( "log" "net/http" ) func Handler(w http.ResponseWriter, r *http.Request) { w.Write([]byte("Hello, world\n")) } func main() { http.HandleFunc("/", Handler) log.Println("Starting HTTP server on :1234") log.Fatal(http.ListenAndServe(":1234", nil)) } And build it for Windows 32- and 64-bit:
GOOS=windows GOARCH=386 go build -o http_example.exe GOOS=windows GOARCH=amd64 go build -o http_example64.exe Checking:
$ file http_example*.exe http_example.exe: PE32 executable for MS Windows (console) Intel 80386 32-bit http_example64.exe: PE32+ executable for MS Windows (console) Mono/.Net assembly I think it is not necessary to say that both binaries are ready to be copied to the target Windows system and will work.
Example 2. Cross-compiling for ARM for a Nokia N9 phone
I’ll say right away that I’m not working tightly with embedded devices, so I don’t know any details - so I’ll try not to go into this topic, but in general I’m following the situation from Go to embedded. Generally speaking, Go was not positioned as a language for embedded platforms, which, however, did not prevent people from actively starting to use it in this area. Perhaps the reason is that the embedded-industry made a leap forward, and now the "embedded" device no longer means a critically small amount of resources, and perhaps the compromises not in favor of saving memory in Go turned out to be much less tangible in practice, but the fact is Go has already created a lot of projects like Gobot (robotics-framework for a whole heap of platforms - from Arduino, Raspberry PI and Beaglebone Back to LeapMotion, Pebble and ArDrone) or EMBD (framework for working with hobby boards), and PayPal has been using for a couple of years Go in your beacon-device for wireless checks and payments .
For example, take the Nokia N9 (or N950, lucky one) - and collect the above example for it:
GOOS=linux GOARCH=arm go build -o http_example_arm scp http_example_arm developer@192.168.2.16:/home/user/ Just like that, yes.
For ARM platforms, in fact, you may still need to specify the GOARM flag, but here, if the default version is not suitable, the binary on the target platform will produce a clear message, like this:
runtime: this CPU has no floating point hardware, so it cannot run this GOARM=7 binary. Recompile using GOARM=5. Automating the process
It would seem that it may be easier to specify one variable before go build. But there are situations when the code needs to be collected and deployed to different platforms 100 times a day. For such tasks there are several projects to automate the process of preparing toolchains and, directly, assembling code for the required platform.
Gox
Link: github.com/mitchellh/gox
Installation and preparation of all possible toolchains at once:
go get github.com/mitchellh/gox gox -build-toolchain ... Now, instead of “go build”, we write “gox”:
$ gox Number of parallel builds: 4 --> darwin/386: github.com/mitchellh/gox --> darwin/amd64: github.com/mitchellh/gox --> linux/386: github.com/mitchellh/gox --> linux/amd64: github.com/mitchellh/gox --> linux/arm: github.com/mitchellh/gox --> freebsd/386: github.com/mitchellh/gox --> freebsd/amd64: github.com/mitchellh/gox --> openbsd/386: github.com/mitchellh/gox --> openbsd/amd64: github.com/mitchellh/gox --> windows/386: github.com/mitchellh/gox --> windows/amd64: github.com/mitchellh/gox --> freebsd/arm: github.com/mitchellh/gox --> netbsd/386: github.com/mitchellh/gox --> netbsd/amd64: github.com/mitchellh/gox --> netbsd/arm: github.com/mitchellh/gox --> plan9/386: github.com/mitchellh/gox You can specify a specific package or a specific platform:
gox -os="linux" gox -osarch="linux/amd64" gox github.com/divan/gorilla-xmlrpc/xml The remaining command line arguments are identical to go build. Intuitive enough.
Gocx
GoCX is one of the most famous wrappers around cross-compilation features, but with an emphasis on packaging (it can even do .deb) and various buns for automated builds. I did not use it myself, therefore, for whom it is interesting, see the site and the documentation.
github.com/laher/goxc
Understanding CGO
If someone watched the video from the GopherCon 2014 conference, which was held last spring in Denver, you may remember Alan Shreve’s “Build Your Developer Tools in Go” speech - and one of the things he says quite categorically: “Do not use cross compile compile natively. " Next comes the explanation - the reason for the Cgo. If you don't need to use cgo, everything is ok. And in fact, a very small part of very specific code in Go needs third-party C-libraries. What is the problem?
The problem is that some functions of the standard library depend on cgo. Ie, if we collect Go with CGO_ENABLED = 0, they will simply not be available and at the compilation stage we will get an error. Despite the fact that there is a very convenient and beautiful workaround, let's see what exactly in the standard library depends on cgo.
Fortunately, this is easy to do:
# cd $(go env GOROOT)/src/ # grep -re "^// +build.*[^\!]cgo" * crypto/x509/root_cgo_darwin.go:// +build cgo net/cgo_android.go:// +build cgo,!netgo net/cgo_linux.go:// +build !android,cgo,!netgo net/cgo_netbsd.go:// +build cgo,!netgo net/cgo_openbsd.go:// +build cgo,!netgo net/cgo_unix_test.go:// +build cgo,!netgo os/user/lookup_unix.go:// +build cgo runtime/crash_cgo_test.go:// +build cgo Briefly go through these files:
- crypto / x509 / root_cgo_darwin.go - implements one function to get root X.509 certificates in MacOS X. If you don’t use this feature explicitly - no problem, without cgo everything will work for you.
- net / cgo_android / linux / netbsd / openbsd / cgo_unix_test.go is the code required to use the native DNS resolver in different unixes. Just below the details.
- os / user / lookup_unix.go - functions from the os / user package - to get information about the current user (uid, gid, username). Used by getpwuid_r () to read passwd entries
- runtime / crash_cgo_test.go - test file for handling kreshy, nothing relevant
Now more about the DNS resolver.
Each file from that list (which is compiled only for its platform thanks to // + build tags) contains the implementation of a single function cgoAddrInfoFlags (), which, in turn, is used in cgoLookupIP (), which is used in dnsclient_unix.go, in which we we find the goLookupIP () function, which serves as a fallback option in the absence of a cgo-enabled code, and immediately find an explanation:
// goLookupIP is the native Go implementation of LookupIP.
// Used only if cgoLookupIP refuses to handle the request
// (that is, only if cgoLookupIP is the stub in cgo_stub.go).
// Normally we let the C library resolver instead of
// depending on our lookup code
// answers.
goLookupIP is actually resolvable only by the Hosts file and the DNS protocol, which for most systems is approx. But there can be problems if the system uses non-standard name resolving methods. I believe that in 99% of cases, hosts and dns will be more than enough.
The bottom line is - if your code does not use C / C ++ - libraries through Cgo, and does not use the following two things:
- x.509 certificate validation that should work on MacOS X
- guaranteed to receive system information about the current user
then all the problems with Cgo can be scored .
The first part (from X.509) is actually not so rare. If I understand correctly, this code is needed, if your program uses standard net / http.StartAndListenTLS () - and you use real certificates that you really need to check.
Therefore, in brief about a simple workaround around this topic - it is called gonative , and does one simple thing - download the official golang binary versions from the official website of the required version for the desired platform, which already has compiled binaries of all standard packages and, in fact, completes the process “build toolchain with cgo-code ".
All you need to do is install it (go get github.com/inconshreveable/gonative) and execute one simple command:
gonative And then use the standard cross-compilation procedures, as before, with pens or via gox / gocx.
Read more about gonative here: inconshreveable.com/04-30-2014/cross-compiling-golang-programs-with-native-libraries
Practical use
Now about the main thing - the application in practice. I used in production so far only three schemes - “build on darwin / amd64 -> deploy on linux / 386”, “linux / amd64 -> linux / 386” and “linux / amd64 -> windows / amd64”. These are products that have been working for more than a year. The third case (deployed on windows) then took me completely by surprise - there was a server running successfully on Linux, and then suddenly it took to run it on Windows. And "that's urgently needed." Remembering the sleepless nights of experience with cross- - yes, there’s cross, just compiling Qt for deploying to Windows - the 60-second process “google it as done → build toolchain → recompile project → deploy to windows” - was just a shock, I even did not believe his eyes.
But this is where the next moment arises - once the cross-compilation and the deployment become so simple and fast, there is an incentive for all the dependencies on the files — be it configs, certificates or anything else — to be embedded in the binary too. However, this is a fairly simple task, even for third-party libraries, thanks to the effective use of the io.Reader interface and the go-bindata package , but this is already a topic for a separate article.
I hope I missed nothing of the main thing.
But overall, this is actually a very significant difference with all previous cross-build experiences. To be honest, I'm still not used to this change. No more virtual machines with configured dev-environment, no need for docker-images to build - yes in general dev-environment disappears as such. This is a too sharp game changer to get used to so quickly.
Links
dave.cheney.net/2012/09/08/an-introduction-to-cross-compilation-with-go
blog.hashbangbash.com/2014/04/linking-golang-statically
www.limitlessfx.com/cross-compile-golang-app-for-windows-from-linux.html
Source: https://habr.com/ru/post/249449/
All Articles