Code Generation in Go
Translation of the article by Rob Pike from the official Go blog on automatic code generation using go generate. The article is a bit outdated (it was written before the release of Go 1.4, in which go generate appeared), but it explains go generate well.
One of the properties of the theory of computability - Turing completeness - is that a program can write another program. This is a powerful idea that is not as appreciated as it deserves, although it occurs quite often. This is quite a significant part of the definition of what compilers do, for example. Also, the go test command also works on the same principle: it scans the packages that need to be tested, creates a new Go program in which the necessary body kit for the tests is added, then compiles and runs it. Modern computers are so fast that such a seemingly expensive sequence of actions is executed in a split second.
There are plenty of other examples where programs write programs. Yacc , for example, reads a grammar description and produces a program that parses this grammar. The “compiler” Protocol Buffers reads the interface description and provides definitions for structures, methods, and other code. A variety of configuration utilities work in a similar way, extracting metadata from the environment and creating custom launch commands.
Thus, programs that write programs are an important element in software development, but programs like Yacc that create source code must be integrated into the build process so that their output can be passed to the compiler. When an external build system is used, like Make, this is usually easy to do. But in Go, in which the go utility gets all the necessary information about the build from the source code, this is a problem. It simply has no mechanism to start Yacc with the go tool.
')
Up to this point, in a sense.
The latest release of Go, 1.4, includes a new command, go generate, which allows you to run these utilities. It's called go generate , and when it starts, it scans the code for special comments that indicate which commands to run. It is important to understand that go generate is not part of go build . It does not analyze dependencies and must be run before go build. It is intended for the author of the Go package, and not for its users.
The go generate command is very easy to use. To warm up, here’s how to use it to generate a Yacc grammar. Let's say you have an input Yacc file, called gopher.y, that defines the grammar of your new language. To generate Go code to parse this grammar, you would usually run the standard go version of yacc, like this:
The -o option here specifies the name of the resulting file, and -p - the name of the package.
To shift this process to go generate, you need to add a comment in any ordinary (non-autogenerated) .go file in this directory:
This text is the same command, but with a comment added at the beginning that go generate recognizes. The comment should begin at the beginning of the line and have no spaces between // and go: generate. After this marker, the remainder indicates which go generate command should run.
Now run it. Go to the source directory and run go generate, then go build and so on:
And that's all you need. If there are no errors, then go generate will call yacc, which will create gopher.go, at this point the directory will contain all the necessary go-files that we can collect, test and work with them normally. Every time gopher.y changes, just restart go generate to re-create the parser.
If you are interested in more details on how go generate works inside, including parameters, environment variables, and so on, see the design document .
Go generate does nothing that could not be done using Make or another build mechanism, but it comes out of the box in the go command — you don’t need to install anything further — and it fits well with the Go ecosystem. Most importantly, remember that this is for the authors of the package, not for users, at least for the reason that the program that will be called may be absent on the user's machine. Also, if the package is supposed to be used with go get, do not forget to transfer the generated files to the version control system, making them available to users.
Now, let's see how you can use this for something new. As a radically different example where go generate can help, there is a new stringer program in golang.org/x/tools repository. It automatically generates String () string methods for sets of numeric constants. It is not included in the standard Go set, but it is easy to install:
Here is an example from the stringer documentation. Imagine that we have some code, with a set of numeric constants defining different types of drugs:
For debugging purposes, we would like these constants to give their name nicely, in other words, we want a method with the following signature:
It is easy to write it manually, for example, something like this:
There are several ways to write this function, of course. We can use a slice of lines indexed by Pill, or a map, or some other technique. One way or another, we must maintain it every time we change a set of medications, and we must verify that the code is correct. (Two different names for paracetamol, for example, make this code a bit more tricky than it could be). Plus, the very question of choosing a method of implementation depends on the types of values: signed or unsigned, dense and scattered, starting from zero or not, and so on.
The stringer program takes care of this. Although it can be started manually, it is intended to run through go generate. To use it, add a comment to the source, most likely in code with a type definition:
This rule indicates that go generate must run the stringer command to generate the String method for the Pill type. The output will automatically be written to the pill_string.go file (the output can be redefined using the -output flag).
Let's run it:
Every time we change the definition of Pill or constants, all we have to do is run
to update the String method. And of course, if we have several types in one package that need to be updated, go generate will update them all.
It goes without saying that the generated code is ugly. This is OK, however, since people will not work with this code; auto-generated code is often ugly. He tries to be as effective as possible. All names are combined together in one line, which saves memory (just one line for all names, even there are a myriad of them). Then the array, _Pill_index, finds a type matching with the name using a simple and very efficient technique. Notice that _Pill_index is an array (not a slice; one header is smaller) of values ​​of type uint8, the smallest possible integer type capable of containing the desired values. If there are more values, or are negative, the type of the generated _Pill_index array may change to uint16 or int8, depending on what works best.
The approach used in the methods generated by the stringer varies, depending on the properties of the set of constants. For example, if the constants are discharged, it can use map. Here is a simple example based on a set of constants representing powers of two:
In summary, the automatic generation of the method allows us to solve the problem better than a human would.
Go source has a ton of other examples of using go generate. This includes generating Unicode tables in the unicode package, creating efficient methods for encoding and decoding arrays in encoding / gob, creating a timezone data set in the time package, and the like.
Please use go generate creatively. He is here to encourage experimentation.
And even if not, use stringer to add String methods to your numeric constants. Let the computer do the work for you.
One of the properties of the theory of computability - Turing completeness - is that a program can write another program. This is a powerful idea that is not as appreciated as it deserves, although it occurs quite often. This is quite a significant part of the definition of what compilers do, for example. Also, the go test command also works on the same principle: it scans the packages that need to be tested, creates a new Go program in which the necessary body kit for the tests is added, then compiles and runs it. Modern computers are so fast that such a seemingly expensive sequence of actions is executed in a split second.
There are plenty of other examples where programs write programs. Yacc , for example, reads a grammar description and produces a program that parses this grammar. The “compiler” Protocol Buffers reads the interface description and provides definitions for structures, methods, and other code. A variety of configuration utilities work in a similar way, extracting metadata from the environment and creating custom launch commands.
Thus, programs that write programs are an important element in software development, but programs like Yacc that create source code must be integrated into the build process so that their output can be passed to the compiler. When an external build system is used, like Make, this is usually easy to do. But in Go, in which the go utility gets all the necessary information about the build from the source code, this is a problem. It simply has no mechanism to start Yacc with the go tool.
')
Up to this point, in a sense.
The latest release of Go, 1.4, includes a new command, go generate, which allows you to run these utilities. It's called go generate , and when it starts, it scans the code for special comments that indicate which commands to run. It is important to understand that go generate is not part of go build . It does not analyze dependencies and must be run before go build. It is intended for the author of the Go package, and not for its users.
The go generate command is very easy to use. To warm up, here’s how to use it to generate a Yacc grammar. Let's say you have an input Yacc file, called gopher.y, that defines the grammar of your new language. To generate Go code to parse this grammar, you would usually run the standard go version of yacc, like this:
go tool yacc -o gopher.go -p parser gopher.y The -o option here specifies the name of the resulting file, and -p - the name of the package.
To shift this process to go generate, you need to add a comment in any ordinary (non-autogenerated) .go file in this directory:
//go:generate go tool yacc -o gopher.go -p parser gopher.yThis text is the same command, but with a comment added at the beginning that go generate recognizes. The comment should begin at the beginning of the line and have no spaces between // and go: generate. After this marker, the remainder indicates which go generate command should run.
Now run it. Go to the source directory and run go generate, then go build and so on:
$ cd $GOPATH/myrepo/gopher $ go generate $ go build $ go test And that's all you need. If there are no errors, then go generate will call yacc, which will create gopher.go, at this point the directory will contain all the necessary go-files that we can collect, test and work with them normally. Every time gopher.y changes, just restart go generate to re-create the parser.
If you are interested in more details on how go generate works inside, including parameters, environment variables, and so on, see the design document .
Go generate does nothing that could not be done using Make or another build mechanism, but it comes out of the box in the go command — you don’t need to install anything further — and it fits well with the Go ecosystem. Most importantly, remember that this is for the authors of the package, not for users, at least for the reason that the program that will be called may be absent on the user's machine. Also, if the package is supposed to be used with go get, do not forget to transfer the generated files to the version control system, making them available to users.
Now, let's see how you can use this for something new. As a radically different example where go generate can help, there is a new stringer program in golang.org/x/tools repository. It automatically generates String () string methods for sets of numeric constants. It is not included in the standard Go set, but it is easy to install:
$ go get golang.org/x/tools/cmd/stringer Here is an example from the stringer documentation. Imagine that we have some code, with a set of numeric constants defining different types of drugs:
package painkiller type Pill int const ( Placebo Pill = iota Aspirin Ibuprofen Paracetamol Acetaminophen = Paracetamol ) For debugging purposes, we would like these constants to give their name nicely, in other words, we want a method with the following signature:
func (p Pill) String() string It is easy to write it manually, for example, something like this:
func (p Pill) String() string { switch p { case Placebo: return "Placebo" case Aspirin: return "Aspirin" case Ibuprofen: return "Ibuprofen" case Paracetamol: // == Acetaminophen return "Paracetamol" } return fmt.Sprintf("Pill(%d)", p) } There are several ways to write this function, of course. We can use a slice of lines indexed by Pill, or a map, or some other technique. One way or another, we must maintain it every time we change a set of medications, and we must verify that the code is correct. (Two different names for paracetamol, for example, make this code a bit more tricky than it could be). Plus, the very question of choosing a method of implementation depends on the types of values: signed or unsigned, dense and scattered, starting from zero or not, and so on.
The stringer program takes care of this. Although it can be started manually, it is intended to run through go generate. To use it, add a comment to the source, most likely in code with a type definition:
//go:generate stringer -type=PillThis rule indicates that go generate must run the stringer command to generate the String method for the Pill type. The output will automatically be written to the pill_string.go file (the output can be redefined using the -output flag).
Let's run it:
$ go generate $ cat pill_string.go // generated by stringer -type Pill pill.go; DO NOT EDIT package pill import "fmt" const _Pill_name = "PlaceboAspirinIbuprofenParacetamol" var _Pill_index = [...]uint8{0, 7, 14, 23, 34} func (i Pill) String() string { if i < 0 || i+1 >= Pill(len(_Pill_index)) { return fmt.Sprintf("Pill(%d)", i) } return _Pill_name[_Pill_index[i]:_Pill_index[i+1]] } $ Every time we change the definition of Pill or constants, all we have to do is run
$ go generate to update the String method. And of course, if we have several types in one package that need to be updated, go generate will update them all.
It goes without saying that the generated code is ugly. This is OK, however, since people will not work with this code; auto-generated code is often ugly. He tries to be as effective as possible. All names are combined together in one line, which saves memory (just one line for all names, even there are a myriad of them). Then the array, _Pill_index, finds a type matching with the name using a simple and very efficient technique. Notice that _Pill_index is an array (not a slice; one header is smaller) of values ​​of type uint8, the smallest possible integer type capable of containing the desired values. If there are more values, or are negative, the type of the generated _Pill_index array may change to uint16 or int8, depending on what works best.
The approach used in the methods generated by the stringer varies, depending on the properties of the set of constants. For example, if the constants are discharged, it can use map. Here is a simple example based on a set of constants representing powers of two:
const _Power_name = "p0p1p2p3p4p5..." var _Power_map = map[Power]string{ 1: _Power_name[0:2], 2: _Power_name[2:4], 4: _Power_name[4:6], 8: _Power_name[6:8], 16: _Power_name[8:10], 32: _Power_name[10:12], ..., } func (i Power) String() string { if str, ok := _Power_map[i]; ok { return str } return fmt.Sprintf("Power(%d)", i) } In summary, the automatic generation of the method allows us to solve the problem better than a human would.
Go source has a ton of other examples of using go generate. This includes generating Unicode tables in the unicode package, creating efficient methods for encoding and decoding arrays in encoding / gob, creating a timezone data set in the time package, and the like.
Please use go generate creatively. He is here to encourage experimentation.
And even if not, use stringer to add String methods to your numeric constants. Let the computer do the work for you.
Source: https://habr.com/ru/post/269887/
All Articles