Translation of the blog post and the Russ Cox report from GopherCon 2017, with an appeal to the entire Go community to help with discussion and planning of Go 2. The video of the report will be added immediately after publication.

On September 25, 2007, after Rob Pike, Robert Grismier, and Ken Thompson had been discussing the idea of ​​creating a new language for several days, Rob suggested the name "Go."

The following year, Ian Lance Taylor and I joined the team and the five of us created two compilers and a standard library that were publicly opened on November 10, 2009.

In the next two years, with the help of the new open-source community of gophers, we experimented and tried various ideas, improving Go and leading it to the planned release of Go 1 , proposed on October 5, 2011.

With even more active help from the Go community, we revised and implemented this plan, eventually releasing Go 1 on March 28, 2012.

The Go 1 release marked the culmination of almost five full years of creative and frantic efforts that led us from choosing a name and discussing ideas to a stable, ready-made language. He also marked a clear transition from change and inconstancy to stability.

In the years preceding Go 1, we changed our language by breaking other people's Go programs almost weekly. We understood that this kept Go from being used in production, where no one would rewrite programs every week, synchronizing with changes in the language. As it is written in a blog post with the announcement of Go 1 , the main motivation of the language was to provide a stable foundation for creating reliable products, projects and publications (blogs, tutorials, reports and books), giving users the confidence that their programs will be compiled and work without having to change them even after many years.

After Go 1 was released, we knew that we had to spend some time in actual use of Go in the production environment for which it was created. We obviously moved from language changes to using Go in our projects and improving the implementation: we ported Go to many new systems, we rewrote almost all the parts that are critical to performance to make Go even more efficient and add key tools like a race detector .

Today we have 5 years of real experience in using Go to create huge, high-quality production systems. It gave us a sense of what works and what doesn't. And now is the time to start a new stage in the evolution and development of Go. Today, I ask all of you, the Go developer community, whether you are here in the GopherCon room or watching a video or reading this in the Go blog, work with us as we plan and implement Go 2.

Next, I will tell and explain the tasks that go before Go 2; our limitations and barriers; the process itself as a whole; the importance of describing your experience with Go, especially if it relates to problems that we can try to solve; possible solutions; how we will implement Go 2 and how you can all help with this.

Tasks

The tasks for Go today are exactly the same as they were in 2007. We want to make programmers more efficient in managing two types of scalability: systems scalability, especially multi-threaded (concurrent) systems that interact with many other servers — widely represented as servers for the cloud, and development scalability, especially large code bases that many programmers work on , often remotely - like, for example, a modern open-source development model.

These types of scalability are present today in companies of all sizes. Even a startup of 5 people can use large cloud-based API services provided by other companies and use more open-source software than software they write themselves. Scalability of systems and scalability of development are also relevant for a startup, as well as for Google.

Our goal for Go 2 is to fix the major flaws in Go that interfere with scalability.

(If you want to learn more about these tasks, see Rob Pike's 2012 article “ Go at Google: Language Design in the Service of Software Engineering ” and my report from GopherCon 2015 “ Go, Open Source, Community ”.)

Obstacles

Our tasks for Go have not changed, but obstacles have changed. The main one is the existing use of Go. According to our estimates, there are now at least half a million Go programmers in the world, which means about a million Go source files and at least a billion lines of Go code. These programmers and this source code represent the success of Go, but at the same time are the main obstacle for Go 2.

Go 2 should promote all these developers. We must ask them to forget their old habits and learn new ones only if the benefits of this are really worth it. For example, before Go 1, the error interface type method was called a String . In Go 1, we renamed it Error to distinguish types for errors from other types, which a formatted string representation can simply have. Once I implemented a type that satisfies the error interface, and without thinking, I called the method String instead of Error , which, of course, did not compile. Even after 5 years, I still have not completely forgotten the old way. This example of clarifying renaming was an important and useful change for Go 1, but would be too destructive for Go 2 without a really good reason.

Go 2 should also be good friends with existing Go 1 code. We should not split Go ecosystem. Mixed programs in which packages are written in Go 2 and import packages in Go 1 or vice versa, should work seamlessly during a transition period of several years. We have yet to figure out exactly how to achieve this; automatic fix and static analysis tools like go fix will definitely play a role here.

To reduce the destructive effect, every change will require very careful thinking and planning, as well as a toolkit, which, as a result, will limit the number of changes we can do at all. Maybe we can do two or three, but definitely not more than five.

At the same time, I do not consider minor auxiliary changes, like, perhaps, resolving identifiers in more natural languages ​​or adding literals for numbers in binary form. Such minor changes are also important, but they are much easier to do right. Today, I will focus on possible major changes, such as additional support for error handling or the addition of immutable or read-only values, or the addition of some form of generics, or some other important yet voiced theme. We can only make a few such major changes. And we will have to choose them very carefully.

Process

This raises an important question. What is the process of developing Go in general?

In the early days of Go, when there were only five of us, we worked in a couple of adjacent offices separated by a glass wall. It was very easy to gather everyone in the same room, discuss a problem, return to their seats and immediately implement a solution. If there was any difficulty during implementation, it was easy to get together and discuss again. In the office of Rob and Robert there was a small sofa and a whiteboard, and usually one of us would drop by and start writing an example on the blackboard. As a rule, by the time the example was written, everyone else found a moment at which it was possible to pause the current task, and were ready to sit down and discuss the code. Such an informal approach, of course, cannot be scaled to the size of the Go community today.

Part of our work after the release of Go in open-source was porting this informal process into a more formal world of mailing lists and task trackers for half a million users, but it seems to me that we never explicitly told how the whole process works. Perhaps we never even fully consciously thought about it. Looking back, however, I think that the basic plan of the process that Go has followed since its inception looks like this:

The first step is to use Go to gain experience with it.

The second step is to identify the problem in Go, which probably needs a solution and express it, explain it to others, present it in writing.

The third step is to propose a solution to the problem, discuss it with others and reconsider the solution based on this discussion.

The fourth step is to implement the solution, test it and improve it based on the results of the verification.

And finally, the fifth step is to implement the solution by adding it to the language or standard library or to the toolkit that people use every day.

The same person does not have to take all these steps by himself. In fact, there are usually many different people involved at every step and many solutions can be proposed for the same problem. Also, at each stage we can decide not to go ahead and go back a step.

And although I do not think that we have ever talked about this process entirely, but we explained it in parts. In 2012, when we released Go 1 and said it was time to start using Go and stop changing, we explained the first step. In 2015, when we introduced changes to the proposal process for Go, we explained steps 3, 4, and 5. But we never explained the second step in detail, and I would like to do it now.

(For more information on the development of Go 1 and on the termination of changes in the language, see the report of Rob Pike and Andrew Gerrand on OSCON in 2012 “ The Path to Go 1. ”. For more details on the process of proposals can be found in the report of Andrew Gerrand on GopherCon in 2015 “ How Go was Made ” and in the documentation for the process itself )

Problem explanation

The explanation of the problem consists of two parts. The first part, the easy one, is simply to voice what the problem actually is. We, the developers, are generally able to do this quite well. In the end, every test we write is a statement of a problem that needs to be solved, and written in such an exact language that even a computer will understand. The second part — the difficult one — is to describe the importance of the problem well enough for everyone else to understand why we should spend time solving it and supporting it. Unlike the exact formulation of the problem, we do not often describe their importance and we are not very good at it. The computer will never ask us “Why is this test case important? Are you sure that this is exactly the problem that you have to solve? Is the solution to this problem exactly the most important task you need to do? ” Perhaps one day it will be, but certainly not today.

Let's take a look at the old example from 2011. Here is what I wrote about renaming os.Error into error.Value , when we planned Go 1.

error.Value
(rsc) The problem we have in low-level libraries is that everything depends on “os” because of os.Error, so it’s difficult to do things that the os package itself could use (as an example with time.Nano below) . If it were not for os.Error, there would not be so many other packages that depend on the os package. Especially computational packages like hash / * or strconv or strings or bytes could do without it, for example. I plan to investigate (without offering anything so far) determine the error package with something like this API:

package error
type Value interface {String () string}
func New (s string) Value

It starts with a brief one-line statement of the problem: in low-level libraries, everything imports “os” for the sake of os.Error . Further, there are 5 lines that I highlighted, describing the significance of the problem: packages that “os” uses cannot use the error type in their API, and other packages depend on os for reasons not related to the operating system.

Will these 5 lines convince you that the problem is worth attention? It depends on how well you can fill the context that I left out of the box: to be understood, you need to be able to predict what other people know. For my audience at the time — ten other people on the Google team working on Go who read this document — those 50 words were enough. To present the same problem to the audience at the GothamGo conference last fall — an audience with much more diverse experiences — I have to provide more context and I have already used 200 words, plus examples of real code and a diagram. And it is a fact that the modern Go community, which is trying to explain the importance of any problem, should add context, illustrated by specific examples, which could be excluded in a conversation with your colleagues, for example.

To convince others that the problem is really important is a key step. If the problem seems not so important, then almost any solution looks too expensive. But for a really important problem, there are almost always some not so expensive solutions. When we disagree about whether or not to make a decision, it usually means that we disagree on the importance of the problem being solved. This is such an important point that I want to show two recent examples, well illustrating it, at least in retrospect.

Example: leap seconds

My first example is about time.

Imagine that you want to measure how long an event takes. You first memorize the start time, trigger the event, record the end time, and then subtract the start time from the end time. If the event took 10 milliseconds, the subtraction operation will return you exactly 10 milliseconds, possibly plus or minus a small measurement error.

  start := time.Now() // 3:04:05.000 event() end := time.Now() // 3:04:05.010 elapsed := end.Sub(start) // 10 ms 

This obvious procedure may not work during a “ leap second ” ( leap second ). When our watches are not exactly synchronized with the Earth's day rotation, a special leap second - officially it is 23:59 and 60 seconds - is inserted right before midnight. Unlike a leap year, leap seconds have no easily predictable pattern, which makes it difficult to automate their accounting in programs and APIs. Instead of introducing a special, 61st, second, the operating systems usually implement a leap second by translating the clock a second back exactly before midnight, so that at 23:59 it happens twice. This clock shift looks like a reversal of time, and our measurement of a 10-millisecond event can now be a negative value of 990 milliseconds.

  start := time.Now() // 11:59:59.995 event() end := time.Now() // 11:59:59.005 (really 11:59:60.005) elapsed := end.Sub(start) // –990 ms 

Since normal clocks are inaccurate for measuring the duration of events during such time shifts, operating systems provide a second type of clock — monotonous clocks, which simply count seconds and never change or shift.

Only with this non-standard clock shift, monotonous watches are not particularly better than regular watches, which, unlike monotonous ones, are able to show the current time. Therefore, for the sake of simplicity, the time package API in Go 1 only has access to regular computer clocks.

In October 2015, a bug-report appeared that the Go programs incorrectly return the duration of events during such clock shifts, especially in the case of a leap second. The proposed solution was also a report headline: “Add a new API for accessing monotonous watches”. Then I argued that the problem was not significant enough to create a new API for her. A few months before that, for a leap second in mid-2015, Akamai, Amazon, and Google learned to slow down their watches in such a way that this extra second “smeared out” throughout the day and there was no need to turn the clock back. Everything went to the fact that the widespread use of this “smearing second” approach would get rid of the watch transfer in general and the problem would disappear by itself. For contrast, adding a new API to Go would add two new problems: we would have to explain these two types of watches, train users when to use which one and convert a lot of existing code, and only for a situation that is very rare and, rather everything, will disappear all by itself.

We did what we always do when the solution to the problem is not obvious - we waited. Waiting gives us more time to gain more experience and deepen our understanding of the problem, plus more time looking for a good solution. In this case, the wait added an understanding of the seriousness of the problem, in the form of a failure in Cloudflare, thankfully insignificant . Their Go code measuring the duration of DNS requests during a leap second at the end of 2016 returned a negative value, similar to the example of -990 milliseconds above, and this led to a panic on their servers, breaking about 0.2% of all requests at the peak of the problem.

Cloudflare is exactly the type of cloud system for which Go was created, and they crashed in production due to the fact that Go could not measure time correctly. Further, and this is the key point here, Cloudflare wrote about their experience - John Graham-Cumming published a blog post “How and why a leap second influenced DNS Cloudflare” . By telling the specific and incident details and their experience with Go, John and Cloudflare helped us understand that the problem of inaccurate metering during a leap second was too important to leave unresolved. Two months after the publication of the article, we developed and implemented a solution that will appear in Go 1.9 (and, by the way, we did it without adding a new API ).

Example: aliases

My second example is about alias support in Go.

Over the past few years, Google has put together a team focused on large-scale code changes, such as API migrations and bug fixes throughout the code base, consisting of millions of source code files and billions of lines of code written in C ++, Go, Java, Python and others. languages. One of the things that I learned from their work was that when replacing the old name in the API with a new one, it is important to be able to make changes step by step, and not all at once. To do this, it must be possible to declare that the old name implies a new one. In C ++, there is #define, typedef and the use of declarations allow this, but there was no such mechanism in Go. And since one of the main tasks for Go is the ability to scale in large code bases, it was obvious that we need some kind of mechanism for switching from old names to new ones during refactoring, and that other companies will also face this problem as their code increases. bases on go.

In March 2016, I began to discuss with Robert Grismier and Rob Pike how Go could cope with multi-step refactoring of code bases, and we came up with the idea of ​​alias declarations, which were exactly the right mechanism. At that moment I was very pleased with how Go developed. We have been discussing the idea of ​​aliases since the early days of Go - in fact, the first draft of the Go specification contains an example using aliases - but, every time we discuss aliases, and, a little later, type aliases, we didn’t really understand what they might be important for. so we put off the idea. Now we proposed to add aliases to the language not because they were a directly elegant concept, but because they solved a very serious practical problem, besides they helped Go to better solve the task of development scalability. I sincerely hope that this will serve as a good model for future changes in Go.

A little later that spring, Robert and Rob wrote a proposal , and Robert presented it in a short report (lightning talk) at GopherCon 2016 . The next few months were rather vague, and certainly cannot serve as an example of how to make changes in Go. One of the many lessons we then learned was the importance of describing the significance of the problem.

A minute ago, I explained to you the essence of the problem, giving some minimal information about how and why this problem may arise, but without giving specific examples about how you can decide whether the problem will touch you or not. That proposal and report was operated on with abstract examples, including the C, L, L1, and C1..Cn packages, but nothing concrete with which programmers could associate the problem. As a result, much of the response from the community was based on the idea that aliases solve the Google problem, and that is not relevant to the rest.

Just as we at Google did not at first understand the importance of correctly handling a leap second, we also did not effectively convey Go to the community the importance and the need to be able to cope with the gradual migration and correction of code bases during large-scale refactoring.

In the fall we started anew. I gave a talk and wrote an article explaining the problem in detail , using many concrete examples from real open-source projects showing that this problem is relevant for everyone, not just Google. Now, after more people understand the problem and appreciate its importance, we were able to start a productive discussion about which solution would work best. The result of this is that the type aliases will be included in Go 1.9 and help Go to scale better in larger and larger code bases.

Experience stories

One of the lessons here is that it is difficult, but it is crucial to describe the importance of the problem in an understandable way so that other people working in a different environment and environment can understand it. In order to discuss major changes in Go in the community, we will need to pay special attention to this process in detail describing the importance of each problem we will try to solve. The best way to do this is to show how the problem affects real programs or real systems, like in the blog post of Cloudflare or my article on refactoring .

Such stories about the experience of using transform problems from abstract to concrete and allow us to understand its significance. : .

, (generics), , Go . , — , generic-, , (receiver). , , .

, , error , Go , , Go . , .

. Go , , Go - . , Go, , , .

Go 2, , Go. , , . Medium , Github Gist ( .md Markdown), Google doc , . , , Wiki: https://golang.org/wiki/ExperienceReports

Solutions

, , , , , , , .

, , , , , , Go . 2013 , (“comma-ok”) . , x y , , uint32 , lo, hi = x * y 32 , 32 . , , . .

, Go 1.9 math/bits, :

  package bits // import "math/bits" func LeadingZeros32(x uint32) int func Len32(x uint32) int func OnesCount32(x uint32) int func Reverse32(x uint32) uint32 func ReverseBytes32(x uint32) uint32 func RotateLeft32(x uint32, k int) uint32 func TrailingZeros32(x uint32) int ... 

, , . math/bits , , , , , , math/bits . , .

, Go 1 , (shared) (races) Go , . , , , - , , , . , Go . , — , (race detector) Go. runtime , .

, , .

Go 2

, Go 2?

, - Go 2 , , Go 1. . -, Go 1, , . -, Go 1 Go 2. -, Go 1 Go 2, . -, , . -, , -.

, - Go 1, , , - , Go 1.12 . .

- , Go 1.20, - Go 2. , - , , Go 1.20 Go 2. , Go 1.X Go 2.X, Go 1.X .

, , , Go 1 , , , Go 1 , .

Go 2 , , . , .

. , , Go , , , . , , . - . , , , Go , Go.

Thank.

Russ Cox, 13 2017