About the author: Tammy Butov is the technical head of the infrastructure for developers at Dropbox. This code flow control is the complete cycle of using Go in Dropbox, from programming to release. She spoke at the GopherCon 2017 conference on how Dropbox developers create and maintain large-scale Go services.

How Dropbox Came to Use Go

Tammy quotes Rob Pike's article “ Go at Google: language design in software development ” from 2012, because it generally conveys well why Go works well in Dropbox:

“Go is an efficient, scalable and productive language. Some programmers enjoy working with it; others find it prosaic, even boring. In this article we will explain why all these positions do not contradict each other. Go is designed to solve the problems that arise in software development at Google, which led to the creation of a language that is not a breakthrough from a research point of view, nevertheless it is a great tool for developing large software projects. ” - Rob Pike, 2012

Dropbox scale is impressive:
')

• Over 500 million users
• 200,000 business users
• 500 petabytes of user data
• Multi-byte Go Storage System

As a result, Dropbox requires a lot from its systems, languages, and developers. The guiding principles and requirements are as follows:

• Create reliable systems
• Create secure systems
• Combine reliability and security in the original architecture
• Reliability 99.9999999999% (twelve nines)
• Availability 99.99%

Go State in Dropbox

Today, most of the Dropbox infrastructure is written in Go. In particular:

• Go server repositories - 150 unique contributors (out of 500 developers in the company)
• More than 15 groups create and maintain Gobox services in Dropbox
• 1.3 million lines on Go are written throughout the company in Dropbox

Some of the key systems written in Go are:

• RAT: rate limiting and choking (muffling) of traffic
• HAT: replacement memcached
• AFS: file system to replace the global Zookeeper
• Edgestore: distributed database
• Bolt: for messages
• DBmanager: to automate and monitor over 6000 Dropbox databases
• Jetstream, Telescope, block routing and more

Many of these are successors of previous systems not on Go.

How did Dropbox start using Go?

Tammy shared several stories of how Dropbox naturally went over to Go.

The prototype of the speed limiter on Go from the hacker week
Before the hacker week that happened once, the developers of Dropbox implemented a speed limit and throttling separately for each service where it was required. But for this hacker week, one Dropbox engineer decided to create a single implementation of these functions. This is how RAT (Rate limiting And Throttling) was born.

The first RAT prototype was created in four days and is shown on the fifth. Within a few weeks after the creation of the RAT, information about it spread throughout the company. Another Dropbox engineer wrote a suggestion to Tammy to see how they can use RAT from the Python project. The integration went smoothly, the service was accepted naturally - and soon RAT began to bring benefits. Now several Dropbox teams use RAT.

Dbmanager
Dropbox has more than 6,000 databases, and such a large system requires automation and monitoring. At any given time, there is database replication, crashes, transfer from a regular replica to a primary replica, and so on.

To manage this process, Dropbox engineer developed Dbmanager - UI on the web to quickly view the status of all more than 6000 databases. Status information is also transmitted to other systems.

Update Go versions to Dropbox
With hundreds of developers, Dropbox neatly coordinates the update of major versions of Go. Tammy didn’t mention any specific issues, indicating a smooth process!

Some interesting facts:

• Dropbox recently finished migrating production services from Go 1.5 to 1.6.
• To track the update process, they created a simple Dropbox Paper document, where the owner of each service reports on progress and asks for help if necessary.
• Dropbox skips Go 1.7 version and immediately switches to Go 1.8 when the migration to Go 1.6 is complete (including non-production services).

How Dropbox engineers master Go

Every Dropbox engineer goes through the same rigorous Go process, which consists of the following steps:

• Reading Infrastructure Topology, Go Style Guides and Protobuf Style Guides
• Strict but friendly code analyzes
• Creating a small application (in the App Store) on Go
• Exploring Bazel to Build and Test Go Code

For an experienced programmer, the process takes about a week.

What went smoothly with the introduction of Go at Dropbox? And what not?

Overall, the use of Go in Dropbox's has been very successful.

• Go is easy to show high productivity.
• Go is easy to write and use services. (And people like both!)
• The standard library is very good.
• Debugging tools (mostly!) Work well.

Here, one of the important facts is that Dropbox does not attempt to rewrite Go services in other languages. This is a sign that people are generally satisfied. (Tammy gave an intriguing detail: Dropbox uses Rust a bit. But he is not considered a replacement for Go).

What complicated Dropbox found in Go?

The greatest difficulty Tammy called the work with the state of the race.

• The data race condition is the most difficult bug to debug, detect, fix, etc.
• A few Dropbox engineers are especially good at finding such bugs, and the rest rely on their experience.
• A race state detector in Go does not always help. You need to understand when he is helpless.
• It is important to carefully design Go programs if simultaneous data access is required.

Dropbox is recruiting engineers who care about data reliability and durability, so it was easy for them to understand (although simultaneous access is difficult all the time).