Go: Two years in production
I would like to share our experience of using Go for two years in the production of Iron.io. We are one of the first companies to use Go (golang) in high-load services. When it was decided in Iron.io to use this language, we did not know what to expect in the long term, but so far everything is going fine.
I already wrote a little about this in a previous post about switching to Go from Ruby. But now I would like to talk about specific things for which we love this language, which we learned during its use. . Here they are in random order:
')
When we first made a decision in Iron.io on which language we would use, we first did some research and wrote tests for our product, the message queue. I wrote a partial beanstalkd clone on Go that used the beanstalkd protocol so that existing clients could be used to validate it. The tests showed high performance, almost the same as the official version in C. And as a nice addition, it was surprisingly easy to write on Go.
You can find criteria for comparing Go with other languages ​​on the Computer Language Benchmarks Game website. The graph below compares it with Java (which would probably be the language we would use if it weren’t Go):
This graph shows that Go is slightly faster in some cases, but in general it loses Java. However, this is not so bad for a language that is only a few years old. I have no doubt that he will catch up with competitors after some time. You can also see that the amount of memory used is much smaller.
Just for the sake of interest, compare the differences between Go and Ruby below. Go is superior to Ruby in performance and memory usage.
For two years, Go has never been a bottleneck in our products, it has always been a database.
Since there is no virtual machine or interpreter in Go, it runs quickly and uses a small amount of memory. IronMQ starts at about 6.5 KB of resident memory, including loading configuration, creating connections, etc. After it has been running for some time, memory consumption increases, mainly due to caching. Right now, our production servers are using around 400 MB (I understand that this comparison may not be appropriate in the context of your application).
For two years, we have never had a memory leak or memory problems.
Parallelism is an extremely important part of Go, and a variety of high-level abstractions make it easy to use. I have been using Java for many years and felt very comfortable with the java.util.concurrency package, which provides a pretty good set of tools for parallelization, but this is not at all the same thing as Go, because of its simplicity of the basic implementation. Go has goroutines for parallel operations and communication channels between them. Goroutines are especially interesting:
“Goroutines are part of what simplifies the use of parallelization. An idea that has existed for a long time is a set of independently executed coroutines on multiple threads. When a coroutine is blocked, for example, by calling a blocking system call, the runtime automatically moves other coroutines in the same operating system thread to another executable thread so that they are not blocked. The programmer sees none of this, which is a clear advantage. As a result, we get goroutines. They are lightweight, and in the absence of large time spent on long-term system calls will cost a little more memory for a stack that fits just a few kilobytes. To keep stacks small, Go runs at runtime using segmented stacks. A new goroutine is given a few kilobytes, which is almost always enough. Otherwise, the segments are automatically expanded at run time. Overhead costs on average about three simple instructions for calling a function. It is advisable to create hundreds of thousands of goroutines in the same address space. If the goroutines were just streams, system resources would end much faster. " Source
The important thing we had to do was limit concurrency in order to make sure that we did not overload our database or other parts of the system during load peaks. We implemented this with a simple semaphore using Go channels.
Reliability is difficult to quantify, but in the end we came to the conclusion that our Go applications are very stable. I do not think that we had failures or errors that were not related to some external problems (database, errors in third-party libraries). In general, open source libraries for Go are very high quality libraries. We did not find memory leaks or major errors in the main libraries.
I also tend to think that our code is of higher quality only due to the fact that it is written in Go. I can not say exactly why, but when I look at the IronMQ code, it seems to me “warm and tube”. Perhaps the point is a very strict compiler, which forces us to delete imports and unused variables. Perhaps this is due to the fact that sometimes it is enough to write a small piece of code to solve a large task. Probably, I will soon understand what the point is and share the details with the audience.
Go compiles to one static binary file. Deployment is simply uploading a file to a server and launching it. No additional dependencies are required. There is no program runtime (you do not need to install Go on the server). The executable file is small - the size of the IronMQ is ~ 6 MB.
If something goes wrong after deployment and there is a need for a rollback, then you can always just stop the executable file and replace it with the previous one. There is no need to worry about dependencies, since the entire program is combined into a single binary file.
Two years ago, we took a big risk in choosing Go - then there were few people who knew the language. We were the first company to publish a vacancy in the golang mailing list, and we were puzzled by the qualifications of the people who responded. We received applications from developers with vast experience from the best technology companies, others had Ph.D. degrees, and still others worked on really hardcore projects. Most of them were not full-time go developers, but worked with him enough to gain experience and a desire to share their knowledge. I'm not sure if what we are trying to create matters, or if they just wanted to work with Go.
Our first Go programmer Evan Shaw , who works with us so far, is one of the developers of Golang.
After two years of working with Go, I can say with confidence that we have made the right choice. If we started Iron.io today, Go would still be the best option. Many other companies use it now, including Google and Heroku. A similar opinion is expressed by + Rob Pike , one of the creators of Go:
“We realized that the software we build at Google is not always well served by the available languages. Robert Griesemer, Ken Thompson and I decided to make the language most suitable for writing programs that we develop at Google. ”
Derek Collison , founder of Apcera , recently stated in an article for Wired :
“Which control layers and infrastructure layers of the latest technologies will be used to ensure the functioning of cloud services?” Within two years, most of them will be written on Go. ”
Go is the next generation language that we have been waiting for. It may be too early for such a categorical statement, but it is certainly a good start.
I already wrote a little about this in a previous post about switching to Go from Ruby. But now I would like to talk about specific things for which we love this language, which we learned during its use. . Here they are in random order:
- Performance
- Memory
- Parallel computing
- Reliability
- Deployment
- Talented specialists
')
Performance
When we first made a decision in Iron.io on which language we would use, we first did some research and wrote tests for our product, the message queue. I wrote a partial beanstalkd clone on Go that used the beanstalkd protocol so that existing clients could be used to validate it. The tests showed high performance, almost the same as the official version in C. And as a nice addition, it was surprisingly easy to write on Go.
You can find criteria for comparing Go with other languages ​​on the Computer Language Benchmarks Game website. The graph below compares it with Java (which would probably be the language we would use if it weren’t Go):
This graph shows that Go is slightly faster in some cases, but in general it loses Java. However, this is not so bad for a language that is only a few years old. I have no doubt that he will catch up with competitors after some time. You can also see that the amount of memory used is much smaller.
Just for the sake of interest, compare the differences between Go and Ruby below. Go is superior to Ruby in performance and memory usage.
For two years, Go has never been a bottleneck in our products, it has always been a database.
Memory
Since there is no virtual machine or interpreter in Go, it runs quickly and uses a small amount of memory. IronMQ starts at about 6.5 KB of resident memory, including loading configuration, creating connections, etc. After it has been running for some time, memory consumption increases, mainly due to caching. Right now, our production servers are using around 400 MB (I understand that this comparison may not be appropriate in the context of your application).
For two years, we have never had a memory leak or memory problems.
Parallel computing
Parallelism is an extremely important part of Go, and a variety of high-level abstractions make it easy to use. I have been using Java for many years and felt very comfortable with the java.util.concurrency package, which provides a pretty good set of tools for parallelization, but this is not at all the same thing as Go, because of its simplicity of the basic implementation. Go has goroutines for parallel operations and communication channels between them. Goroutines are especially interesting:
“Goroutines are part of what simplifies the use of parallelization. An idea that has existed for a long time is a set of independently executed coroutines on multiple threads. When a coroutine is blocked, for example, by calling a blocking system call, the runtime automatically moves other coroutines in the same operating system thread to another executable thread so that they are not blocked. The programmer sees none of this, which is a clear advantage. As a result, we get goroutines. They are lightweight, and in the absence of large time spent on long-term system calls will cost a little more memory for a stack that fits just a few kilobytes. To keep stacks small, Go runs at runtime using segmented stacks. A new goroutine is given a few kilobytes, which is almost always enough. Otherwise, the segments are automatically expanded at run time. Overhead costs on average about three simple instructions for calling a function. It is advisable to create hundreds of thousands of goroutines in the same address space. If the goroutines were just streams, system resources would end much faster. " Source
The important thing we had to do was limit concurrency in order to make sure that we did not overload our database or other parts of the system during load peaks. We implemented this with a simple semaphore using Go channels.
Reliability
Reliability is difficult to quantify, but in the end we came to the conclusion that our Go applications are very stable. I do not think that we had failures or errors that were not related to some external problems (database, errors in third-party libraries). In general, open source libraries for Go are very high quality libraries. We did not find memory leaks or major errors in the main libraries.
I also tend to think that our code is of higher quality only due to the fact that it is written in Go. I can not say exactly why, but when I look at the IronMQ code, it seems to me “warm and tube”. Perhaps the point is a very strict compiler, which forces us to delete imports and unused variables. Perhaps this is due to the fact that sometimes it is enough to write a small piece of code to solve a large task. Probably, I will soon understand what the point is and share the details with the audience.
Deployment
Go compiles to one static binary file. Deployment is simply uploading a file to a server and launching it. No additional dependencies are required. There is no program runtime (you do not need to install Go on the server). The executable file is small - the size of the IronMQ is ~ 6 MB.
Rollback
If something goes wrong after deployment and there is a need for a rollback, then you can always just stop the executable file and replace it with the previous one. There is no need to worry about dependencies, since the entire program is combined into a single binary file.
Talented specialists
Two years ago, we took a big risk in choosing Go - then there were few people who knew the language. We were the first company to publish a vacancy in the golang mailing list, and we were puzzled by the qualifications of the people who responded. We received applications from developers with vast experience from the best technology companies, others had Ph.D. degrees, and still others worked on really hardcore projects. Most of them were not full-time go developers, but worked with him enough to gain experience and a desire to share their knowledge. I'm not sure if what we are trying to create matters, or if they just wanted to work with Go.
Our first Go programmer Evan Shaw , who works with us so far, is one of the developers of Golang.
Conclusion
After two years of working with Go, I can say with confidence that we have made the right choice. If we started Iron.io today, Go would still be the best option. Many other companies use it now, including Google and Heroku. A similar opinion is expressed by + Rob Pike , one of the creators of Go:
“We realized that the software we build at Google is not always well served by the available languages. Robert Griesemer, Ken Thompson and I decided to make the language most suitable for writing programs that we develop at Google. ”
Derek Collison , founder of Apcera , recently stated in an article for Wired :
“Which control layers and infrastructure layers of the latest technologies will be used to ensure the functioning of cloud services?” Within two years, most of them will be written on Go. ”
Go is the next generation language that we have been waiting for. It may be too early for such a categorical statement, but it is certainly a good start.
Source: https://habr.com/ru/post/195464/
All Articles