Golang to AeroFS
Translation of the AeroFS engineer’s article on the transfer of their microservice architecture from Java to Go.
TLDR; By porting some of our microservices from Java to Go, we reduced the memory usage by several orders of magnitude .
')
The AeroFS Appliance architecture consists of many microservices, and the vast majority of them are written in Java. This has never caused us problems, the entire system serves thousands of users from different clients without any performance problems.
However, after our transition to Docker, we noted a sharp increase in memory usage by our system. Having tried several fashionable tools for monitoring dockers, we stopped at this somewhat geeky, but very useful script:
It lists the running containers, sorted by the amount of resident memory used. Example script output:
Investigating the problem, we found that some Java services used a surprisingly large amount of memory, often without any correlation with their complexity or lack thereof. We identified several major factors that led to this use of memory.
Being a man of the old school, accustomed to writing in assembler for the Z80 for devices with 64kb of memory on board, I was very inspired by the idea of how to return hundreds of megabytes of valuable RAM. On a happy occasion, our next hackathon was just a few days later, and it was a great chance to focus on a problem and a great excuse to try new tools.
(comment of the translator: greasefire - “the fire caused by the ignition of oil / fat on the stove”)
My main goal of this hackathon was to burn through the layers of metaphorical fat in order to reduce the total amount of AeroFS system memory used.
In particular, my criteria for success were:
In the spirit of the hackathon, I also wanted to try new languages and tools, so just correcting existing services was not an option.
And in order to increase the likelihood of obtaining a result that can be shown, and possibly even closed, it was important to choose an adequate size and complexity target. The obvious choice was TeamServer probe service (team-servers on the appliance status page) - a small tomcat servlet with a single HTTP call and very clear internal logic.
As a result, the goal was the following - to create a server:
To meet the criteria for CPU and memory, the main candidates were compiled languages created for system programming. And although the hackathon did not mean that the result would be immediately usable, I still adhered to the fact that the code should be easily maintained and leave darker alternatives.
The pool of candidates quickly narrowed to two participants - Go and Rust. Both compile the code quite easily into small static binary files, ideally sharpened to run in minimal containers. Both promised adequate performance, memory integrity, good support for concurrent programming, and, what was especially important for me, less memory usage than in the case of the JVM.
The intricate Rust type system looked especially intriguing. But at the same time, Rust was much less mature than Go, at that time it had not even reached version 1.0. The choice of Rust was also hampered by the lack of good libraries for HTTP and low-level networking.
Earlier we tried to port one of our services to Go, in the year 2013, but at that time we got into some kind of memory leak, and decided to stop the experiment. Two years later, Go looked much more mature and was chosen as the most suitable candidate for our experiment.
Go is quite similar to the languages from the C-family, which allows it to be picked up very quickly, but at the same time it contains enough features that require the first few days of constant switching between code and documentation. Fortunately, the documentation is well written and very successfully incorporated into the source code of the standard library, which was incredibly useful for clarifying various points and understanding the idiomatic code.
I was also very pleased to have a single standard for formatting the language, and the gofmt magic utility that enforces it and integrates very easily with a text editor like vim (a small insider: this hackathon was also my first attempt to use vim for something more than one-line editing)
It took me about a day to get acquainted with Go and port a simple service chosen for the hackathon. The results were very promising:
Until the hackathon, there was still almost a whole day, and I noticed another service, the Certificate Authority (ca on the appliance status page). This service accepts certificate signing requests from internal services and desktop clients and returns signed certificates that are used to encrypt the transfer of both peer-to-peer content between clients and for client-server communication.
When this new CA finally replaced its Java equivalent, a few days after the end of the hackathon, it reduced the memory usage to an incredible 100x !
This project won the nomination "Technical Steepness" (orig. "Technical Amazingness") , and turned into ongoing efforts to reduce the memory usage of the entire system.
By version 1.1.5, four more services were ported to Go — 6 as a whole — and the total memory savings was 1 GB . In each case, we received a similar reduction in code size, and in some cases we even got a significant reduction in processor usage or better throughput.
- Hugues & the AeroFS Team.
TLDR; By porting some of our microservices from Java to Go, we reduced the memory usage by several orders of magnitude .
In the beginning was Java
')
The AeroFS Appliance architecture consists of many microservices, and the vast majority of them are written in Java. This has never caused us problems, the entire system serves thousands of users from different clients without any performance problems.
However, after our transition to Docker, we noted a sharp increase in memory usage by our system. Having tried several fashionable tools for monitoring dockers, we stopped at this somewhat geeky, but very useful script:
for line in `docker ps | awk '{print $1}' | grep -v CONTAINER`; do \ echo $(( `cat /sys/fs/cgroup/memory/docker/$line*/memory.usage_in_bytes` / 1024 / 1024 ))MB \ $(docker ps | grep $line | awk '{printf $NF" "}') ; \ done | sort -n It lists the running containers, sorted by the amount of resident memory used. Example script output:
46MB web 66MB verification 74MB openid 82MB havre 105MB logcollection 146MB sp 181MB sparta Investigating the problem, we found that some Java services used a surprisingly large amount of memory, often without any correlation with their complexity or lack thereof. We identified several major factors that led to this use of memory.
- increasing the number of running JVMs as each tomcat servlet ran in a separate container
- a reduced opportunity for several JVMs to share read-only memory: the JVM itself, all dependent libraries, and, of course, many JARs used by different services
- memory isolation in some cases confused the heuristics of memory calculations, which led to large cache allocations in some services
Being a man of the old school, accustomed to writing in assembler for the Z80 for devices with 64kb of memory on board, I was very inspired by the idea of how to return hundreds of megabytes of valuable RAM. On a happy occasion, our next hackathon was just a few days later, and it was a great chance to focus on a problem and a great excuse to try new tools.
Code Name: Greasefire
(comment of the translator: greasefire - “the fire caused by the ignition of oil / fat on the stove”)
My main goal of this hackathon was to burn through the layers of metaphorical fat in order to reduce the total amount of AeroFS system memory used.
In particular, my criteria for success were:
- CPU usage should not increase significantly
- stability and memory security should remain
- resident memory usage should decrease 2 or more times
In the spirit of the hackathon, I also wanted to try new languages and tools, so just correcting existing services was not an option.
And in order to increase the likelihood of obtaining a result that can be shown, and possibly even closed, it was important to choose an adequate size and complexity target. The obvious choice was TeamServer probe service (team-servers on the appliance status page) - a small tomcat servlet with a single HTTP call and very clear internal logic.
As a result, the goal was the following - to create a server:
- with completely identical API
- packed in docker image
Trying new tools
To meet the criteria for CPU and memory, the main candidates were compiled languages created for system programming. And although the hackathon did not mean that the result would be immediately usable, I still adhered to the fact that the code should be easily maintained and leave darker alternatives.
The pool of candidates quickly narrowed to two participants - Go and Rust. Both compile the code quite easily into small static binary files, ideally sharpened to run in minimal containers. Both promised adequate performance, memory integrity, good support for concurrent programming, and, what was especially important for me, less memory usage than in the case of the JVM.
The intricate Rust type system looked especially intriguing. But at the same time, Rust was much less mature than Go, at that time it had not even reached version 1.0. The choice of Rust was also hampered by the lack of good libraries for HTTP and low-level networking.
Earlier we tried to port one of our services to Go, in the year 2013, but at that time we got into some kind of memory leak, and decided to stop the experiment. Two years later, Go looked much more mature and was chosen as the most suitable candidate for our experiment.
Go is quite similar to the languages from the C-family, which allows it to be picked up very quickly, but at the same time it contains enough features that require the first few days of constant switching between code and documentation. Fortunately, the documentation is well written and very successfully incorporated into the source code of the standard library, which was incredibly useful for clarifying various points and understanding the idiomatic code.
I was also very pleased to have a single standard for formatting the language, and the gofmt magic utility that enforces it and integrates very easily with a text editor like vim (a small insider: this hackathon was also my first attempt to use vim for something more than one-line editing)
results
It took me about a day to get acquainted with Go and port a simple service chosen for the hackathon. The results were very promising:
- Code size was halved , from 175 lines to 96
- Resident memory usage dropped from 87MB to just 3MB, 29x decrease !
- The resulting docker image has decreased from 668MB to 4.3MB - this is 155x reduction ! I agree that the largest layers of docker-images are still reused by different services, so the real reduction in disk usage was much less with the use of many Java services. Nevertheless, these figures are very pleasing to the eye.
Until the hackathon, there was still almost a whole day, and I noticed another service, the Certificate Authority (ca on the appliance status page). This service accepts certificate signing requests from internal services and desktop clients and returns signed certificates that are used to encrypt the transfer of both peer-to-peer content between clients and for client-server communication.
When this new CA finally replaced its Java equivalent, a few days after the end of the hackathon, it reduced the memory usage to an incredible 100x !
This project won the nomination "Technical Steepness" (orig. "Technical Amazingness") , and turned into ongoing efforts to reduce the memory usage of the entire system.
By version 1.1.5, four more services were ported to Go — 6 as a whole — and the total memory savings was 1 GB . In each case, we received a similar reduction in code size, and in some cases we even got a significant reduction in processor usage or better throughput.
- Hugues & the AeroFS Team.
Source: https://habr.com/ru/post/264251/
All Articles