Users convinced GitLab not to leave the cloud

Image source

At the end of 2016, Gitlab announced that it was going to leave the cloud (we did a translation of this article to Medium). A very detailed hardware purchase plan was also presented. Users followed the events with interest, actively commented on the published articles and eventually convinced GitLab to abandon this idea.

This story has an additional intrigue. GitLab, which itself is essentially a cloud service provider (although providing an application to users, rather than computing resources), suddenly decided that, as a consumer, such a scheme of work no longer suits, but still changed its mind .

GitLab experts seriously thought about leaving the cloud in November 2015, when task # 727 was created to switch to its own hardware.

The main disadvantage the company saw was the insufficient disk subsystem capacity for GitLab in Azure, as well as the need to share the cloud infrastructure with other users. This problem was planned to be solved by switching to our own server hardware, which also had to give substantial savings.

Describing the financial side of the issue, the company reported that hosting GitLab.com in Azure (not including GitLab CI) at that time cost about $ 200k per month. At the same time, the purchase of own equipment was equal in value to three months of cloud hosting. The colocation of their servers was estimated at $ 10k per month. That is, two and a half years would have been achieved tenfold savings. A detailed description of the costs can be found here in this publicly available spreadsheet .

However, users rightly pointed out that GitLab’s core competency is software development, and the company greatly underestimates the level and number of specialists, as well as the costs required for self-maintenance of infrastructure of this scale, especially in the event of emergency situations.

The company’s proposals for specific components of the server equipment that were planned for purchase were also analyzed in detail. As a result, much has been criticized: the architecture, the number of servers, hard drives, etc., as well as the presence of many spaces and undisclosed topics.

With regard to performance, the general conclusion was as follows: the additional costs of virtualization do not exceed 3%, so leaving the cloud is unlikely to solve the problems of lack of bandwidth I / O system. In addition, the lack of bandwidth could be more related to the lack of efficiency of NFS, CephFS and, in particular, the git version control system, which is not optimized for the parallel operation of a large number of users. As a result, it was decided to abandon CephFS and develop a solution to optimize access to git-repositories, which was called Gitaly .

Well, in this round of struggle “cloud vs own computer” white and fluffy won again. The process of increasing the complexity of the infrastructure and the narrowing of areas of specialization in the field of IT continues - to be a master of all trades every year is more difficult.

We should also note the openness of GitLab, which, thanks to this value, during the incident with the accidental deletion of the database, managed to turn defeat into a victory, but this time, in the matter of leaving the cloud, it gained from its weakness - lack of competence. Thanks to a broad public discussion, the company, on the one hand, was able to get an abundance of qualified community help and make a better decision, and on the other hand, to share experiences with people who may have similar choices or are simply interested in these issues. It is always curious to see how important tasks are solved in other organizations, how they think and what tools they use to facilitate their work.

References:

1. Issue # 727 in the GitLab.com bugtracker, where the problem was originally announced: https://gitlab.com/gitlab-com/infrastructure/issues/727 .
2. GitLab Infrastructure Update, which discusses performance issues (there is a link to the entertaining video of a working call): https://about.gitlab.com/2016/09/26/infrastructure-update/ .
3. The article about leaving the cloud: How it knit it and the cloud: Gitlab: we understood that it was time to leave the cloud .
4. Article about the planned server equipment purchase: Proposed server purchase for GitLab.com .
5. Electronic plate with calculations on equipment and hosting: https://docs.google.com/spreadsheets/d/1XG9VXdDxNd8ipgPlEr7Nb7Eg22twXPuzgDwsOhtdYKQ/edit#gid=894825456 .
6. Discussion of the proposal to purchase equipment on Hacker news: https://news.ycombinator.com/item?id=13153031 .
7. Post in issue # 727, explaining why the company changed its mind to leave the cloud: https://gitlab.com/gitlab-com/infrastructure/issues/727#note_20044060 .
8. Gitlab's blog post, where a point is put on the issue of switching to your hardware and the user comments that seemed most helpful to the staff of GitLab are given: https://about.gitlab.com/2017/03/02/why-we-are-not-leaving- the-cloud / .