Instagram "coming down from the clouds" Amazon to Facebook data centers
No matter how many reasons exist to move applications from ordinary data centers to cloud ones, there are at least some reasons to do the opposite. An illustrative example of this transfer was the recent transfer of Instagram servers from the Amazon public cloud service to Facebook data centers.
To carry out such an operation, Instagram engineers had to think hard, using a non-standard approach in overcoming unexpected difficulties. This work of theirs is a reminder that placing capacity in the cloud remains a rather difficult task, both for providers and for manufacturers of equipment for the data center.
')
Facebook founder and CEO Mark Zuckerberg noted that Instagram had the opportunity to appreciate the benefits of engineering solutions and the Facebook infrastructure in 2012 after acquiring an online photo-hosting service from the Internet giant.
The team decided to move in order to facilitate integration with Facebook systems and in order to use the entire Facebook infrastructure created by engineers to manage their large-scale server capacity. After implementing these actions, the engineering team found several points of contact with the Facebook infrastructure, which provided even more opportunities for product development and increased security.
The migration project was not as simple as it was first expected. “At first, moving seemed simple: set up a secure network between the Amazon Elastic Compute Cloud (EC2) and Facebook data center and gradually move services in parts,” said Instagram engineers Rick Branson, Pedro Kahuati and Nick Shotway in a blog .
However, they quickly realized that it was not as easy as it seemed at first. The main problem was the conflict of IP addresses of Facebook and EC2. The solution was the initial migration to Amazon's Virtual Private Cloud (VPC) and only then to the Facebook data center using Amazon Direct Connect.
Direct Connect is a service offered by Amazon in the data center, which is essentially the link between user services and public cloud services. Primarily aimed at enterprises, it was created in order to circumvent the Internet public, maintain productivity and ensure security. “Amazon's VPC is flexible enough to avoid conflicts with the Facebook network being buried!” Said the engineers.
However, moving applications from a cloud infrastructure to a private cloud is also not as easy as it seems at first. Instagram has thousands of processes a day. To reduce the risk of failure and facilitate these processes as much as possible, the team needs EC2 and VPC cloud services at the same time — and that’s the problem.
“AWS does not provide the ability to exchange security groups, nor overlap the closed EC2 and VPC networks,” they write. "The only way to interact between two closed networks is to create a public address space." Python and Zookeeper wrote a special application for handling dynamic IP addresses, called Neti, which provided a secure group activity and a separate address for each specific request.
Thus, after three weeks, a move was made, called the fastest of its kind. The stack was ready for placement at a new location - in Facebook data centers.
This stage of the process was more complicated, since The Instagram team wanted to retain all management tools created for production systems hosted on the EC2. For this, Chef configurators and a tool called Fabric were used to perform various functions - from hosting applications to promoting databases.
To put the tools in a well-tuned Linux-based Facebook environment, the team placed all the backup tools in Linux-containers (LXC), which is exactly the way they are hosted on Facebook's native servers now. “Facebook's backup tools are used to create the base structure, and Chef inside containers installs and configures specific Instagram software,” the Instagram team reported.
Such a project can not pass without the need to explore one or two innovations, and the Instagram team also had to deal with this. Here are just a few of the new ideas:
To carry out such an operation, Instagram engineers had to think hard, using a non-standard approach in overcoming unexpected difficulties. This work of theirs is a reminder that placing capacity in the cloud remains a rather difficult task, both for providers and for manufacturers of equipment for the data center.
')
Facebook founder and CEO Mark Zuckerberg noted that Instagram had the opportunity to appreciate the benefits of engineering solutions and the Facebook infrastructure in 2012 after acquiring an online photo-hosting service from the Internet giant.
Not as easy as it first seemed
The team decided to move in order to facilitate integration with Facebook systems and in order to use the entire Facebook infrastructure created by engineers to manage their large-scale server capacity. After implementing these actions, the engineering team found several points of contact with the Facebook infrastructure, which provided even more opportunities for product development and increased security.
The migration project was not as simple as it was first expected. “At first, moving seemed simple: set up a secure network between the Amazon Elastic Compute Cloud (EC2) and Facebook data center and gradually move services in parts,” said Instagram engineers Rick Branson, Pedro Kahuati and Nick Shotway in a blog .
First tried to move to a closed cloud Amazon
However, they quickly realized that it was not as easy as it seemed at first. The main problem was the conflict of IP addresses of Facebook and EC2. The solution was the initial migration to Amazon's Virtual Private Cloud (VPC) and only then to the Facebook data center using Amazon Direct Connect.
Direct Connect is a service offered by Amazon in the data center, which is essentially the link between user services and public cloud services. Primarily aimed at enterprises, it was created in order to circumvent the Internet public, maintain productivity and ensure security. “Amazon's VPC is flexible enough to avoid conflicts with the Facebook network being buried!” Said the engineers.
EC2 is not very "friendly" with VCP Amazon
However, moving applications from a cloud infrastructure to a private cloud is also not as easy as it seems at first. Instagram has thousands of processes a day. To reduce the risk of failure and facilitate these processes as much as possible, the team needs EC2 and VPC cloud services at the same time — and that’s the problem.
“AWS does not provide the ability to exchange security groups, nor overlap the closed EC2 and VPC networks,” they write. "The only way to interact between two closed networks is to create a public address space." Python and Zookeeper wrote a special application for handling dynamic IP addresses, called Neti, which provided a secure group activity and a separate address for each specific request.
Thus, after three weeks, a move was made, called the fastest of its kind. The stack was ready for placement at a new location - in Facebook data centers.
Linux containers made custom tools portable
This stage of the process was more complicated, since The Instagram team wanted to retain all management tools created for production systems hosted on the EC2. For this, Chef configurators and a tool called Fabric were used to perform various functions - from hosting applications to promoting databases.
To put the tools in a well-tuned Linux-based Facebook environment, the team placed all the backup tools in Linux-containers (LXC), which is exactly the way they are hosted on Facebook's native servers now. “Facebook's backup tools are used to create the base structure, and Chef inside containers installs and configures specific Instagram software,” the Instagram team reported.
Migration is experience
Such a project can not pass without the need to explore one or two innovations, and the Instagram team also had to deal with this. Here are just a few of the new ideas:
- try almost no effect on the environment to be introduced;
- apply such "crazy" ideas like the Neti application that simplify the work;
- create your own applications to avoid surprises;
- use familiar concepts and processes to make work as easy as possible.
Source: https://habr.com/ru/post/234809/
All Articles