What should we build an open data center? Some details about the structure of Facebook DC in Altoona

Over the past year, Facebook has brought many interesting things to the standards for the development of network equipment. Moreover, if the majority of developers leave their projects proprietary, then Facebook opens up innovations for others. Actually, for a company whose purpose is to provide information to the public, such a model of work makes sense. At the same time, an open work scheme allows Facebook to save money. According to James Taylor, vice president of infrastructure, for the last 3 years, Facebook has saved more than $ 2 billion, allowing members of the Open Compute Project to work on their own specifications.

In addition, the Wedge , an open top-of-rack switch developed by the OCP community, attracts attention. This was followed by 6-Pack , FBOSS and OpenBNC . Facebook has built its new data center based on the development of the Open Compute Project in Altoona, Iowa, USA. At the same time, the company provided all the necessary information about this project . There are several ideas that can be used in data centers of other companies, moreover, DC of any size.

Facebook cluster design

The first image shows the aggregated cluster design of the DC in Altoona. The developers call the 4-post architecture of the data center. Here, up to 255 racks can be combined via ToR switches to high density cluster switches (CSW). RSW can have up to 44 10G downlinks and 4 or 8 10G uplinks. Four CWS and connected RSWs make up a cluster.
')


Four aggregated “FatCat” (FC) switches unite clusters into a single system. Each CSW has a 40G connection with one of the four FCs. A protective 80G ring combines CWS within each cluster, and the FCs are connected to a protective 160G ring.

This is a really good structure for several reasons, including the reliability and practicality of the system. However, for Facebook this was not enough. The fact is that many problems in this kind of architecture are a consequence of the need to install very large switches for CSW and FC.

What is with Altoona?

In the architecture of the data center of the new generation from Altoona, most of the problems of cluster architecture are solved, while maintaining the best features of this type of architecture.

For example, not a few big switches are used here, but many small ones. In addition, each switch is responsible for a small percentage of the load, and the failure of one switch is not a significant problem;

Also in such a DC, capital and operating costs are reduced;

Increasing the size and power of a data center of this type is made in a very short time, much cheaper than in conventional DCs.

The network topology of such a DC is shown in the following image, where you can quickly recognize Clos . Instead of working with hundreds of racks in a cluster design, here each of the topological units is responsible for 48 racks.





Below is a volume diagram of the topology of this kind of data center.



Facebook representatives argue that the modular design of the data center allows you to very quickly change the structure of the DC, adding or removing certain elements. All changes are made in minimal time with minimal cost. This point of view is explained in more detail here:



The advantage of Facebook's new type of data center is the ability to use small switches in architecture, which allows the object to be scaled to any size without the need to change the basic blocks.

Switches can be used from Accton, Quanta, Celestica, Dell and some other companies. At the same time, the cost of a switch from Quanta with 32 40G ports is 7495 dollars, and Juniper QFX5100 with 24 40G ports costs a little less than 30 thousand.

Hyperscaling - what is it?

Most professionals working in the telecommunications sector apply this term only to giants like Amazon, Google, Facebook. However, the term means the ability to change the scaling in a very short time. A hyper-scaled data center may be relatively small, but it can be enlarged at any time without making fundamental changes to the infrastructure. Also in this case it should be possible to use the same switches and connections that were used initially.

In DC there can be only a few racks, but it can already be hyperscale-DC.

Another misconception in terms of hyper-scaling is the belief that DCs of this type are “sharpened” for optimal performance with one or several main applications. This is not entirely true. Ideally, hyperscale design means the ability to support hundreds of business applications with the same ease with which DCs work with big-data, search applications or social media.

As for DC Facebook - here you can add additional blocks and layers without any problems, and as much as you need at a particular moment.