Therascale OCS Solution
In the previous article about the Open Compute Project, we talked about Facebook's initiative to spread an open platform for building a hyper-data center. The idea is simple - existing solutions for exploitation are not enough, you need to create something that suits you.
In his blog, Microsoft talks a lot about how to use the Azure cloud service built on their infrastructure. The infrastructure, as it turned out, was also developed from scratch under the vision of Microsoft.
The process of publishing its platform turned out to be contagious, Microsoft joined the initiative and shared its vision of the optimal infrastructure, and we will tell about it in detail.
')
Key Features
View full rack
Details about the architecture
Structure
The chassis is a complex shaped trough in which baskets with universal backplanes for servers and disk shelves, a power distribution board, power supplies, a remote control controller and a wall oftext fans are installed.
Back view
Six large fans (140 * 140 * 38) are combined into a single unit on the hinged door, control comes from the chassis controller. The door closes the I / O ports and the space for cables, where special channels are laid for neat wiring and installation.
Microsoft uses standard 19 "racks with a height of 50U, of which 48U are occupied directly by servers, and another 2U is allocated for switches. Power supplies and PDUs are located on one side of the rack and do not take up space, unlike OCP.
Each chassis has a built-in service module (more about it later) and 6 power supply units (exactly the same as in our RS130 G4 , RS230 G4 ) are located according to the 5 + 1 scheme.
Rear view of the basket
A good idea - regardless of the modules used, the set of ports on the basket is the same. Two SAS ports, two SFP + ports and two RJ45 ports are always displayed, and what will be connected to them from the second side is determined on the spot.
Format Comparison
Cooling system efficiency
Control system
Chassis manager
The management server is inserted and extracted “on hot”, the software is built (not surprisingly) on Windows Server 2012. The source code is available to the general public and everyone can make their own edits (the toolkit is free).
general idea
Directly fee
Functional:
Tire routing
How does it go through the connectors
Not forgotten, and the software part:
Logic circuit
Out-of-band part
In-Band part
Facebook relies on Intel ME capabilities, Microsoft is using the familiar
IPMI using BMC.
Blades
Two types of blades are developed, one with a server, the other JBOD for 10 discs.
Possible combinations
You can combine, of course, as you like — even two servers, at least both JBODs.
Cables stay in place
The cables are connected to the backplane, so to replace the blade, simply remove the basket (a significant difference from OCP). This approach reduces maintenance time and eliminates possible error when reconnecting cables.
PCIe slot is designed for a very specific purpose - the use of PCIe Flash cards.
So divorced mezzanine boards
Disk shelf
Network infrastructure
All network connections are 10G, gigabit is used only for control and not everywhere. The important point is that Microsoft is very actively working on popularizing Software Defined Networking (SDN) technologies, their own services are based on software-configured networks.
Not for nothing on the past the Ethernet Summit their SDN monitoring product, DEMon, received six of the six possible awards , and the company is on the list of OpenDaylight Platinum Sponsors .
At the same time, we remind you that we have long announced a product for hybrid networks with SDN support - an Eos 410i switch with a port cost below $ 100 :)
Total
The largest companies in the industry rely on Software Defined Datacenter and Microsoft is no exception. The development of Ethernet technologies has led to the fact that 10G and RDMA (using RDMA over Converged Ethernet, RoCE) make it possible to do without a separate FC network (the gain from RDMA was discussed here: habrahabr.ru/company/etegro/blog/213911 ) without loss in performance . Windows Storage Spaces capabilities are such that hardware storage systems are gradually being replaced by solutions on Shared DAS / SAS networks (it was written in detail here habrahabr.ru/company/etegro/blog/215333 and then habrahabr.ru/company/etegro/blog/220613 ).
MS advantages:
Chassis-based design reduces cost and power consumption
All signal and power lines are transmitted through fixed connectors.
Network and SAS cables on a single backplane
Secure and scalable management system
Conclusion
The invention of the next incarnation of blades took place. Key difference from the rest
- fully open platform specifications and use of standard network switches. In traditional blades from a well-known circle of companies, the choice of the network part is limited by the imagination of the developers; in an open solution, you can use any of the existing products, from used Juniper to SDN switches of its own design.
Well, we presented a solution based on Open CloudServer in the product line .
In his blog, Microsoft talks a lot about how to use the Azure cloud service built on their infrastructure. The infrastructure, as it turned out, was also developed from scratch under the vision of Microsoft.
The process of publishing its platform turned out to be contagious, Microsoft joined the initiative and shared its vision of the optimal infrastructure, and we will tell about it in detail.
')
Key Features
- Width standard, 19 inches. 12U chassis.
- Unified cooling and power system.
- Unified server management system.
- All maintenance does not affect cabling (unless the problem is in the switch).
- Standard PDUs are used.
- Power supplies match our rack-mount line in form factor.
View full rack
Details about the architecture
Structure
The chassis is a complex shaped trough in which baskets with universal backplanes for servers and disk shelves, a power distribution board, power supplies, a remote control controller and a wall of
Back view
Six large fans (140 * 140 * 38) are combined into a single unit on the hinged door, control comes from the chassis controller. The door closes the I / O ports and the space for cables, where special channels are laid for neat wiring and installation.
Microsoft uses standard 19 "racks with a height of 50U, of which 48U are occupied directly by servers, and another 2U is allocated for switches. Power supplies and PDUs are located on one side of the rack and do not take up space, unlike OCP.
Each chassis has a built-in service module (more about it later) and 6 power supply units (exactly the same as in our RS130 G4 , RS230 G4 ) are located according to the 5 + 1 scheme.
Rear view of the basket
A good idea - regardless of the modules used, the set of ports on the basket is the same. Two SAS ports, two SFP + ports and two RJ45 ports are always displayed, and what will be connected to them from the second side is determined on the spot.
Format Comparison
| Open rack | Open cloudserver | Standard 19 "rack | |
| Internal width | 21 ” | nineteen" | nineteen" |
| Rack height | 2100 mm (41 OU) | 48-52U, 12U / Chassis | 42-48U |
| OU / RU | OU (48 mm) | RU (44.45 mm) | RU (44.45 mm) |
| Cooling | In the system (HS / Fixed) | Wall fans (6 pieces) | In system |
| Fan size | 60 mm | 140 mm | Varies |
| Power bus | 3 pairs | - | - |
| Power Zones / Budget | 3, budget up to 36 kW | 4, budget up to 28 kW | - |
| Power shelf | In front | Behind | - |
| Power shelf size | 3 OU | 0 U | 0 U |
| Built-in UPS | Yes | Yes (coming soon) | No (but no one bothers to deliver) |
| Rack management | Yes, RMC (but not fully resistant) | Yes CM | - |
| Blade Identification (Sled ID) | Not | Yes (in the chassis) | Not |
| Cables | In front | Behind | Front or back |
Cooling system efficiency
| Total system consumption of 7200 W (24 blades x 300 W) | ||||
| External temperature (° C) | Fan speed (%) | Fan consumption (W) | Fan consumption (W) | The percentage of consumption from the rack (%) |
| 25 | 66 | 19.2 | 115.2 | 1.6 |
| 35 | 93 | 40.2 | 241.2 | 3.35 |
Control system
Chassis manager
The management server is inserted and extracted “on hot”, the software is built (not surprisingly) on Windows Server 2012. The source code is available to the general public and everyone can make their own edits (the toolkit is free).
general idea
Directly fee
Functional:
- Input / Output (I / O):
- 2 x 1GbE Ethernet (to access the network or manage ToR switches)
- 4 x RS-232 (Connect to switches to control their loading)
- Power management (one input, three outputs for connecting a PDU or another CMM) - Windows Server 2012
- Hot swap module
- Iron:
- Built-in Atom S1200 processor
- 4GB memory with ECC
- 64GB SSD - TPM module as an option
- Built-in multiplexers for blade communications
- Blade power control is derived separately (more reliable)
Tire routing
How does it go through the connectors
Not forgotten, and the software part:
Logic circuit
Out-of-band part
- CM has a REST-like API and a CLI command line for scaling the management system
- Fan Control / Monitor
- PSU Control / Monitor
- Chassis Power Management ON / OFF / Reset
- Serial Console Redirect
- Blade Identify LED
- TOR Power Control ON / OFF / Reset
- Power capping
In-Band part
- BMC blades work with standard IPMI over KCS interface
- Windows - WMI-IPMI
- Linux - IPMItool
- Identification, Chassis Power, Event Logging, Temperature Monitor, Power Management, Power Capping, HDD and SSD monitoring.
Facebook relies on Intel ME capabilities, Microsoft is using the familiar
IPMI using BMC.
Blades
Two types of blades are developed, one with a server, the other JBOD for 10 discs.
Possible combinations
You can combine, of course, as you like — even two servers, at least both JBODs.
Cables stay in place
The cables are connected to the backplane, so to replace the blade, simply remove the basket (a significant difference from OCP). This approach reduces maintenance time and eliminates possible error when reconnecting cables.
| Specification | OCS server |
| CPU | 2x Intel Xeon E5-2400 v2 per node, up to 115 W |
| Chipset | Intel C602 |
| Memory | 12 DDR3 800/1066/1333/1600 ECC UDIMM / RDIMM / LRDIMM slots per server |
| Discs | 4 3.5 "SATA per server |
| Expansion slots | [options = "compact"] * 1 PCIe x16 G3 * SAS mezz * 10G mezz |
| Control | BMC-lite connected to Chassis Manager via I2C bus |
PCIe slot is designed for a very specific purpose - the use of PCIe Flash cards.
So divorced mezzanine boards
| Specification | OCS JBOD |
| Controllers | 1 SAS Interface Module (SIM) |
| External ports | 2 6Gb / s mini-SAS ports |
| Discs | 10 3.5 "SAS / SATA hot-swappable whole basket |
| Control | SCSI Enclosure Service (SES-2) |
Disk shelf
Network infrastructure
All network connections are 10G, gigabit is used only for control and not everywhere. The important point is that Microsoft is very actively working on popularizing Software Defined Networking (SDN) technologies, their own services are based on software-configured networks.
Not for nothing on the past the Ethernet Summit their SDN monitoring product, DEMon, received six of the six possible awards , and the company is on the list of OpenDaylight Platinum Sponsors .
At the same time, we remind you that we have long announced a product for hybrid networks with SDN support - an Eos 410i switch with a port cost below $ 100 :)
Total
The largest companies in the industry rely on Software Defined Datacenter and Microsoft is no exception. The development of Ethernet technologies has led to the fact that 10G and RDMA (using RDMA over Converged Ethernet, RoCE) make it possible to do without a separate FC network (the gain from RDMA was discussed here: habrahabr.ru/company/etegro/blog/213911 ) without loss in performance . Windows Storage Spaces capabilities are such that hardware storage systems are gradually being replaced by solutions on Shared DAS / SAS networks (it was written in detail here habrahabr.ru/company/etegro/blog/215333 and then habrahabr.ru/company/etegro/blog/220613 ).
MS advantages:
Chassis-based design reduces cost and power consumption
- Standard rack EIA 19 "
- Modular design simplifies deployment: mounted sidewalls, 1U baskets, high performance power supplies, large fans for efficient control, hot-swappable control board
- Up to 24 servers per chassis, optional JBOD shelf
- Optimized for contract manufacturing
- Reduced costs up to 40% and 15% better energy efficiency compared to traditional enterprise servers
- Expected savings of 10,000 tons of metal per million servers (there is clearly bent, 10 Kg per server a bit too much)
All signal and power lines are transmitted through fixed connectors.
- Separating server and chassis simplifies deployment and recovery
- Cable-free design reduces human error during maintenance
- Reducing the number of incidents caused by loose cables
- Reduced deployment and maintenance time reaches 50%
Network and SAS cables on a single backplane
- Passive PCB simplifies design and reduces the risk of loss of signal integrity
- Various types of network and cables, 10 / 40G Ethernet and copper / optical cables are supported.
- The cables are routed to the chassis once, during assembly.
- Cables do not touch during operation and maintenance.
- Save hundreds of miles of cable on every million servers.
Secure and scalable management system
- Server in each chassis
- Multiple security levels: TPM at boot, SSL transport for commands, role-based authentication from AD
- REST API and CLI for scaling management system
- 75% more convenient to use than regular servers
Conclusion
The invention of the next incarnation of blades took place. Key difference from the rest
- fully open platform specifications and use of standard network switches. In traditional blades from a well-known circle of companies, the choice of the network part is limited by the imagination of the developers; in an open solution, you can use any of the existing products, from used Juniper to SDN switches of its own design.
Well, we presented a solution based on Open CloudServer in the product line .
Source: https://habr.com/ru/post/222625/
All Articles