132 servers in a rack or how to pack up and not burst

Two years ago, I first told a blog about the DELL servers that we use in our work. It's time to talk about the new equipment, during this time a lot has changed. The post did not want to be written for a long time, the photos had time to dust, but it is better late than never.

In this post, I will tell you about the new DELL PowerEdge C series servers (C from Cloud), as well as how, thanks to new equipment, we doubled the capacity of our site in the DC and at the same time kept the operating costs at the same level.

')

Dull non-rubber world (patience, photos further)

About a year ago, Rusonix was seriously faced with the problem of infrastructure growth. We put the servers in a cool, but very expensive DC (Comcor / Akado on Warsaw). Each rack is like an airplane, but the platform allows you to allocate a lot of electricity, up to 10-15 kW per rack. For comparison, in a typical Moscow economy-class data center, between 3 and 5 kW is allocated per rack. There are other advantages, but this is not about now.

So, for us it was very expensive to grow extensively, in breadth. In addition, in all that we do, we try to use available resources as efficiently as possible. Moving to an inexpensive DC is not our method; corporate clients simply won’t understand. In addition, we really wanted to launch a project for the rental of branded Della servers.

Thus, we faced the following tasks:

• infrastructure to ensure the growth of the customer base;
• start a project for renting DELL servers;
• try to avoid increasing the fixed operating costs (rack rental and server maintenance).

We implemented the zero option immediately - we threw the old stuff into the garbage.



There were several serious solutions:

1. moving to an inexpensive data center;
2. partial transfer of non-critical services to an average data center with reasonable prices;
3. compaction of equipment at the existing site.

The first option was not seriously considered - too high costs, risks and unpredictable results.

Having considered the second option and weighing all the pros and cons, we came to the conclusion that the game is not worth the candle. There are more points of failure, risks are increasing, fixed costs are increasing.

There was a scenario with the compaction of equipment on the existing site. Unfortunately, rubber racks have not yet been delivered to our data center. I had to work with what is. In the rack 42 units - no more, no less. And we started working on Fridays - shuffling servers between racks, replacing switches and other interesting, but not very efficient things.

Happily for us, DELL in 2011 announced the PowerEdge C5220 and C6100 servers. And then the fun began.

Thank God you came (pictures even further)

C5220 is 8 or 12 servers / traces (sled in DELL terminology) of level R210 in one 3U package.

Photos from dell.com

The C6100 is 4 servers of the R610 level in one 2U package (spoiler: a beast, not a machine. The C6100 deserves a separate post).

Photos from dell.com

We accidentally learned about the imminent release of the new line and immediately asked the Russian representative for information about the possibility of supplying such servers. Servers are not that not available - they were not even on the US site DELL. And of course, the platforms were not certified in the Russian Federation. Long months of waiting for prices, quotas for delivery and certification have been pulled.

The Russian DELL worked very quickly and here in November the first servers on the C5000 platform came to us. One of the first in the world and the first in the Russian Federation. Needless to say, the miracle glands have gathered around us both our engineers and representatives of Wirth, our distributor.

The C5000 is a case with central power and cooling. "Cartridges" are inserted into the case - traces. There may be 8 or 12. Now we use only 12-trace configuration. In 8-track, you can install more powerful processors and expansion cards. The first batch of 8-tracks has already passed customs and will be with us from day to day.



Each track is a separate server with its own disks, memory, network interface and KVM.



There is only one processor slot: Xeon E3 series with TDP up to 65W (up to 95W in 8-trace configuration) are supported. We put E3-1260L, in our opinion, this is the most optimal processor from the modern E3 line, combining low power consumption, high performance (4 cores, 8 threads, TurboBoost from 2.4 to 3.3 GHz) and reasonable cost.
There is a similar platform for AMD processors, but we don’t use the products of this vendor, so I can’t say anything about them.



4 memory slots are recommended to be fully occupied for optimal cooling and speed. The maximum server supports up to 32GB DDR3 ECC UDIMM (1333 MHz).



The track has room for two 3.5-inch disks or four 2.5-inch disks. We put on two large disks. Installation of four disks for similar applications seems to us rather wasteful and not rational. But if there is a demand, we will consider purchasing such configurations.



Each server has two network interfaces. One we use to communicate with the outside world, and the second for the backnet. The backnet is needed for our management system: we install and configure software on it, migrate users and, if necessary, make backups. In addition, the chassis has a single control center that performs the role of KVM to access the low-level settings of each of the servers (IPMI 2.0, Virtual Media, and other remote administration delights). And of course, through the center, it is possible to remotely monitor the technical condition of all hardware installed in the chassis.



Despite the wide possibilities of remote administration, it is possible that you need to physically connect to any of the servers. To do this, each of them is equipped with a D-SUB connector and two USB 2.0 ports. Since the front panel area is extremely small, it was very important to leave more space for ducts. Therefore, all connectors are integrated into one universal.



It's all about the magic postings.



Traces are inserted into the chassis about as cartridges in the baby consoles of the early 90s.



Through these connectors is the power supply and data exchange with the control center.



At the rear of the chassis, N + 1 redundant cooling units are installed. In fact, one row of coolers is enough for cooling. The second is for reservations.



Power is supplied through two hot-swappable units, one of them is backup. The power supply units are connected to two different beams, the benefit of the data center allows it. Thus, the servers are protected both from internal failure - the failure of one of the units, and from an external factor - power off on one of the beams.



The attentive reader will notice how cool the power supply lock is. The unit cannot be pulled out without lowering the dog locks, and the handle for lowering the locks (green in the photo) is locked until the power supply is de-energized.



For us, the C-series servers have become a great gift from DELL engineers, since they allowed us to solve the complex problem of compacting and cheapening the infrastructure while maintaining the quality of services. On the one hand, such servers are much cheaper to operate and maintain, and on the other hand, they provide greater reliability and redundancy than conventional single-link DELL PowerEdge R210 with similar characteristics.

The servers worked perfectly with our Parallels Automation management system and Parallels Virtuozzo Containers (PVC) virtualization. Thus, we were able to integrate them with our existing infrastructure and use ready-made sets of customized software for installation (control panel, fine-tuning for various CMS, etc.). As a result, accounts on the new VPS of the 2012th tariff year when upgrading the server rental are automatically migrated from the VPS node to a separate machine. Transparent migration between servers is a separate interesting topic, which I will definitely discuss in one of the following posts.

What is the result?

Thanks to the c5000 platform, the density of servers for rent in conventional 42U cabinets was 132 servers per cabinet (more than 3 times higher than usual) . In addition, using servers on the C6100 platform (now C6220), we increased the density of production servers used for virtualization by 2 times.

Taking into account our standard equipment DELL PowerEdge R-series, still in operation, over the year we increased the number of physical and virtual servers by more than 2 times and at the same time avoided the growth of fixed costs for operating a data center.

When the article was already written, I was surprised to find that we do not have any good full-length photo of the track. That's all that was found.



Be sure to re-photograph when the next batch comes.

PS If you need someone, you can rent a server from us.