PhosAgro: data center for four months
Today we would like to tell you about one of the latest projects to create a corporate data center. This is the Cherepovets Datacenter of PhosAgro, whose main activity is the production of phosphate fertilizers, phosphate raw materials, as well as feed phosphates, nitrogen fertilizers and ammonia. PhosAgro Group includes Apatit JSC and its Balakovo branch, PhosAgro-Cherepovets JSC, Metakhim JSC, PhosAgro-Trans LLC, PhosAgro-Region LLC and NIUIF JSC. The group is the largest European producer of phosphate fertilizers, as well as the world's largest producer of high-grade phosphate raw materials.
The main goals of creating a new data center were centralization of the PhosAgro server complex in one place, as well as the reorganization of the network core. The project was implemented in just four months, and, according to the classical technology, with the overhaul of the premises. How did you manage to achieve this? First of all, thanks to an efficient team, timely delivery of equipment, coordinated installation of various subsystems. And, of course, a competent choice of technical solutions helped. Details below.
Before dwelling on the construction of its own data center using the classic technology, PhosAgro experts also considered other options for solving existing IT problems. One of them is the use of services of commercial data centers. But after analyzing the proposals of several providers, we realized that high operating costs (OPEX) lead to the fact that this option is unprofitable (above TCO) with a large planning horizon. In addition, the concentration of the KTsOD in Moscow and St. Petersburg meant the need for a difficult and expensive transfer of the server complex to one of the capitals - one more minus of such an option.
Next, consider the possibility of installing a container data center. However, the benefit was not obvious due to technical circumstances and the fact that there are enough resources in the existing building, and its modernization was not so expensive. Such a solution is required in cases where there is a shortage of physical space for equipment placement or the possibility of relocating it to another place is assumed - neither is relevant for PhosAgro.
')
As a result, it was decided to quickly create a data center in a three-storey IT building in the territory of PhosAgro.
The main criterion for creating a data center was fault tolerance, so it was decided to purchase all the main engineering equipment from one vendor. The monovendor solution also ensures 100% compatibility of all components of the engineering infrastructure, allowing to reduce the cost of operation and maintenance. It eliminates disputes between contractors about the causes of problems and contributes to their prompt resolution. Another advantage of the mono-vendor data center is a single service contract. It is easier to react to changes and deviations from the norm - the vendor discovers the reasons, he also makes a decision, for example, on the replacement of the failed component. No need to look for, on whose side the problem.
Thus, PhosAgro considered only mono-vendor solutions and almost immediately the Schneider Electric offer was chosen. The quality and accumulated positive operating experience of Schneider Electric solutions contributed to this decision. Schneider Electric equipment is successfully used at the PhosAgro site in Kirovsk, and the staff has specialists trained by this company.
At the next stage, a tender was held, in which seven major integrators with high Schneider Electric partner status and experience in implementing similar projects participated. Won the company IBS Platformix, which offered the best price and timing of the project.
The project needed to be implemented in a short time, because the state of engineering systems in existing server rooms no longer allowed to provide the required reliability indicators. Thanks to the well-coordinated work of a team of specialists of the customer, manufacturer and integrator, efficient delivery and installation of equipment, the data center was built and put into operation in a record short time. The results of the tender were summed up in August 2016, and the main work on the project was completed before the end of winter.
Interestingly, the work of specialists of different profiles in the construction of the data center were carried out simultaneously. For example, the delivery time of chillers was coordinated with the timing of construction and installation works - the equipment was installed in its place literally “from the wheels”, without additional movements around the object. A clear synchronization of work together with representatives of the vendor has reduced the time for their implementation.
Despite the operational creation of a new site, the PhosAgro data center cannot be called typical. It consists of two rooms, one of which is designed for standard IT equipment, the second for racks where data storage systems and servers of non-standard form factor are located. In the first hall, there were 28 racks (6 racks of 7 kW each, the rest 3.5 kW each), and the insulation of the hot corridor was implemented.
EcoAisle Hot Corridor Air Insulation System
In the second hall, where the IT equipment of non-standard dimensions is located, a slightly different scheme is implemented: a cold corridor is also separated between two rows of racks, but it is not isolated. The area of ​​the data center allows to place all the equipment available in the Group and leaves a reserve for growth.
The estimated load of the data center for IT equipment is 250 kW, the same is consumed by engineering systems. The power supply to the data center is supplied from a medium voltage substation with a total power of 1 MW (two transformers of 500 kW each). At the site, two 500 kW UPSs are deployed according to a mutual backup scheme and can power not only IT, but also critical elements of the engineering infrastructure, and the modular architecture eliminates the presence of a single point of failure. Installed devices Symmetra PX is a modular uninterruptible power supply system with high efficiency - 95.6%, which reduces the power costs of both the UPS and the air-conditioning system serving it (for this purpose, a standard freon split-system was used).
Symmetra PX Schneider Electric modular three-phase uninterruptible power system
For a short-term autonomous power supply for the IT load, a set of rechargeable batteries is installed, designed to provide an autonomous power supply of 10-15 minutes - this time is sufficient to guarantee the start of the diesel. But while the data center is not loaded at full capacity, the batteries are able to guarantee autonomy for more than an hour. DGU is used as a backup source.
Another important requirement for the new data center has become high energy efficiency. The PhosAgro enterprise in Cherepovets is well supplied with energy, therefore there were no problems with obtaining the required power (although careful “cleaning” of the electricity coming from the plant was required). Energy efficiency was important, not so much because of the reduction in energy consumption, as for the sake of reducing the impact on the environment. This is all the more symbolic, since 2017 is the year of ecology in Russia. IT managers of PhosAgro wanted to prove that you can quickly and efficiently build a reliable and energy-efficient facility.
Energy efficiency is influenced by many factors, including, in particular, the efficiency of installed UPS systems. But the main contribution to reducing energy consumption makes the active use of natural cooling (free cooling). The new data center has an innovative chilling system Aquaflair, in which the function of free cooling is involved from October to April. In this mode, only the circulating pump, pumping the coolant (ethylene glycol), works, at the same time consumes a minimum of electricity.
Chillers Schneider Electric Uniflair Series TRAF with free cooling function
To improve the cooling efficiency, the already mentioned EcoAisle hot corridor insulation system and InRow in-row air conditioners in a compact package (only 300mm wide) are provided. The cooling capacity of this air conditioner is up to 62 kW with a power consumption of 1.8 kW. Thus, the efficiency ratio of the air conditioner is more than 34 kW of cold per 1 kW of electricity.
In-row air conditioning system InRowtm APC by Schneider Electric
The new data center is built in accordance with the requirements for reliability and fault tolerance imposed on Tier III objects, which, in particular, allows you to disable the components of the engineering infrastructure for their preventive maintenance and repair without interrupting the work of the data center.
For some of the key components, even triple redundancy is provided. For example, three chillers are installed: one of them is in operating mode, the second is in standby, and the third is in standby mode. Every few hours they change roles. This increases the fault tolerance and ensures a uniform development of equipment life. Any unit can be taken out of service for maintenance without disrupting the operation of services.
In addition, the concept of building the PhosAgro IT system envisaged the creation of two geographically dispersed data centers for disaster recovery - in Cherepovets and Kirovsk (Murmansk region). The site in Kirovsk, now backup data center was built three years ago. It is similar to the new, basic, but it uses cooling systems on the freon. In the data center of Kirovsk there are also two halls, but they are designed for a smaller number of racks. Currently, PhosAgro IT systems are separated by these two data centers.
StruxureWare Data Center Expert is used to monitor engineering systems in a new data center. It is a scalable monitoring system that is independent of the equipment of a specific manufacturer, which collects data from various sensors and “intelligent” devices, provides alarm generation and other functions, forming a unified view of the complex physical infrastructure of the facility.
The centralized management system for all engineering infrastructure objects provides end-to-end monitoring of all levels of engineering systems and allows to increase the efficiency of operation services. StruxureWare not only monitors various environmental parameters, but also stores the information obtained for later analysis.
Even before the main tender, at the design stage, Schneider Electric experts carried out an expert review of the project, made their comments, prepared equipment specifications. Representatives of the manufacturer were present at the site at all stages: installation of a hot corridor, electricity and cooling systems, including the most difficult stage of commissioning and commissioning equipment.
Schneider Electric promptly completed installation supervision, conducted equipment testing. Under the contract, she will serve both sites for three years. The vendor conducts regular inspections of equipment (cooling and power supply systems) for scheduled preventive maintenance. At a predetermined time, a Schneider Electric engineer conducts a comprehensive survey of cooling systems twice a year and power supply systems once a year.
In StruxureWare, notifications are set up to send alarm messages to the Help Desk, cases start up automatically, and the engineer quickly takes action. All components of engineering systems conduct self-diagnostics on a regular basis, and any errors in their work are displayed in the monitoring system, which allows you to immediately respond to emergency situations.
In recent years, PhosAgro has invested heavily in IT development, automation of production and business. The centralization of IT resources in Cherepovets contributed to the implementation of such projects as a single electronic archive with the exchange of documents between the holding sites. Business-critical applications now run on separated clusters. The venues are connected by a reserved 400 Mbit / s channel. User access in the new data center is organized at a speed of 1 Gbit / s, and for other company sites in the five largest cities of Russia - 200 Mbit / s.
It is clear that the processes of “digital transformation” are impossible without modern technologies. And with the help of the platform provided by the Schneider Electric engineering infrastructure for the PhosAgro data center, reliable and efficient functioning and sustainable development of IT systems is implemented. The data center has allowed IT equipment to provide high-quality uninterrupted power supply, to create the necessary climatic conditions for its operation. Now the data center load on power supply is only 15-20% - engineering systems were designed with a view to the future growth of the server park in the next 5-7 years.
The main goals of creating a new data center were centralization of the PhosAgro server complex in one place, as well as the reorganization of the network core. The project was implemented in just four months, and, according to the classical technology, with the overhaul of the premises. How did you manage to achieve this? First of all, thanks to an efficient team, timely delivery of equipment, coordinated installation of various subsystems. And, of course, a competent choice of technical solutions helped. Details below.
Option Analysis
Before dwelling on the construction of its own data center using the classic technology, PhosAgro experts also considered other options for solving existing IT problems. One of them is the use of services of commercial data centers. But after analyzing the proposals of several providers, we realized that high operating costs (OPEX) lead to the fact that this option is unprofitable (above TCO) with a large planning horizon. In addition, the concentration of the KTsOD in Moscow and St. Petersburg meant the need for a difficult and expensive transfer of the server complex to one of the capitals - one more minus of such an option.
Next, consider the possibility of installing a container data center. However, the benefit was not obvious due to technical circumstances and the fact that there are enough resources in the existing building, and its modernization was not so expensive. Such a solution is required in cases where there is a shortage of physical space for equipment placement or the possibility of relocating it to another place is assumed - neither is relevant for PhosAgro.
')
As a result, it was decided to quickly create a data center in a three-storey IT building in the territory of PhosAgro.
Vendor selection
The main criterion for creating a data center was fault tolerance, so it was decided to purchase all the main engineering equipment from one vendor. The monovendor solution also ensures 100% compatibility of all components of the engineering infrastructure, allowing to reduce the cost of operation and maintenance. It eliminates disputes between contractors about the causes of problems and contributes to their prompt resolution. Another advantage of the mono-vendor data center is a single service contract. It is easier to react to changes and deviations from the norm - the vendor discovers the reasons, he also makes a decision, for example, on the replacement of the failed component. No need to look for, on whose side the problem.
Thus, PhosAgro considered only mono-vendor solutions and almost immediately the Schneider Electric offer was chosen. The quality and accumulated positive operating experience of Schneider Electric solutions contributed to this decision. Schneider Electric equipment is successfully used at the PhosAgro site in Kirovsk, and the staff has specialists trained by this company.
At the next stage, a tender was held, in which seven major integrators with high Schneider Electric partner status and experience in implementing similar projects participated. Won the company IBS Platformix, which offered the best price and timing of the project.
In a short time
The project needed to be implemented in a short time, because the state of engineering systems in existing server rooms no longer allowed to provide the required reliability indicators. Thanks to the well-coordinated work of a team of specialists of the customer, manufacturer and integrator, efficient delivery and installation of equipment, the data center was built and put into operation in a record short time. The results of the tender were summed up in August 2016, and the main work on the project was completed before the end of winter.
Interestingly, the work of specialists of different profiles in the construction of the data center were carried out simultaneously. For example, the delivery time of chillers was coordinated with the timing of construction and installation works - the equipment was installed in its place literally “from the wheels”, without additional movements around the object. A clear synchronization of work together with representatives of the vendor has reduced the time for their implementation.
"Anatomy" of the data center
Despite the operational creation of a new site, the PhosAgro data center cannot be called typical. It consists of two rooms, one of which is designed for standard IT equipment, the second for racks where data storage systems and servers of non-standard form factor are located. In the first hall, there were 28 racks (6 racks of 7 kW each, the rest 3.5 kW each), and the insulation of the hot corridor was implemented.
EcoAisle Hot Corridor Air Insulation System
In the second hall, where the IT equipment of non-standard dimensions is located, a slightly different scheme is implemented: a cold corridor is also separated between two rows of racks, but it is not isolated. The area of ​​the data center allows to place all the equipment available in the Group and leaves a reserve for growth.
The estimated load of the data center for IT equipment is 250 kW, the same is consumed by engineering systems. The power supply to the data center is supplied from a medium voltage substation with a total power of 1 MW (two transformers of 500 kW each). At the site, two 500 kW UPSs are deployed according to a mutual backup scheme and can power not only IT, but also critical elements of the engineering infrastructure, and the modular architecture eliminates the presence of a single point of failure. Installed devices Symmetra PX is a modular uninterruptible power supply system with high efficiency - 95.6%, which reduces the power costs of both the UPS and the air-conditioning system serving it (for this purpose, a standard freon split-system was used).
Symmetra PX Schneider Electric modular three-phase uninterruptible power system
For a short-term autonomous power supply for the IT load, a set of rechargeable batteries is installed, designed to provide an autonomous power supply of 10-15 minutes - this time is sufficient to guarantee the start of the diesel. But while the data center is not loaded at full capacity, the batteries are able to guarantee autonomy for more than an hour. DGU is used as a backup source.
Energy efficiency
Another important requirement for the new data center has become high energy efficiency. The PhosAgro enterprise in Cherepovets is well supplied with energy, therefore there were no problems with obtaining the required power (although careful “cleaning” of the electricity coming from the plant was required). Energy efficiency was important, not so much because of the reduction in energy consumption, as for the sake of reducing the impact on the environment. This is all the more symbolic, since 2017 is the year of ecology in Russia. IT managers of PhosAgro wanted to prove that you can quickly and efficiently build a reliable and energy-efficient facility.
Energy efficiency is influenced by many factors, including, in particular, the efficiency of installed UPS systems. But the main contribution to reducing energy consumption makes the active use of natural cooling (free cooling). The new data center has an innovative chilling system Aquaflair, in which the function of free cooling is involved from October to April. In this mode, only the circulating pump, pumping the coolant (ethylene glycol), works, at the same time consumes a minimum of electricity.
Chillers Schneider Electric Uniflair Series TRAF with free cooling function
To improve the cooling efficiency, the already mentioned EcoAisle hot corridor insulation system and InRow in-row air conditioners in a compact package (only 300mm wide) are provided. The cooling capacity of this air conditioner is up to 62 kW with a power consumption of 1.8 kW. Thus, the efficiency ratio of the air conditioner is more than 34 kW of cold per 1 kW of electricity.
In-row air conditioning system InRowtm APC by Schneider Electric
Failure and disaster recovery
The new data center is built in accordance with the requirements for reliability and fault tolerance imposed on Tier III objects, which, in particular, allows you to disable the components of the engineering infrastructure for their preventive maintenance and repair without interrupting the work of the data center.
For some of the key components, even triple redundancy is provided. For example, three chillers are installed: one of them is in operating mode, the second is in standby, and the third is in standby mode. Every few hours they change roles. This increases the fault tolerance and ensures a uniform development of equipment life. Any unit can be taken out of service for maintenance without disrupting the operation of services.
In addition, the concept of building the PhosAgro IT system envisaged the creation of two geographically dispersed data centers for disaster recovery - in Cherepovets and Kirovsk (Murmansk region). The site in Kirovsk, now backup data center was built three years ago. It is similar to the new, basic, but it uses cooling systems on the freon. In the data center of Kirovsk there are also two halls, but they are designed for a smaller number of racks. Currently, PhosAgro IT systems are separated by these two data centers.
Monitoring and Management
StruxureWare Data Center Expert is used to monitor engineering systems in a new data center. It is a scalable monitoring system that is independent of the equipment of a specific manufacturer, which collects data from various sensors and “intelligent” devices, provides alarm generation and other functions, forming a unified view of the complex physical infrastructure of the facility.
The centralized management system for all engineering infrastructure objects provides end-to-end monitoring of all levels of engineering systems and allows to increase the efficiency of operation services. StruxureWare not only monitors various environmental parameters, but also stores the information obtained for later analysis.
One contract, one area of ​​responsibility
Even before the main tender, at the design stage, Schneider Electric experts carried out an expert review of the project, made their comments, prepared equipment specifications. Representatives of the manufacturer were present at the site at all stages: installation of a hot corridor, electricity and cooling systems, including the most difficult stage of commissioning and commissioning equipment.
Schneider Electric promptly completed installation supervision, conducted equipment testing. Under the contract, she will serve both sites for three years. The vendor conducts regular inspections of equipment (cooling and power supply systems) for scheduled preventive maintenance. At a predetermined time, a Schneider Electric engineer conducts a comprehensive survey of cooling systems twice a year and power supply systems once a year.
In StruxureWare, notifications are set up to send alarm messages to the Help Desk, cases start up automatically, and the engineer quickly takes action. All components of engineering systems conduct self-diagnostics on a regular basis, and any errors in their work are displayed in the monitoring system, which allows you to immediately respond to emergency situations.
In recent years, PhosAgro has invested heavily in IT development, automation of production and business. The centralization of IT resources in Cherepovets contributed to the implementation of such projects as a single electronic archive with the exchange of documents between the holding sites. Business-critical applications now run on separated clusters. The venues are connected by a reserved 400 Mbit / s channel. User access in the new data center is organized at a speed of 1 Gbit / s, and for other company sites in the five largest cities of Russia - 200 Mbit / s.
It is clear that the processes of “digital transformation” are impossible without modern technologies. And with the help of the platform provided by the Schneider Electric engineering infrastructure for the PhosAgro data center, reliable and efficient functioning and sustainable development of IT systems is implemented. The data center has allowed IT equipment to provide high-quality uninterrupted power supply, to create the necessary climatic conditions for its operation. Now the data center load on power supply is only 15-20% - engineering systems were designed with a view to the future growth of the server park in the next 5-7 years.
Source: https://habr.com/ru/post/339290/
All Articles