Secrets of the effective management of data centers around the world: from Tokyo to St. Louis, from Sydney to London
We at Acronis protect data from more than 5 million users and 500,000 companies in 150 countries. These are dozens and hundreds of petabytes of data that are stored in our data centers located in Tokyo and St. Louis, in Frankfurt am Main and Sydney, in Moscow and London. In total, our data is located in 14 data centers located in different countries, different time zones and in different parts of the world. All this "farm" every day must be managed. And I must admit that this is an extremely interesting case, and therefore we decided to share with you our experience in this matter, and prepared a small guide for the novice data center manager.
The working day of the head of the department should begin with checking the status of the work of its data center. If you have only one server in one data center, this should not be a big problem. But if we are talking about several data centers in several countries of the world, it will be a bit more complicated. At Acronis, we use automated systems and dashboards that allow real-time tracking of what is happening in each of the data centers, evaluate statistics on their completion, and based on this, adapt your task list to the day as needed. First of all, it is necessary to check the current state of the network, the state of the server equipment and its loading. Another very important parameter is the growth rate of server load. Knowing it, you can avoid such a common problem as the drop of servers under load, and properly plan the commissioning of new equipment. In general, kapasiti planning at our growth rates becomes a very nontrivial task that requires creative approach and daily attention.
')
After you are convinced that everything is in order with your data center and there are no urgent tasks, you can proceed to “administrative issues”, deal with the accumulated e-mail and make important phone calls. We cannot say that morning is the most ideal time for this, but for a good manager it is important to always be in touch, promptly respond to requests and requests from your colleagues, keep in touch with suppliers and follow the latest news from the professional field.
When the “administrative tasks” are finished, it’s time to move on to the execution of the task list formed at the beginning of the day. Plan updates, order new equipment, form requests for management companies of data centers, etc. Usually, tasks from the list take place all day, but if you work in a global company and / or you have servers in other time zones, for example, in North America, after lunch, when the Western Hemisphere wakes up, you will come across “ administrative tasks. " You answer mail and phone calls again, synchronize your actions with colleagues from other countries, and after this, as a rule, the second half of the day passes.
Sometimes, especially when you have employees who are in the minus ten hours from you, the second half of the day may go unnoticed on the first day of the next. And what to do, the attention of the staff, the more distant, must always be, otherwise the spirit spirit may weaken.
Something like this is the routine of the data center manager: checking equipment, planning money, parsing mail and phone calls, working on current tasks, and again picking up mail and phone calls.
But not always everything goes according to the plan described above, sometimes various unpleasant situations happen, such as server crashes, DDOS attacks, and other amenities of the modern tech world. If your company has a well-designed infrastructure, with at least one backup critical element, like Acronis, and you use backup and disaster recovery systems, then most likely it will help you avoid fatal problems and quickly get out of the existing difficulties. . If the well-thought-out infrastructure and backup systems are not about you, then “we are singing a song to the madness of the brave,” you will have to get a pair of gray hair.
First of all, you need to try to “revive” the server remotely, IPMI to help. If the server cannot be restored remotely, then you have no choice but to write a request to the technical support of the data center, which will explain in as much detail and as simple as possible what happened. It often happens that technical support responds to such calls in an hour or two, so if this data center is close to you and it’s vital to restore its performance, then you should go there yourself and solve all the problems on the spot yourself (if you don’t thought about a good support contract).
If the data center is located in another region or another country, and it’s not possible to quickly reach it yourself, then you can follow our example and sign a contract with external specialists who will become your eyes and hands. By placing our equipment in a new data center in another area or country, we always try to find an external specialist who, in the event of an emergency situation, can quickly get to the desired data center and eliminate the problem that has appeared. We are looking for such people among the leaders of small local IT companies who are technically savvy enough and can independently diagnose and solve such problems.
When the infrastructure is critical for your company and your business, the data center does not rely on the technical support of the data center, as I noted above, it takes too long to wait for an answer, and there are times when it is wrong to understand what it is from they were required, they only made it worse. Therefore, we recommend finding such external specialists, concluding all necessary agreements with them (contract, NDA, etc.), and keeping in touch with them. You should take them as insurance, not at all the fact that you will need their services. However, if such a situation occurs, they will save you time and nerves.
We faced situations when it was necessary to promptly replace the hard disk in the server or install new hardware. The standard time for performing such requests by a data center is usually hours or even days, and with the help of an external specialist, we carried out these actions within one hour. And this is very important for us, because when your services are used by several million people, time is the decisive factor. Even minutes go by.
In addition to the time that you always want to speed up, at the global level, standardization and unification of the used hardware and software greatly helps. It sounds simple and easy, but in fact, in situations where the company is growing and developing rapidly, is actively involved in M ​​& A transactions, the support of a unified infrastructure is a difficult task. But if the task is completed, then the overall manageability of global data centers becomes much more trouble-free.
And, of course, you should always remember about the triad that allows you to sleep at night - capacity planning, redandancy and backup. They are good, without them bad. The team, and the team in Acronis Data Center Operations is excellent, penetrates this immediately, and some of the elements of the triad even go into nicknames / nicknames. There are, for example, Vladimir nicknamed Redundancy. The entire network infrastructure that Vladimir is building in our data centers is fully consistent with the nickname)
The tasks of the head include not only maintaining the data center in working condition, but also participating in the procurement of various equipment and services for its operation. There is almost no information on specialized resources about how to find suppliers, negotiate with them and enter into contracts - this is delicate information, which often differs depending on the region, so it will not be superfluous to tell a little about it.
Based on my experience, I can say that the most important thing in procurement is building good relationships with suppliers. If you buy a lot and often, you can always count on more favorable conditions: at a price, by payment method, delivery, etc., rather than in a situation where you just come "from the street." So, for example, one of our regular suppliers provides us with the latest hardware testing, which we can test in the conditions we require and under the loads we require. After all, there are often cases when one equipment does not work correctly together with another, and buying hundreds of thousands of hard drives, costing many, many dollars, we expect them to work well on our servers and with our software
And since the cases are different, it is impossible not to appreciate such a relationship, when under a “honest word” a supplier almost overnight can send a server to another end of the world when it is very necessary (real case: from London to Tokyo).
Another good way to conclude a lucrative contract is the ability to openly identify your own equipment needs and confirm them. Remember that suppliers are always looking for opportunities for long-term cooperation, so they can more intelligently manage their own inventory without freezing money in a product, effectively form their financial flows and accumulate a base of regular customers. Therefore, if you can sufficiently accurately identify your needs for equipment and services for the long term, you can safely expect a good discount, which in some cases will be significantly higher than 50% of the retail price.
Personal contact and long-term contract is not the only way to conclude a lucrative contract, there is another, third, opportunity to get a discount. In the West, there is such a “magic phrase” - “target price”. What it is? Any manufacturer directly concludes contracts only with major distributors who already bring the goods to the market, where we are buying them. It will not be a secret for anyone that a distributor purchases a product from a manufacturer at a much lower price than it sells on the market, and the difference between the purchase price and the sale price forms its income. It’s not always possible to find out for what exact price the distributor buys the goods from the manufacturer, but, after a simple market analysis, you can determine the average cost of equipment and services, and from this determine your target price. Can it be lower than the market average? Of course, but by demanding a price that is 70-80% lower than the market average, you can simply offend the supplier and not achieve anything at all! Will a distributor sell your product price? Not necessarily, but you should always keep it in your head when you go to negotiate. At the negotiations, you will most likely have bargaining, after which you will be able to conclude a good contract. Practice shows that this works especially well with the prolongation of expiring contracts, when it is possible to revise prices especially effectively, reaching its “target price”.
Using these three techniques, you can always enter into profitable contracts and build long-term cooperation with your suppliers. You might think that only large companies can use these techniques, but this is not entirely true. Even a small start-up company, guided by the recommendations written above, will be able to conclude good contracts. Yes, maybe it will not be about 40-50% discounts, but she can get a 20% discount and lay the foundation for a good long-term relationship.
The last topic on the account, but not least, is the issue of IT innovation. In times of crisis, companies most often begin to reduce costs from IT, and at such times the demand for rational use of IT infrastructure increases dramatically. And if the head of the data center independently makes a rationalization proposal to the management of the company, it will be only a plus for you.
First, let's see what is the rational use of IT infrastructure. As I noted above, any infrastructure must have “excess” capacities, such as to cover ever-increasing demands for capacity. But this “surplus” will be necessary at some future moment in time, and when exactly it will come is not always known. Some companies follow the simplest path, buying “here and now” the most top-level hardware, in the hope that in the next year or two they will be able to use it to the full. A year goes by, new, more sophisticated “glands” come out, the purchased equipment is morally obsolete, but still far from its full load. The second year passes, the equipment has time to become obsolete already and physically, but again the load is not complete. It turns out that during these two years the company spent extra money on equipment and maintaining it in working condition, and as a result could not use it under the “full program”. An additional problem could be the fact that, having bought the very top right away once and “with a reserve”, the company could not get a discount by telling about its needs for the future, which we talked about earlier. How to rationalize this situation? First of all, daily monitor the needs for new capacities and build graphics. Yes, yes, the same kapasiti planning, not once mentioned. It is true, true and relevant for everything related to data centers: communication channels, utilization of all resources, all equipment according to its roles and functions, firewall throughput (and suddenly your company decides to be certified by PCI DSS tomorrow, and you will suddenly be asked to enable IDS / IPS, which will reduce the throughput of the firewall by 3 times?). Very rarely, the need for power in a short period of time is doubled and kept at this level, as a rule, it grows gradually. Having made a schedule, considering what kind of equipment is needed now, how quickly it can be purchased and installed, and when you need to put new equipment into operation, you can go to the equipment suppliers, talk with them and enter into long-term contracts, following my recommendations above.
There is also another situation, when for different tasks a company uses similar, largely duplicating equipment. On the one hand, some especially “overcapacity” is not created, on the other hand, this equipment takes up space in racks and consumes electricity. It is in this situation that we appeared not so long ago. The two main Acronis products are Acronis Backup Cloud and Acronis Disaster Recovery. To provide services within these products, different “hardware” sets differing in their specifications are used. Despite this, there was an understanding that there is room for optimization, and we decided that it was necessary to compare the characteristics and specifications of these sets of hardware in each of our American data centers. Based on this analysis, we were able to identify four main specifications that can be used and that meet the main requirements: maximum resources per rack unit and the lowest possible power consumption. Since the 36-month depreciation cycle of existing equipment was coming to an end, we decided to centrally purchase equipment based on the specifications received and update our data centers in the United States by the end of the year. According to our calculations, the number of occupied places in the racks should be reduced by about 2 times (hooray, we reduce OPEX to the collocation!), And the amount of resources - storage, the RAM of the processor cores - increases significantly.
What I want to say at the end of this post. Managing a data center is a very important and interesting task, especially when you realize that there are important documents, photos from family archives, sketches of poems, or something like that are hidden behind petabytes of data. That the files we store are elements of someone’s digital identity. In international companies such as Acronis, when data centers are scattered around the world, managing your IT infrastructure is also a serious professional challenge. In the morning you work with Tokyo, in the afternoon - with Strasbourg, and in the evening - with Dallas. Each of the data centers has its own characteristics, we have already talked about this in one of the previous posts, and in one working day you make a kind of “journey around the world”, getting acquainted with the “cultures” of different countries and peoples. It goes without saying that such work has its own specifics: a working day can start at 6.00 and last until 22.00, having experienced your nerves and your wits for strength, but such challenges make us true professionals of our work.
https://www.linkedin.com/pulse/senior-linux-system-administrator-wanted-alexander-ragel?trk=prof-post
http://www.acronis.com/ru-ru/company/employment/vacancy/
How to beginner department to build your work day?
The working day of the head of the department should begin with checking the status of the work of its data center. If you have only one server in one data center, this should not be a big problem. But if we are talking about several data centers in several countries of the world, it will be a bit more complicated. At Acronis, we use automated systems and dashboards that allow real-time tracking of what is happening in each of the data centers, evaluate statistics on their completion, and based on this, adapt your task list to the day as needed. First of all, it is necessary to check the current state of the network, the state of the server equipment and its loading. Another very important parameter is the growth rate of server load. Knowing it, you can avoid such a common problem as the drop of servers under load, and properly plan the commissioning of new equipment. In general, kapasiti planning at our growth rates becomes a very nontrivial task that requires creative approach and daily attention.
')
After you are convinced that everything is in order with your data center and there are no urgent tasks, you can proceed to “administrative issues”, deal with the accumulated e-mail and make important phone calls. We cannot say that morning is the most ideal time for this, but for a good manager it is important to always be in touch, promptly respond to requests and requests from your colleagues, keep in touch with suppliers and follow the latest news from the professional field.
When the “administrative tasks” are finished, it’s time to move on to the execution of the task list formed at the beginning of the day. Plan updates, order new equipment, form requests for management companies of data centers, etc. Usually, tasks from the list take place all day, but if you work in a global company and / or you have servers in other time zones, for example, in North America, after lunch, when the Western Hemisphere wakes up, you will come across “ administrative tasks. " You answer mail and phone calls again, synchronize your actions with colleagues from other countries, and after this, as a rule, the second half of the day passes.
Sometimes, especially when you have employees who are in the minus ten hours from you, the second half of the day may go unnoticed on the first day of the next. And what to do, the attention of the staff, the more distant, must always be, otherwise the spirit spirit may weaken.
Something like this is the routine of the data center manager: checking equipment, planning money, parsing mail and phone calls, working on current tasks, and again picking up mail and phone calls.
A Sound of Thunder!
But not always everything goes according to the plan described above, sometimes various unpleasant situations happen, such as server crashes, DDOS attacks, and other amenities of the modern tech world. If your company has a well-designed infrastructure, with at least one backup critical element, like Acronis, and you use backup and disaster recovery systems, then most likely it will help you avoid fatal problems and quickly get out of the existing difficulties. . If the well-thought-out infrastructure and backup systems are not about you, then “we are singing a song to the madness of the brave,” you will have to get a pair of gray hair.
First of all, you need to try to “revive” the server remotely, IPMI to help. If the server cannot be restored remotely, then you have no choice but to write a request to the technical support of the data center, which will explain in as much detail and as simple as possible what happened. It often happens that technical support responds to such calls in an hour or two, so if this data center is close to you and it’s vital to restore its performance, then you should go there yourself and solve all the problems on the spot yourself (if you don’t thought about a good support contract).
If the data center is located in another region or another country, and it’s not possible to quickly reach it yourself, then you can follow our example and sign a contract with external specialists who will become your eyes and hands. By placing our equipment in a new data center in another area or country, we always try to find an external specialist who, in the event of an emergency situation, can quickly get to the desired data center and eliminate the problem that has appeared. We are looking for such people among the leaders of small local IT companies who are technically savvy enough and can independently diagnose and solve such problems.
When the infrastructure is critical for your company and your business, the data center does not rely on the technical support of the data center, as I noted above, it takes too long to wait for an answer, and there are times when it is wrong to understand what it is from they were required, they only made it worse. Therefore, we recommend finding such external specialists, concluding all necessary agreements with them (contract, NDA, etc.), and keeping in touch with them. You should take them as insurance, not at all the fact that you will need their services. However, if such a situation occurs, they will save you time and nerves.
We faced situations when it was necessary to promptly replace the hard disk in the server or install new hardware. The standard time for performing such requests by a data center is usually hours or even days, and with the help of an external specialist, we carried out these actions within one hour. And this is very important for us, because when your services are used by several million people, time is the decisive factor. Even minutes go by.
In addition to the time that you always want to speed up, at the global level, standardization and unification of the used hardware and software greatly helps. It sounds simple and easy, but in fact, in situations where the company is growing and developing rapidly, is actively involved in M ​​& A transactions, the support of a unified infrastructure is a difficult task. But if the task is completed, then the overall manageability of global data centers becomes much more trouble-free.
And, of course, you should always remember about the triad that allows you to sleep at night - capacity planning, redandancy and backup. They are good, without them bad. The team, and the team in Acronis Data Center Operations is excellent, penetrates this immediately, and some of the elements of the triad even go into nicknames / nicknames. There are, for example, Vladimir nicknamed Redundancy. The entire network infrastructure that Vladimir is building in our data centers is fully consistent with the nickname)
How to work with suppliers?
The tasks of the head include not only maintaining the data center in working condition, but also participating in the procurement of various equipment and services for its operation. There is almost no information on specialized resources about how to find suppliers, negotiate with them and enter into contracts - this is delicate information, which often differs depending on the region, so it will not be superfluous to tell a little about it.
Based on my experience, I can say that the most important thing in procurement is building good relationships with suppliers. If you buy a lot and often, you can always count on more favorable conditions: at a price, by payment method, delivery, etc., rather than in a situation where you just come "from the street." So, for example, one of our regular suppliers provides us with the latest hardware testing, which we can test in the conditions we require and under the loads we require. After all, there are often cases when one equipment does not work correctly together with another, and buying hundreds of thousands of hard drives, costing many, many dollars, we expect them to work well on our servers and with our software
And since the cases are different, it is impossible not to appreciate such a relationship, when under a “honest word” a supplier almost overnight can send a server to another end of the world when it is very necessary (real case: from London to Tokyo).
Another good way to conclude a lucrative contract is the ability to openly identify your own equipment needs and confirm them. Remember that suppliers are always looking for opportunities for long-term cooperation, so they can more intelligently manage their own inventory without freezing money in a product, effectively form their financial flows and accumulate a base of regular customers. Therefore, if you can sufficiently accurately identify your needs for equipment and services for the long term, you can safely expect a good discount, which in some cases will be significantly higher than 50% of the retail price.
Personal contact and long-term contract is not the only way to conclude a lucrative contract, there is another, third, opportunity to get a discount. In the West, there is such a “magic phrase” - “target price”. What it is? Any manufacturer directly concludes contracts only with major distributors who already bring the goods to the market, where we are buying them. It will not be a secret for anyone that a distributor purchases a product from a manufacturer at a much lower price than it sells on the market, and the difference between the purchase price and the sale price forms its income. It’s not always possible to find out for what exact price the distributor buys the goods from the manufacturer, but, after a simple market analysis, you can determine the average cost of equipment and services, and from this determine your target price. Can it be lower than the market average? Of course, but by demanding a price that is 70-80% lower than the market average, you can simply offend the supplier and not achieve anything at all! Will a distributor sell your product price? Not necessarily, but you should always keep it in your head when you go to negotiate. At the negotiations, you will most likely have bargaining, after which you will be able to conclude a good contract. Practice shows that this works especially well with the prolongation of expiring contracts, when it is possible to revise prices especially effectively, reaching its “target price”.
Using these three techniques, you can always enter into profitable contracts and build long-term cooperation with your suppliers. You might think that only large companies can use these techniques, but this is not entirely true. Even a small start-up company, guided by the recommendations written above, will be able to conclude good contracts. Yes, maybe it will not be about 40-50% discounts, but she can get a 20% discount and lay the foundation for a good long-term relationship.
Rationalization and optimization!
The last topic on the account, but not least, is the issue of IT innovation. In times of crisis, companies most often begin to reduce costs from IT, and at such times the demand for rational use of IT infrastructure increases dramatically. And if the head of the data center independently makes a rationalization proposal to the management of the company, it will be only a plus for you.
First, let's see what is the rational use of IT infrastructure. As I noted above, any infrastructure must have “excess” capacities, such as to cover ever-increasing demands for capacity. But this “surplus” will be necessary at some future moment in time, and when exactly it will come is not always known. Some companies follow the simplest path, buying “here and now” the most top-level hardware, in the hope that in the next year or two they will be able to use it to the full. A year goes by, new, more sophisticated “glands” come out, the purchased equipment is morally obsolete, but still far from its full load. The second year passes, the equipment has time to become obsolete already and physically, but again the load is not complete. It turns out that during these two years the company spent extra money on equipment and maintaining it in working condition, and as a result could not use it under the “full program”. An additional problem could be the fact that, having bought the very top right away once and “with a reserve”, the company could not get a discount by telling about its needs for the future, which we talked about earlier. How to rationalize this situation? First of all, daily monitor the needs for new capacities and build graphics. Yes, yes, the same kapasiti planning, not once mentioned. It is true, true and relevant for everything related to data centers: communication channels, utilization of all resources, all equipment according to its roles and functions, firewall throughput (and suddenly your company decides to be certified by PCI DSS tomorrow, and you will suddenly be asked to enable IDS / IPS, which will reduce the throughput of the firewall by 3 times?). Very rarely, the need for power in a short period of time is doubled and kept at this level, as a rule, it grows gradually. Having made a schedule, considering what kind of equipment is needed now, how quickly it can be purchased and installed, and when you need to put new equipment into operation, you can go to the equipment suppliers, talk with them and enter into long-term contracts, following my recommendations above.
There is also another situation, when for different tasks a company uses similar, largely duplicating equipment. On the one hand, some especially “overcapacity” is not created, on the other hand, this equipment takes up space in racks and consumes electricity. It is in this situation that we appeared not so long ago. The two main Acronis products are Acronis Backup Cloud and Acronis Disaster Recovery. To provide services within these products, different “hardware” sets differing in their specifications are used. Despite this, there was an understanding that there is room for optimization, and we decided that it was necessary to compare the characteristics and specifications of these sets of hardware in each of our American data centers. Based on this analysis, we were able to identify four main specifications that can be used and that meet the main requirements: maximum resources per rack unit and the lowest possible power consumption. Since the 36-month depreciation cycle of existing equipment was coming to an end, we decided to centrally purchase equipment based on the specifications received and update our data centers in the United States by the end of the year. According to our calculations, the number of occupied places in the racks should be reduced by about 2 times (hooray, we reduce OPEX to the collocation!), And the amount of resources - storage, the RAM of the processor cores - increases significantly.
What I want to say at the end of this post. Managing a data center is a very important and interesting task, especially when you realize that there are important documents, photos from family archives, sketches of poems, or something like that are hidden behind petabytes of data. That the files we store are elements of someone’s digital identity. In international companies such as Acronis, when data centers are scattered around the world, managing your IT infrastructure is also a serious professional challenge. In the morning you work with Tokyo, in the afternoon - with Strasbourg, and in the evening - with Dallas. Each of the data centers has its own characteristics, we have already talked about this in one of the previous posts, and in one working day you make a kind of “journey around the world”, getting acquainted with the “cultures” of different countries and peoples. It goes without saying that such work has its own specifics: a working day can start at 6.00 and last until 22.00, having experienced your nerves and your wits for strength, but such challenges make us true professionals of our work.
https://www.linkedin.com/pulse/senior-linux-system-administrator-wanted-alexander-ragel?trk=prof-post
http://www.acronis.com/ru-ru/company/employment/vacancy/
Source: https://habr.com/ru/post/309688/
All Articles