The budget for the operation of the data center: instructions for compiling

⋅⋅

In the last post I told what and why you need to do yourself in the operation of the data center. Slowly I will share experiences on every aspect. I'll start with budgeting.

During the leadership of the maintenance service, I fell out of reality for several days, and sometimes nights. Brewing stronger tea and plunged headlong into the world of numbers and statistics, so that the operation of the data center had something to live next year. That's where the day really feeds you: you need to calculate and paint all the planned maintenance, replacement of consumables, as well as predict all possible repairs. Yes, and at the most exemplary exploitation everything ever breaks. After you need to perform a separate exercise - to prove to the leadership the expediency of everything that you wrote. Today I will tell you how to put together a budget so as to arrive at the desired “Agreed” by the most direct route.

All costs for operating the server / data center are divided into two types:

• operating - money that will go on repairs and maintenance of engineering systems and equipment;
• investment - the cost of upgrading systems, replacing large equipment.

Someone may also include personnel here (the salaries of the people responsible for the operation, the cost of their training), but I will not touch it here.

Let's look at how to collect each of the budgets.

Operational budget

These are fixed costs for spare parts, consumables, repairs and maintenance, which are laid out from year to year.

STEP 1. We compile a list of engineering systems and the most important equipment.

I will show on the example of the NORD-4 data center:

• Uninterrupted power supply: 33 UPS.
• Guaranteed power supply: 12 DGU.
• Cold supply system: 108 air conditioners.
• CCTV and fire suppression system.

The last two are not always included in the responsibility zone of the operation service, but in our case we also take these systems into account.

Making such a list, it is important not to forget about:

• auxiliary systems, such as monitoring, ventilation, irrigation, cargo lifts and gates;
• that, it would seem, quite indirectly relates to the operation - cleaning services, transportation costs, disposal of waste materials (oil, battery).

STEP 2. Determine the cost and quantity of consumables and spare parts for each system.

If it is clear about the cost of parts, we simply request from suppliers or we monitor supply on the market, then the question: “How much do we need this?” Is more difficult. Here we can use statistics on breakdowns and repairs, which I am talking about, probably, at each of my seminars .

Life hacking for those who stubbornly refuse this useful practice: it is theoretically possible to collect information backdating on invoices / certificates from contractors for repairs and the purchase of spare parts.

I foresee the question: what to do if the server and data center are new? The first year or two can be held under warranty from the supplier of the equipment, but all this time we are collecting statistics.
To calculate the number of necessary parts, I use the coefficients. Here's how to get them. Let's say you need to understand how many fans we need for external units of air conditioners. We turn to statistics and see that 10 fans have failed in the previous year. Only 100 air conditioners. Accordingly, we need 0.1 fan for 1 air conditioner.

Here such a plate can be obtained at the exit for the rest of the parts and consumables of the air conditioning system:

⋅⋅

Having calculated such coefficients for one year, they can be used for subsequent ones. My experience shows that the second year from the third, the third from the fourth will differ minimally: for subsequent years, it remains only to check the statistics for the correction of coefficients, and simply substitute the actual cost of a spare part or consumable into the formula. If in some year the coefficient has greatly increased, then it is worth thinking about modernization, but more on that later.

There are nontrivial consumables and spare parts for which the coefficients do not work very well. For example, fuel for DGU. On the one hand, it’s difficult to predict how many times Mosenergo will let us down next year, but on the other hand, we have such a predictable thing as testing. If DGS are regularly tested, then we know the frequency of launches, their duration. We also know diesel consumption and load. Based on this we calculate the volume of fuel. If there is an unplanned shutdown of city power, then the nearest testing is not carried out, since it has already been tested in actual conditions.

STEP 3. We calculate the cost of work on each system by contractors.

If we have an option when everything is included in the contract, including equipment repair, then we simply request an updated KP, hold a tender or extend the contract for next year without any unnecessary movements (here you can try to bargain with existing contractors to make them cheaper, or they bargain with us to make it more expensive :)). In the end, we get some fixed price for next year. It's all clear.

If only maintenance is specified in the contract, it is necessary to additionally calculate the cost of repairs and emergency trips. There is the same as with consumables and spare parts: look at the statistics on the work done over the past year.

It is important to consider what our contract is with postpay or prepay, so as not to miss the month of payment. For example, we are planning to wash the external units of air conditioners in April, which means it will go in March with prepayment, and with post-payment in May. For someone's accounting, it will be not just different quarters, but different financial years.

STEP 4. We summarize all the numbers into one excel file.

First, we take our costs for MOT and paint them in the form of a schedule by month. To this we add in a separate tab the cost of repairs and the cost of spare parts and consumables.

About the schedule MOT. I recommend that all works be “spread out” evenly over the calendar year, so that several works do not fall into one month. Then you will not have to lay out in one month immediately a tidy sum of money, and people who control the work will not be overloaded.

Investment budget

These are large expenses for modernization and replacement of equipment. To compile it, we need the following:

• data on failures and accidents last year. We look at the statistics and identify weak links. For example, we have a constantly breaking air conditioner, which in the last year was replaced 2 times by a compressor. Or another option: on all air conditioners steam humidifiers break down. In this case, think about whether it is better to replace this air conditioner entirely or upgrade the entire subsystem, the further you spend money and energy on repairs.

  
  

• data on obsolete elements of the system. The simplest example is rechargeable batteries. In my experience, the average battery life is 0.7 from that stated by the manufacturer: if they promise 10 years, after 7 you need to prepare for replacement. If we change, then immediately the whole line. For example, we have 12 UPS and 12 racks. We split the entire replacement into a year: every month we change one rack, starting with the deadly ruler.

  
  
• everything that we were wrapped up last year with the words: “Let's go back to this issue in a year”. Do not forget about these promises and enter into the new budget. If this has not lost relevance, but should not.

Next, we calculate the cost of upgrades and replacements with the contractor. We bring in the budget.

Then we go with all this to the authorities, and ...

they offer us to think about how to multiply this by 0.8. Do not panic. There is a place for maneuvering.

What can you save

Thoughtful reading of the contract for maintenance by the contractor provides great opportunities to save money. I have already written about the intricacies of drawing up such contracts. Add a little more wisdom. So, we check:

• Do not duplicate the same work under different formulations.

  
  

• Can we do some work that does not require high qualification, to perform on our own. For example, replace air conditioning filters.

  
  

• Are there any irrelevant work for your systems? Here are a couple of cases from our practice: the standard contract was to check the oil level in the air conditioner compressor, and we have a chiller circuit, where the compressor is only in the chiller. Checking steam humidifiers for all air conditioners, although we only had two air conditioners with a steam humidifier.

  
  

• There are no “empty” checks of equipment without subsequent troubleshooting. It is preferable that the wording be as follows: “Verification and, if necessary, correction of faults”. Therefore, if the contract states: “Control of the level of refrigerant and oil”, then we supplement: “Control of the level of refrigerant and oil, if necessary - refueling”.

  
  

• Frequency MOT. It is clear that the more the better the contractor. Do not be lazy to consult with the manufacturer of the equipment to determine the optimal amount. Well, look at your practice. For example, we abandoned the annual oil change for one of the DGU models, since the oil was completely clean. After consultation with the vendor, we began to change the oil every two years. Approximately the same situation we had with air conditioners filters, with which we switched to replacements as needed, and not every time MOT was carried out.

  
  
• Service level and contractor response time to incidents. You can register everything to the maximum and get the corresponding value of the contract. Here it is necessary to separate the flies from cutlets critical from non-critical. For example, decide for yourself whether the arrival of a contractor to the site is really necessary within three hours if the backup air conditioner broke down. In general, here you can reasonably play with the values ​​and bargain for your discount.

A good way to cut the budget is to optimize the cost of spare parts and consumables . Here it is attentive: I appeal to save NOT by refusing to purchase spare parts. Do the following:

• We buy directly from vendors, and not through contractors and other intermediaries.
• We use economical analogues of consumables (I'm not talking about Chinese fakes on aliexpress).
• We unify consumables for equipment of different vendors.

The most energy-consuming, but effective way to save - to find and eliminate errors in operation, which regularly lead to breakdowns, and therefore to the cost of repairs. An audit of engineering systems will help identify improper operation of equipment, prevent future breakdowns. Here are my top possible weak points:

• poor quality or late conduct of MOT;
• incorrect equipment operation algorithms, for example, frequent stops and starts of compressors;
• underestimated settings on air conditioners due to incorrectly installed server hardware;
• load imbalance in phases and rays.

Problems in operation may be inherited from those who made mistakes at the design or construction stage.
Unlike previous techniques, an audit will help not only to spend less, but also save the data center from potential accidents.

What can not be saved

Gladly deleting articles from the budget, do not even think about cutting the next expenses.

Testing. If you want to sleep well, then combat exercises are necessary. Let better the malfunction in the equipment or system (and sometimes in the process) manifest itself in all its glory under our control, than it will happen on December 31, when the right people will dance around the Christmas tree.
You need to test correctly. In my understanding, this is complex and regular. Even when we are testing diesel generators, we have the opportunity to simultaneously test the uninterrupted power supply system, monitoring, and much more (read more here ). If the last 44 tests were successful, it does not mean that the 45th can be postponed or canceled.

Seasonal preparatory work . First of all it concerns the cooling system. Before the summer season, at the latest - in April, you need to flush all the external units and chillers, check the installed IT equipment in the halls, and ensure that the backup air conditioners are working ( here he described in detail how to prepare the data center for the summer).
We are preparing for winter in October: we merge everything that can freeze, check the heating of the diesel generator set.

Purchase of spare parts. My experience shows that a contractor or your engineer can be quickly pulled to the site at night, on weekends or holidays, but to bring the necessary part from the warehouse is just as quickly difficult. Warehouses live according to their own schedule, not 24x7. And there just may not be the necessary details. Then the spare parts will have to wait for weeks, if not months.
It is not necessary to duplicate all the infrastructure in the warehouse. It is enough to identify those parts that often have to be changed (again, we look at the statistics), and those due to which you may lose the reserve or even stay with inactive equipment / system. When we let stocks into the business, we immediately buy new ones, although the accounting department will be surprised to ask: “Why do you need a new one, did you just change it?”
And another point: the list of spare parts is not a constant. Every time something new will break, do not forget to add it to the list of future purchases for permanent storage in your warehouse.

If you are not convinced, then here is an instructive case for you , after which we began to purchase and store spare parts and tools.

Warehouse with spare parts for cooling systems at the site OST.

And the last farewell.

If budgeting is a science, then reconciliation is more like art, which, however, can also be mastered. On defense, management likes to ask tricky questions, such as: “Why is there 150, not 149 thousand?” Or: “Why do you need so much for the fans?” If you, without regaining consciousness, can answer with examples from the statistics, where each figure, then repulse.