Checklist of troubles that may occur when moving

Most recently, we opened a cloud in St. Petersburg. Equipment for it was transported from Moscow. While preparing this move, I remembered many stories and “rakes” from my experience and decided to prepare a small overview article on this topic.

In the format of a checklist, flavored with examples from life, I will tell you what surprises and pitfalls may lie in wait for newcomers when moving and installing server hardware in a new territory. For clarity, I used photographs from our montage in St. Petersburg.
')
Come under the cat, let's talk about how to move and not go crazy.

The process of moving is full of routine and is arranged almost equally in the companies of our industry. I will not dwell on issues such as choosing a contractor for transportation or loading and unloading (although they are also important!).

Most of the article I have devoted to all sorts of difficulties that await when planning a move or installation of equipment.

Training

It is important to calculate everything in advance to avoid stupid mistakes when moving.

• Rehearse . Always try to assemble the equipment on your site before you take it to a new one. This is what we did, and it saved us from surprises. All equipment was mounted in Moscow in the rack and marked. Cross-logs and layout were created. Why - read below. After that, the equipment was dismantled and packed in special shipping boxes to the size of the carrier’s car body, which guarantee the safety of the equipment during transportation.
  
• Get to know Familiarize yourself with the new place in advance, it is best to visit it in person, look around and talk to the staff. Get samples of all the documents you will need to provide to enter and open applications in the helpdesk. Learn the rules of unloading: where is the ramp, how to get to it, how to call an engineer to accompany, at what time can be unloading. Specify the time of the warehouse! We were in a situation where installation in the data center is possible 24 hours a day, and all the equipment is placed in advance at the warehouse. But the warehouse worked from 10 to 18-30 :)
  
• Count it . Before moving prepare a comprehensive inventory of the equipment transported, literally from the rack to the nut. You must rewrite all models, serial numbers, etc.
  
• Label . Make a log of cross-overs, mark all the equipment (at least from two sides) so that you do not understand in a new place in a panic what is what. Measure your rack, including its useful size (depth, width, height) - you will understand a little later why this is important. Marking follow the same standard. Be sure to mark all the ports on the equipment with a permanent marker.
  
• Write down . Agree on all documents and powers of attorney that you will need to take with you. The power of attorney must be issued in the form of the receiving organization. Do not rely on employees, check and recheck their work. Even a small mistake can break your deadlines.
  
• Get together . Take along a disturbing briefcase: laptop with power supply, screwdrivers (straight, cross, short cross, powerful), pliers, stationery knife, scissors, velcro for fixing wires, ties (not for wires!), Marker, power adapter C14 to euro , a proven USB-COM wire, a USB flash drive, an APC rack key, a notepad, a pencil, a marking printer, or labels for labeling, enough sheets to write. Contact your colleagues and contractors on paper and the opportunity to call them not by cellular communication (it may not be in the data center). Just take the word!
  

Measure seven times ... Measure again!

Despite all the measures taken, incidents occur when moving. I'll tell you about the most typical.

• Cables too long . In the delivery were 10 m sas cables, 4 pieces per rack. This length was remarkably enough for us in Moscow data centers, but in St. Petersburg the wire tray was lower and the cable length was redundant. The problem is small, we found a way to gently lay the wires.
  
  
  
• New configuration . The two hundred kilogram upper storage shelf of 3U in size, which was perfectly located in the 20th unit on the equipment circuit, unexpectedly :) it was necessary to push it forward to its full length, for which it was necessary to remount it downwards. This means that the storage controller must be moved down, under the shelf, in order to provide the possibility of increasing capacity. This led to the redesign of the installation of other equipment. From such situations, no one is insured, it is important to be able to solve them on the fly. Therefore, trust the move to the most experienced engineers.
  
• Power connectors . Here is another example: among other things, we brought the Power Distributor Unit (s) - the power modules to which the servers are connected. These modules are single- and three-phase and require, respectively, single- or three-phase power. There are also “pears”, connectors for connecting the remote control, they are also different: 16 and 32 amps. You request a 3-phase food, it is organized according to the application. And here, it would seem: there is a three-phase plug, in front of you is a three-phase socket. But they do not connect, because the plug is 16 amps, and the socket is 32. The difference between them is literally half a millimeter, not visible to the eye. Even before the move, we clarified this point, took what was needed, but for safety purposes we were still equipped with mating connectors.
  
• Measure again . Some patches appear quite unexpectedly: we brought equipment to the object 980 mm deep for a rack per 1000 mm, and its actual installation depth did not coincide with the nominal one - it turned out that the working space in it was 750 mm. A rack of 1200 mm after installing the equipment 970 mm left 3 free centimeters instead of 23. Again, be sure to measure everything that you install and transport.
  
• Point of attachment . Sometimes our engineers have to be creative on site. If you used to work with standard APC racks, then you probably know that for placing the remote control there are special holes in them where the cushions fit perfectly. And if the stand is made by another vendor and there are no holes for the remote control, you can be mistaken: do not pick it with a drill, the right word! Fortunately, the engineers took with them ordinary metal corners, which turned out neat and reliable fasteners.
  
  
  
• Attention oversized! Another potential problem is that the pear power connectors may suddenly end up not under the raised floor, but inside the racks, in the “paid” zone. That is, taking the place of units how much in vain. Suppose it was planned to place 42 units in a rack, but in fact, 2 units are already occupied by “pears”. And we have to place below the minimum dimensional equipment, which has no communications behind. Such situations can break the entire equipment circuit. In our case, the problem was solved by placing boxes for spare parts at the bottom of the rack, which have no electricians or other wires in the back.
  
  
  Those most sudden "pears". Picture from montage.
  
• Primordially Russian dexterity . Racks - this is in principle a problem thing. They are of two types: a little bit already and a little wider than necessary. The documentation says: 19 inches, but in fact the width can walk by ± 5 mm, and we have to put up with this, as for some reason, the adjusting washers may not be provided by the manufacturer.

Tips from "Gregory Oster":

• In a situation where the stand is a few millimeters narrow, you will need strong men's hands in the amount of four pieces. For example, one engineer and one programmer. You place a more subtle programmer inside the rack and ask to stand up. At this time, the muscular engineer installs the first server. As a rule, one repetition is enough for the rack to acquire the “correct” width, and then the equipment is installed as it should.
  
• The server inside the rack is on special rails. And if the rack is slightly wider than necessary, it will not be possible to hook it on both sides at the same time. To solve this problem, we need a specialized tool for precise manipulation, known in narrow circles as a mount. It can be borrowed from installers or prudently to take with you. One person with a tool slightly tightens either side of the rack to the center, and the other catches the equipment.

Just in case: there is nothing supernatural in this process, and the equipment does not suffer - the discrepancy in width is slightly smaller than the tetrad cell.

Stock up

Another thing that seems obvious, and therefore they always forget about it: take fasteners with a large margin. Even if it seems that it should be enough, take another. Search or buy them on the spot - more expensive.

We needed to mount the equipment in the facility in two racks. We took with us three sets of 50 bolt nuts each, which we felt should have been enough with our head. But this was not the case: in this configuration, the equipment, plugs and decorative panels are fastened with the help of nuts and crackers, and not on the latches, as we used to. As a result, in order to secure everything firmly, I had to buy another set, which was almost enough but very close.

The same goes for the wires: the more their choice on the site, the better, the extra ones you bring home. Take as much as possible: power, network cables, and Ethernet of different lengths. The tray through which you pass the wires may hang at different heights above the rack, and if there are at least seven-meter cables, inside or on the roof of the rack you will get a tangle of wires that you cannot work with. You need a guarantee that it will be possible to lay everything nice and neat and the server room will not resemble a snake's nest during the mating period.

Life hacking:
Be sure to take with you a wizard who knows how to stack “beautifully”. According to rumors, this is a superpower that only a few people perfectly master.

And again about labeling

Installation of the rack is completed, no engineer was hurt. But that's not all. Ideally, after installation, you need to draw a detailed layout of the equipment. In this case, the "portraits" are made both in front and behind. Each has a signed rack number. It is necessary not to confuse what is on the left and what is on the right.

Remember the beautiful wires? In addition to aesthetics, they are useful during operation and reduce the risk of error. For example, you need to reconnect only one wire, and after him from the coil stretch another four. As a result, important services will be disabled.

Therefore, even if it is very lazy and everything seems (to you now) simple and understandable, be sure to mark all the wires and all the equipment. At least with the help of a marker and paper tape. Engineers who will support you after the move will thank you.

Even better, in critical nodes the wires will have not only signatures, but also different colors. Suppose an engineer gets a task to disconnect a particular black wire. While all the wires are black. What is the probability of error? ... We use green wires to connect the left power supply, and orange - for the right. Power supplies from different sides are connected to different power beams.


Designate different food rays in different colors. A long black power cord on the left is temporarily connected to a laptop - a photo from the installation.

Behavior rules

Finally, I remind you: all data centers have their own rules of behavior, and your employees should know them before starting work.

Do not forget to check with the attendant in advance the following points:

• Where can I / can not walk, how does the access control system in general work?
• How fast to go out?
• Where to throw garbage?
• How to bring back the remaining parts?
• Where are the restrooms?
• Where is water and food (dining, vending machines, etc.)?
• Where to get fresh boot covers?
• How to quickly contact the attendant?
• Will it be possible to leave the hall independently, without the attendant?
• Where is the console with the monitor?
• Is there internet access via Wi-Fi? Get login and password.
• Is it possible to make inter-rack crossings?

And a couple of special things that you need to know in advance:

• Be sure to find out what time the passes are valid. Regularly engineers remain at the facility after midnight. The security system may reset passes at 00:00, and people will not be able to exit.
  
• When installing the equipment may require electrical approval. For this work, you must have a 2nd admission group, and without certifying the state sample you will not be allowed into the data center.
  

If all items are checked and there are no problems, you can proceed to installation on the site.

Here is a checklist I have. Not a single move is 100% smooth, but by following these points you will save yourself a couple of nerve cells.
That's all, ask questions and share your experiences in the comments.

PS When I first told this story to an experienced CIO, he got an old DLINK switch in the rack from the wardrobe: “This is the first equipment that comes from us to the new data center at 23:45. With it, we check what paper we need to deposit, whether the rack is ready, whether standard equipment can be mounted into it, power supplied, external network crossings ready, how to ask “remote hands” to reload your equipment, what you need for dismantling and export equipment. And only when DLINK successfully returns home, we consider that we are ready to move. ” A good way to try out a new rake without loss.

Happy New Year and all the best!