Weekdays data center: unobtrusive trivia for 7 years of operation. And the continuation of the rat

I’ll say right away: the rat in the brought server, which we otpoili tea with a couple of years ago after an electric shock, most likely ran away. Because we saw somehow on the detour her friend. And immediately decided to put ultrasonic repellents.

Now there is a cursed land around the data center: no birds sit on the building, and probably all moles and worms have fled. Worried that the sound can cause a failure of the HDD, but checked, the frequencies are not the same.
')
The next story is much more fun. We somehow got a piece of iron for a couple of million rubles in a box with tilt, vibration and humidity sensors. Everything is whole. Carefully removed the packaging, and the bent piece of iron. Mystic.

The body is straight arc. Beautiful so.

Detective

We would not have given it any attention, because the metal body was curved was almost a design image. Beautifully so without chips. And if it were not for other similar pieces of iron nearby, we would not even think on unpacking that something was wrong. But there were the same ones, only of a more regular geometric shape.

Fortunately, the unpacking of such iron is removed on video (I recommend everyone to make such a habit), so we managed to prove to the manufacturer that she came. The whole package and neatly bent body is not a blow from the movers. Most likely, it was damaged before leaving for Russia.

Vendor says: "Uh, guys, and let's give it to you, we will change it under the guarantee". And here we were waiting for an epic ambush.

The fact is that customs allows us to import such equipment according to documents without the right to export. That is, you can bring, but you can not resell someone outside of Russia. When we send back a burnt power supply, for example, everything is clear. This is a spare part, power supply.

And then it took to send back everything:
- Guys, look, we send the piece of iron back to the manufacturer.
- Equipment entirely?
- Yes.
- Model such and such?
- Yes.
- Can work?
- Do not know, did not include.
- So, this is the whole equipment.
- Well, she does not work.
- Well, look, the whole equipment is such a model. Without the right of re-export. Not allowed.

In general, there were many squats before we figured out that we do not export it, but give it back. In the end, managed to do everything.

There were also bakhilnitsy

At first, many years ago, we had the first automatic dream of the admin. You load the boot covers there with a bundle, she unpacks them herself, opens and places them in a position where you just have to step on. Chpok-chpok and ready.

About six months later, she cut about a hundred packs of shoe covers and choked. It turned out that there are so many moving parts that it is necessary either to repair it once a month on our workload (we have a lot of customer engineers on the object, because we are a commercial data center), or we need to buy a new one.

The second problem is that later on at one of the regular cleanups we somehow found a “blue rag” dangling on the grate of one of the stands of our testing. A forensic scientist in the face of an engineer of the X-team identified a fragment of the body of the boot cover. It turned out that it was convenient in the polyclinic with shoe covers: half an hour was like - and that's it. And some engineers can work all day with iron. Shuffling feet. Shuffling hard. And the boot covers at the same time wear out into these small patches that fly along the main hall.

We almost immediately bought a new bahilnaya. They took a thermowlm: this is the machine into which the film is being charged, and she neatly heat-shrinks this film over the shoe. Beautiful, effective, durable. Less shatters. For a long time, it stood with us, but it was necessary to change the shrink film about once every 1-2 hours, because the sole had the ability to fall off by itself.

At first we thought that we were not lucky, and people somehow solved this problem. But no. Western colleagues were asked - the same story. As a result, they began to think how to do it normally. Returning from the turbine hall for new boot covers - the idea is, frankly, so-so. Found industrial cleaners for construction sites and industries. This is something like the tracks along which a shift comes into the workshop. The tracks with a bunch of rollers clean everything, and make it the way you want it or not - catch and clean it. They cost half a million to a million rubles. We dig deeper and found the same for 200 thousand, but we have to poke our foot in it ourselves. In terms of dimensions like a machine for cleaning shoes. You come up, shove your foot there, she chews it and gives it back clean. Put at the entrance to the data center.

Works fine with the exception of two nuances. First, it quickly became clear that it was us, the engineers, who were normal. But in practice, a variety of people come to the data center, including the tops of large companies. With leather shoes from the ass dragon. And even for putting cream on shoes, they have a brush that is more expensive than my training sneakers, they pick up a special nap there. They refused to shove their feet in our miracle device. The second problem arose in the winter: when the shoes are right here in the mash, it may not get everything out of the deep tread. Then you walk around the hall, leaving traces of ectoplasm.

Just decided. They placed a rolled bakilnitsa nearby. Anyway, we all need to duplicate the standard.

There was a new problem. Observing the behavior of customer engineers, we saw the following picture: they first stuck their feet in the cleaning machine, and then they hooked the boot covers from the roll shoe covers. Now they hang a sign that something is either different, and it is better to clean it, but if the vital principles prohibit cleaning shoes, then put on shoe covers. Like the ticket, which was for a couple of days, and stretched out for a long time, closed. Here is the device:

Two times "ku"

According to the requirements of PCI DSS, you need to be able to visually distinguish the roles of people in the data center. Without looking at the pass and reading something there, but directly visually, as the military distinguish each other by epaulettes, only brighter. We decided not to show off and used the good old chatlansky method - color differentiation of pants. Specifically, they began to make ribbons of different colors. Green as your favorite immediately took our admins.

It sounds simple, but it caused three unexpected effects:

Retractors were needed to automatically retract these passes while wearing (these are the things that regulate the length of the tape themselves). We wrote a ToR, where we took into account all the wishes of all departments. It was a strategic mistake. Color, format, material, the retractor is not plastic, metal fishing line to put the logo so that it is sewn into the tape. The pieces turned out so expensive that they had to cut the requirements and change the format.
When the differentiation of pants earned, it became very convenient. The contractors have some tapes, the external administrators have others, our administrators have the third. It is clear who has what role. In electrics - only gray, in air conditioning - blue. And here we needed ribbons for drivers (this is a separate role, they can enter the unloading zone, but they cannot leave it except outside). In this case, drivers do not need passes. At first we gave them ribbons without gaps. Then the guards decided that it was very strange and degraded the human dignity of drivers. They have their own military logic, so now the drivers come to get a pass immediately with the tape, but this pass does not allow anywhere. From the point of view of security, a marker is obtained that the security guard of this person has checked.
One of our engineers suggested making green sweaters instead of ribbons. And sent an innovation proposal. They did it halfway: they left the passes with the ribbon, plus they really did sew green uniform sweaters. Now we have the admin uniform. The security officers supported the joke and added it to the regulations. Now she is obligatory (pants, shirt, sweater, but the sweater can be removed).

Our customers also often complained about the curved routes in the maps before entering our Compressor data center. You score the address, and the road is shown incorrectly. As a result, visitors taxed the wrong way, because there was a railway, and there was a traffic jam behind it, and there was no turning around. At first we wanted to hang signs over the road. There is such a service in the city - to place yellow extras under usual signs, they are considered advertising. And the price for them is like an advertisement: on the Enthusiasts Highway one sign costs a million rubles a year. In parallel, we wrote in Yandex, and they even suddenly answered. And they stopped behaving. You can even specify the gate diodes: entry through one, exit through the other.

Google, if you are reading us, then know: you still have a problem, and we don’t know who to say about it so that we can be heard.

In the letter of invitation, the links were inserted not just to the address, but to the address with the route being laid by the user's geo-location. As a result, the miss have become less.

Gobo projectors and other little things

Do you know what gobo projectors are? We did not know either. Somehow thought how to mark the rows of racks. The stands themselves, of course, are marked with special quick-release markings, but they can be seen from a distance of 1-2 meters. The hall itself is 500 square meters, so getting lost there is where. Therefore, still risen to the marking of the rows. Brainstorm began. How to label what and where? On the floor, on the wall, signposts on the ceiling, etc. And then our colleague saw that there were wearable stickers on the floor at Ikea, and then light arrows appeared. Well, we decided to reverse it in a simple way: go to Ikea and turn out one of the projectors. It was not possible to get it: while we were wearing chairs, the seller asked what we were doing. And immediately helped, saying that it was a gobo. It turns out that this is not the projector itself, but a plate or lens for a color image. This light filter is a gobo. One projector costs from 40 thousand rubles (there is a powerful lamp for daytime use), and we have 14 rows in each of the four machine rooms. Therefore, pasted stickers.

We also have schemes on the walls fade over the years. We changed them to laminated ones, with “embedded” special pockets for revizorro. In our case, Revizorro is the chief engineer, whose duties include checking for the relevance of all circuits placed in the data center. So, all the schemes should be annually checked and signed by such auditorro. And the presence of a special small journal in the pocket of the circuit makes this procedure easier and does not require replacing the circuit itself every three years. Profit!

They made a rotary cleaning of the raised floor outside. We have regular cleaning, there are techniques and time for cleaning. But the wheels of heavy racks remain traces. Made a purge. Now we are nervous: we didn’t seem to be much more careful, but there were some highlights from certain angles for certain people, well, who have their own markers to their taste. Now we are thinking and looking for some kind of chemistry that will whiten the floor and give it a gloss. To even the selected questions were not.

Have you seen console racks? These are the traveling tables, only instead of drinks there is a terminal for connecting to the rack. So at these console racks, the wheels fall off and wedge like carts in a supermarket. It got us incredibly. As a result, the only possible way to reanimate is to buy a new wheel. And the wheels to get specifically to our models no longer work, we interviewed all contractors. As a result, they themselves designed the rack for themselves, with an emphasis on ease of movement on the main hall and maintainability. It turned out very well.

There was a story with synthetic socks. There is such a thing - anti-static bracelets. This is when you come to the rack, connect the bracelet to the ground on the rack, and it communicates with the potential equalization system. So, the rack is grounded, but it may turn out that the engineer is not grounded. Colleagues from past jobs told us how they saw sparks a couple of times on video surveillance, and we decided to commit ourselves to sin, according to the rules, to use everyone.

Critical Incidents

From the serious - there was a situation when all the chillers were cut off at once. Our chillers are not protected by UPS, because we believe in physics, and as a reserve for the temperature we have a pool with chilled water. If something turns off, you do not need batteries to power the chillers that cool the water, but simply the cold water itself, which is already ready. Convenient and simple, but there is a nuance. Chillers are equipped with protective automatics, which in case of hazardous parameters of the electrical network disconnects them. If the input is disabled, we turn on the diesel generator set, and then the chillers are powered by them. Everything would be fine if we did not live in Russia. We had network drawdowns many times, but everything was fine. But once the jump was sharp first down, then sharply up, then down again - in a few seconds, the input parameters changed about 4 times. Chillers shut down naturally. We first tried to turn them on remotely, but they protected themselves very reliably, in an emergency way. I had to go to change the legs on the roof and turn them on manually. What is important, according to the TierIII standard, such a situation is the legitimate reason for stopping the data center. We did not have a stop, because the people on the ground with a head, and there is a drill with teachings. For this, we UI just interrogated regularly, to be sure about the TIII Operational. If anything, we passed UI recertification to TIII Gold - Operational Sustainability. There is nothing cooler on the Russian commercial data center market, except for ours, only one data center has the same accessibility. I note that it is harder to re-certify than to get a certificate from scratch, because they check the past period as not being in themselves, and much more evidence is needed.

There was an interesting incident with the cameras. We decided to recalculate the blind zones just in case, drew intersections, built diagonals of viewing angles on the plan, and suddenly found a dead zone of about 30 centimeters by 15 meters right in the center of one of the halls. Narrow and long. In the next room there is no such. It turned out that the turning camera slowly moved over the years so that it began to show approximately one and a half degrees more to the left than it should be in the extreme position.

Another big case was in the post about repair replacement DDIBP .