How we built our mini data center. Part 2 - Hermoson

Continuing Part 1 - Colocation

The next part - Moving

Hello, friends! First of all, thank you for your comments and interest in our last article. As we have said, in the construction of our center, we took from Habr a lot of useful information, which, in total, was very useful to us. Therefore, this cycle of articles is a kind of gratitude to the site and the authors for their articles, without them it would be many times more difficult. Thank!

So, we finished the last part at the moment with the search for a new room for the "construction" of our own data center. As we already wrote, it was vital for our young project, which had difficulties at times of serious problems in the data center. I will show several frames of the data center building. At the same time I want to say that we are still working with the owners in other directions and parted good friends. Now (according to rumors from colleagues who are posted there) - everything is fine.

Attention! Lots of images. There are not compressed photos of panoramas, because when they are compressed, the quality is very much lost.


')
The building itself, as you see, I did not lie, it is really a former industrial complex. I specifically insert enough photos so that you understand the need to move. By the way, the containment is quite good, as is the technical approach in general, good repair, etc.



These are two photos of the stairs to the data center and the door of the lower floor (abandoned).




In general, how it happened, we began to search for premises. There were some requirements that we had to meet.

- premises derived from the housing stock (important for official activities).
- preferably the first or ground floor (we are very tired of wearing iron on the fifth floor on the hump).
- access to telecom operators (within a radius of 2-3 km.).

The remaining items, such as repairs or something else, worried us less, we understood that there would be enough work. We reviewed about 20 different objects and each had some problems. We chose several small objects and chose the best. We understood that we don’t have the means to rent a hangar and make an industrial data center out of it, which means we need to save money. But the economy should be economical, which means we need to do everything as professionally as possible (albeit on a small scale, under an extension) so that later there will be no problems.

An option was found in the city center (on the sixth floor if I am not mistaken), but with freight elevators. The data center was already prepared and made very seriously (the former striped carrier was located). There is made and reinforcement of the floor under the hall of the UPS and the supply of electricians and everything else.

UPS cabinet:



Engine room:



But in fact it turned out all the worse than we drew. The UPS remained 1 locker and the one that is rather dusty. DGU taken, inputs cut off. And the price for the rent was broken as for a spacecraft, plus a mandatory connection to one of the telecom operators, who owns (rents the floor) this room. In general, and it would not repel us if it were not for one BUT. The owners of the building want money for any action, and the format “insert windows on the floor for this”, etc. And so absolutely for everything, whether it is to lay the optics or power supply. We did not like this, it did not want to depend on someone else’s whims, and to be honest with you. Looking further.

And here, quite by chance, in the neighborhood with my office (for another type of activity) I see an advertisement about renting out a room. The area is residential, quiet. The first floor, two entrances, a ramp, is derived from the housing stock, for a long time and for sane money. They examined, calculated - they decided to take it. This room, though not large, but initially met our requirements completely.

We were given the keys (how long did we understand what was happening), there are really a lot of keys:



Initially, the room looked like this (except for fire and security alarms and trifles, we have already installed them). Panoramas.

Future engine room:



Future office for attendants:





So, the countdown started, and we had less than two weeks to complete the move. And we did not have, absolutely nothing. First of all, we started to make a room in which we will be at the time of minor repairs in the data center.

Kavrolin laid out and let him vymeryat:



We bought various trivia, stickers, tablets, clasps:





And another vacuum cleaner and various useful things:



Started laying kavrolina and baseboards. We did everything ourselves:







The beautiful half is also with us and helps:



They brought a cooler for some water, dragged tables, and brought Cisco IP phones:





As a result, it was something like this. Thanks guys, very helpful:

Repair work in the engine room

One of the main criteria that was important for us is the insulation of the data center. They chose for a long time how to do it, consulted, but decided to do it anyway. To break the old plaster (it was very pity but necessary), to lay 100 mm. cotton wool, glue all metal joints, profile with special vibration insulation tape and only then re-assemble everything. The same goes for the ceiling. The process has begun. Vatu, by the way, was chosen specifically non-flammable.

The first purchase of building materials:



Rushim wall (unfortunately shot only the first). By the way, we found the secret wiring of past owners of the premises. Cut off:







We lay wool and plaster:







We lay wool under the entire ceiling:



Actually the same operations were performed with the other walls and the entire ceiling. Noise isolation an important point. Then we moved on to painting and all sorts of final works:

Data Center Infrastructure

Now that the dirty part of the work is over, we can think about the equipment that we plan to install and the complete infrastructure for its deployment. I have to say, the air conditioner in the room was what helped us a lot. Industrial, real, huge air conditioning. He eats the truth more than we need, but these are trifles. We have highlighted important points in our plan and have begun to implement them.

We started with the power supply. We were striving for the ideal in order to succeed - albeit well. We connected the power supply from different substations, made wiring, changed the power cables, laid the cables in the machine room:



A phase switch was also installed (in case the 1 phase disappears, the system switches it to another itself):



UPS. It was harder here. We considered completely different solutions, but the power we needed was either not available or it cost more than we could afford. It was also important for us that this solution could be scaled, because for a start we need not much, but it is important that it be and perform its functions well.

So we found an inverter of 5 kW, which runs on batteries (48 volt battery circuit) and can even work on solar panels. To be honest - we were tortured with its connection well, we even thought about going to pass, as at one moment we found the button that needs to be “clamped” to turn on. At the same time, nobody knew this in the store either (they forgot to say this in the instructions) and offered to return the goods. But we did not give up. Connect the battery.

Here, too, their moments: the battery at 12 volts. They can be connected either in parallel or in series, from which we get either 48 volts instead of 12 or multiplied amperage. We needed 48 volts, so we hooked it up like in the photo below. Batteries by the way, we could choose two types. Automotive and special. The first ones have a higher starting current, but they cannot be discharged to zero, the second has the opposite, a smaller current, but is not afraid of discharging. We chose automobiles, the price was very different, and there was no difference for us (we still need to change them in a year):





Probably a separate post deserves shelf for batteries and UPS. As you understand it all weighs decently. They chose an expensive regiment in one system mage, it was clearly written on it, it was assembled in 15 minutes. We cursed this regiment for half a day, until we collected it. In fact, it turned out that there are fewer parts than necessary, and I had to constantly invent something. We were not happy with the shelf, but that is - that is:



Install the UPS:



Connected the UPS. The connection is simple. 220 is included in the UPS, he straightens it and lets it to the cabinet from the exit. In the case of the fall of the main line - it delivers 220 of the batteries. It works instantly, turned off, turned on, tested:



A generator arrived. Calculated power 5 kW. Petrol. The system is built into the generator AVR (automatic input to the reserve), i.e. when the light disappears, it automatically starts itself. Also attached to the remote, if you want to play. It would seem that the most difficult thing is over, which could be simpler - to drag, check and connect, but it didn’t happen as it seemed to us at the beginning ...





The most difficult for us - it turned out his connection with the UPS and the main input. After all, in fact we get. 220 incoming voltage, 220 voltage from the UPS (suppose it gives it ONLY when light is lost) and 220 from the generator. And here we made a fatal mistake. We simulate a situation:

- light disappears
- the inverter maintains the equipment while the generator is running
- the generator starts and gives voltage to the cabinets
- the inverter does not know what voltage is applied to the cabinets and it turns out the oncoming
- turn on the main power 220 and it turns out a counter with the generator

No matter how we try, we can’t connect the inverter and the generator and the network - it’s not, where and then the opposite voltage is obtained. And in the network to connect the generator - we will feed the whole house.

So that this did not happen and we never met two 220 (from UPS, generator or network) - we developed a system of magnetic starters, to be honest, it was a huge haemorrhoids, because finding specialists in a few hours is not easy, but moving just “to nose. "

Here's how it turned out in the end:





And then the cabinets arrived. Cabinets we took the same as in the data center where they were located on colocation, the company APC. There were several reasons for this. The first is that they are really very comfortable. The second is that our controlled outlets become perfect in them. Third - to quickly move, because in the same cabinets easier to move.

Another one thing that we did not take into account is the delivery of cabinets. She left about 50% of their value (we took 2 cabinets to start with). And one more important moment, how did we rejoice at our ramp when we pulled cabinets (they are on wheels), for DC, this is an important part of the infrastructure. Cabinets by the way are not so much heavy as not comfortable:











Patch panels and switching are a nice bonus to the cabinets from the supplier:

Rated power

According to our calculations, the data center becomes at least 10 cabinets of 42U height with a total capacity of 25-40 kW. We laid 2.5 kW of reserved power for our cabinets. In general, even now when we have enough space, we consume no more than 3.5 kW for both cabinets.

The data center was designed with the possibility of expansion (since we wanted to make a quality project, but with limited finances), namely:

- there is an adjacent room that can be bought / rented, approximately the same in size
- it is possible to quickly add a UPS + battery, the entire setup will not take more than an hour
- it is possible to increase the power of the generator by replacing it.

In the next, final part of our story there will be connecting fiber optic cables (Internet), with which there were a lot of problems, moving together with server equipment (how couldn’t it be easy to drag) and setup, installing SCS, lifting BGP tunnels with providers, testing iLO / KVM switches, purchasing desktop equipment, comparing it with HP, calling an ambulance to our employee during connection, and much more. Also separately, we will devote time to the video surveillance system, its implementation in our case.

It will be interesting! Thanks for attention.