Servers and data centers in the Netherlands - operating experience
We at HOSTKEY keep more than 400 of our own servers in DC Certerius, which is located in the middle of the beautiful fields in the Netherlands, in the city of Dronden - just north of Amsterdam. I want to share my impressions and cross-cultural differences from more than 2 years of experience placing our servers in Cerverius, how not to step on a rake.
The whole story began about 3 years ago, when we decided to open the site in the Netherlands. We drove around a dozen data centers in Amsterdam, Rotterdam and the surrounding area. Negotiated, collected offers and contracts.
All of them could be divided into two categories:
The difference ends - the technological solutions are almost the same. I saw these technological solutions in Moscow only in the coolest local data centers. Everything is perfect, optimized to never touch, what would work without human intervention. Full automation. Total outsourcing.
')
In general, from 5-6 real proposals, we then chose Servius and did not regret. For all the time we have not had a single accident due to their fault.
As a rule, the entire data center personnel for 100–200 racks consists of 5 people: the owner, he is also the chief engineer and seller; chief finance officer; technical support chief; a couple of junior engineers to help him. Salaries are high, taxes are high - there is little staff. All that can be endured is being outsourced. All stick to the conscience (unemployment of 20% and mortgage work wonders).
Consequence: there is no round-the-clock shift, as everyone is used to. You have to get used to the fact that the Dutch come to work at exactly 9 in their own time and leave at exactly 18. Overtime - 150 euros per hour, for this amount the owner himself will sit on the bike and come to see what happened to your server. We have never used.
We must pay tribute, if they began to work - then they will finish. In Germany, a Bulgarian engineer can dump at 18-00, leaving the case in the middle, and the server dismantled.
Consequence: No one will go anywhere to change disks, buy memory, etc. Your supplier should do all this. All issues are resolved only through KPN / DHL, and the courier arrives at the data center.
Moral: all servers must be equipped with remote access modules. All drives must be server class. Everything should be under warranty.
We here in Moscow got used to the fact that planning is practically a nonsensical thing and we are in a hurry to resolve issues. We do not trust anyone, we take the goods daily from the warehouse.
In Holland this is not the case. If you order something, the supplier honestly says - it will be ready in a week. And exactly one week + 1 day it arrives at the data center. Nobody holds anything in stock. All collectors order from distributors, this is brought by mail from different places and then collected and sent by mail too. So it is accepted and so everyone works.
The topic “I need a server tomorrow or today,” no one will understand. You order, you are called a term and in this period everything comes. Do not forget - if something breaks under warranty, the courier is called and the item is sent to the supplier. For example, if the server stops responding to IPMI, it is sent to the supplier as a whole. Disks accumulate until they have a box and go to the same supplier. A month later comes the same box with swap disks.
In the data center there is an SLA - paid. Since the data center is not a monster, and you are not the smallest customer - your requests will be serviced faster.
It is necessary to formulate accurately.
A stumbling block to the use of regional data centers in the Russian Federation was the impossibility of sending servers to the data center without its own forwarder-engineer, which destroyed the entire economic effect of the price. At Certerius, our supplier simply sends the server by mail with the HOSTKEY mark, the data center accepts them, configures IPMI, mounts into racks, and connects the servers to the network. Similarly, they reconfigure servers, send parts under warranty, and so on. All work that can be done in 15 minutes is considered small and not separately paid. There is no problem for them to throw memory, replace discs, throw cars between racks.
It is very important - there will be no favors, describe everything in the contract. The contract is the law.
In Russia, we are accustomed to the formula 42U and 5 kW. In Holland, usually in the price of 42U rack include a little electricity, about 1.5 kW. The base price is around 400-500 euros and the excess costs 15 euro cents. About 1000 Euros come out on a full rack. The consequence is simple - put economical equipment and pay less. Turn off the servers that are waiting for the client. The most disadvantageous thing you can do is keep the old servers, they will cost significantly more expensive to operate.
Read the contracts very carefully - in Holland and Germany, if you have ordered something for a year, you will have to pay for it for exactly one year, whether it is important or not . To abandon the rack ahead of time will not work, but sometimes it is necessary to warn for 3 months. So accepted. There are also five-year contracts - carefully look at the term for which you are signing. Then do not jump off.
Typically, the data center gives its Internet, already delivered to the site. Be wary and run when there is none - installation costs and long-term contracts will make your venture muddy. We have a price on the guaranteed Internet of about 800 euros per gigabit band. Again, carefully read the contracts - we are accustomed to in the Russian Federation that the excess is usually proportional to mandatory consumption, i.e. if gigabit is bought for $ 2,000, then the excess will go for $ 2 or less. In Holland, the excess can be at 50-70 Euro / megabit, you have to increase the guaranteed consumption! And it will not be able to quickly reduce it, the annual contract. Otherwise, everything is traditional, billing 95% burstable - 5% of the top monthly load values ​​are discarded.
The data center itself does not control the Internet - as a rule, it is outsourced to professionals. The data center can not afford to keep a few super-networkers, these are the offices that hire them.
Often there is a “recommended” provider. The price for dark optics is such that it becomes advantageous to make the route to AMS-IX only after 10G band in that direction. At the same time, little depends on the distance - the route inside Amsterdam and the Amsterdam-DPC route in the region will cost almost the same.
By experience, the Supermicro servers have shown themselves to be the best. Delivery time 7-10 days, 3 years warranty for everything.
Single-processor we take in the composition of Microcloud for 8-12 blades. Processors E3-1230v2, disks WDC RE4 500Gb 7200 rpm, 2Tb 7200 RPM. All machines with remote control module. 10 Microcloud systems in this configuration consume about 4-5 kW, and these are 80 servers.
We order larger machines from standard 1U cases with filling from the latest E5-2630 models and others. Since we do not resell, we can order from the supplier all that we want from fresh. We try to stick to the following popular articles on older servers:
Processors E5-2630, 6 cores 2,3GHz, E5-2670 8 cores, 2,6GHz
Memory - only 16Gb DDRIII ECC REG LV DIMM from Kingston, no warranty case
Mothers - Supermicro X9DRD, 2xs2011, 16xDDRIII (256Gb RAM)
Controllers: LSI Megaraid 9260-4i, Megaraid 9260-8i, HBA 9211-8i
Drives: SATA WDC RE4 500Gb, 3Tb. SAS - 300Gb 15K, 600Gb 15K, 2Tb 7.2K - Seagate / Hitachi
Supermicro enclosures 4, 12 and 36 slots for 3.5 "drives.
Server class SSD: Samsung 940 Pro, Intel 520 128 and 256 GB in size. You can order more cool models.
Total, 15 part numbers - everything is interchangeable.
The coolest thing that we took to order is the 2xE5-2630 / 384Gb / 36x300Gb SAS 15K / 5xLSI 9211-8i HBA system.
Any Tsisk comes as a side dish - from simple switches for VLANs to ASA5530 with all the bells and whistles. The purchase price is three times lower than in the Russian Federation.
We are connected through two 10G interfaces using our modern Brocade CER-2024 routers. This allows us to overlook rare DDOS attacks that have gone unnoticed through the upstream monitoring system. Inside, the entire network is made on Cisco Catalyst 1000 / 100Mbps. Everything is controlled and monitored centrally.
Everything is very simple - suppliers give us installments on servers in equal parts for 2 or 3 years without an advance. To do this, we are carefully conducting a white Dutch company, we have a credit history and we are playing a white game. We add the cost of data center, Internet, margin and get a price comparable to Hetzner or other players. Unlike well-known discounters, HOSTKEY can assemble any necessary config from the details described above and maintain it in the best possible way. Any clusters, virtualization systems, storage systems - everything you want, including the collocation of your client servers.
This approach allows us to manage one person in Amsterdam, who is busy with finances and interaction with suppliers. Everything else for the equipment, we decide through the Moscow office by the duty shift and one senior engineer who is responsible for the Netherlands site. These are well-established, polished business processes.
I hope this post will seem useful to someone, worthy of being added to favorites and save you from mistakes.
Author - director and co-owner of Hostkey.ru
The whole story began about 3 years ago, when we decided to open the site in the Netherlands. We drove around a dozen data centers in Amsterdam, Rotterdam and the surrounding area. Negotiated, collected offers and contracts.
All of them could be divided into two categories:
- "Cool data centers" / telehouses from 10000m2 with security, parking and all things with a price for a rack of about 2000 euros and a bunch of restrictions. You communicate with the manager, everything is beautiful, but there are full ambushes in the contract - I will tell about it later.
- “Normal data centers” for hosting, 100-200 racks - in the region, in a former warehouse or a former telephone exchange or something like a hangar. The price for a rack around 1000 with electricity - well, and how to agree. The contract is also not easy, but you communicate with the owner.
- The data centers of large hosters - as a rule they are looking for resellers, the reseller share in the Netherlands is hard - a margin of 5-20 euros per car, an obligation to order 10-20 servers per month to have a good discount. We did not consider such.
The difference ends - the technological solutions are almost the same. I saw these technological solutions in Moscow only in the coolest local data centers. Everything is perfect, optimized to never touch, what would work without human intervention. Full automation. Total outsourcing.
')
In general, from 5-6 real proposals, we then chose Servius and did not regret. For all the time we have not had a single accident due to their fault.
There is nothing more expensive than the staff
As a rule, the entire data center personnel for 100–200 racks consists of 5 people: the owner, he is also the chief engineer and seller; chief finance officer; technical support chief; a couple of junior engineers to help him. Salaries are high, taxes are high - there is little staff. All that can be endured is being outsourced. All stick to the conscience (unemployment of 20% and mortgage work wonders).
Consequence: there is no round-the-clock shift, as everyone is used to. You have to get used to the fact that the Dutch come to work at exactly 9 in their own time and leave at exactly 18. Overtime - 150 euros per hour, for this amount the owner himself will sit on the bike and come to see what happened to your server. We have never used.
We must pay tribute, if they began to work - then they will finish. In Germany, a Bulgarian engineer can dump at 18-00, leaving the case in the middle, and the server dismantled.
Consequence: No one will go anywhere to change disks, buy memory, etc. Your supplier should do all this. All issues are resolved only through KPN / DHL, and the courier arrives at the data center.
Moral: all servers must be equipped with remote access modules. All drives must be server class. Everything should be under warranty.
No rush
We here in Moscow got used to the fact that planning is practically a nonsensical thing and we are in a hurry to resolve issues. We do not trust anyone, we take the goods daily from the warehouse.
In Holland this is not the case. If you order something, the supplier honestly says - it will be ready in a week. And exactly one week + 1 day it arrives at the data center. Nobody holds anything in stock. All collectors order from distributors, this is brought by mail from different places and then collected and sent by mail too. So it is accepted and so everyone works.
The topic “I need a server tomorrow or today,” no one will understand. You order, you are called a term and in this period everything comes. Do not forget - if something breaks under warranty, the courier is called and the item is sent to the supplier. For example, if the server stops responding to IPMI, it is sent to the supplier as a whole. Disks accumulate until they have a box and go to the same supplier. A month later comes the same box with swap disks.
In the data center there is an SLA - paid. Since the data center is not a monster, and you are not the smallest customer - your requests will be serviced faster.
It is necessary to formulate accurately.
Lack of contrived formalities
A stumbling block to the use of regional data centers in the Russian Federation was the impossibility of sending servers to the data center without its own forwarder-engineer, which destroyed the entire economic effect of the price. At Certerius, our supplier simply sends the server by mail with the HOSTKEY mark, the data center accepts them, configures IPMI, mounts into racks, and connects the servers to the network. Similarly, they reconfigure servers, send parts under warranty, and so on. All work that can be done in 15 minutes is considered small and not separately paid. There is no problem for them to throw memory, replace discs, throw cars between racks.
It is very important - there will be no favors, describe everything in the contract. The contract is the law.
Rack pricing
In Russia, we are accustomed to the formula 42U and 5 kW. In Holland, usually in the price of 42U rack include a little electricity, about 1.5 kW. The base price is around 400-500 euros and the excess costs 15 euro cents. About 1000 Euros come out on a full rack. The consequence is simple - put economical equipment and pay less. Turn off the servers that are waiting for the client. The most disadvantageous thing you can do is keep the old servers, they will cost significantly more expensive to operate.
Read the contracts very carefully - in Holland and Germany, if you have ordered something for a year, you will have to pay for it for exactly one year, whether it is important or not . To abandon the rack ahead of time will not work, but sometimes it is necessary to warn for 3 months. So accepted. There are also five-year contracts - carefully look at the term for which you are signing. Then do not jump off.
the Internet
Typically, the data center gives its Internet, already delivered to the site. Be wary and run when there is none - installation costs and long-term contracts will make your venture muddy. We have a price on the guaranteed Internet of about 800 euros per gigabit band. Again, carefully read the contracts - we are accustomed to in the Russian Federation that the excess is usually proportional to mandatory consumption, i.e. if gigabit is bought for $ 2,000, then the excess will go for $ 2 or less. In Holland, the excess can be at 50-70 Euro / megabit, you have to increase the guaranteed consumption! And it will not be able to quickly reduce it, the annual contract. Otherwise, everything is traditional, billing 95% burstable - 5% of the top monthly load values ​​are discarded.
The data center itself does not control the Internet - as a rule, it is outsourced to professionals. The data center can not afford to keep a few super-networkers, these are the offices that hire them.
Often there is a “recommended” provider. The price for dark optics is such that it becomes advantageous to make the route to AMS-IX only after 10G band in that direction. At the same time, little depends on the distance - the route inside Amsterdam and the Amsterdam-DPC route in the region will cost almost the same.
What we use in our daily work
By experience, the Supermicro servers have shown themselves to be the best. Delivery time 7-10 days, 3 years warranty for everything.
Single-processor we take in the composition of Microcloud for 8-12 blades. Processors E3-1230v2, disks WDC RE4 500Gb 7200 rpm, 2Tb 7200 RPM. All machines with remote control module. 10 Microcloud systems in this configuration consume about 4-5 kW, and these are 80 servers.
  |  |
We order larger machines from standard 1U cases with filling from the latest E5-2630 models and others. Since we do not resell, we can order from the supplier all that we want from fresh. We try to stick to the following popular articles on older servers:
Processors E5-2630, 6 cores 2,3GHz, E5-2670 8 cores, 2,6GHz
Memory - only 16Gb DDRIII ECC REG LV DIMM from Kingston, no warranty case
Mothers - Supermicro X9DRD, 2xs2011, 16xDDRIII (256Gb RAM)
Controllers: LSI Megaraid 9260-4i, Megaraid 9260-8i, HBA 9211-8i
Drives: SATA WDC RE4 500Gb, 3Tb. SAS - 300Gb 15K, 600Gb 15K, 2Tb 7.2K - Seagate / Hitachi
Supermicro enclosures 4, 12 and 36 slots for 3.5 "drives.
Server class SSD: Samsung 940 Pro, Intel 520 128 and 256 GB in size. You can order more cool models.
Total, 15 part numbers - everything is interchangeable.
The coolest thing that we took to order is the 2xE5-2630 / 384Gb / 36x300Gb SAS 15K / 5xLSI 9211-8i HBA system.
Any Tsisk comes as a side dish - from simple switches for VLANs to ASA5530 with all the bells and whistles. The purchase price is three times lower than in the Russian Federation.
Network
We are connected through two 10G interfaces using our modern Brocade CER-2024 routers. This allows us to overlook rare DDOS attacks that have gone unnoticed through the upstream monitoring system. Inside, the entire network is made on Cisco Catalyst 1000 / 100Mbps. Everything is controlled and monitored centrally.
Why such a price
Everything is very simple - suppliers give us installments on servers in equal parts for 2 or 3 years without an advance. To do this, we are carefully conducting a white Dutch company, we have a credit history and we are playing a white game. We add the cost of data center, Internet, margin and get a price comparable to Hetzner or other players. Unlike well-known discounters, HOSTKEY can assemble any necessary config from the details described above and maintain it in the best possible way. Any clusters, virtualization systems, storage systems - everything you want, including the collocation of your client servers.
This approach allows us to manage one person in Amsterdam, who is busy with finances and interaction with suppliers. Everything else for the equipment, we decide through the Moscow office by the duty shift and one senior engineer who is responsible for the Netherlands site. These are well-established, polished business processes.
I hope this post will seem useful to someone, worthy of being added to favorites and save you from mistakes.
Author - director and co-owner of Hostkey.ru
Source: https://habr.com/ru/post/181173/
All Articles