Purchase of a modern Intel server with NVMe P4800X
September 17th, I bought for personal use a server costing $ 7680 from a supplier in a vice.
Delivered to the Moscow data center tech.ru and began operation.
Photo report available. The article consists of 5 parts:
In August 2015, circumstances were well established for me to start making an online game.
I had a fairly large selection of how long-term construction to do. For several reasons, I chose an online game.
')
He began to do it, and for half a year of hard work, he advanced well, but suddenly his savings ran out. Nevertheless, I went very far in this question and then continued to refine the online game in my spare time. The game requires a lot of server resources - to calculate ratings.
For 10 years I have a personal server in the Kiev data center hvosting.ua. I got it by chance: “Do you want a server like that for 750 euros and only pay for a place in the rack?”, Well, then I once had extra 750 euros and I agreed. The existing server was enough for about 500 people online. Therefore, I had to ask all the players "do not tell anyone about the game." And now I would not like to show anyone the current version, 5 people are actively playing - more than enough at the moment.
It became clear that you need to take or rent a new server.
I discarded the rental option, because I have a positive experience in placing my
server duration of 10 years.
There is a negative experience of closing a successful startup due to expensive hosting. Well, that is, you had to pay $ 600 per month for hosting, and the income at some point was less than $ 600 per month.
Accordingly, I could not show it in the portfolio - and I think it was a terrible damage to my income as a whole.
It is because of these two reasons, I decided to buy a server, and not to rent.
That is, if an online game becomes financially untenable, it will still remain online and be available for the portfolio. And the cost of maintaining it will be quite feasible, even with zero income from the game itself.
In addition, I need a server for several non-commercial sites and teamspeak servers for a large guild in a well-known online game.
Overview of all components in this server, everything is still in the boxes, now there will be unpacking
I decided that the server should pull 50,000 players online and 1,000,000 registered players.
When these indicators are achieved, the startup will grow into a business and there will be an opportunity to buy new necessary equipment. But in general, why exactly 50,000, and not 60,000, we will not consider, the article is not about that.
Further configuration selection - selection of parameters, under the required parameters.
At the time of the beginning of this story I was rather poorly versed in the server hardware and what I really needed.
I believe that the train of thought and the logic of decision-making will be very interesting to all involved in the topic. Or decision makers.
All components in their boxes, now will begin unpacking and installation
To reduce financial risks, it is necessary that the monthly expenses be minimal. Accordingly, the server must be 1U in size.
The server part of the game is designed in such a way that almost everything can be run in multi-threading. That is, it is even possible to share the task of rendering the game world among several servers.
This means that when choosing "several productive cores" vs "many mid-performing cores" - one should make a choice in favor of the second option.
Having made some measurements, I decided that for 50,000 people online, 50Gb of RAM will be needed (again, the article is not about why so many).
Suddenly it turned out that the game makes a lot of records on the disc. So it should be ssd and quite powerful.
There should be some hdd for backups. I am one of those who are "already doing backups."
Processor manufacturer: Intel, AMD, MCST.
AMD - I tried to use the official AMD site, but unfortunately it is impossible to understand there:
The decision to abandon the purchase of AMD - a very uncomfortable site and outdated processors.
The MCST is certainly very interesting, the very fact of the existence of Russian-made processors is very interesting. Which is even possible to buy.
The inability to buy on behalf of a private individual is very strained, but the issue is resolved.
The inability to study the full characteristics of the processor, there are no tests in the public domain. Let even synthetic. I also note the lack of support for DDR4 - but this is excusable (for AMD this is not excusable). The decision to abandon the purchase of the MCST was after realizing the impossibility of using the “apt-get upgrade” command.
Intel - only this option remains. There is support for DDR4 memory in the latest line of processors. There is a special section of the site in which they dump a sheet of characteristics of the processor - I can find there all the characteristics that interest me.
Turn off the processor radiators
Turn off the processor radiators
The time of installation of the processor E5-2630 v4
Installed processor E5-2630 v4 close-up
In the free market there are server options 1x, 2x, 4x, 8x.
The option to buy a motherboard with multiple connectors and eventually install a complete set there - I decided not to consider. Why? To buy equipment this is a lot of time, nerves and money. This is different from a home computer.
Variants of 4x-8x processors turned out to be incredibly expensive and even in the most modest configuration they were with an extremely heavy price for me. Single-processor and dual-processor systems were for different niches of the market. Here is a list of processors to consider (those that start at "E5-4" are uninteresting, they are for 4 processor systems).
Consider the options "E5-1" - this is a processor for single-processor systems.
Immediately striking is the high base frequency and high burst, but the number of cores is very small.
For example, the “E5-1660 v4” 8 cores - $ 1113, and the processor for a dual-core system, but with a lower frequency “E5-2620 v4” 8 cores - $ 422.
The price is 3 times less in favor of a dual-processor system! I remember that more weak nuclei are much more useful for the task than few strong nuclei. This decision was made to buy a dual-processor system.
Both E5-2630 v4 processors are installed in their slots
Applying thermal paste to the processor
Thermal paste spreading on the processor
All the same, there remains a fairly large selection of processors, what choice to make. I pay attention to prefixes like "L" or "W" or "A".
“L” is low power consumption. For me it is not interesting. If I would have my own desk in the data center - it would matter, and since I have 1U and no data center wants to put a counter to my server, this is important.
"W" - increased power consumption, "A" - a very good copy in the line. Pretty quickly you can see that the remaining processors are divided into 2 groups. A small number of cores + high frequency, a large number of cores + low frequency.
Drop the first group. Total left (cost above $ 1500 apiece, I decided not to consider):
E5-2603, E5-2609, E5-2620, E5-2630, E5-2640, E5-2650, E5-2660.
Immediately rejected too weak processors: E5-2603, E5-2609, because the money just enough for 2pcs E5-2620.
There are 5 options: E5-2620, E5-2630, E5-2640, E5-2650, E5-2660.
Upon closer examination, it turned out that they can be divided into 2 groups according to the maximum permissible memory frequency:
E5-2620, E5-2630, E5-2640 - 2133Mhz
E5-2650, E5-2660 - 2400Mhz
Consider 2 processors from this group:
The frequency E5-2630 vs E5-2650 do not differ. The difference is only +2 cores (and the memory frequency is a bit more) and the price is 2 times more (or + $ 600)! The second group is no longer due to the price.
Now consider the E5-2630 vs E5-2640:
Only the base frequency + 10% is different, and the price is one and a half times higher. I clean option E5-2640. Incomprehensible overpayment for me.
There are only E5-2620 vs E5-2630:
The price difference is $ 250, and for them we get + 10% to the frequency and +2 cores.
I am ready for such a price to receive such a premium.
Total selection is made: Intel Xeon Processor E5-2630 v4
Installing the radiator on the processor
Spin the heatsink on the processor
How processors are visible in bios
I remind you: at the idea level - I decided that I need ~ 50Gb of memory. On the Internet, I found that collecting a server with a memory that is not a multiple of two is quite acceptable.
But still. For reasons that all memory should work in the same way — so that I didn’t need to be soared to which kernel how difficult the task was to submit - I decided that the amount of memory should be a multiple of two.
Accordingly, it turns out that you need to put 64Gb of memory. The server part of the game is written in such a way - that on the server side the whole game process is completely calculated.
Thus, cheating is impossible. However, if suddenly there is not enough memory to fully fit in the memory of all players online - there will be a small disaster, the game simply will not start, and I will need to urgently make some kind of hotfix. For an opportunity to dump intermediate calculations on a disk.
It also turned out that if many memory dies are installed, the speed of the processor’s communication with the memory drops.
Therefore, if I buy 64Gb now, and then I buy the same amount, it will be much worse than if I buy 128Gb right away.
The difference in price turned out to be quite feasible, maybe it would be enough for 256Gb, but there was no extra $ 1000, the entire reserve eventually went to ssd.
Installing the memory dies, attentive gopher here can see 2400 memory. In fact, this memory is here for photos, the necessary memory arrived at the very last moment.
I already knew that if you put too many memory dies, it would work significantly slower than if you put fewer memory dies. So what - the best option is to put one plate on 128Gb?
It turns out that processors have such a thing as “memory channels”. The best option is one memory plate in each memory channel. This rule turns out to work for gaming / work computers. Surprisingly, all sorts of bloggers do not convey this simple idea to the public.
My case has 2 processors, and each processor has 4 memory channels. That is, for my case it is best to put 8 dies of memory.
Total selection: 8 x 16Gb
Installing Dies
The choice was not great. Firstly, the choice at all at the moment is 2133 and 2400. But the processor pulls 2133. Does it make any sense to take 2400?
It turns out not only makes no sense, but also harmful (for my case). It turns out that memory access is performed in 17 clock cycles at 2400 memory, and in 15 clock cycles at 2133 memory.
But my processor cannot work with 2400 memory, it will work with it anyway at 2133. However, if the processor were E5-2650 v4 or higher, then on the contrary, it would be more advantageous to install 2400 memory.
The next factor is LRDIMM. This technology allows you to install more memory on the server and when installing a maximum of brackets - this memory is faster than if without LRDIMM.
LRDIMM is the usual server memory + some additional chips. Accordingly, the price is + 20% approximately.
For me, respectively, LRDIMM is not needed, because I do not install memory in each slot.
Now the manufacturer. Among the server memory is the manufacturer does not matter. I now do not understand how this is possible - but true. Synthetic tests are the same. The memtest data is about the same. For Kingston memory, I found an option: 2 options for chip saturation, but even in this case there is no functional difference.
General view of the installed processors and memory
How is memory visible in bios
When I first made the desired server configuration - I wanted 1 ssd for 240gb and one hdd for 2tb.
As I found out what I needed, the understanding of what I really needed changed.
I had to learn that ssd disks are very, very different. What servers better to use raid1.
What appeared new technology NVMe and it has a lot of options. Raid controllers are very different for different tasks.
Let's take it in order. Starting with "and what is needed for the task?". You need a 200Gb ssd disk capable of withstanding a lot of rewriting. And 2Tb hdd for reliable backups. "A lot of rewriting" - with very approximate calculations it is 1-2Tb per day. Reliable backups - this means raid1 configuration. A hardware or software raid is a question of the amount of effort, qualification and responsibility when repairing a raid.
In terms of management, the choice is as follows:
Accordingly, Tiscom advised the cheapest of the modern hardware raid controllers, which would allow to repair possible damage just like that.
This: ark.intel.com/ru/products/82030/Intel-Integrated-RAID-Module-RMS3CC040
I note that this is a raid controller for an Intel motherboard. That is, if I would choose a non-Intel platform, then I would have to take something else and take up a whole pci slot.
I have no idea what it is. Or something related to encryption or something related to the raid card
Preparing a raid card for installation
Installing a raid card
How raid controller is visible in bios
I was going to buy Western Digital due to the fact that I have a very positive experience using such hdd for home computers. But the Vice stated that they could only make a replacement within a week (with or without warranty). And they offered HGST - they said they could make a replacement within 24 hours.
I began to search the Internet for what they write about the reliability of HGST - like normal reviews, and agreed with this choice of disk manufacturers.
Next volume. 2Tb is necessary, good. And what is more expensive? + 1Tb is $ 70 more expensive, well, then I won’t forgive myself what I saved on the volume, and if I suddenly need more space later. Well and further along the line, the volume increases a little bit and the price increases slightly. Between 6Tb and 8Tb, there was a border between my budget possibilities.
Total selection: 2pcs 6Tb HGST HDD.
2 HDD HGST 6Tb
Installing some part of the raid controller for the HDD recycle bin
Connecting the HDD basket to the Raid card RMS3CC040
The first option: just put 1pc ssd 200gb. Next, I learned that the more ssd - the more it has a rewrite resource (other things being equal). And so that this rewrite could be performed faster.
For example, I will have 200gb of data and a volume of 240gb vs a volume of 1000gb. So, all other things being equal, a large-volume rewriting resource will be 4 times larger.
Let the experts forgive me, but for making a decision it is not at all obvious.
Further, I understood, you need to use raid1 and you need 2 such ssd. Yes, even if backups are made at least every hour and it will not be terrible to lose a single ssd. But there OS, settings of everything. Current DB. We have to urgently spend a lot of time and effort on recovery.
Ssd withstand 2tb per day overwriting is very expensive. And it is also necessary to have some stock for this indicator.
Then I learned about the NVMe technology. This is simple: the ssd drive is connected not as usual, but through a pci slot.
The data band is bigger and faster - this is important for my task. Immediately the question arose - is it possible to combine them into a raid?
It turns out you can, but there are no controllers like this for raid. There are prototypes with an unknown price.
In general, at the time of purchase - it was an unacceptable option.
Another factor in the decision: let's say I bought 2 expensive ssd disks, combined them into a raid. But the resource rewriting they equally quickly consumed. And what will I get? What they will fail at the same time?
Next, I found out that most often ssd fails not because of a rewrite resource, but because of a controller breakdown in ssd. Yes, for this raid will be relevant.
And the choice is between pci-ssd (NVMe) and raid1-ssd. And taking into account the above considerations, for me, definitely better than pci-ssd. Well, it's just stupid there is not another $ 2000 for backup ssd, and if there were, then you need to take a 2U system and buy another basket for $ 400. Well, or buy 10 times worse ssd.
Well, yes, I am ready to trust a well-known brand (in this case, Intel), in the hope that they have a sufficiently reliable device so that the claimed period and resource will work without failure.
NVMe P4800X installed in the riser
In ssd, I am interested in recording iops, the speed of a single operation, volume, rewriting resource.
Revising the characteristics of thousands of ssd, I believe that in my case, you must first look at the stated number of rewrites.
Almost all ssd can be divided into groups by the number of overwrites of its volume per day, for 5 years. Groups: 0.3, 1, 3, 10, 30, 100.
Almost always the exact match between the rewrite resource and iops to write. Separately stand out ssd, reading from which is performed with an incredible amount of iops, but relatively weak to write. Such to me for this server are not interesting.
Manufacturers: Intel, HGST, Huawei, HP, OCZ. All sorts of different ssd - an incredible amount. Honestly - I have little idea how people objectively choose ssd at all. There are so many Ssd, they are so different, the price range is so huge.
From my point of view, Intel products were best grouped. They managed to break all the ssd into groups so that I could form an opinion which device I need and which one I don’t.
3 lines of NVMe pci-ssd: P3500, P3600, P3700. 400Gb each:
P3700 - 75000 iops - $ 1000, 10 rewrites per day
P3600 - 30000 iops - $ 650, 3 rewrites per day
P3500 - 23000 iops - $ 600, 0.3 rewriting per day
Actually it is obvious that the P3500 immediately disappears. For $ 50 a tenfold increase in an important parameter to me.
Am I ready for $ 350 to get ~ one and a half times more iops and 3 times more rewriting resource? Kaneshna, yes.
So I was going to buy a P3700 for 400Gb, but a couple of months before the purchase, the P4800X went on sale.
It costs almost 2 times more expensive than the P3700. Slightly less volume. iops on the record 500000. Resource rewriting 30.
That is, for a total of $ 800, I get 8 times more iops and 3 times more rewriting resources.
Well, honestly speaking in terms of iops - it’s just beautiful and in spite of the fact that it wasn’t easy to get an extra $ 800 - I decided to buy such ssd.
The bonus is a faster recording of each individual team.
Choice: www.intel.com/content/www/us/en/products/memory-storage/solid-state-drives/data-center-ssds/optane-dc-p4800x-series/p4800x-375gb-aic-20nm. html
Raiser with NVMe P4800X install
NVMe P4800X installed
How NVMe is visible in bios
Separately, I want to note that there is a type ssd SLC / MLC / TLC / QLC. The meaning of the difference is clear: Several data bits are written to a single physical cell at once. Accordingly, SLC - only one. MLC - two. TLC - three. QLC - four. From left to right - from expensive to cheap. Due to the speed of writing, reading, rewriting resource.
SLC - the most reliable, the fastest and ... monstrously expensive. For example, "Micron RealSSD P320h" is a hairy monster with a volume of 350Gb for only $ 5000. Declared iops to write 350000. Unlimited rewriting resource.
MLC - are very different. Well, at the factory, they make a bunch of memory chips, something turned out well - they thrust into the server version, something not very much - they thrust into the home version of ssd.
TLC - they used to do frankly fast failures, now they seem to be doing good ones. And they are trying to make an idea that TLC is good (so that the old ones will buy up stupid buyers).
QLC - tried to do, but apparently the commodity option no one could do.
Here Intel recently released 3D-XPoint, this is some kind of new memory. As for me - puff cake from ssd chips.
For servers, in contrast to home computers, the choice of motherboard is completely different. For a home PC, the size of the actual computer is not important, but for the server, you have to put everything in 1U (This is 4.3 centimeters!).
From here another approach to the selection of motherboards immediately appears. Not motherboards, but platforms are on sale. That is, the motherboard + case. The one who sells only motherboards separately - immediately loses in the competition.
Platforms for this processor come from different manufacturers. With the smallest possible modifications, on the sites, you can specify a link only to a specific modification (I will say a bit about the modification later):
HP and Dell manufacturers - I will not consider because of the monstrous overpayment is not clear why.
- Supermicro
- Asus
- Intel
- Separately Intel
The choice of these four options is the cheapest option suitable for all requirements from each of the listed manufacturers. The price they have about the same turns out. The R1304WFTYS option disappears as soon as I see the 1100W power supply (why this is important - more on that later).
I thought about ASUS so that they recently climbed onto the server motherboard market, it is not clear what I would risk.
About supermicro - they are actually only server platforms and have been doing for a very long time. But ... raid controller. Either I take the Intel platform or I need another controller and occupy a PCI slot.
Therefore, I decided to choose the Intel platform R1304WTTGSR. But I note that it is 10% more expensive than a similar platform from supermicro.
Modification: with 10 gbit ethernet. The price difference is $ 100, and if I suddenly need a fat channel, I would not want to touch the server and install something there. And I probably need a fat channel for the start of the game.
Platform R1304WTTGSR boxed
This is what we see when we open the new platform R1304WTTGSR for the first time.
All components are installed, it remains to close the cover
Such information about the platform is visible in bios
Of course, first of all I turned to hvosting.ua, because I worked with them for more than 15 years. And the volume of transactions exceeds the cost of the server being discussed. But hosting a server in Kiev after the well-known events of 2014 is unacceptable for security reasons.
hvosting.ua offered me accommodation at the ttc teleport data center in Prague.But the discussion stopped on the question "I want a piece of paper that is legally binding that I am the owner of the server." Well, forgive me, I am no longer a student completely trusting everyone.
However, if someone really needs to host something in / in Ukraine - I recommend hvosting.ua uptime ~ 1200 days, then the planned transfer of the server to the neighboring data center and another uptime 700 days. Then I tried to do “apt-get upgrade” without rebooting the server :) It didn’t work for debian, it couldn’t update itself in 2000 days - it’s not possible to fix some kind of dependency. Hvosting helped to settle.
In general, for 15 years there was not a single power failure and the channel fell for 15 minutes every 4 months, 2 times the channel fell for several hours.
Especially delivered broken oauth authorization vk (they have vk blocked there).
There was a question about buying in Russia and placing also in Russia. Why purchase and placement in the Russian Federation? And so I would be legally protected. There was a pleasant moment, at the very beginning of the search - the choice of the best supplier, no recommendations, no bias - all are equal. I remember how I started with the banal search “datacenter in Russia” (later I found out that in the Russian Federation this is called the data center). And "buy a server in Russia."
I think it will be correct to tell in chronological order.
12/05/2016
In the beginning there was a very constructive conversation, with an interesting proposal. I was studying the possibility of the raid1-NVMe variant, asked if they could put raid1-NVMe on the intel platform?
Suddenly I received this question: a rude refusal to answer (by the way, I would be satisfied with just the answer "no", I just really was not aware of this matter). Then I did not want to communicate with them.
They first saw the P4800X on sale and therefore I decided to ask them for the purchase details.
The conversation ended with the fact that without my personal presence in Moscow, it was impossible to make a deal.
12/05/2016:
They write on their site that 12 years of experience. And other cool performance. DNS whois shows that the domain was recently registered (3 months prior to review). I immediately have a question for them: “why such discrepancies?”. It turns out before that there was another company to which there were reviews on the Internet of the type “paid for a long time, there is neither a server nor money”. It would be possible to forgive specialists who created a new company after working with a bad director. But unfortunately, what they wrote - did not agree with the history of the DNS. I decided that the risks were unacceptable, although they offered an interesting price.
12/08/2016:
Citizens trade in used equipment. Then I still considered such options. Over time, I decided not to consider it, because DDR4 is not used. sdd-hdd B / Y is better not to take anyway. Well, it turns out they have nothing to offer me.
12/08/2016
This is a regional equipment supplier. Principle is a good option - they are also nearby. Then I still thought that there could be a decent data center in Krasnodar, but no - if there are decent ones, then they are packed to capacity. The paradigm turned the prices up and did not answer my stupid questions.
12.12.2016:
Here is such a link still working: server.trinitygroup.ru/expert/25736
As I recall, I was confused by the price. Honestly, even now I look at this price and do not understand where such a high price comes from.
12.12.2016
They were recommended by the data center tech.ru type they themselves cooperate with them. I listened to all my Wishlist - picked up exactly what I wanted of my time. At a great price. Then he began to dig up the DNS whois and reviews on the Internet.
DNS whois corresponded to the one described in the correspondence. During the investigation many curious details were revealed.
It turned out how the company was founded, by whom, how it changed. Found how to change the site and asked for feedback about it somewhere to leave.
There were employees of the company. There were positive and negative reviews. All this said that the company is old and is actively engaged in something constantly.
I also checked the Intel certificate - it was at least outdated, but real. I checked some other random certificate - also a real one. For this I was very pleased.
There is also https. Also adds points in the direction of choosing their company.
From the very beginning, it looked like a very good option.
12.12.2016
The first communication is positive. But the price - just no words how big. Until now, there hangs an example brutalized by greed . Choose 400Gb NVMe 75k iops. This corresponds to our Intel P3700. It turns out that everyone is selling this piece of iron for 60000 ±, and they are pushing it for 90000.
Okay, maybe this new site with some kind of bug? .. But no, this is not a bug. Here we find the same on the main site with the old design:
stss.ru/products/servers/T-series/TX217.4-004LH.html?config=
Here, in the appropriate place, choose the same piece of hardware and also get +90000 to the cost.
50% of the bonus money is definitely very profitable, but I'm sorry, we will take the other way.
12.12.2016
In general, it was a very decent option and the DNS whois and the reviews looked good.
But the question “can they, without my presence, deliver the server to the data center?” Was not answered.
So it would be vserez would choose between Tisky and Desten. If they had a more patient support - maybe I would have bought a server in the end.
12/13/2016
They just sent me the configuration and price - I was horrified and did not answer. Now I look - the price is adequate, and why I was horrified earlier - because I did not understand the meaning of the word “DWPD”. Apparently, I had a telephone conversation with them.
12/14/2016
Citizens tried to sell me Dell equipment. Again, I see an overpayment for the brand is generally not clear why. An employee explains to me that there is an overpayment for operational assistance. But I do not understand something - the server is in the data center tier3 rack. No one will kick him down. In addition to the factory marriage - nothing can happen there. Plus, they said that they couldn’t install a server like a Tis without my presence - it was not clear what to say next.
01/10/2017
Now I see on their site a poor server configurator. Then they had only a business card site. And there it was written that they were considering options for installments, and I decided to ask about this topic. They said that such small amounts for installments are not interesting. Servers have DELL / HP - that IMHO overpay do not understand why.
Allow me to justify my lack of interest by these companies.
www.regard.ru/catalog/?query=2620+v4
Someone will explain to me the difference between "Intel Xeon E5-2620 v4 OEM" and "Dell Xeon E5-2620 v4 (338-BJEU)"? I now see only the difference in price by 2 times! The processors themselves make Intel chips. Ready to believe that they put the solder on the other, well, ok + $ 50 is ready to understand. Explain to me stupid for what the company DELL in this example takes + $ 500?
Apparently somehow a reminder worked for them successfully and they called me a week before the final agreement on the configuration with Tisk. They tried to intercept me as a client at the last moment. But ... the price was higher than the Tisky.
As well as a parcel post, ComputerUniverse, ogg and other sites. The option that delivers me to the outback, and then somehow I deliver to Moscow - it looks terrible. Not only that all profit from refusal from intermediaries comes at the cost of delivery. So also time will eat incredibly much. And most importantly - the risks of damage appear (strike about something for example).
The option with delivery to Moscow - you need somewhere a room where I can build a server plus a personal presence. Plus transport between point of issue, assembly point and data center. Plus, the risk of nosyachit when assembling (if there is a secondary posting is not so connect - there will be no way to fix it).
I noticed that the price varies for each piece of iron in different stores. That is, you can run around Moscow to buy money in wounded shops. But again the risks of an assembly curve. Personal presence, a place to build and transport around the city. And the question is - where is the profit from this event?
In the outback, of course, there are shops and they are ready for a long time to deliver anything at a reasonable price. However, as it turned out, they somehow have a friend’s brain work. I explain to them that here ssd dear, I need this here. And they - we do not have it, so take another with similar characteristics. From "similar" only volume is considered.
But again - they brought it, it is necessary to collect (there is a risk that something will be crooked), then delivery, personal presence, etc.
Very interesting option. The person collects computers to order. Speaks directly: we take 4% of the commission. All additional costs are at your expense. I really liked this position, but there are no guarantees that it will not disappear after receiving the money. I now do not understand.What prevented him to say, they say that there is an option with the contract, everything is legalized correctly. This option will increase the cost here so much. That would be a very interesting option. Here is the basic price, here are my 4% surcharge, here are the additional costs associated with delivery, here are the expenses on legal issues.
I never spoke with these guys, but I spent a lot of time on their site.
I have to admit - they have the most convenient configurator from all sites. Convenient not from the point of view of prettiness, but from the point of view of how “how can you figure out all these characteristics?” I didn’t communicate with them because they have a full 15% comparison with all the rest. They have all sorts of certificates - for myself, I concluded that the municipalities of the Moscow Region or Moscow region are the best supplier of equipment. But for me there is no - primarily because of the surcharge.
Their configurator is so beautiful that you deserve special attention. Consider this example.. Perhaps they should sell it as a service to other companies? Type download their prices and what is there, and users themselves are assembling what they need. Well ... if Tisky had such a configurator, that would be super fine.
At first glance - bad because of the design. But we are not here a beauty salon.
I would also note that in spite of the seeming simplicity there is a huge amount of small and useful functionality hidden in it. Platform. Thanks to this site, it finally reached me how motherboards for servers are selected (in the form of a platform and their modifications). Also here you can immediately see all the modifications. It is seen that for ssd and for hdd. It can be seen that 10GbE and 1GbE. It immediately becomes clear that if I want to set myself hdd - then 2 modifications do not suit me at all. And the price difference on 10GbE and 1GbE is 1GbE + 10% = 10GbE. And that's all, no more thinking and comparing, asking stupid questions to the posters. Looking at it - everything is clear right away.
Next, a block with processors - you can clearly see which processors can be plugged into this platform. A processor with a bunch of characteristics — at the “I don’t understand anything, but I want to understand” stage — dumping all the different characteristics on the bars is very useful and very reasonable. This section helped me a lot to understand which characteristics of the processor are important for making a decision, and which are not.
Then a block with memory and it suddenly also helped me a lot to figure out what characteristics to look for in order to make a decision. Here it is very clearly seen that we must take into account the frequency of the memory and RDIMM / LRDIMM. I wrote about this before.
Then I understood how to build raid1-nvme. Let's see here. There you can select a basket for ssd-nvme. After that, you can select nvme-ssd in this basket. And then any modern raid controller will be suitable at the rate of 4x pci lines for each nvme device.
So just about the most important thing - I'm probably so stupid, I spent more than 20 hours to understand 3 lines above.
Yes, it is not yet possible to assemble a raid1-nvme hardware in modern 1U Intel. Due to the lack of appropriate baskets.
I spent a lot of time here choosing configurations and comparing prices. The prices here are good, but they are constantly popping up "out of stock" - which makes it pretty difficult to plan. I did not contact the store. Because what is actually talking about? Here is the product, here is the price - pay and take. As it is necessary to take it in person, the option is not impermeable.
I tried to build a server, but they were constantly missing something. And what was - not at the most interesting price.
Of all the options listed, it is obvious that the Vice is best for me. Before buying agreed on the final configuration. Signed a contract remotely, then after purchase - in paper form. On the 4th day after the payment, as planned, the server was delivered to tech.ru and in the evening I began to set it up for the same. Through bios I checked that all the equipment is in full and in accordance with the agreement.
Everything went flawlessly, even nothing to complain about.
In the above photo flashes hand. On this hand is a blue armband with a wire. This is not for fun, but grounding. This is an absolute plus to the qualifications of employees Tisk.
I have found correspondence in history with only 4 data centers, but I remember exactly - there were many more of them. I remember exactly - there were different phone calls. I remember exactly - there were site reviews.
Pretty quickly you can find out that there are different levels of the data center. tier1, tier2, tier3, tier4. Or even without this brand. tier4 in the Russian Federation there is only one. For some reason, I can't find the offsite now, but I remember I found it and saw that it was impossible to get through with my 1U. Yes, and probably the price would be too high for me.
Working equipment in the data center tech.ru
tier3 is a lot of options in the Russian Federation. I took this list and began to look for compliance. I will now take this list and go from top to bottom:
Possibility to rent only racks. Would write here a list of intermediaries, through whom you can shove 1U to them.
Clearly just for the needs of the bank itself.
Not for resale.
Only racks trade. There are no intermediaries.
These are already offering something for 1U equipment. I remember that I looked at this place before, but did not like the prices and therefore did not even write to them. They did not write anything on the site about the cost of communication channels - and even without communication channels, the unacceptable price is already obtained. I was also embarrassed by the lack of photos. I'm also confused by the lack of https. I understand that they built the site this way, that they now need wild-ssl, which is significantly more expensive than the usual ssl. But there is no money for a certificate or there is not enough intelligence to properly organize the site - it is also not a plus for them.
This bank also built its own data center.
I somehow missed such a data center. Now, if I was looking, I would love to contact them. At prices nothing is known - all through the "send request". Perhaps this confused me half a year ago. The absence of https is surprising - they spent several million dollars on the construction of a data center and there was no $ 10 to buy a certificate. Attention to the question: what else did they save, or what else did they lose sight of?
This is something public.
This is the data center of the mobile operator.
Only racks trade. There are no intermediaries.
Akado - Internet provider has built a data center.
On closer examination, it is clear that they will soon expire a tier3 certificate. This means that they have designed the data center in accordance with tier3, but for some reason have not yet implemented it. Https no, no pictures. The prices are very good, why I didn’t write them - I find it difficult to answer.
Total from the data center, ofcocially relevant tier3 - for me, no one was acceptable. What to do next? I tried to search in different forms on the Internet sites of different data centers:
February 14, 2017
Apparently they first caught me in advertising or searching. I remember exactly that I was very stressed - that the addresses of each data center were not indicated, now I can already see it. I also remember the terrible layout on the site. Even now, debian + firefox looks awful. When discussing prices, I had "stupid" questions, such as I do not understand what to pay 30000r / month. They decided not to answer one of these questions. Well what can you do - mine and the truth I am not their client. They specialize in payment systems there.
Surely I found more than one data center, but something I did not like. I remember that I began to learn "and what is the correct data center?". Watch all the videos on the Internet about the data center and got the idea of ​​what I should see in the pictures.
I tried to get into the local data center Rostelecom. Because my provider is Rostelecom. And the organization is decent. Spent $ 3000 fiber - very good call quality.
Offtopic: Rostelecom has the most terrible native DNS - most probably, most users have a terrible opinion about this provider. But they do not offer placement services for their equipment, at all, nowhere.
Then I just called them in support of the central and asked - "and what will you recommend me?". This employee recommended me “hc.ru” and “tech.ru”.
12/16/2016
Patiently answered all my stupid questions (I now see and understand that questions are stupid). They offered a bundle of supplier + data center, besides them, only the data center in the Baltic States have proposed this. They have a modern website, as you saw in the analysis above - not everyone has that. Photo equipment - consistent with the idea of ​​what should be the correct data center (actually all who posted photo-video match, but not all posted). There is an unusual channel "burstable rate". Prices from my point of view a little lower than at all. Well, made a special offer in the form of discounts on power supply (PSU).
It is necessary to explain a little why all the data center was brutalized with this indicator of the maximum power consumption of the PSU. As a consumer, it’s not at all clear to me why, when paying, they look at the dial-inscription on the BP. It would be logical to look at the power consumption or in the end - the maximum possible power consumption. In my case, if you collect all the glands to the heap - you get 420W the maximum possible consumption and the actual consumption now is probably about 100W. And on my PSU 750W is drawn - it means I have to pay for it 1900 rubles a month for exceeding the power of 400W. And they are not the only ones - it turned out all the polls are so strange. Why?
It turned out that for the case of shutting down the servers — and then simultaneously turning them on, the maximum possible current is consumed.
Well, it turns out that in a critical situation - the operator or simply click "turn on everything." Or if there are a lot of people like me, I will have to turn on each server in turn. Respectively longer downtime for some customers. I didn’t find any more reasons, but this reason also completely explains the oddity.
The next oddity is the connection to the 100 Mbps / 1 Gbps / 10 Gbps ports. Even if very little data is being chased through this port - sorry, this is very expensive. And this is in no way obvious to a person who is not aware of these nuances.
Login to the data center tech.ru
12/16/2016
They wrote to me in support “Write to us, please, when the server is ready for installation in DC”. Well, that's how I perceive it? How polite non-cooperation? After 2 months, I asked them if they had a food restriction? Suddenly it turned out that there is. Well, what the hell they told me to first buy a server, and then find out what you had to buy? At this time, I decided that it is unacceptable to communicate with such support.
01.18.2016
I asked them whether there is an opportunity to order equipment. They offered only a collection of dedicated servers to order.
The choice was apparently made in favor of tech.ru. First of all, because they work closely with Tiscom. Secondarily due to good support. Thirdly, due to nutritional discounts.
The fact that there is a burstable rate, a well-designed site, https, photos, and other trifles was also a plus.
For the month of use, 2 downtime channels were recorded approximately 15 minutes and 3 minutes, respectively. When connected, the ip address routing was mixed up (7 ip addresses per 10 Mbps <-> 1 ip address per 100 Mbps channel). The rest is all fine.
I can also come up with the fact that there is no service type API to find out how much time UPS has before shutdown. Something like this is in hetzner.
An example of a rack in tech.ru with working customer equipment
When they found out that I was going to write an article about them, they asked me to leave this phone here: +79055628383
I took out separately the problems that I had while setting up this server.
Not standard problems from my point of view. That is, a solution which is not easy to google.
The first problem is how to make NVMe disk bootable.
I started with a simple one — I tried to set it as it is. Did not work out. Well, that is, Debian was installed and made NVMe bootable, but the BIOS did not see this boot option, well, no way. I didn’t want to touch in the BIOS what I didn’t understand, but from what I understood, I tried everything. From the boot options found UEFI option. But he did not want to run boot-NVMe through this option. Having rummaged in UEFI - I was convinced that from the command line it is really possible to get information about the installed NVMe. And this means that it will really somehow boot from it.
Actually it became obvious that you need to register something in this UEFI.
Photos from the UEFI excavation site. Mass Storage Controller is our client.
Dug-dug into the console of this UEFI - I realized that some kind of this is an incredibly difficult way, it should be easier.
Went to the Debian site and there actually was an answer on the surface. All you had to do was:
In general, frankly speaking, this is IMHO a strong joint from the side of Debian - well, obviously, Debian is installed on the NVMe device - in such a case it would be correct to offer the method of loading in non-expert mode and by default via UEFI.
The Debian installer marked out and installed the UEFI bootloader on a piece of NVMe disk (as many as 500mb it took for this, well, okay). Through that console, UEFI independently made the settings and the choice of boot from this section of NVMe. It remains for me to choose this new boot method as the main one in the BIOS.
That's the kind of markup you need to do for a bootable NVMe device.
This is the key point when installing Debian on NVMe, this is exactly what should be in a non-expert installation mode for installing on NVMe. And the default is yes.
The second sudden problem is iops counting. I naively believed that zabbix can do this and it will draw me how many iops my NVMe produces. It turned out that he could not. As for me, this is incredibly strange.
It turned out that there is a third-party add-on for zabbix that can collect such data.
But this third-party add-on does not know how to work with NVMe devices. I had to finish a little, the benefit is all opensource. And the data collection method is not compatible with the latest version of zabbix, and indeed contained an error in data collection. Also corrected. Pull request here .
There was a task - to unpack the archive with a huge number of small files. This NVMe is so fast that it coped with the archive of 100 gigs per 10,000 files in a few seconds. Nevertheless, zabbix recorded this minute: The maximum value that I saw is 5000iops per record when recycling is 5%. That is a very roughly, this device will accurately give out 100000iops for recording.
In general, such power was bought for the database and what this piece of hardware is capable of will be known only after the combat launch of the online game, which is being done. In the meantime, here is a synthetic simple test.
I also want to note that HDDs are capable of issuing 800iops per record when recycling 100% within 5 minutes and then 500iops per record - 15 minutes (also at 100% recycling).
There was also a funny moment when I pulled off the 90-bit archive by mistake on the NVMe device, but I had to pull it down, unpack it and put it on the HDD. Well, I think now I will unpack it on HDD, measure iops for reading, it is clear that there will not be much, but still. And suddenly I see iops reading equal to zero. Well, I think exactly some kind of error in the zabbix scripts. I rechecked everything carefully, made it right - I was convinced that there was no mistake. How is it possible then? And the trick is that it was the first hard operation on this server and Debian is watching - 90gigovy file and 120gov free operatives.
Well, I completely stuck this file into RAM. Then, when accessing this file, I read it from RAM, not from NVMe.
There is a saying: “How you call a ship, you will sail on it”. The server received the name "nova". Brief entry form in English from "Hypernova Star". A Hyperstar is a very bright flash lasting several hundred years. Instant by the standards of space. So the server should perfectly cope with the launch of an online game in a “short” period of time.
It will take at least a year before everything is completed, the content has been completed, everything has been thoroughly tested, publication materials are ready, legal issues have been resolved and there are more dark issues - and then it will be time to break out this hypernova. Well this is such a rarity today - the publication of the completed game.
In the meantime, I am ready to conduct synthetic tests NVMe p4800x. For example, for mysql to write some kind of task in 20 threads, so that this ssd tormented very fiercely. And yet, find out the real limit of the capabilities of this device. I am pleased to execute your load scripts on this server and publish the test results.
I am also ready to consider options with investments in the game itself, from $ 100,000 and, properly, everything is arranged and carefully arranged.
Delivered to the Moscow data center tech.ru and began operation.
Photo report available. The article consists of 5 parts:
- Why do I need a server.
- Configuration selection.
- Choose from whom to buy.
- The choice of who to post.
- Problems configuring this server.
Why do i need a server
In August 2015, circumstances were well established for me to start making an online game.
I had a fairly large selection of how long-term construction to do. For several reasons, I chose an online game.
')
He began to do it, and for half a year of hard work, he advanced well, but suddenly his savings ran out. Nevertheless, I went very far in this question and then continued to refine the online game in my spare time. The game requires a lot of server resources - to calculate ratings.
For 10 years I have a personal server in the Kiev data center hvosting.ua. I got it by chance: “Do you want a server like that for 750 euros and only pay for a place in the rack?”, Well, then I once had extra 750 euros and I agreed. The existing server was enough for about 500 people online. Therefore, I had to ask all the players "do not tell anyone about the game." And now I would not like to show anyone the current version, 5 people are actively playing - more than enough at the moment.
It became clear that you need to take or rent a new server.
I discarded the rental option, because I have a positive experience in placing my
server duration of 10 years.
There is a negative experience of closing a successful startup due to expensive hosting. Well, that is, you had to pay $ 600 per month for hosting, and the income at some point was less than $ 600 per month.
Accordingly, I could not show it in the portfolio - and I think it was a terrible damage to my income as a whole.
It is because of these two reasons, I decided to buy a server, and not to rent.
That is, if an online game becomes financially untenable, it will still remain online and be available for the portfolio. And the cost of maintaining it will be quite feasible, even with zero income from the game itself.
In addition, I need a server for several non-commercial sites and teamspeak servers for a large guild in a well-known online game.
Overview of all components in this server, everything is still in the boxes, now there will be unpacking
Configuration selection
I decided that the server should pull 50,000 players online and 1,000,000 registered players.
When these indicators are achieved, the startup will grow into a business and there will be an opportunity to buy new necessary equipment. But in general, why exactly 50,000, and not 60,000, we will not consider, the article is not about that.
Further configuration selection - selection of parameters, under the required parameters.
At the time of the beginning of this story I was rather poorly versed in the server hardware and what I really needed.
I believe that the train of thought and the logic of decision-making will be very interesting to all involved in the topic. Or decision makers.
All components in their boxes, now will begin unpacking and installation
General indicators
To reduce financial risks, it is necessary that the monthly expenses be minimal. Accordingly, the server must be 1U in size.
The server part of the game is designed in such a way that almost everything can be run in multi-threading. That is, it is even possible to share the task of rendering the game world among several servers.
This means that when choosing "several productive cores" vs "many mid-performing cores" - one should make a choice in favor of the second option.
Having made some measurements, I decided that for 50,000 people online, 50Gb of RAM will be needed (again, the article is not about why so many).
Suddenly it turned out that the game makes a lot of records on the disc. So it should be ssd and quite powerful.
There should be some hdd for backups. I am one of those who are "already doing backups."
CPU selection
Processor manufacturer: Intel, AMD, MCST.
AMD - I tried to use the official AMD site, but unfortunately it is impossible to understand there:
- What processors do they have in general? It was possible to understand studying a non-official site (all other sites).
- What processors are the most relevant. At the time of the study, the most current server processors were without DDR4 support. I did not study further.
The decision to abandon the purchase of AMD - a very uncomfortable site and outdated processors.
The MCST is certainly very interesting, the very fact of the existence of Russian-made processors is very interesting. Which is even possible to buy.
The inability to buy on behalf of a private individual is very strained, but the issue is resolved.
The inability to study the full characteristics of the processor, there are no tests in the public domain. Let even synthetic. I also note the lack of support for DDR4 - but this is excusable (for AMD this is not excusable). The decision to abandon the purchase of the MCST was after realizing the impossibility of using the “apt-get upgrade” command.
Intel - only this option remains. There is support for DDR4 memory in the latest line of processors. There is a special section of the site in which they dump a sheet of characteristics of the processor - I can find there all the characteristics that interest me.
Turn off the processor radiators
Turn off the processor radiators
The time of installation of the processor E5-2630 v4
Installed processor E5-2630 v4 close-up
Number of processors
In the free market there are server options 1x, 2x, 4x, 8x.
The option to buy a motherboard with multiple connectors and eventually install a complete set there - I decided not to consider. Why? To buy equipment this is a lot of time, nerves and money. This is different from a home computer.
Variants of 4x-8x processors turned out to be incredibly expensive and even in the most modest configuration they were with an extremely heavy price for me. Single-processor and dual-processor systems were for different niches of the market. Here is a list of processors to consider (those that start at "E5-4" are uninteresting, they are for 4 processor systems).
Consider the options "E5-1" - this is a processor for single-processor systems.
Immediately striking is the high base frequency and high burst, but the number of cores is very small.
For example, the “E5-1660 v4” 8 cores - $ 1113, and the processor for a dual-core system, but with a lower frequency “E5-2620 v4” 8 cores - $ 422.
The price is 3 times less in favor of a dual-processor system! I remember that more weak nuclei are much more useful for the task than few strong nuclei. This decision was made to buy a dual-processor system.
Both E5-2630 v4 processors are installed in their slots
Applying thermal paste to the processor
Thermal paste spreading on the processor
Processor model
All the same, there remains a fairly large selection of processors, what choice to make. I pay attention to prefixes like "L" or "W" or "A".
“L” is low power consumption. For me it is not interesting. If I would have my own desk in the data center - it would matter, and since I have 1U and no data center wants to put a counter to my server, this is important.
"W" - increased power consumption, "A" - a very good copy in the line. Pretty quickly you can see that the remaining processors are divided into 2 groups. A small number of cores + high frequency, a large number of cores + low frequency.
Drop the first group. Total left (cost above $ 1500 apiece, I decided not to consider):
E5-2603, E5-2609, E5-2620, E5-2630, E5-2640, E5-2650, E5-2660.
Immediately rejected too weak processors: E5-2603, E5-2609, because the money just enough for 2pcs E5-2620.
There are 5 options: E5-2620, E5-2630, E5-2640, E5-2650, E5-2660.
Upon closer examination, it turned out that they can be divided into 2 groups according to the maximum permissible memory frequency:
E5-2620, E5-2630, E5-2640 - 2133Mhz
E5-2650, E5-2660 - 2400Mhz
Consider 2 processors from this group:
The frequency E5-2630 vs E5-2650 do not differ. The difference is only +2 cores (and the memory frequency is a bit more) and the price is 2 times more (or + $ 600)! The second group is no longer due to the price.
Now consider the E5-2630 vs E5-2640:
Only the base frequency + 10% is different, and the price is one and a half times higher. I clean option E5-2640. Incomprehensible overpayment for me.
There are only E5-2620 vs E5-2630:
The price difference is $ 250, and for them we get + 10% to the frequency and +2 cores.
I am ready for such a price to receive such a premium.
Total selection is made: Intel Xeon Processor E5-2630 v4
Installing the radiator on the processor
Spin the heatsink on the processor
How processors are visible in bios
Memory selection
I remind you: at the idea level - I decided that I need ~ 50Gb of memory. On the Internet, I found that collecting a server with a memory that is not a multiple of two is quite acceptable.
But still. For reasons that all memory should work in the same way — so that I didn’t need to be soared to which kernel how difficult the task was to submit - I decided that the amount of memory should be a multiple of two.
Accordingly, it turns out that you need to put 64Gb of memory. The server part of the game is written in such a way - that on the server side the whole game process is completely calculated.
Thus, cheating is impossible. However, if suddenly there is not enough memory to fully fit in the memory of all players online - there will be a small disaster, the game simply will not start, and I will need to urgently make some kind of hotfix. For an opportunity to dump intermediate calculations on a disk.
It also turned out that if many memory dies are installed, the speed of the processor’s communication with the memory drops.
Therefore, if I buy 64Gb now, and then I buy the same amount, it will be much worse than if I buy 128Gb right away.
The difference in price turned out to be quite feasible, maybe it would be enough for 256Gb, but there was no extra $ 1000, the entire reserve eventually went to ssd.
Installing the memory dies, attentive gopher here can see 2400 memory. In fact, this memory is here for photos, the necessary memory arrived at the very last moment.
Select the number of memory plates
I already knew that if you put too many memory dies, it would work significantly slower than if you put fewer memory dies. So what - the best option is to put one plate on 128Gb?
It turns out that processors have such a thing as “memory channels”. The best option is one memory plate in each memory channel. This rule turns out to work for gaming / work computers. Surprisingly, all sorts of bloggers do not convey this simple idea to the public.
My case has 2 processors, and each processor has 4 memory channels. That is, for my case it is best to put 8 dies of memory.
Total selection: 8 x 16Gb
Installing Dies
Choosing a memory model
The choice was not great. Firstly, the choice at all at the moment is 2133 and 2400. But the processor pulls 2133. Does it make any sense to take 2400?
It turns out not only makes no sense, but also harmful (for my case). It turns out that memory access is performed in 17 clock cycles at 2400 memory, and in 15 clock cycles at 2133 memory.
But my processor cannot work with 2400 memory, it will work with it anyway at 2133. However, if the processor were E5-2650 v4 or higher, then on the contrary, it would be more advantageous to install 2400 memory.
The next factor is LRDIMM. This technology allows you to install more memory on the server and when installing a maximum of brackets - this memory is faster than if without LRDIMM.
LRDIMM is the usual server memory + some additional chips. Accordingly, the price is + 20% approximately.
For me, respectively, LRDIMM is not needed, because I do not install memory in each slot.
Now the manufacturer. Among the server memory is the manufacturer does not matter. I now do not understand how this is possible - but true. Synthetic tests are the same. The memtest data is about the same. For Kingston memory, I found an option: 2 options for chip saturation, but even in this case there is no functional difference.
General view of the installed processors and memory
How is memory visible in bios
Disk subsystem selection
When I first made the desired server configuration - I wanted 1 ssd for 240gb and one hdd for 2tb.
As I found out what I needed, the understanding of what I really needed changed.
I had to learn that ssd disks are very, very different. What servers better to use raid1.
What appeared new technology NVMe and it has a lot of options. Raid controllers are very different for different tasks.
Let's take it in order. Starting with "and what is needed for the task?". You need a 200Gb ssd disk capable of withstanding a lot of rewriting. And 2Tb hdd for reliable backups. "A lot of rewriting" - with very approximate calculations it is 1-2Tb per day. Reliable backups - this means raid1 configuration. A hardware or software raid is a question of the amount of effort, qualification and responsibility when repairing a raid.
In terms of management, the choice is as follows:
- Software raid - need a qualification to fix raid, the slightest error and data is lost. But cheaper.
- Hardware raid - just replace the failed disk and that's it. But expensive.
Accordingly, Tiscom advised the cheapest of the modern hardware raid controllers, which would allow to repair possible damage just like that.
This: ark.intel.com/ru/products/82030/Intel-Integrated-RAID-Module-RMS3CC040
I note that this is a raid controller for an Intel motherboard. That is, if I would choose a non-Intel platform, then I would have to take something else and take up a whole pci slot.
I have no idea what it is. Or something related to encryption or something related to the raid card
Preparing a raid card for installation
Installing a raid card
How raid controller is visible in bios
HDD model
I was going to buy Western Digital due to the fact that I have a very positive experience using such hdd for home computers. But the Vice stated that they could only make a replacement within a week (with or without warranty). And they offered HGST - they said they could make a replacement within 24 hours.
I began to search the Internet for what they write about the reliability of HGST - like normal reviews, and agreed with this choice of disk manufacturers.
Next volume. 2Tb is necessary, good. And what is more expensive? + 1Tb is $ 70 more expensive, well, then I won’t forgive myself what I saved on the volume, and if I suddenly need more space later. Well and further along the line, the volume increases a little bit and the price increases slightly. Between 6Tb and 8Tb, there was a border between my budget possibilities.
Total selection: 2pcs 6Tb HGST HDD.
2 HDD HGST 6Tb
Installing some part of the raid controller for the HDD recycle bin
Connecting the HDD basket to the Raid card RMS3CC040
SSD type selection
The first option: just put 1pc ssd 200gb. Next, I learned that the more ssd - the more it has a rewrite resource (other things being equal). And so that this rewrite could be performed faster.
For example, I will have 200gb of data and a volume of 240gb vs a volume of 1000gb. So, all other things being equal, a large-volume rewriting resource will be 4 times larger.
Let the experts forgive me, but for making a decision it is not at all obvious.
Further, I understood, you need to use raid1 and you need 2 such ssd. Yes, even if backups are made at least every hour and it will not be terrible to lose a single ssd. But there OS, settings of everything. Current DB. We have to urgently spend a lot of time and effort on recovery.
Ssd withstand 2tb per day overwriting is very expensive. And it is also necessary to have some stock for this indicator.
Then I learned about the NVMe technology. This is simple: the ssd drive is connected not as usual, but through a pci slot.
The data band is bigger and faster - this is important for my task. Immediately the question arose - is it possible to combine them into a raid?
It turns out you can, but there are no controllers like this for raid. There are prototypes with an unknown price.
In general, at the time of purchase - it was an unacceptable option.
Another factor in the decision: let's say I bought 2 expensive ssd disks, combined them into a raid. But the resource rewriting they equally quickly consumed. And what will I get? What they will fail at the same time?
Next, I found out that most often ssd fails not because of a rewrite resource, but because of a controller breakdown in ssd. Yes, for this raid will be relevant.
And the choice is between pci-ssd (NVMe) and raid1-ssd. And taking into account the above considerations, for me, definitely better than pci-ssd. Well, it's just stupid there is not another $ 2000 for backup ssd, and if there were, then you need to take a 2U system and buy another basket for $ 400. Well, or buy 10 times worse ssd.
Well, yes, I am ready to trust a well-known brand (in this case, Intel), in the hope that they have a sufficiently reliable device so that the claimed period and resource will work without failure.
NVMe P4800X installed in the riser
NVMe model selection
In ssd, I am interested in recording iops, the speed of a single operation, volume, rewriting resource.
Revising the characteristics of thousands of ssd, I believe that in my case, you must first look at the stated number of rewrites.
Almost all ssd can be divided into groups by the number of overwrites of its volume per day, for 5 years. Groups: 0.3, 1, 3, 10, 30, 100.
Almost always the exact match between the rewrite resource and iops to write. Separately stand out ssd, reading from which is performed with an incredible amount of iops, but relatively weak to write. Such to me for this server are not interesting.
Manufacturers: Intel, HGST, Huawei, HP, OCZ. All sorts of different ssd - an incredible amount. Honestly - I have little idea how people objectively choose ssd at all. There are so many Ssd, they are so different, the price range is so huge.
From my point of view, Intel products were best grouped. They managed to break all the ssd into groups so that I could form an opinion which device I need and which one I don’t.
3 lines of NVMe pci-ssd: P3500, P3600, P3700. 400Gb each:
P3700 - 75000 iops - $ 1000, 10 rewrites per day
P3600 - 30000 iops - $ 650, 3 rewrites per day
P3500 - 23000 iops - $ 600, 0.3 rewriting per day
Actually it is obvious that the P3500 immediately disappears. For $ 50 a tenfold increase in an important parameter to me.
Am I ready for $ 350 to get ~ one and a half times more iops and 3 times more rewriting resource? Kaneshna, yes.
So I was going to buy a P3700 for 400Gb, but a couple of months before the purchase, the P4800X went on sale.
It costs almost 2 times more expensive than the P3700. Slightly less volume. iops on the record 500000. Resource rewriting 30.
That is, for a total of $ 800, I get 8 times more iops and 3 times more rewriting resources.
Well, honestly speaking in terms of iops - it’s just beautiful and in spite of the fact that it wasn’t easy to get an extra $ 800 - I decided to buy such ssd.
The bonus is a faster recording of each individual team.
Choice: www.intel.com/content/www/us/en/products/memory-storage/solid-state-drives/data-center-ssds/optane-dc-p4800x-series/p4800x-375gb-aic-20nm. html
Raiser with NVMe P4800X install
NVMe P4800X installed
How NVMe is visible in bios
IMHO about ssd
Separately, I want to note that there is a type ssd SLC / MLC / TLC / QLC. The meaning of the difference is clear: Several data bits are written to a single physical cell at once. Accordingly, SLC - only one. MLC - two. TLC - three. QLC - four. From left to right - from expensive to cheap. Due to the speed of writing, reading, rewriting resource.
SLC - the most reliable, the fastest and ... monstrously expensive. For example, "Micron RealSSD P320h" is a hairy monster with a volume of 350Gb for only $ 5000. Declared iops to write 350000. Unlimited rewriting resource.
MLC - are very different. Well, at the factory, they make a bunch of memory chips, something turned out well - they thrust into the server version, something not very much - they thrust into the home version of ssd.
TLC - they used to do frankly fast failures, now they seem to be doing good ones. And they are trying to make an idea that TLC is good (so that the old ones will buy up stupid buyers).
QLC - tried to do, but apparently the commodity option no one could do.
Here Intel recently released 3D-XPoint, this is some kind of new memory. As for me - puff cake from ssd chips.
Motherboard selection
For servers, in contrast to home computers, the choice of motherboard is completely different. For a home PC, the size of the actual computer is not important, but for the server, you have to put everything in 1U (This is 4.3 centimeters!).
From here another approach to the selection of motherboards immediately appears. Not motherboards, but platforms are on sale. That is, the motherboard + case. The one who sells only motherboards separately - immediately loses in the competition.
Platforms for this processor come from different manufacturers. With the smallest possible modifications, on the sites, you can specify a link only to a specific modification (I will say a bit about the modification later):
HP and Dell manufacturers - I will not consider because of the monstrous overpayment is not clear why.
- Supermicro
- Asus
- Intel
- Separately Intel
The choice of these four options is the cheapest option suitable for all requirements from each of the listed manufacturers. The price they have about the same turns out. The R1304WFTYS option disappears as soon as I see the 1100W power supply (why this is important - more on that later).
I thought about ASUS so that they recently climbed onto the server motherboard market, it is not clear what I would risk.
About supermicro - they are actually only server platforms and have been doing for a very long time. But ... raid controller. Either I take the Intel platform or I need another controller and occupy a PCI slot.
Therefore, I decided to choose the Intel platform R1304WTTGSR. But I note that it is 10% more expensive than a similar platform from supermicro.
Modification: with 10 gbit ethernet. The price difference is $ 100, and if I suddenly need a fat channel, I would not want to touch the server and install something there. And I probably need a fat channel for the start of the game.
Platform R1304WTTGSR boxed
This is what we see when we open the new platform R1304WTTGSR for the first time.
All components are installed, it remains to close the cover
Such information about the platform is visible in bios
Who to buy from
Of course, first of all I turned to hvosting.ua, because I worked with them for more than 15 years. And the volume of transactions exceeds the cost of the server being discussed. But hosting a server in Kiev after the well-known events of 2014 is unacceptable for security reasons.
hvosting.ua offered me accommodation at the ttc teleport data center in Prague.But the discussion stopped on the question "I want a piece of paper that is legally binding that I am the owner of the server." Well, forgive me, I am no longer a student completely trusting everyone.
However, if someone really needs to host something in / in Ukraine - I recommend hvosting.ua uptime ~ 1200 days, then the planned transfer of the server to the neighboring data center and another uptime 700 days. Then I tried to do “apt-get upgrade” without rebooting the server :) It didn’t work for debian, it couldn’t update itself in 2000 days - it’s not possible to fix some kind of dependency. Hvosting helped to settle.
In general, for 15 years there was not a single power failure and the channel fell for 15 minutes every 4 months, 2 times the channel fell for several hours.
Especially delivered broken oauth authorization vk (they have vk blocked there).
Market Review
There was a question about buying in Russia and placing also in Russia. Why purchase and placement in the Russian Federation? And so I would be legally protected. There was a pleasant moment, at the very beginning of the search - the choice of the best supplier, no recommendations, no bias - all are equal. I remember how I started with the banal search “datacenter in Russia” (later I found out that in the Russian Federation this is called the data center). And "buy a server in Russia."
I think it will be correct to tell in chronological order.
12/05/2016
Brigo
In the beginning there was a very constructive conversation, with an interesting proposal. I was studying the possibility of the raid1-NVMe variant, asked if they could put raid1-NVMe on the intel platform?
Suddenly I received this question: a rude refusal to answer (by the way, I would be satisfied with just the answer "no", I just really was not aware of this matter). Then I did not want to communicate with them.
They first saw the P4800X on sale and therefore I decided to ask them for the purchase details.
The conversation ended with the fact that without my personal presence in Moscow, it was impossible to make a deal.
12/05/2016:
Unit Solution
They write on their site that 12 years of experience. And other cool performance. DNS whois shows that the domain was recently registered (3 months prior to review). I immediately have a question for them: “why such discrepancies?”. It turns out before that there was another company to which there were reviews on the Internet of the type “paid for a long time, there is neither a server nor money”. It would be possible to forgive specialists who created a new company after working with a bad director. But unfortunately, what they wrote - did not agree with the history of the DNS. I decided that the risks were unacceptable, although they offered an interesting price.
12/08/2016:
ItTelo
Citizens trade in used equipment. Then I still considered such options. Over time, I decided not to consider it, because DDR4 is not used. sdd-hdd B / Y is better not to take anyway. Well, it turns out they have nothing to offer me.
12/08/2016
It paradigma
This is a regional equipment supplier. Principle is a good option - they are also nearby. Then I still thought that there could be a decent data center in Krasnodar, but no - if there are decent ones, then they are packed to capacity. The paradigm turned the prices up and did not answer my stupid questions.
12.12.2016:
Trinity
Here is such a link still working: server.trinitygroup.ru/expert/25736
As I recall, I was confused by the price. Honestly, even now I look at this price and do not understand where such a high price comes from.
12.12.2016
Vice
They were recommended by the data center tech.ru type they themselves cooperate with them. I listened to all my Wishlist - picked up exactly what I wanted of my time. At a great price. Then he began to dig up the DNS whois and reviews on the Internet.
DNS whois corresponded to the one described in the correspondence. During the investigation many curious details were revealed.
It turned out how the company was founded, by whom, how it changed. Found how to change the site and asked for feedback about it somewhere to leave.
There were employees of the company. There were positive and negative reviews. All this said that the company is old and is actively engaged in something constantly.
I also checked the Intel certificate - it was at least outdated, but real. I checked some other random certificate - also a real one. For this I was very pleased.
There is also https. Also adds points in the direction of choosing their company.
From the very beginning, it looked like a very good option.
12.12.2016
stss
The first communication is positive. But the price - just no words how big. Until now, there hangs an example brutalized by greed . Choose 400Gb NVMe 75k iops. This corresponds to our Intel P3700. It turns out that everyone is selling this piece of iron for 60000 ±, and they are pushing it for 90000.
Okay, maybe this new site with some kind of bug? .. But no, this is not a bug. Here we find the same on the main site with the old design:
stss.ru/products/servers/T-series/TX217.4-004LH.html?config=
Here, in the appropriate place, choose the same piece of hardware and also get +90000 to the cost.
50% of the bonus money is definitely very profitable, but I'm sorry, we will take the other way.
12.12.2016
Desten
In general, it was a very decent option and the DNS whois and the reviews looked good.
But the question “can they, without my presence, deliver the server to the data center?” Was not answered.
So it would be vserez would choose between Tisky and Desten. If they had a more patient support - maybe I would have bought a server in the end.
12/13/2016
Server-gid
They just sent me the configuration and price - I was horrified and did not answer. Now I look - the price is adequate, and why I was horrified earlier - because I did not understand the meaning of the word “DWPD”. Apparently, I had a telephone conversation with them.
12/14/2016
Itelon
Citizens tried to sell me Dell equipment. Again, I see an overpayment for the brand is generally not clear why. An employee explains to me that there is an overpayment for operational assistance. But I do not understand something - the server is in the data center tier3 rack. No one will kick him down. In addition to the factory marriage - nothing can happen there. Plus, they said that they couldn’t install a server like a Tis without my presence - it was not clear what to say next.
01/10/2017
Server-city
Now I see on their site a poor server configurator. Then they had only a business card site. And there it was written that they were considering options for installments, and I decided to ask about this topic. They said that such small amounts for installments are not interesting. Servers have DELL / HP - that IMHO overpay do not understand why.
Allow me to justify my lack of interest by these companies.
www.regard.ru/catalog/?query=2620+v4
Someone will explain to me the difference between "Intel Xeon E5-2620 v4 OEM" and "Dell Xeon E5-2620 v4 (338-BJEU)"? I now see only the difference in price by 2 times! The processors themselves make Intel chips. Ready to believe that they put the solder on the other, well, ok + $ 50 is ready to understand. Explain to me stupid for what the company DELL in this example takes + $ 500?
Apparently somehow a reminder worked for them successfully and they called me a week before the final agreement on the configuration with Tisk. They tried to intercept me as a client at the last moment. But ... the price was higher than the Tisky.
Ebay
As well as a parcel post, ComputerUniverse, ogg and other sites. The option that delivers me to the outback, and then somehow I deliver to Moscow - it looks terrible. Not only that all profit from refusal from intermediaries comes at the cost of delivery. So also time will eat incredibly much. And most importantly - the risks of damage appear (strike about something for example).
The option with delivery to Moscow - you need somewhere a room where I can build a server plus a personal presence. Plus transport between point of issue, assembly point and data center. Plus, the risk of nosyachit when assembling (if there is a secondary posting is not so connect - there will be no way to fix it).
Samou run around Moscow
I noticed that the price varies for each piece of iron in different stores. That is, you can run around Moscow to buy money in wounded shops. But again the risks of an assembly curve. Personal presence, a place to build and transport around the city. And the question is - where is the profit from this event?
Local shop
In the outback, of course, there are shops and they are ready for a long time to deliver anything at a reasonable price. However, as it turned out, they somehow have a friend’s brain work. I explain to them that here ssd dear, I need this here. And they - we do not have it, so take another with similar characteristics. From "similar" only volume is considered.
But again - they brought it, it is necessary to collect (there is a risk that something will be crooked), then delivery, personal presence, etc.
Russianhardware
Very interesting option. The person collects computers to order. Speaks directly: we take 4% of the commission. All additional costs are at your expense. I really liked this position, but there are no guarantees that it will not disappear after receiving the money. I now do not understand.What prevented him to say, they say that there is an option with the contract, everything is legalized correctly. This option will increase the cost here so much. That would be a very interesting option. Here is the basic price, here are my 4% surcharge, here are the additional costs associated with delivery, here are the expenses on legal issues.
team.ru
I never spoke with these guys, but I spent a lot of time on their site.
I have to admit - they have the most convenient configurator from all sites. Convenient not from the point of view of prettiness, but from the point of view of how “how can you figure out all these characteristics?” I didn’t communicate with them because they have a full 15% comparison with all the rest. They have all sorts of certificates - for myself, I concluded that the municipalities of the Moscow Region or Moscow region are the best supplier of equipment. But for me there is no - primarily because of the surcharge.
Their configurator is so beautiful that you deserve special attention. Consider this example.. Perhaps they should sell it as a service to other companies? Type download their prices and what is there, and users themselves are assembling what they need. Well ... if Tisky had such a configurator, that would be super fine.
At first glance - bad because of the design. But we are not here a beauty salon.
I would also note that in spite of the seeming simplicity there is a huge amount of small and useful functionality hidden in it. Platform. Thanks to this site, it finally reached me how motherboards for servers are selected (in the form of a platform and their modifications). Also here you can immediately see all the modifications. It is seen that for ssd and for hdd. It can be seen that 10GbE and 1GbE. It immediately becomes clear that if I want to set myself hdd - then 2 modifications do not suit me at all. And the price difference on 10GbE and 1GbE is 1GbE + 10% = 10GbE. And that's all, no more thinking and comparing, asking stupid questions to the posters. Looking at it - everything is clear right away.
Next, a block with processors - you can clearly see which processors can be plugged into this platform. A processor with a bunch of characteristics — at the “I don’t understand anything, but I want to understand” stage — dumping all the different characteristics on the bars is very useful and very reasonable. This section helped me a lot to understand which characteristics of the processor are important for making a decision, and which are not.
Then a block with memory and it suddenly also helped me a lot to figure out what characteristics to look for in order to make a decision. Here it is very clearly seen that we must take into account the frequency of the memory and RDIMM / LRDIMM. I wrote about this before.
Then I understood how to build raid1-nvme. Let's see here. There you can select a basket for ssd-nvme. After that, you can select nvme-ssd in this basket. And then any modern raid controller will be suitable at the rate of 4x pci lines for each nvme device.
So just about the most important thing - I'm probably so stupid, I spent more than 20 hours to understand 3 lines above.
Yes, it is not yet possible to assemble a raid1-nvme hardware in modern 1U Intel. Due to the lack of appropriate baskets.
Regard
I spent a lot of time here choosing configurations and comparing prices. The prices here are good, but they are constantly popping up "out of stock" - which makes it pretty difficult to plan. I did not contact the store. Because what is actually talking about? Here is the product, here is the price - pay and take. As it is necessary to take it in person, the option is not impermeable.
AND systems
I tried to build a server, but they were constantly missing something. And what was - not at the most interesting price.
Conclusion
Of all the options listed, it is obvious that the Vice is best for me. Before buying agreed on the final configuration. Signed a contract remotely, then after purchase - in paper form. On the 4th day after the payment, as planned, the server was delivered to tech.ru and in the evening I began to set it up for the same. Through bios I checked that all the equipment is in full and in accordance with the agreement.
Everything went flawlessly, even nothing to complain about.
In the above photo flashes hand. On this hand is a blue armband with a wire. This is not for fun, but grounding. This is an absolute plus to the qualifications of employees Tisk.
Data Processing Centers
I have found correspondence in history with only 4 data centers, but I remember exactly - there were many more of them. I remember exactly - there were different phone calls. I remember exactly - there were site reviews.
Pretty quickly you can find out that there are different levels of the data center. tier1, tier2, tier3, tier4. Or even without this brand. tier4 in the Russian Federation there is only one. For some reason, I can't find the offsite now, but I remember I found it and saw that it was impossible to get through with my 1U. Yes, and probably the price would be too high for me.
Working equipment in the data center tech.ru
tier3 is a lot of options in the Russian Federation. I took this list and began to look for compliance. I will now take this list and go from top to bottom:
Dataspace
Possibility to rent only racks. Would write here a list of intermediaries, through whom you can shove 1U to them.
Sberbank Datacenter
Clearly just for the needs of the bank itself.
CROC
Not for resale.
Datapro
Only racks trade. There are no intermediaries.
IT park
These are already offering something for 1U equipment. I remember that I looked at this place before, but did not like the prices and therefore did not even write to them. They did not write anything on the site about the cost of communication channels - and even without communication channels, the unacceptable price is already obtained. I was also embarrassed by the lack of photos. I'm also confused by the lack of https. I understand that they built the site this way, that they now need wild-ssl, which is significantly more expensive than the usual ssl. But there is no money for a certificate or there is not enough intelligence to properly organize the site - it is also not a plus for them.
VTB
This bank also built its own data center.
Data line
I somehow missed such a data center. Now, if I was looking, I would love to contact them. At prices nothing is known - all through the "send request". Perhaps this confused me half a year ago. The absence of https is surprising - they spent several million dollars on the construction of a data center and there was no $ 10 to buy a certificate. Attention to the question: what else did they save, or what else did they lose sight of?
LLC “Inforesource St-Petersburg”
This is something public.
Mobile TeleSystems PJSC
This is the data center of the mobile operator.
Avantage
Only racks trade. There are no intermediaries.
Akado - Internet provider has built a data center.
Stack telecom
On closer examination, it is clear that they will soon expire a tier3 certificate. This means that they have designed the data center in accordance with tier3, but for some reason have not yet implemented it. Https no, no pictures. The prices are very good, why I didn’t write them - I find it difficult to answer.
Total tier3
Total from the data center, ofcocially relevant tier3 - for me, no one was acceptable. What to do next? I tried to search in different forms on the Internet sites of different data centers:
February 14, 2017
Store Data
Apparently they first caught me in advertising or searching. I remember exactly that I was very stressed - that the addresses of each data center were not indicated, now I can already see it. I also remember the terrible layout on the site. Even now, debian + firefox looks awful. When discussing prices, I had "stupid" questions, such as I do not understand what to pay 30000r / month. They decided not to answer one of these questions. Well what can you do - mine and the truth I am not their client. They specialize in payment systems there.
Random search did not help
Surely I found more than one data center, but something I did not like. I remember that I began to learn "and what is the correct data center?". Watch all the videos on the Internet about the data center and got the idea of ​​what I should see in the pictures.
I tried to get into the local data center Rostelecom. Because my provider is Rostelecom. And the organization is decent. Spent $ 3000 fiber - very good call quality.
Offtopic: Rostelecom has the most terrible native DNS - most probably, most users have a terrible opinion about this provider. But they do not offer placement services for their equipment, at all, nowhere.
Then I just called them in support of the central and asked - "and what will you recommend me?". This employee recommended me “hc.ru” and “tech.ru”.
12/16/2016
Tech ru
Patiently answered all my stupid questions (I now see and understand that questions are stupid). They offered a bundle of supplier + data center, besides them, only the data center in the Baltic States have proposed this. They have a modern website, as you saw in the analysis above - not everyone has that. Photo equipment - consistent with the idea of ​​what should be the correct data center (actually all who posted photo-video match, but not all posted). There is an unusual channel "burstable rate". Prices from my point of view a little lower than at all. Well, made a special offer in the form of discounts on power supply (PSU).
It is necessary to explain a little why all the data center was brutalized with this indicator of the maximum power consumption of the PSU. As a consumer, it’s not at all clear to me why, when paying, they look at the dial-inscription on the BP. It would be logical to look at the power consumption or in the end - the maximum possible power consumption. In my case, if you collect all the glands to the heap - you get 420W the maximum possible consumption and the actual consumption now is probably about 100W. And on my PSU 750W is drawn - it means I have to pay for it 1900 rubles a month for exceeding the power of 400W. And they are not the only ones - it turned out all the polls are so strange. Why?
It turned out that for the case of shutting down the servers — and then simultaneously turning them on, the maximum possible current is consumed.
Well, it turns out that in a critical situation - the operator or simply click "turn on everything." Or if there are a lot of people like me, I will have to turn on each server in turn. Respectively longer downtime for some customers. I didn’t find any more reasons, but this reason also completely explains the oddity.
The next oddity is the connection to the 100 Mbps / 1 Gbps / 10 Gbps ports. Even if very little data is being chased through this port - sorry, this is very expensive. And this is in no way obvious to a person who is not aware of these nuances.
Login to the data center tech.ru
12/16/2016
Hosting Center
They wrote to me in support “Write to us, please, when the server is ready for installation in DC”. Well, that's how I perceive it? How polite non-cooperation? After 2 months, I asked them if they had a food restriction? Suddenly it turned out that there is. Well, what the hell they told me to first buy a server, and then find out what you had to buy? At this time, I decided that it is unacceptable to communicate with such support.
01.18.2016
Miran
I asked them whether there is an opportunity to order equipment. They offered only a collection of dedicated servers to order.
Total
The choice was apparently made in favor of tech.ru. First of all, because they work closely with Tiscom. Secondarily due to good support. Thirdly, due to nutritional discounts.
The fact that there is a burstable rate, a well-designed site, https, photos, and other trifles was also a plus.
For the month of use, 2 downtime channels were recorded approximately 15 minutes and 3 minutes, respectively. When connected, the ip address routing was mixed up (7 ip addresses per 10 Mbps <-> 1 ip address per 100 Mbps channel). The rest is all fine.
I can also come up with the fact that there is no service type API to find out how much time UPS has before shutdown. Something like this is in hetzner.
An example of a rack in tech.ru with working customer equipment
When they found out that I was going to write an article about them, they asked me to leave this phone here: +79055628383
Problems configuring this server
I took out separately the problems that I had while setting up this server.
Not standard problems from my point of view. That is, a solution which is not easy to google.
NVMe configuration - as the main disk of the system
The first problem is how to make NVMe disk bootable.
I started with a simple one — I tried to set it as it is. Did not work out. Well, that is, Debian was installed and made NVMe bootable, but the BIOS did not see this boot option, well, no way. I didn’t want to touch in the BIOS what I didn’t understand, but from what I understood, I tried everything. From the boot options found UEFI option. But he did not want to run boot-NVMe through this option. Having rummaged in UEFI - I was convinced that from the command line it is really possible to get information about the installed NVMe. And this means that it will really somehow boot from it.
Actually it became obvious that you need to register something in this UEFI.
Photos from the UEFI excavation site. Mass Storage Controller is our client.
Dug-dug into the console of this UEFI - I realized that some kind of this is an incredibly difficult way, it should be easier.
Went to the Debian site and there actually was an answer on the surface. All you had to do was:
- Select the expert mode for installing Debian.
- When choosing a boot method (which is available only in expert mode) UEFI / GRUB -> Select UEFI.
- Mark in a special way a small piece of NVMe disk for the UEFI bootloader.
In general, frankly speaking, this is IMHO a strong joint from the side of Debian - well, obviously, Debian is installed on the NVMe device - in such a case it would be correct to offer the method of loading in non-expert mode and by default via UEFI.
The Debian installer marked out and installed the UEFI bootloader on a piece of NVMe disk (as many as 500mb it took for this, well, okay). Through that console, UEFI independently made the settings and the choice of boot from this section of NVMe. It remains for me to choose this new boot method as the main one in the BIOS.
That's the kind of markup you need to do for a bootable NVMe device.
This is the key point when installing Debian on NVMe, this is exactly what should be in a non-expert installation mode for installing on NVMe. And the default is yes.
Configuring Zabbix-iops
The second sudden problem is iops counting. I naively believed that zabbix can do this and it will draw me how many iops my NVMe produces. It turned out that he could not. As for me, this is incredibly strange.
It turned out that there is a third-party add-on for zabbix that can collect such data.
But this third-party add-on does not know how to work with NVMe devices. I had to finish a little, the benefit is all opensource. And the data collection method is not compatible with the latest version of zabbix, and indeed contained an error in data collection. Also corrected. Pull request here .
There was a task - to unpack the archive with a huge number of small files. This NVMe is so fast that it coped with the archive of 100 gigs per 10,000 files in a few seconds. Nevertheless, zabbix recorded this minute: The maximum value that I saw is 5000iops per record when recycling is 5%. That is a very roughly, this device will accurately give out 100000iops for recording.
In general, such power was bought for the database and what this piece of hardware is capable of will be known only after the combat launch of the online game, which is being done. In the meantime, here is a synthetic simple test.
I also want to note that HDDs are capable of issuing 800iops per record when recycling 100% within 5 minutes and then 500iops per record - 15 minutes (also at 100% recycling).
There was also a funny moment when I pulled off the 90-bit archive by mistake on the NVMe device, but I had to pull it down, unpack it and put it on the HDD. Well, I think now I will unpack it on HDD, measure iops for reading, it is clear that there will not be much, but still. And suddenly I see iops reading equal to zero. Well, I think exactly some kind of error in the zabbix scripts. I rechecked everything carefully, made it right - I was convinced that there was no mistake. How is it possible then? And the trick is that it was the first hard operation on this server and Debian is watching - 90gigovy file and 120gov free operatives.
Well, I completely stuck this file into RAM. Then, when accessing this file, I read it from RAM, not from NVMe.
Conclusion
There is a saying: “How you call a ship, you will sail on it”. The server received the name "nova". Brief entry form in English from "Hypernova Star". A Hyperstar is a very bright flash lasting several hundred years. Instant by the standards of space. So the server should perfectly cope with the launch of an online game in a “short” period of time.
It will take at least a year before everything is completed, the content has been completed, everything has been thoroughly tested, publication materials are ready, legal issues have been resolved and there are more dark issues - and then it will be time to break out this hypernova. Well this is such a rarity today - the publication of the completed game.
In the meantime, I am ready to conduct synthetic tests NVMe p4800x. For example, for mysql to write some kind of task in 20 threads, so that this ssd tormented very fiercely. And yet, find out the real limit of the capabilities of this device. I am pleased to execute your load scripts on this server and publish the test results.
I am also ready to consider options with investments in the game itself, from $ 100,000 and, properly, everything is arranged and carefully arranged.
Source: https://habr.com/ru/post/340638/
All Articles