How we almost lost 5,000,000 hryvnia per month due to the “wrong” hosting: client history

Clouds are an intriguing and scary thing at the same time, but not so many “our-real” examples of how people work with cloud providers.

Under the cut, the real story of our client with real numbers and queries, about working with local hosting providers, what problems he encountered, how his entire database was “accidentally” deleted during the transfer, and that it’s important not WHERE from whom to rent power, the service provider is important. This case is an example of the fact that good marketing does not guarantee good service. So, the word to the client:
')
It all started with the fact that a large pharmacy network needed a website and hosting for it. Everything would be fine, only we had a large catalog with a huge number of photos. Contrary to expectations, it was possible to start your website store only from the “third hosting provider” :), and then there was a server rental, and then we went “to the cloud” .... But first things first.

A few words about us: a network of pharmacies " DS ", 111 pharmacies, we serve 25,000 customers daily, we work mainly in western Ukraine - in Lviv, Volyn, Ternopil, Chernivtsi, Ivano-Frankivsk, Khmelnytsky and Transcarpathian regions. On the scale of the business better tell the numbers:



After working for several years, at some point we decided to expand. Among other things, we decided to make a website that will present the entire catalog of products offered by our pharmacy chain. At that time (2012) it was about 6000 titles. As a presentation tool online, the site had to tell customers about the drugs in our pharmacies. Each position of the range was supplemented with instructions and a photo.

Since the sale of drugs on the Internet is prohibited in Ukraine, we have allocated products for online sales in a separate category. This list includes cosmetics, non-medicinal drugs (vitamins, for example), medical equipment, etc. Thus, the first task of the site was informative - instructions for using + photos, the second - selling an allowed range of goods online.

The site was implemented on Bitrix and integrated with 1C. For hosting for the site, we turned to one of the Ukrainian hosters. During testing, we have never managed to unload the entire directory. After the failure, we thought that the problem was that "we got such hosting". Ok, we tried another vendor - and once again, uploading the catalog did not work out even once. Suspicions that something was wrong with our software quickly disappeared (tested on your virtual server - it all worked).

We had to search for a hosting provider whose capacity could withstand the load of our extensive product catalog. After 2 unsuccessful attempts with the Ukrainian hosting, it was decided to try hosting from a European provider. Having made this decision, we have never regretted it!
When downloading the catalog already at the SIM-Networks hoster, we did everything from the first time without problems and delays. Since then, four years of work with this hosting have been successful.



Leasing other parts of our IT structure

In addition to the site, referring to - say, politically correct - “some actions” of the controlling bodies (recall 2013), we wanted to secure our infrastructure, our ERP network, our data. To do this, they decided to try taking infrastructure away to the “cloud”. Our servers could not cope with the peak load and had to increase capacity. In particular, it was about streaming download documents (more on this below). Initially, there was an idea on our own servers to switch from SAS disks to SSD, but, having calculated the cost of such a transition, we realized that it was much more profitable to rent the necessary capacities. With the growth rates like ours, we will need to purchase more expensive equipment every year. Naturally, this strategy is not profitable at all.

Having monitored prices, initially we tried to “go to the cloud” to one of the domestic hosting providers - due to lower prices. We took the power to the test, downloaded a copy of our system, conducted the most necessary tests, in our opinion, everything was fine. But, having run the system into a product, they were very disappointed. Throughout 2014, we observed data access failures, which led to huge delays in work.

Dips in access from the Ukrainian hosting. How it was

It was "impressive." For example, infrastructure performance dropped to zero for about 1-3 minutes, and then quickly “rose”. And this, it seems, a small delay caused a snowball of even greater temporary delays and became the cause of serious violations of our business processes.



In a concrete situation, the problem looked like this: a drop of “only 3 minutes” on the hosting lined up a long queue to our database (all users hung up), the 1C system could unlock the lock in those 3 minutes and all packages of outlets and departments were discarded. Did this inconvenience?

Yes, everyone was just “happy” - they stamped their feet and swore at what the world was worth! At the level of a specific employee, everything developed as a good thriller. Suppose a certain manager downloaded several invoices and did not know if a failure occurred - was the wiring done or not? Like many other colleagues at the same time, this manager had to delete all downloadable transactions and manually drive unloaded invoices. It was such an eternal, tedious, annoying and unnecessary process. The monthly turnover of our network was about UAH 50,000,000, and because of such failures, the company could lose up to 10%, i.e. up to 5 000 000 UAH per month!

Not wanting to do anti-advertising, I will not call these hosting companies.

Our peak load is approximately 700 batch-loaded documents in the morning, from 9.30 to 11.00, and the same in the afternoon from 15.00 to 17.00. Every day (every working day) about 15,000 documents were issued. The entire volume of documents was loaded at the same time and was subjected to different processing - check 1 invoice could be carried out on 20 (!) Items.

It is necessary to say about the kind of seasonality of such failures on the Ukrainian hosting: as a rule, they happened at the beginning-end of the month (about 10 days), the middle of the month (obviously, when everyone handed over the documents to the pension fund), on the 20th (the time of tax documents).

Comment by SIM Networks
Such delays, most often occur when a server designed for, say, 5 clients, the provider places 10 clients, in (naive) hope that they will never use all their (already purchased by them!) Capacities. We believe that if a client bought a place, bought a supply of power, then it is empty or not, this is already a client, and he can do whatever he wants with resources.

It was noticeable that the hoster did not count on the simultaneous use of all rented capacity by all customers. As a result, a resource guaranteed by a vendor, ordered and paid for by us was not provided! There were no failures, obviously, only when our hosting neighbors did not use servers.

And so, we very long proved to a hoster that this problem lives on their party - started monitoring, wrote letters on failures. And only after a year of such torment, the hoster company agreed to buy an SSD-shelf to its data center “specially for us”. But it was too early to relax ...

The migration of our data to this shelf was terrible. The provider promised to move the base in just a day, at the weekend. On Sunday at lunch, the restructuring of the new shelf was not finished yet and we asked to return everything back, for which we received an answer - and we moved, we looked, the system went up, and therefore the old database was deleted. The hoster decided to delete this data, since he believed that the migration had already taken place and it was no longer necessary to store it! - we were shocked ... Ie even before the new SSD regiment was completely restructured, they killed the entire original database (!), and we already need to launch our entire retail network.

Due to this state of emergency, we deployed our data from backups. The complete restructuring of our “new” SSD-shelves at the hosting provider took a week! And all this time we had to work much slower on our spare infrastructure. Can you imagine what it means for a retail network to stop for a whole week ?!

We were ready to rent physical equipment from this hosting provider, to which the hoster told us that we needed about $ 85,000 and offered to buy this equipment ourselves for installation in their data center. This option, of course, did not interest us.

In the search for a solution, we tested different hosting providers — both ours and European ones — and saw the difference.

On our own servers, 100 invoices were loaded in 16 minutes. If you plant 5 people who will upload 100 invoices - this stretched to 2 hours and the 16 minutes turned into 40 minutes - 1 hour.

On the SSD hosting of a domestic provider, the same package of documents was downloaded in 9 minutes.



With SSD hosting in a European data center, everything was loaded in about 4 minutes. The main task that we solved was the elimination of queues to the database, and at the facilities leased from the provider SIM-Networks we managed to achieve the desired result.

After accumulating an unsuccessful, but extremely rewarding experience, we decided to rent a physical infrastructure rather than a virtual one from a European hosting provider. To her, the company moved not only the main 1C server. After a successful experience of use, we additionally rented 2 backup servers, plus transferred a cluster including IT infrastructure, file server, Microsoft exchange, etc. In general, we rented several servers and a shelf in the data center.

On the economic feasibility of leasing capacity for IT infrastructure

At the time when we thought about switching to SSD, our servers had already served us for 3 years, the warranty from the manufacturer was just ending (next business day) and it needed to be extended. This guarantee annually required about 300,000 hryvnia (approximately $ 25,000 at that time) plus the cost of electricity.

Calculations showed that if you independently purchase all the necessary equipment, we will reach 0 to cover capital expenditures only after 3.5 years. Given the continuous development of the company, the amount of the cost of increasing the capacity of our IT infrastructure could increase exponentially.

As a result, renting the same equipment from a hosting provider did not come out much more expensive, but this is without worries about:
- extension and purchase of a guarantee;
- increase physical capacity (equipment upgrade);
- electricity costs.

We go to the clouds!

We want to expand again, as we no longer have enough power, and seriously consider the cloud option. We don’t have fears of moving to the cloud, as we’ve seen from our experience - it’s not so important WHERE to rent, it’s important to rent someone, it’s important to choose a reliable service provider!

Andrey Semenyuk
CIO
The network of pharmacies DS (Market Universal LTD)

The story of our adventures with hosting and transition to the clouds:

2012 - hosting the dska.com.ua site, unsuccessful attempts to upload the entire directory to Ukrainian hosting, hosting the site in SIM Networks

2013 - plans to switch from own SAS to own SSD (economically unprofitable), transfer of the company's database to the Ukrainian data center, failures during operation - violation of business processes - during the year

2014 - allocation of the SSD shelf in the Ukrainian Data Center, backup for a week during database transfer, transfer to SSD hosting in SIM Networks - transfer of database, 1C servers

2015 - after a successful experience of use, we rented several backup servers, moved a cluster (IT infrastructure), a file server, Microsoft exchange, etc.

2016 - plans to fully switch to cloud / leased infrastructure in SIM Networks

---

SIM-CLOUD - Fail-safe cloud in Germany

Dedicated servers in reliable data centers in Germany!
Any configuration, quick build and free installation

Our fast SSD hosting for sites on CMS 1C-Bitrix . We are also ready to offer dedicated servers and VPS optimally configured to work with 1C-Bitrix products.