The evolution of IT-infrastructure of our retail - consistent "molting"

Starting a retail or large online store without a strong IT side in terms of at least data analysis is a very bad option. But you can not immediately use the full functionality - you will bury the complexity alive.

Therefore, I will now tell you how we have consistently "shed" from simple systems to more complex ones. The general principle is that you first do it cheaply and angrily so that it works. Then - correctly. Then - optimal, but expensive. And all this is for reinvested profits, and not immediately in capital costs.
')
So, when we didn’t have a store, from the IT infrastructure there was only a cell phone, a personal laptop with Yandex.Maps for rendering deliveries and an Excel spreadsheet for bookkeeping. We hosted our site with 20 games on Masterhost, we didn’t think about any database replication or anything like that - everything was on static HTML.

1C

Around the second store, we became so many that only manual operations were no longer channelized. A large general journal was required at that time for the call center (at first it was paper, then moved to Google.Docs), in retail it was necessary to analyze too much data, plus there was a threat of abuse inside the company.

So we discovered 1C. First trade, then all its other varieties. First glance was like a combine: just a bunch of unnecessary things. Guess what we started to do? That's right, they put the developer to “sharpen” it for us, that is, to remove all those functions that “slowed down the work”.

The database began to quickly acquire crutches for tasks. When you don’t have a code to maintain, it's simple: hrenak-hrenak - and in production. It is now to do, for example, a gift certificate, you need half a year to work with lawyers, accounting and train the entire network throughout the country. Then it would take two hours.

Having driven a dozen crutches, we were faced with the fact that some very smart-ass staff learned how to steal. We found them and fired them, but the investigation of the incidents showed that in order to maintain strict accounting, it was not at all necessary to throw out the functionality that seemed superfluous to us. Guess what we started to do. That's right, they put the developer to restore everything as it was out of the box.

Then it became more fun. It turns out that these cumbersome complicated and non-obvious procedures, which exist by default in 1C, were not invented by any sadists in order to drive users out (I still miss the 7th version a little bit, where everything was clear and understandable in terms of requests - but not at all obvious, for example, for sellers by GUI). It turns out that all this kit was needed.

A year later we realized that if we didn’t analyze a huge amount of data on goods, sales and much more, efficiency drops. The first mechanisms for calculating logistics and additional delivery of goods to stores were considered empirically, almost by sensation. Then it turned out that, in general, the mathematical apparatus has been around for a long time; it simply begins to reveal its power from large numbers. We have grown to these large numbers (there are more goods than the purchasers could manually track) and began to use automation to calculate the same composition of supplies to stores based on demand, supplier reliability, optimal storage space, and so on. Each formula lay on top of the database and required additional data.

So we came to the conclusion that, without exception, these companies need to be stored in a single space from where they can be picked up by a robot. In a couple of years, this will become pathetically called Big Data and will acquire methodology, but for us it was just an unstructured collection of tables in a large database. Everything was stored - from the data of purchases and prices to statistics for the past year and even the description of the goods on the site. By the way, yes, the site takes all the information from one of the read-only copies of such database tables - this, among other things, allowed showing the availability of goods in stores in real time without special dances with a tambourine. Not without excesses, really. There was a case when we brought to the site the reason for the lack of goods. This reason was written by buyers in a special field. About two o'clock on one of the product pages there was an entry: “There are no games, because the supplier is a deer”.

Now the base 1C - the main source of information for all within the company. Through it, everything is linked to each other. 1C has become overgrown with a bunch of our modules: from solutions for a call center to managing shares on the site and even managing sorting of goods on the site.

At first, there were problems in the regions with 1C - in the mode we need, it wants at least half a megabyte, which was often not very realistic in a shopping center somewhere in the cities with 400 thousand inhabitants. Now even the cellular channel allows you to keep such a band, so everything is in order (and we began to do asynchronous replication).

Ticket Office

Initially, all cash registers were not connected to 1C. We used a special program, there were a lot of mistakes in the reports, someone constantly punched checks in the wrong section, wrote explanatory, the accounting department swore strongly. Now all cash desks are connected to 1C, various buns appeared, such as a check with information about the accumulated discount, the ability to see quickly the balance of cash, how many returns, how much money was removed to the safe, quickly find a check for returning “day-to-day”, and In the evening, quickly fill out a report for management, since the system fills almost all the data itself.

Updates

We have branches throughout the country, the most distant one is in Yuzhno-Sakhalinsk. Therefore, there is no single time when all stores are offline. The scatter of time zones suggests that there are only a couple of hours at night, when you can quickly roll up the update 1C. It turns out that at least once a week (more often in the development cycle in the off-season), at least two of our IT specialists, who check everything, are going to take part in a remote night service. It is necessary to meet at 1 o'clock in order to have time to roll back, if that.

I remember, there was a case when a large-scale reinstallation was needed in several stages. First, the provider in the data center added equipment to the server, then the system administrator did the work on his part, then the turn of the 1C department followed, all this happened from 2-4 o'clock at night. At 3 nights, 1C specialists lost contact with the sysadmin. Half of my life flashed before our eyes, our people clearly showed how we would explain to the shops why we couldn’t sell anything. Fortunately, the admin just got the phone, and he quickly got in touch again.

After that, we started to make a backup database not once a day, but asynchronous replication. And, it should be noted, despite the internal workings of IT, the work of the company's departments over the past three years has never stopped.

Our 1C specialists tell one more fun fact: the less users know about changes or system updates, the quieter it is to work. It was noticed many times that if users were warned about a nightly update, in the morning they would begin to take a closer look at the system with special attention. And there will be at least 3 calls from users who have something broken and they know for sure that this is due to the update. It is because of this.

Site

First, a simple static site on Masterhost. A little later, when there are more games, a more complicated version is there. By next November, we first saw attendance that could put the site. Moved to another site, but still lay down. For a long time they suffered from different sites in the Russian Federation, then they simply took it and moved to Amazon. It took several instances - the main site, a backup stub with a ping service, several base replicas for different purposes (shops, site cache, other retail services), plus the base itself.

The site still fell several times. The second major fall is my first habraeffect when we found out that deploying a new instance is done incorrectly. The third is lightning in a data center in Ireland (many people remember it here). Then we dropped almost half a day (specifically, the order form, the rest worked) the RetailRocket script, which did not respond - since then we have been afraid of this.

Internal resources

Here the development is relatively standard. First - personal laptops, then the first desktops in the local network, with an increase in the number of stores - server farm and terminals in the field. We accumulated a lot of internal data, and we added it to our network storage, where mirrored RAID was raised for reliability. When one day the store died, we learned that:

1. The last backup of heavy data (like a photo) is a month ago.
2. Nobody checked the second piece of RAID for operability, and for three months nothing has been written there, and there is no indication.
3. And our disk with the necessary data went to the country of eternal hunting due to the controller's failure.

Fortunately, the patient underwent a transplantation of the controller, and we collected data from it.

Inside the company we constantly communicate by mail, messengers are not raised at all. It used to be ICQ, now - Skype for chat. Intercity for a call center - its own lines, for purchasers - cellular unlimited, for wholesale - Skype. Broadcasts in the corporate blog. Trackers are optional in departments, somewhere there are, Google calendars are used instead of them, somewhere there is a 1C functional for repetitive tasks. There is no single tracker.

All important - mail. We change files through dropboxes, many, especially in the development, where there are a lot of pictures, large accounts are bought. At some point in the office, they almost completely switched to tablets in addition to desktops - this determined more attention to Google services and cloudiness in general.

Transaction letters (“your order is confirmed”) are sent from our Amazon servers, mailings - first, also, then learned through Unisender. It is important to separate, because if a person unsubscribes from the mailing list, the order letters will not come to him either - I know a travel agency that was badly burned at this. Two years ago they switched over to external service, when it was necessary to fight for the percentage of delivery. Previously sent from their servers and almost through the console, so not without incident: there was, for example, “hello, this is a test” for 50 thousand people in the first year of operation.

Iron

Our sysadmin at some point took and founded his business. Now it is a company that supports us and a few more retailers. The first dialogue with the new unfamiliar and terribly shy admin was this: “I have something the computer does not turn off ... Yes, I tried. Yes, I took it out of the socket. No, it does not turn off. Yes, Windows was on a virtual machine. Yes, the laptop. "

The most important for us was the transition to business processes, which resulted in an SLA for each joint on the one hand, and a clear, separate payment on the other. For 5 years, no questions for the guys: silently come, do and leave. When our girls caught an encryption Trojan on personal laptops (“Oh, I don’t open something, I’m sending you, please see”), one of the “sweaters” hit me deeply, disassembling the malware, finding there is an error in the random number generator and, four hours later, having reformatted codes for decoding to the result. In general, there are no tasks impossible.

I remember how a sysadmin called in support for the day early in the morning, said that it was full of trash, frenzy and needed urgent repair, opened the connection to his desktop and asked him to connect the most experienced one. When he logged out - he saw his congratulations.

General philosophy

If we knew in the beginning how many crutches and “molts” would be needed, we could have done it right away. But I think it would not be very effective. As I have already said, it is optimal to work on what is, albeit crooked and unscaled. There was a need and a profit - took a profit, invested in a new level. And so round after round. Yes, it is more expensive, but all capital costs turn into operating rooms, which makes you wildly happy for a retail model.

We are often asked how to organize this or that piece of IT. It seems to me - do as it is convenient now, not thinking about cool technologies, the future and advice in the spirit of "do not forget to do this or that, otherwise you will have to correct it" written by the theorists. When you have a dedicated IT department, then its head will already think about such matters. At the moment of starting a business and the first 1-2 years of development, the main thing is that everything should work here and now. And some rakes are very useful, because if they are not stepped on, while they are small, then the error will cost much more.