Import data to online stores

What do we mean by importing data into online stores? To explain, we describe the scenario that is most typical for online store owners.
Usually, online store owners want to import relevant data from suppliers. They need it in order to update the status of the database of their online store, so that they quickly have new items, so that those products that are no longer sold by the supplier are not available for order in their online store. And they want to repeat this process as often as possible.


Supplier data is:
')
- categories (with or without their hierarchy),
- products with descriptions, parameters and attributes, images, their combinations, if any,
- availability status, quantity in stock, purchase price, sale price, discounts, minimum order.

It is about these data will be discussed. There are two options for working with this data:
1) the online store is empty and we do not need to take into account what is in it;
2) the online store is already full.

We will consider the second option, because it most fully reflects the specifics of the data import process for online stores.

Requirements



What does an online store owner usually want? His requirements for automating data import are as follows:

- if a new category has appeared on the supplier, it is necessary that it also be created in his store;
- so that goods that are not in his store and present at the supplier appear (meaning with descriptions, characteristics, pictures, color options, etc.) in the desired category and (or) of the “New” category;
- if the product is present in the customer’s store, but is not in the supplier’s warehouse, then you need to disable (deactivate) this product or change the status to “out of stock” (this is necessary so that the product becomes unavailable for ordering);
- if the product is present in the store and at the supplier, then you need to update the prices and quantity (if such data is available).

Hidden requirements



In fact, an online store manager wants more from imports. These are the so-called hidden requirements, about which no one speaks out loud, but, as it were, everyone means already when accepting a project or operating the first month.
These hidden requirements for importing data into an online store are the following items:
- so that the products that the user brings to the online store independently, the process of automatic import does not “change” (did not delete, did not deactivate);
- so that editing the descriptions of products that are updated from the supplier’s warehouse, which he does with the help of copywriters, does not disappear (this is important for SEO);
- for new categories and products to form correct and beautiful CNCs - “URL rewrite”;
- to change the price taking into account its margin, which depends on the category and popularity of the product;
- receive reports on what new products have appeared, and which are no longer on sale;
- so that the process of updating data in the online store is simple and takes little time (a few or 1 click);
- if the product is present in the supplier’s system in more than one category, then you need to link this product to the appropriate categories on the side of the online store;
- if the goods that we import into the online store are combination goods (for example, the shoe is “red, 42 size, 2300 rubles” and “black, 43 size, 3100 rubles”, etc.), they were processed correctly.

Solutions



All the requirements described above are caused by the specifics of e-commerce, and the algorithm that will import the data must take them into account in full.

How to fulfill these requirements, what tools can be used?
- use the standard CMS import (usually from files from csv / xls / xml / yml, etc.);
- import via the 1C-Bitrix exchange mechanism (1C-Bitrix, UMI, HostCMS, etc., where there is an exchange with 1C);
- program your own import mechanism.

An analysis of each of the proposed options will be presented below.

Importing data via csv is a very simple thing in itself and should work reliably. However, as practice shows, this is not always the case. There are questions if you need to import something more complicated than the set “name, description, picture, article, price”. What to do with features, options, etc.? Today, this option of importing data into the CMS is not always well programmed in most implementations, and always requires improvements. Plus, these are intermediate files (they need to be generated), hosting requirements (restrictions). And the main disadvantage is the manual work, or rather, a lot of boring and of the same type of manual work, which few people like to do.

Data import through the 1C-Bitrix exchange mechanism. The idea itself is very good. 1C is at all, and if it is not, then the protocol itself can be emulated. But the implementation on the side of online stores makes asking many questions. If you look towards the benchmark for all Beatrix, then you can simply say that on the volumes of more than 10 thousand products this mechanism already fails. Malfunctions are different. For example, it may not support ZIP. There may be problems with the amount of available memory for the php process. Speed ​​is the most sensitive topic for importing into Bitrix through 1C. Why is it so difficult to say so slowly, but today it is so. For example, the default bitrix imports not into a standard information block of goods, but into an information block that is specially created for 1C. Plus, when working through 1C, we have a loss in time, that is, the data is loaded twice - the first time in 1C, the second time in Beatrix. If you look at other CMS, then there the implementation of this protocol is in a state where ordinary people cannot use it without the help of programmers. In more detail about the import of goods into Bitrix, we wrote in this article: habrahabr.ru/post/133993 .

Programming your own mechanism. It is difficult and expensive to import this way, but the implementation gives a lot of advantages that are not available today from the box in the proposed solutions. In our opinion, if the task is to fulfill the points indicated in the requirements for the import of data in full, then only this option is able to meet the requirements.

The following factors are key when choosing a tool for import:
- data update rate,
- the amount of data transmitted,
- data update rate,
- details of information
- budget
- competence and team workload.

Those. It turns out such a schedule for selection.



Regarding budgets, the situation is ambiguous and depends primarily on the cost of the integrator team. The graph shows the arrangement of budgets and proposals on the Russian market. The overestimated cost of integration of 1C-Bitrix is ​​due to the lack of specialists of the required level of 1C and Bitrix in the market, which causes the corresponding rise in the price of development and support of the solution. CSV integration is possible and acceptable with small amounts of data or in the absence of a budget, or with a small granularity of data import.

How to implement support for “direct import into the database” of more than 10 popular CMS online stores will be discussed in the following articles.

Nikolay Kekish,
Director of CatalogLoader.com