Import data into online stores: an algorithm that will make your life easier
The previous article habrahabr.ru/post/209040 addressed general issues of data import into online stores, the associated features, advantages and disadvantages of the methods. It can be concluded that the only effective way in the long term is to develop your own data import modules for each online store (meaning direct import into a specific CMS database). Why? First of all, the code of the import modules is fully controlled, you can always make changes quickly. The second, which is also very important, is the integration of the modules into the software line. What is special about import modules? This is a large number of parameters that affect the data import process. Below is a simplified diagram of the data import algorithm in the CatalogLoader modules.
The following image shows the standard download dialog from the CatalogLoader to OpenCart software:
')
Standard options for OpenCart:
- Disable products (determines the ability to disable the "disappeared" products).
- Prefix articles that do not need to be updated.
- Database prefix.
- Path to the Catalogloader_handler.php file.
- The path to the file delete_cache.php (typical for OpenCart).
It is worth noting that the standard import module, which is provided to users, does not have the entire set of parameters available in the algorithm. This is done deliberately in order not to overload the interface and so that users do not have additional questions. These parameters are sufficient for the vast majority of projects. And only parts need special options that are available in a special mode.
The following image shows the standard dialog for uploading to Bitrix via the 1C-Exchange protocol:
Features:
- 1C-Exchange protocol support.
- Support for debug mode. Often this mode is the only option to import into Bitrix.
The principle of the algorithm is based on the following idea. The input data for the algorithm is the local database with the new state of the online store (we call this the KLP base). Those. after the import process is completed, the CLP-database “flows” into the required CMS database. Completely or not, with some features or not - it already depends on the settings of the data import process that the user defines.
Below are technical information that will be of interest to integrators and developers - parameters that affect the process of importing data.
CmsEngine string variable [default ("")]
Defines a CMS online store. Possible values: bitrix, commerceml, magento, insales, prestashop, vamshop, opencart, hostcms, shopcms, shopscript, virtuemart, joomshoping, simpla, etc.
ModeIsDisable Boolean variable [default value (false)]
Includes a mode that will deactivate products that are not in our KLP database.
DbPrefix string variable [default value ("")]
Specifies the prefix of the online store database.
HttpTunnelLink string variable [default value ("")]
Defines a link that serves as a data transfer interface between the CatalogLoader and the online store.
The options are: a link to the script CatalogLoader_hanpler.php, a link to the script 1c_exchange.php for transmission over the CommerceML protocol.
HttpTunnelDelay numeric variable [default value (0)]
Defines the delay between requests that go to the interface defined in HttpTunnelLink. This is necessary if the site is located on a weak hosting and active actions from the side of CatalogLoader (frequent calls) stop the work of the site (CPU limit, for example).
HttpImageWebDownloaderLink string variable [default value ("")]
Defines a link to the script for uploading images to the online store hosting.
For cms such as Bitrix, insales is not required.
HttpImageWebDownloaderLinkSaveToFile Boolean variable [default value (false)]
Prevents downloading of images via the interface defined in HttpImageWebDownloaderLink, and saves all the parameters for calling this interface to a file in the folder with the program CatalogLoader. This is necessary when it is necessary to postpone the loading of images for the future.
SqlConnectionString string variable [default value ("")]
Defines a ConnectionString variable of the online store database. This allows you to significantly increase the speed of loading data.
ShopUser string variable [default ("")]
Specific user login (usually admin) in the target online store.
Available for: bitrix in 1c-exchange mode, insales.
ShopPassword is a string variable [default ("")]
The specific user password (usually the admin) in the target online store.
Available for: bitrix in 1c-exchange mode, insales.
CategoryNameForImportAll string variable [default value ("")]
Defines the category in which all “new” products for an online store will be defined. Sometimes it is required to import all new products into the “New” category, and simply update old products.
Note: does not work for bitrix, commerceml.
PriceUpdate Boolean variable [default value (true)]
Specifies the behavior of the product update function, in which the price of the product is not changed by the CatalogLoader program. This is necessary when the site administrator independently sets prices for the goods.
MissedProductIsDisabled Boolean variable [default value (true)]
Defines the deactivation mode of products. If the value is “true”, then deactivation of the product means real deactivation (the product remains in the database and is marked as inactive), otherwise the product will be deleted from the CatalogLoader software store. (I DID NOT GET THAT)
MissedProductProcess Boolean [default value (true)]
Determines the behavior of the product deactivation function. If the value is “true”, the goods that are present in the online store database and are not in the LPP file will be deactivated, otherwise the product deactivation function will not be applied to any goods that are missing.
Boolean variable MissedCombinationIsDisabled [default value (true)]
Determines the behavior of the function deactivate combinations of goods. If the value is “true”, then combinations of goods that are present in the online store database and are not in the LPP file will be deactivated, otherwise the removal from the database follows.
CustomParameters string variable [default value (““)]
Parameters specific to a particular CMS are determined. For example, for prestashop, the value of this parameter can be id_lang = 7, meaning that you need to import the text data of the goods into the language of the online store at number seven.
CallUrlOnStart string variable [default value ("")]
Specifies the http link that will be called from the CatalogLoader software (get request) before the import process starts. Usually used to create a backup copy of the database.
CallUrlOnFinish string variable [default value ("")]
Specifies the http link that will be called from the CatalogLoader software (get request) after the import process is complete. Usually used to clear the cache or create a backup copy of the database.
ProductsUpdateSkip boolean variable [default value (false)]
Defines the behavior of the product update function. If set to “false”, the update function is applied to the product, otherwise it is not.
CustomerProductSkuTemplate string variable [default value ("")]
Defines a pattern (regular expression) with which you can determine a product that was entered into the online store manually, and not through the CatalogLoader software. If a pattern is defined, then the goods, the article of which matches the pattern, are excluded from the selection of products to which the “Deactivation” function should be applied.
For example, CustomerProductSkuTemplate = CUS_
ImagesUpdate Boolean variable [default (false)]
Defines the behavior of the image update function. If the value is “false”, the images are not updated, otherwise they are fully updated, and all images of the processed product are automatically deleted beforehand. Usually, this parameter should be set once to “true” if the photos are poorly filled in previous update iterations.
ImagesUpdateMissed Boolean variable [default value (false)]
Defines the behavior of the image update function. If the value is “false”, the images are not updated, otherwise they are updated in the add mode. First, the program determines the images that the product already has and only after that adds the required “new” images to the product. Usually, this parameter must be set to “true” once if the photos are poorly filled in previous update iterations.
Available for: prestashop.
MiminumProductsCountInSet numeric variable [default value (0)]
Defines the minimum quantity of goods. If the value of the parameter is greater than zero, the CatalogLoader software will not start the process of importing goods if there are fewer goods in the processed LPC file than is determined by the parameter.
Of course, there are many nuances everywhere, not always and not for every CMS it turns out to support all the parameters of the algorithm. This may be due to the exchange protocol, the database architecture or the characteristics of the CMS implementation. The presented approach allows a fairly flexible configuration of data import processes in online stores and copes with large amounts of data up to hundreds of thousands of positions.
This approach was implemented to import data into the following CMS:
Nikolay Kekish,
Director CatalogLoader
Algorithm of data import to the online store
Implementation
The following image shows the standard download dialog from the CatalogLoader to OpenCart software:
')
Standard options for OpenCart:
- Disable products (determines the ability to disable the "disappeared" products).
- Prefix articles that do not need to be updated.
- Database prefix.
- Path to the Catalogloader_handler.php file.
- The path to the file delete_cache.php (typical for OpenCart).
It is worth noting that the standard import module, which is provided to users, does not have the entire set of parameters available in the algorithm. This is done deliberately in order not to overload the interface and so that users do not have additional questions. These parameters are sufficient for the vast majority of projects. And only parts need special options that are available in a special mode.
The following image shows the standard dialog for uploading to Bitrix via the 1C-Exchange protocol:
Features:
- 1C-Exchange protocol support.
- Support for debug mode. Often this mode is the only option to import into Bitrix.
Algorithm principle
The principle of the algorithm is based on the following idea. The input data for the algorithm is the local database with the new state of the online store (we call this the KLP base). Those. after the import process is completed, the CLP-database “flows” into the required CMS database. Completely or not, with some features or not - it already depends on the settings of the data import process that the user defines.
Below are technical information that will be of interest to integrators and developers - parameters that affect the process of importing data.
CmsEngine string variable [default ("")]
Defines a CMS online store. Possible values: bitrix, commerceml, magento, insales, prestashop, vamshop, opencart, hostcms, shopcms, shopscript, virtuemart, joomshoping, simpla, etc.
ModeIsDisable Boolean variable [default value (false)]
Includes a mode that will deactivate products that are not in our KLP database.
DbPrefix string variable [default value ("")]
Specifies the prefix of the online store database.
HttpTunnelLink string variable [default value ("")]
Defines a link that serves as a data transfer interface between the CatalogLoader and the online store.
The options are: a link to the script CatalogLoader_hanpler.php, a link to the script 1c_exchange.php for transmission over the CommerceML protocol.
HttpTunnelDelay numeric variable [default value (0)]
Defines the delay between requests that go to the interface defined in HttpTunnelLink. This is necessary if the site is located on a weak hosting and active actions from the side of CatalogLoader (frequent calls) stop the work of the site (CPU limit, for example).
HttpImageWebDownloaderLink string variable [default value ("")]
Defines a link to the script for uploading images to the online store hosting.
For cms such as Bitrix, insales is not required.
HttpImageWebDownloaderLinkSaveToFile Boolean variable [default value (false)]
Prevents downloading of images via the interface defined in HttpImageWebDownloaderLink, and saves all the parameters for calling this interface to a file in the folder with the program CatalogLoader. This is necessary when it is necessary to postpone the loading of images for the future.
SqlConnectionString string variable [default value ("")]
Defines a ConnectionString variable of the online store database. This allows you to significantly increase the speed of loading data.
ShopUser string variable [default ("")]
Specific user login (usually admin) in the target online store.
Available for: bitrix in 1c-exchange mode, insales.
ShopPassword is a string variable [default ("")]
The specific user password (usually the admin) in the target online store.
Available for: bitrix in 1c-exchange mode, insales.
CategoryNameForImportAll string variable [default value ("")]
Defines the category in which all “new” products for an online store will be defined. Sometimes it is required to import all new products into the “New” category, and simply update old products.
Note: does not work for bitrix, commerceml.
PriceUpdate Boolean variable [default value (true)]
Specifies the behavior of the product update function, in which the price of the product is not changed by the CatalogLoader program. This is necessary when the site administrator independently sets prices for the goods.
MissedProductIsDisabled Boolean variable [default value (true)]
Defines the deactivation mode of products. If the value is “true”, then deactivation of the product means real deactivation (the product remains in the database and is marked as inactive), otherwise the product will be deleted from the CatalogLoader software store. (I DID NOT GET THAT)
MissedProductProcess Boolean [default value (true)]
Determines the behavior of the product deactivation function. If the value is “true”, the goods that are present in the online store database and are not in the LPP file will be deactivated, otherwise the product deactivation function will not be applied to any goods that are missing.
Boolean variable MissedCombinationIsDisabled [default value (true)]
Determines the behavior of the function deactivate combinations of goods. If the value is “true”, then combinations of goods that are present in the online store database and are not in the LPP file will be deactivated, otherwise the removal from the database follows.
CustomParameters string variable [default value (““)]
Parameters specific to a particular CMS are determined. For example, for prestashop, the value of this parameter can be id_lang = 7, meaning that you need to import the text data of the goods into the language of the online store at number seven.
CallUrlOnStart string variable [default value ("")]
Specifies the http link that will be called from the CatalogLoader software (get request) before the import process starts. Usually used to create a backup copy of the database.
CallUrlOnFinish string variable [default value ("")]
Specifies the http link that will be called from the CatalogLoader software (get request) after the import process is complete. Usually used to clear the cache or create a backup copy of the database.
ProductsUpdateSkip boolean variable [default value (false)]
Defines the behavior of the product update function. If set to “false”, the update function is applied to the product, otherwise it is not.
CustomerProductSkuTemplate string variable [default value ("")]
Defines a pattern (regular expression) with which you can determine a product that was entered into the online store manually, and not through the CatalogLoader software. If a pattern is defined, then the goods, the article of which matches the pattern, are excluded from the selection of products to which the “Deactivation” function should be applied.
For example, CustomerProductSkuTemplate = CUS_
ImagesUpdate Boolean variable [default (false)]
Defines the behavior of the image update function. If the value is “false”, the images are not updated, otherwise they are fully updated, and all images of the processed product are automatically deleted beforehand. Usually, this parameter should be set once to “true” if the photos are poorly filled in previous update iterations.
ImagesUpdateMissed Boolean variable [default value (false)]
Defines the behavior of the image update function. If the value is “false”, the images are not updated, otherwise they are updated in the add mode. First, the program determines the images that the product already has and only after that adds the required “new” images to the product. Usually, this parameter must be set to “true” once if the photos are poorly filled in previous update iterations.
Available for: prestashop.
MiminumProductsCountInSet numeric variable [default value (0)]
Defines the minimum quantity of goods. If the value of the parameter is greater than zero, the CatalogLoader software will not start the process of importing goods if there are fewer goods in the processed LPC file than is determined by the parameter.
Of course, there are many nuances everywhere, not always and not for every CMS it turns out to support all the parameters of the algorithm. This may be due to the exchange protocol, the database architecture or the characteristics of the CMS implementation. The presented approach allows a fairly flexible configuration of data import processes in online stores and copes with large amounts of data up to hundreds of thousands of positions.
This approach was implemented to import data into the following CMS:
Nikolay Kekish,
Director CatalogLoader
Source: https://habr.com/ru/post/220187/
All Articles