Data migration between different e-commerce engines

Problem

Today there are more than three hundred different engines for online stores, with different functionality, features, cost, installation method. As a rule, this is a set of scripts that are deployed on the hosting (practically setting up the site), most often PHP + MySQL.

Recently, the so-called hosted shopping carts have occupied an ever-growing niche in the market for E-commerce platforms. This means that by registering on the site of the same Shopify, in a few clicks you can get a trial store (store), which is hosted on the Shopify itself. Trial period and limited functionality, as a rule, is quite enough to check card capabilities. In addition, such a solution, of course, removes the headache associated with renting your own hosting, installing the engine and all the necessary PHP card modules, etc. Everything is already deployed, as they say, pay and use.
')
As a result, different platforms are different ways to reach the store data. If an open source card (“not hosted”) can be accessed directly to the database, then SaaS solutions do not provide this feature. Alternative options can be access via API requests (and very well, if the platform allows you to get all the necessary data in this way, because sometimes developers simply do not add all the methods of working with an entity) or export / import data using files (CSV, XML, txt, dat, xls and other formats depending on the engine).

The latter method is supported by almost all platforms (although each has its own limitations) and simplifies the migration between identical maps. But when you need to move from a store based on osCommerce, which exists and has been operating for more than a year, to a fresh version of Magento or BigCommerce, the task becomes much more complicated.

For a developer who has such a goal, there are two possible ways to solve it:

• use a ready-made tool for automatic migration;
• if the previous version cannot satisfy all the requirements - search / develop your own solution: a module or a separate script that at least changes the format of the file exported from the old platform for importing into a new one (if the platforms support working with files).

Of course, you can manually enter data into the new platform through the admin panel, but when the number of products and / or users is measured in three-digit numbers, this option is no longer an option :)

Therefore, we consider the first two cases.

Custom migration

This method of data transfer can be longer and its cost, in comparison with automatic migration, can also be greater.

Why?

It's simple, because you need to hire a programmer who would work on this task. It goes without saying that you have to pay more for the work of a person, not a car. And even if you yourself are a first-class programmer and are quite able to write the necessary script - this will take your time, and, as you know, it will be money. Especially if you are a good programmer.

Another factor worth considering is the time it takes to develop a solution. And if your business project is rather urgent, then you will almost certainly not have enough time.

On the other hand, the increased cost and development time that characterize this approach can be justified by one big advantage — adaptation to specific goals and requirements. However, there may be their limitations, but about them a little later.

Automatic tools, as a rule, support some limited number of platforms. That, on the one hand, is undoubtedly a plus, since such migrations are polished by years of use and thousands of migrations. But at the same time, it is also a minus, because it allows you to cover only a certain amount of data common to these platforms and interpret them in the way it is incorporated in the tool. For example, the essence of the tax (tax) in different maps can be configured and used completely differently, consist of completely different elements and data. Therefore, the migration will take place as it is in the tool. Using a custom tool such things can be minimized, because it will be developed for a specific case.

In order for a custom solution to be as efficient as possible, you should study in more detail the platforms you need to work with. Investigate entities and determine the relationships between them in order to work them out correctly. It should be remembered that if the platforms are different, one of them can support an entity that the other does not have. For example, in Magento there is the concept of product variants, but in OpenCart it is not. Much less often, but still there are cases when the entity is supposedly supported by both platforms, but each is built differently and at the same time can have a completely different purpose.

Particular attention should be paid to the method of access to the data, since it is this part that can become the bottleneck of the future solution. If the shopping card is “not hosted”, that is, a set of scripts is installed with the database, then you have physical access to this database, and therefore you can get all the necessary data in the same way as adding them to this database. The main thing is to understand where and what information is located, what it means and how it is formed. In addition, in such maps you have access to files: images of goods, downloadable data for goods, etc.

In this case, the data migration scheme will look like this:



There are 2 stores on different servers. The migration script is installed on one of the servers and uses direct connection (direct connect) to the database of this server. To connect to another site, you can use direct connect, but for this you need to open access from the outside.

If stor on one server, then additional access is not required. You can use the same ones that the one uses to connect to the database. Usually they can be found in the configuration file of the store (for example, config.php or configuration.php in the root directory of the store).



An example of a class for connecting to a database in PHP:

class Db { private $_link = null; private $_triesCount = 10; public function connect($config) { while (!$this->_link) { if (!$this->_triesCount--) { break; } $this->_link = mysql_connect($config['dbHost'], $config['dbUser'], $config['dbPass']); if (!$this->_link) { sleep(5); } } if (!$this->_link) { die(mysql_error($this->_link)); } mysql_select_db($config['dbName'], $this->_link); } public function query($sql) { $query = mysql_query($sql, $this->_link); if (!$query) { die(mysql_error($this->_link)); } return $query; } public function cntRows($query) { return mysql_num_rows($query); } } 

The migration process consists of the following steps:

1. Extracting data from a source store (source store): from a database; through API; from files.
2. Target data formatting: in the structure of tables or target file.
3. Inserting data on the target stor: in the database, through the API, in the files.

A little different all hosted shopping cards. There is no direct access to the database, since the store is installed on the developer’s servers, and you are only given the opportunity to manage it from the admin panel. To reach the data, some developers of hosted shopping cards (unfortunately, not all) provide a specific API, which includes a set of methods for working with this data.



However, depending on the card, this set can be quite stingy and not satisfy the requirements of the task. These are the limitations mentioned earlier. For example, Shopify, despite the fairly well implemented and documented methods, does not allow to create customer orders. You can only view information about available or change the status of one of them. And Miva Merchant does not have the ability to work through the API at all. Another feature of the API is a limit on the number of requests over a period of time. If a script or program exceeds this limit, then subsequent requests may not be executed at all within a few minutes.

Almost every shopping card has data export / import as files. If the information in such files is sufficient for successful migration, then access to the database is not required. Quite often, the structure of the files exported by the map is intuitively more comprehensible than the structure of the database, and more complete than the content of the responses of API requests. The lack of a direct connection to the database allows you to make the tool more flexible: it will be tailored to a specific file format and will not depend on such things as database access rights or restrictions on the number of API requests. Using files as data carriers between shopping cards, you can not create any own scripts or programs at all. It is enough just to edit these files, bringing them to the required format.

Several file migration schemes:

1. Convert one file format to another:



2. Migration from file to database:



3. Migration from base to file:



The stages of such migrations will be almost the same as in the previous examples, but there may be additional steps:

1. Export data from the source to files.
2. Pulling data from files.
3. Bringing data into a target format: in the structure of a table or a target file.
4. Forming files for target stor.
5. Import generated files to target stor.

By the way, if the option where you need to open access to the database server is unacceptable, then you can get data from the database as files. phpMyAdmin allows you to export the contents of tables as CSV, XML and other files.

As for the resources required for the custom tool, this will depend on the amount of data to be transferred and the implementation of the script itself. For greater reliability and lower resource consumption, it is better to transfer data in chunks, for example, in arrays of 100 entities.

In general, if we consider the version of a PHP script that will work on one of the servers, then it is worth considering the following things:

• the script creates an additional load on the server, which may affect the operation of sites or sites located on the same server;
• the script needs a certain amount of RAM, which directly depends on the implementation of the script, in particular, on the size of the data packet, which will be transferred in one iteration;
• the script creates an additional load on the disk system when working with a database or files;
• the duration of the data transfer process will depend on the amount of this data and can last from several minutes to several hours, sometimes even days.

As for the programming language that will be used to create a custom solution, there are no hard limitations. The main thing is that it allows you to implement the necessary functionality, such as connecting to a database (not necessarily MySQL), performing HTTP requests, working with files and images. Since a significant part of the platforms are written in PHP, it is the one that is most often used. Although the same Java is also suitable for solving the problem.

By the way, there are companies that create such custom tools or scripts. In particular, TransPacific Software Pvt. Ltd. offers to prepare a solution for migrations osCommerce to Magento, X-Cart to OpenCart, Zen Cart to OpenCart or Magento, Loaded Commerce (CRE Loaded) to Magento for a period of up to 25 working days.

Automatic migration

Among the ready-made solutions for transferring data between shopping cards can be divided into 2 groups: services and applications to the cards. The latter are not created for all maps and more narrowly focused. As a rule, they support only the map for which they are intended. Such applications are relatively inexpensive (if not free), being installed on one of the shopping cards, use the resources of the hosting hosting the card.

Such a means of data transfer could be attributed to hand tools, which would be true, given their essence. But also considering that these are still ready-made solutions, in my opinion, they have a place in this category.

Automatic services, in turn, cover a greater variety of shopping cards. The list of supported options includes the largest number of joint entities for different cards. Although if an entity is unique for a particular map, then it is possible that when transferring data between the same platforms, it will also migrate.

Of course, it will not be possible to transfer the data generated by the custom card module (for example, specific reports). At least the standard functionality of the service does not include this. If you still decide to use the service and custom data you need, you can always ask the support service if it is possible to do additional work. Of course it will be + to the price. Speaking of the price of automatic migration: it is usually lower than in the case of custom migration, since it requires minimal human intervention in the process.

Some services offer additional options for migration, for example, migrating SEO data, saving product IDs, orders, etc. The cost of these options is added to the cost of the entire migration.

As an example, consider the Cart2Cart service, which supports about 50 shopping cards. It allows you to transfer data such as products and their categories, manufacturers, customer lists and their orders, taxes, and also has a number of additional options for each specific direction of migration.

The migration scheme is as follows:



To connect to the site database, the service uses a special Connection Bridge, pre-installed in the root directory of the store.

All migrations are performed on Amazon's Elastic Cloud (EC2) servers, which ensures high speed of the process and allows you to cope with large amounts of data. Thus, all processes occur on dedicated servers.

For some shopping cards, migration from files is supported:

Relationships between entities

There are 3 main entities in each shopping map: products (products), customers (customers), orders (orders) . All others are somehow connected with the main ones. When migrating entities, it is important to take into account the relationships between them in order to correctly restore this data to the target location.

Below is a diagram of the connections of the main entities with others. Depending on the shopping card, their list may vary.
An entity may include such entities or be associated with them:

• categories ;
• images (images);
• reviews (reviews);
• options (options), which, in turn, include option values (option items).

An entity user is associated with addresses and may contain multiple addresses.



Entity order is associated with the following entities:

• user (customer);
• billing address;
• shipping address;
• products ;
• total (total);
• order history (history).

By the way, in the order , as a rule, information about users and products is duplicated. That is, in addition to saving the client ID, which made the order and the ID of the ordered products, basic information about these entities is duplicated in separate tables in the database in separate tables. This allows you to save complete order information, even if the user account or products from the catalog have been completely deleted.

Total

The process of data migration between E-commerce platforms can be complex or simple, long or fast. Everything will depend on the chosen path, tools and resources. Given the rapid development of the industry (according to analysts' forecasts, a third of offline stores will close until 2017), automatic tools are being actively improved, providing an increasing number of services, so writing custom scripts to transfer data fades into the background.