Development of an online store of 13,000+ products on the MODX Revolution. Part 1

I already wrote about my shopModx component. And although few people appreciated it, since many are waiting for exactly ready-made solutions with one big “Install and work” button, nevertheless this component is developed taking into account those minuses that exist in MODX, and which MODX developers often run into, taking into account the advantages that are in MODX, but about which the developers do not know, or simply do not use.

I also want to say that this module is not just being developed. It is being developed for two non-small stores (for a start), and the output will be a run-in platform for the implementation of large online stores.

Today I would like to start a series of articles on the development of large online stores on MODX Revolution, with stories about what difficulties one has to face and what solutions to solve these problems are used. As well as the fact that to solve such problems shopModx will already be carried on board, and what methods will allow you to get 100% control over the development of your unique store, without getting into the shopModx code.
')
So, a little about the store that is being worked on: this is an online furniture store. Yesterday imported base. It turned out 13000+ documents, 43000+ TVs and almost 13000 records in modx_shopmodx_products.

Immediately I will say that I expect to receive the page code not even from the cache and with the search by parameters in less than 1 second, and the average load should not exceed 0.3-0.4 seconds.

So, briefly about the first problems and their solutions.

Problem 1. Large cache file and a lot of used memory.

For starters, the input is for pure Reva. Specially downloaded a clean 2.3.0 and looked at the memory usage. The code stuck in the plugin for the OnWebPageComplite event is the most extreme point of the execution of MODX, after exit (), saving the document cache, etc. First run (manually deleted all cache files):

Memory: 13.5409 Mb TotalTime: 0.1880 s 

Further:

 Memory: 10.1396 Mb TotalTime: 0.0640 s 

By the way, just in case the plugin code is: gist.github.com/Fi1osof/5062419
You can modify the access permissions and always see the actual load on the server.

In general, we check the results on the store (by the way, I just want to clarify that the document is not empty, but has 8 related TV parameters, one of which is a picture with a custom media resource). First set

 Memory: 24.1438 Mb TotalTime: 0.4360 s 

Further:

 Memory: 18.4103 Mb TotalTime: 0.0960 s 

That is, we have an increase in the used memory of almost 10 meters at once. This is because we have cached the entire URL map of the context, and we have 13000+ documents there. The context cache file is almost 2 meters.

The obvious solution is to shorten the context cache file. I already wrote
in detail about the subtle points in MODX-caching and about your cacheOptimizer patch. We set it and disable caching of the resource map for the web context. New results:

 Memory: 16.1369 Mb TotalTime: 0.2640 s 

 Memory: 10.4021 Mb TotalTime: 0.0720 s 

That is, in normal mode, we consume almost as much memory as on the bare system.

Problem 2. Page not found (404)

This problem stems straight from the previous solution :-) Since we chopped off caching the URL card, now MODX will not be able to “understand” URL, which page we are accessing when using the CNC. Immediately, I’ll clarify that if you don’t use CNC, then this shouldn’t be a problem for you (although who doesn’t use CNC today?), Or if you don’t have a large store (up to 1000 products), then you can not chop off the page map , an extra megabyte of RAM is not a problem.

So, to solve this problem, I decided to use my own router. I just wrote a new class extending modRequest, and slightly corrected a couple of methods. The logic is as follows: when accessing a page, MODX tries to find the resource id by the requested URL in the cache. (URL is already cleared, that is, without any parameters, etc.). If found, then returns the ID and then everything happens in normal mode. If not, it tries to find a document in the database by uri. Finds - writes id to cache and then returns id. If not, then the standard procedure OnPageNotFound (so that you can still use your plugin to modify the search).

This additional class will be supplied with shopModx, and if someone needs it (if there is a big store), then simply turn it on in the settings (modRequest.class key).

There is also an option to drive all the pages into the cache at once, for example, when updating the cache (use a plugin for the OnSiteRefresh event).

Problem 3. Many cache files

I can imagine how many have read the previous decision, and thought, “Well, what a moron!” :-)

Yes, to produce hundreds of thousands of cache files is a complete insanity. But here is the key word - files. Yes, it is their state (files) that do not give us peace. Therefore, in this case we simply use another cache provider, not a file one. I decided to use memcached, since I had already come across it and installed it on the server, and you can use another one you like. The standard assembly of Revo also comes with memcache and APC.

I argued my choice in favor of the cache mechanism on RAM by simplifying the reset of the cache. Try deleting 1,000,000 files from your hard drive. This will be sooooo long. In the case of memcached, resetting the cache is simple and fast.

 $modx->cacheManager->getCacheProvider()->flush(); 

Another huge plus of memcached is that you can store any type of data, including objects. The only exception is resources (for example, a connection to a database) and objects, among the properties of which are resources. Such objects should be created with the __sleep () and __wakeup () methods, so that before saving they remove all properties-resources, and when they are restored from the cache, they can re-create these properties.

So see the results. First set

 Memory: 15.0709 Mb TotalTime: 0.1040 s 

Further

 Memory: 10.403 Mb TotalTime: 0.0640 s 

In my opinion, it’s very good for a non-cached context for 13,000+ documents.

Problem 4. Mass update of documents when changing system settings

I will not explain why, but it took me to change the container suffix. Changed, and the Ajax-response did not wait ... It was useful to look at the processor / system / settings / updatefromgrid. There is such a method in it checkForRefreshURIs (). In general, if "friendly_urls", "use_alias_path" or "container_suffix" were changed, then it signals that the URLs need to be updated. All right But the problem is that he is trying to update all documents indiscriminately, not even containers. In addition, the sorting condition by menuindex also adds for some reason (although we are interested in the order of nesting, and not the menu index).
In general, this process made the server cry. Added condition isfolder = 1, and then in 6 seconds updated all containers. I won't change suffixes anymore :-)

Summary

In practice, we received full processing of a document on a site with 13000+ documents (in two tables) and 43000+ TV-NIS, in less than 0.3 seconds with an updated cache. From cache - in less than 0.1 sec.
Conventionally, we can assume that at this stage the difference between a large and a small site ends, since further brakes are possible only at the page rendering level, and this already depends on how we write the templates, etc.
I will sign this moment in the next article (most likely tomorrow). But I’ll say at once what I’m going to do on Smarty, as IMHO to do all this on clean chunks and snippets - a lot of problems.

And finally, the results of a local test of 100 customers with 1000 requests each: gist.github.com/Fi1osof/462e1af10ab7b95311df
Time per request: 44.224 [ms] (mean, across all concurrent requests)

PS Package on modx.com: modx.com/extras/package/shopmodx
GitHub Project: github.com/Fi1osof/shopModx
Filled the latest version with the request-class.

PPS It is better to specify the settings of the memcached-provider directly in config.core.php (just take a word).

 $config_options = array ( 'cache_handler' => 'cache.xPDOMemCached', 'cache_prefix' => 'shopmodx_', //          memcached- ); 

The variable $ config_options is already there.