Modx and "restriction" in 5000 documents

Introduction

Modx is a great framework, but on the resources and in the sections devoted to modx, you can read posts about a certain restriction of the 5000 document framework, and customers sometimes ask if the site will work if there are more than 5,000 pages.
You probably already guessed, it will be a question of modx evolution (version 1.0.5).

When there is a task to make the site more business cards, the question arises: how many pages can serve cmf / cms and how fast?

Modx is famous for its flexibility, and for virtually any task you can come up with several solutions, but the bottleneck is caching, specifically we are interested in the assets / cache / siteCache.idx.php file, which contains absolutely everything that can be cached (except for the pages themselves which has its own cache file of the form assets / cache / docid_ <page ID> .pageCache.php).
Bypassing the slight inconvenience that may arise (if you make a portal and store everything as modx documents) in a large site with the current modx caching concept in several ways, which are a little lower.
')

What's wrong with caching

Everything is so with him, but there is one thing - when the cache is cleared, the main cache file siteCache.idx.php should be re-created.

Updating the cache is triggered by events such as any changes to the TV parameter (s), chunks, snippets, patterns, changes in documents (added a new resource or removed from the publication of any other - you need to refresh the array of document identifiers), as a result, if there is a site directory with several hundred documents, more than half of which have a dozen TV parameters — the size of siteCache.idx.php can be several MB and increase, while to form a new cache file you need to run about a dozen sql queries, if not more, this file b The file will be successfully rewritten as long as the allocated memory (memory_limit) is enough for both the operation of generating the main cache file and other requests to form the current page (which we are currently on), otherwise we get an error: Fatal error: Allowed memory size of ... bytes exhausted (tried to allocate ... bytes) in ..assets/cache/siteCache.idx.php on line ...

Common mistakes in large sites

Under a large site we will submit a resource of several thousand documents, the number of which is constantly increasing.

If the site is not properly designed, the main load will fall on the file system, and specifically on the siteCache.idx.php record, consider one of the possible options.

For example, a blog on modx

Suppose there is a blog on modx, the content of which is “generated” by users, each posts his own articles, which are written to the database as a modx document and displayed in the document tree, which is quite logical and even convenient, since there is no need to do additional work: a separate table, controllers for managing data on behalf of the user and the admin (this will take several hours to work, depending on the complexity), and the tools for editing the site by the admin / manager from the fron-end are quite out of the box.

When you add a new document in modx, you need to clear the entire cache, so at a frequency of 2-3 documents per minute the site can slow down, as the number of documents grows, the brakes increase, over time the site does not have enough allocated memory and the server will generate an error that starts: “Allow memory size of ... ".
You can increase the size of the allocated memory, but you can’t increase it indefinitely (we won’t say why now), sooner or later we will have to revise the concept of our blog and rewrite most of the work.

Modx testing

Not to be unsubstantiated, I made a small test at home, simulating the example of a “modx blog”.

Purpose of testing

Check personally that modx can work with more than 5,000 documents, establish under what conditions modx can stop working.

Instruments

Note: CPU Core i3 2.13Ghz, RAM 8Gb DDR3, HDD 500gb 5400 rpm.
Software: Windows 7 x64, Apache 2.2.12, php 5.3.0, mysql 5.1.36
Server settings: normal xampp web server, normal settings, set the memory_limit value to 64Mb.

Work plan

1. install modx evolution 1.0.5
2. create: 30 copies of templates, 50 chunks, 30 snippets, 100 TV parameters, 5.000 documents
3. write a script that partially makes point 2, we execute point 2.
4. if the admin does not work - do clause 6, if it works - do clause 5
5. increases the number of documents by another 5,000, see item 4
6. increase the memory_limit to 128 mb, if there are no errors - see point 5.
7. if we got to this point, it means that we got the error “Allow memory size of ...”, we draw conclusions.

Working process

Established ( start and finish ) framework, along with all chunks and snippets, which are enabled by default.

At the testing stage, it is not essential what data exactly will contain templates, chunks, snippets, TV parameters, documents. We will constantly copy the same element.
For resource copy operations a small script was written:

1. source code
2. copy to assets / snippets / test / test.php
3. run index-ajax.php? q = assets / snippets / test / test.php

In the admin, made a copy of the first document (id = 1), set the document id = 1 as a child for the copied (id = 2), and prohibited displaying the document in the menu (when there are many documents, you do not need to see all of them on the site), the script will copy the created document (id = 2) and all clones will have the first document (id = 1) as the parent.

Copying resources

The initial size of siteCache.idx.php is 161Kb.

Templates, the donor will be “MODxHost” - about the same place is occupied by any other templates in most projects, made 30 copies, also copied 100 TV parameters, 50 chunks (donor was WebLoginSideBar), 30 snippets (donor - AjaxSearch, a lot of code contains what we need).

After these simple operations, the size of siteCache.idx.php increased to 1.370MB, the site and admin works, we create many documents (remember, the script duplicates the document with id = 2, also insert some content for id = 2, for example, from somewhere, I copied hence the piece).

Test results in the table

An attempt to create another 5k documents with memory_limit = 128Mb and the presence of 60k documents was unsuccessful .

When testing, in the admin panel, documents in the document tree stopped loading, although the admin panel was working, this was because all documents were copied to one “parent” (id = 1), and there was not enough allocated memory for selection and output of the necessary child documents.

When creating a cache file, siteCache.idx.php, it is worth considering from which page the main cache file is created, since snippets and plugins that are used on a specific page also consume the memory allocated for PHP, therefore the maximum number of documents depends on how well snippets are written and how requests are built.

The test is over, with memory_limit 64MB and 128MB approximately 15k and 60k, respectively, can be serviced in modx, this is clearly more than the stated limit of 5k documents, of course this is not a standard, but there is an indication that modx, in principle, can work with a large number of documents .

Organization site structure

To avoid the problem of insufficient memory allocation, I can advise principles that should be followed when creating a site or portal on modx:

• do not write a lot of child documents for one parent, try to logically distribute documents into groups;
• dynamic content for huge directories should be organized in a separate table, write the necessary snippets for it, if you need CNC for the data in the tables - you can write a plugin or a snippet, and hang it on page 404, which will produce content if there is data in our table;
• avoid SELECT * FROM <TABLE> queries, you only need to select what we need, no more, to break complex queries into several simple ones;
• not to abuse, for example, Ditto calls, in practice I met with a dozen uncached Ditto calls on one page, which terribly slowed down the site, for a specific task it is better to write a specific snippet that doesn’t do anything extra.

Of course, if the estimated number of pages on the site is several thousand, other tasks arise before the developer, and if modx is used, it is only for static content and as a bridge for data in other tables.

Conclusion

As you can see, modx can work with a large number of documents, evolution can be used without problems to create large portals and online stores, although the approach should be wise. Modx is just a tool; the quality of the created resource depends on how the developer will use these tools.
The article does not mention the modx revolution, how best to make snippets (as a separate file, or store code in the database) and much more, you can write a lot about modx.

I am pleased to read comments or criticism on the article and code.
Especially for Habr .