Invitation to a public discussion of the CMS usability rating

When choosing a CMS, the developer, as a rule, takes into account many of its parameters, such as cost, number of modules, popularity of the system and, of course, the most important parameter - its knowledge of this CMS. However, one very important characteristic falls out of consideration: how convenient it will be for the site’s customer to use it - add news, correct typos, etc.

This issue is particularly acute when developing small corporate or private sites. In many small companies there are no dedicated specialists for working with the site, and often there is not even a full-time “computer scientist”. And the need to add or correct something on such a site turns into a big problem - the site owner is “strangled by a toad” to pay “inadequate” money for small work, and the developer has no desire to do it for free and uninteresting small things.

Periodically doing research on various CMS interfaces, I thought that it would probably be interesting and useful to do something like a convenience rating of different systems. And to focus this rating not on the developers, but on the site owners. He shared this idea with several web studios and found support and a noticeable interest in the topic. Appealed to CMS magazine - and also met with mutual understanding. Now the question is submitted to a broad discussion of all interested parties.

To avoid suspicion of order and jeans , the discussion of all aspects of testing will be as open as possible, we are ready to consider any suggestions and arguments. The results of all tasks will be open, including videos of all sessions. The testing itself will be conducted using eye-tracking eye-tracking technology, which today is probably the most qualitative tool for the study of the usability of sites and interfaces.
')
Of course, we understand that it is impossible to grasp the immensity. Some CMS are convenient for the store, while others are suitable only for a blog; some work well on small sites, but “fall down” under load or when a couple or three hundred pages appear; some have gigantic functionality, but pay for it with complexity, etc. etc.

In order to somehow offset these differences, we propose the following (all points are negotiable, comments and suggestions are welcome):

1. The ranking will test the usability of a CMS for a small corporate site: a dozen pages, company news, a couple of blogs. If the idea turns out to be interesting, then next time it will be possible to test, for example, shops ...
2. We study the convenience of site management . Moreover, the customer will manage this site. Not having any experience with sites. Those. the situation is modeled when the studio made the site, handed it over to the customer, and then he himself has to do something with this site.
3. Tasks that will be solved during testing are recruited during the discussion - send your options. They should be:
   • Elementary. For example, “add news”, “correct a typo on the page”, “change the contact phone number”, “change the size and font color of the header”, “insert a picture on the page”, etc.
   • Have a clear performance indicator. The task is either completed or not.
   • Be related to reality, i.e. Tasks should be the same plan that face the real owners of the sites.
4. For each task, indicators such as the number of respondents who have completed the task will be recorded; lead time; number of errors during execution; as well as data on the movement of the respondents gaze.
5. The team of each CMS makes a website on it and “sharpens” it for the performance of tasks that will be solved during testing. But it “sharpens” with regular means: if a quick access panel is provided in the CMS, then it is possible to place a call for the corresponding modules there; But to write custom scripts that display the button "change the title" to the site is impossible.
6. We invite all CMS developers to participate, as well as free CMS support teams. Also invited studios who are ready to play for some kind of CMS (in the event that the developer refuses to put his team). Send comments about what CMS would like to see in this rating.
7. As respondents will be recruited people who know about the Internet and have seen a computer, but who do not have experience writing sites.
8. Financial questions. Unfortunately, conducting such a large-scale study is rather expensive. So sponsors and advertisers are wanted. If you are interested in participating, write to eyetracker.ru@gmail.com and cmsmagazine@gmail.com

We hope that the results of such a study will be interesting not only as a rating, but also as an assessment of the “usability” of the solutions used in the interfaces of various CMS. In parallel with the rating, usability reports will be prepared for each of the participating CMS, which may be of interest to both their developers and other interface developers.

Ps. Last year, there was already some attempt to objectively compare two popular CMS (albeit on other parameters), which led to a noticeable holivar in the discussion. This time I want to avoid wars, so all constructive suggestions and ideas are accepted.