Testing unloading 200,000 products on the site of 1C

In one of the projects we were faced with the task of unloading a large number of goods from 1C to the site 2 times a day. We want to share the experience gained during the load testing of the project.

We studied several articles on Habré:
http://habrahabr.ru/post/77593/
http://habrahabr.ru/company/ontico/blog/95060/
http://habrahabr.ru/post/113914/

At first three issues had to be resolved:

• What to test?
• How to monitor the results?
• How to test?

Let's start in order.
')

What to test?

Our stand consists of three main elements:

• 1C server (Intel @ Core i3-2125, 8 GB RAM)
• PHP module for processing data from the 1C platform to the site (Back-end REST Service)
• web site interface

The main functionality of the system is the exchange of xml messages between 1C and the data processing module (REST) ​​located on the application server. The processing module receives data from 1C and writes it to the database. The web-interface of the site works with the database.

How to monitor the results?

Zabbix was chosen as the monitoring tool - the most affordable system, which is easy to install and use. Zabbix allowed us not only to monitor the testing process in real-time mode, but also to keep statistics, based on which we can predict the load and calculate the value of RPO (Recovery Point Objective) and RTO (Recovery Time Objective) indicators, as well as system availability ( Agreed Service Time ).

How to test?

Since the system can be divided into two main parts, we will try to identify possible flaws in them. The first is the interaction of the 1C gateway and the XML handler on the web. The second is user interaction with the database through a web interface.
So let's get started:

1. Testing exchange 1C-Site
   To begin, let us take into account that the “narrowest” place of exchange can be detected when you first unload all the objects necessary for the system to work from 1C base to the web. Since there was no 1C base with a sufficient number of objects at hand, we wrote a script to create an extended list of nomenclature and counterparties in 1C with preservation of all necessary connections. As a result of the script, 200 thousand objects appeared in the database.
   
   Paged objects had the following structure:
   

   <> <>8729</> <>2012-12-17T10:46:15</> <>c9358c28-f33f-11e1-b5e6-00155d002500</> <>0</> </> <></> <>e8a71fc3-55bc-11d9-848a-00112f43529a</> <>0</> <></> </> <>e61d7694-cffa-11e1-99dc-0030678e0763</> </> 

   
   
   There are about 32 such structures in total.
   
   So, setting up the exchange and checking everything again, we start a full download from 1C to the web and look at the graphs.
   
   
   
   
   
   Results: The initial unloading of goods lasted at about 9 o'clock. The bottleneck when unloading a large amount of data from 1C is the performance of the 1C gateway, which is directly related to server capacity and platform limitations. They also revealed the problem of exchange, which is to be resolved in the shortest possible time - the absence of unloading from 23:00 to 02:00.
   
   The next stage of testing will be the creation of an ideal stand with unlimited 1C server capacities and a “wide” communication channel. Testing has shown that at this stage it is necessary to pay particular attention to optimizing the 1C code and test the two-way exchange in different load modes. In the near future, a comprehensive study of this issue is scheduled, following which the 2nd article will be written.
   
   
2. Testing the web interface
   
   The purpose of this test is to count the number of simultaneously working users that the system can efficiently maintain.
   To begin, we define the concept of "effective." We assume that “efficiency” is achieved in the case when the loading time of an arbitrary page is no more than one second.
   The ping time to our test server does not exceed 55ms.
   
   For load testing, they decided to use such systems as Apache Benchmark and Tsung.
   
   Apache benchmark
   100 users, 10 of whom simultaneously make a request using the search engine site.
   
   

    ab -n 100 -c 10 -u putdata.txt -e search.csv _ 

   
   
   Putdata.txt - contains words that are entered into the search form. The result writes a search.csv file.
   
   
   Result: 98% of requests were processed within 1 second.
   
   Tsung
   
   
   1. Create a script using the tsung-record utility. Tsung-record records client actions through a proxy server - Tsung Proxy. It remains to register the proxy server address in the browser -127.0.0.1 and port 8090, which will listen to requests.
      
      
      
      And in the terminal execute the tsung-record – star command
      
      
   2. Open the browser and perform the usual user actions:
      ● Browse categories of goods;
      ● We carry out the order of goods;
      ● We use the search;
      ● Add a new product to the order;
      
      Having finished working in the browser, we stop the tsung-recorder with the stop command.
      
      
   3. As a result, a file was created containing the Get requests:
      
      

       <session name='rec20121127-1403' probability='100' type='ts_http'> <request><http url='http:///' version='1.0' method='GET'></http></request> <request><http url='/thumbs/92_width_80.jpg' version='1.0' method='GET'></http></request> <request><http url='/thumbs/93_width_80.jpg' version='1.0' method='GET'></http></request> <request><http url='/thumbs/97_width_80.jpg' version='1.0' method='GET'></http></request> <request><http url='/cron/session/' version='1.0' method='GET'></http></request> </session> 

      
      
   4. Create a test script. Tsung allows you to create several scenarios, as well as set client IP addresses in case there is a load balancer.
      
      

       <?xml version="1.0"?> <!DOCTYPE tsung SYSTEM "/usr/share/tsung/tsung-1.0.dtd"> <tsung loglevel="notice" version="1.0"> <clients> <client host="localhost" maxusers="300" use_controller_vm="true"/> </clients></b> // servers     ,  ,   tsung  571- fatal: {failed_validation,{no_xml_element,server}}   . <b> <servers> <server host="192.168.1.1" port="80" type="tcp"></server> </servers> <load> <arrivalphase phase="1" duration="1" unit="minute"> <users arrivalrate="10" unit="second"/> </arrivalphase> </load> //     (  ). <session name='rec20121127-1403' probability='100' type='ts_http'> <request><http url='http:///' version='1.0' method='GET'></http></request> <request><http url='/thumbs/92_width_80.jpg' version='1.0' method='GET'></http></request> <request><http url='/thumbs/93_width_80.jpg' version='1.0' method='GET'></http></request> <request><http url='/thumbs/97_width_80.jpg' version='1.0' method='GET'></http></request> <request><http url='/cron/session/' version='1.0' method='GET'></http></request> </session> 

      
      
      
   5. Every second 10 new users executed the script:
      
      After performing load testing, you can generate a log report using the tsung-report command.
      
      Charts in Zabbix (time from 14:30 - 14:50):
      
      
      
      
      
      
      Result: The system works stably with 300 users simultaneously working with its interface. At the same time the formation of the page for each user does not exceed 1 second.
      
      The results obtained at this stage of development, in our opinion, are quite acceptable for systems of this kind. However, we identified a number of important problems that need to be solved and once again became convinced of the need to improve the system.