Monitoring the activity of VK groups. We process data on VKScript

Faced the task of monitoring the activity of users of a well-known social network. My task was to collect data on the number of users who are online in a particular group or community.

Instruments

Since I myself am engaged in web development, the tools I used were such

PHP 5 (Zend Framework)
VK API
Cron

I will explain my choice - Vk API - the fact is that you can get the number of users online without an API, and send a search page for users with community filter and tags online, but I chose not to bother with authorization and parsing tags, but to use the program interface.

Architecture

Implementation can be divided into 2 parts. The first is a script that finds the number of users online on the group id and writes it to the database. The second is the admin panel, which allows you to add new groups for monitoring and view statistics on already added groups.
In order for the statistics to be relevant, it is necessary to monitor the state of the group at the current time as often as possible. The script is worth hanging in Cron, let him call us every 5 minutes.

Vk API Overview

If everything is more or less clear with the admin area, then with the statistics collection script it’s not quite. After reviewing the methods provided by the API, I come to the first solution.
')

First decision (incorrect)

Using the groups.getMembers , users.get methods, we get a list of group members and their status - online or offline. Next, count how many users are online. It's simple. However, the apparent simplicity in the result brings a number of problems.
It would be nice if you have small groups (up to 1000 people). Otherwise, we rest on the limitations of the API - only 1000 users can be obtained at a time. That to us this restriction - it is possible to cause a method in a cycle, but no. Making API calls is allowed no more than 3 requests per second.
Calculate the approximate number of requests that will be needed. Take the community habrahabr Vk. It has more than 40,000 users, so we need ~ 40 requests to get community members and 40 requests - their status.
We are going to look for a new solution.

The second decision (true)

We find out in documentation the execute method

A universal method that allows you to run a sequence of other methods, saving and filtering intermediate results.

It accepts as input a line with code written in so-called VKScript (similar to javascript). The only problem is that the imputed documentation on this method and the language itself is missing. Probably a solution has been found, so you can delve into the study of the Vk and VKScript APIs in particular.

Work with API

We download class for work with API, offered by developers. I only brought it to a more acceptable view so that it fits into the coding style used in the Zend Framework.

Class api

<?php class Vkapi_Model_Api { private $_accessToken = null; private $_apiUrl = 'https://api.vk.com/method/'; public function __construct($accessToken) { $this->_accessToken = $accessToken; } public function api($method, $params = array()) { $params['access_token'] = $this->_accessToken; $query = $this->_apiUrl. $method . '?' . $this->_params($params); $responseStr = file_get_contents($query); if(!is_string($responseStr)){ return null; } $responseObj = json_decode($responseStr); return $responseObj; } private function _params($params) { $pice = array(); foreach($params as $k=>$v) { $pice[] = $k.'='.urlencode($v); } return implode('&',$pice); } }

I will not describe authentication and authorization, since it is implemented through OAuth, there is a lot of information in runet, and also on the VK API page.

Perform a test call to the API - get the first 20 posts in the group habrahabr

  public function wallsAction() { //....... $api = new Vkapi_Model_Api($accessToken); $response = $api->api('wall.get',array('owner_id' => '-20629724')); $this->view->walls = $response->response; }

Now we will do the same, only through the execute method

  public function wallsAction() { //....... $api = new Vkapi_Model_Api($accessToken); $code = " var walls = API.wall.get({ owner_id : -20629724 }); return walls; "; $response = $api->api('execute',array('code' => $code )); $this->view->walls = $response->response; }

As a result, we get the same result.
The one bad thing is that we mixed VKScript and PHP code. It looks very bad. Take refactoring.
It would be nice if each script was stored in a separate file and it could be called with one function. It is also necessary to envisage the fact that later we will still need to transfer some data to this script (for example, owner_id, for example, is hard-packed into the code).

We take out VKScript in separate files

In the root of our module, we will create a folder with the name “vkscripts”, into it we will add our scripts (for example getWalls.vks). Let's write the path to the scripts in the application.ini config-file

 vkapi.scripts.path = APPLICATION_PATH "/modules/vkapi/vkscripts"

We need a class that would be convenient for invoking scripts located in this directory. Let's use the features of PHP5, namely the __call magic method. By the name of the called method, we will search for a script with this name.

Class source

 <?php class Vkapi_Model_Executor { private $_api; public function __construct($api) { $this->_api = $api; } public function __call( $methodName, $arguments ) { $script = $this->_getScript($methodName); if(count($arguments)){ $script = $this->_prepareParams($script, $arguments[0]); } $response = $this->_api->api('execute', array('code' => $script)); if( $error = $this->_getError($response) ){ throw new Exception($error->error_msg, $error->error_code); } return $response->response; } private function _getError($response) { if( isset($response->error) ){ $error = $response->error; return $error; } return null; } private function _getScript( $name ) { $scriptsPath = Zend_Registry::get('vkapi_config')->scripts->path; $filePath = $scriptsPath . '/' . $name . '.vks'; if(is_file($filePath)){ $script = file_get_contents($filePath); return $script; } return null; } }

So let's do something with this class.
In the folder vkscripts put the file getWalls.vks with the contents

 var walls = API.wall.get({ owner_id : -20629724 }); return walls;

In the controller:

  public function wallsAction() { //....... $api = new Vkapi_Model_Api($accessToken); $executor = new Vkapi_Model_Executor($api); $response = $executor->getWalls(); $this->view->walls = $response->response; }

We got the same result, only there are significant advantages: we split the code into separate files, made it more readable, simplified the call to execute.
The next step is to add the ability to pass parameters to our script. Let's use for this a certain representation. In the VKScript code, at the beginning, if necessary, we will write something to the input as follows:

 var groupId = %GROUP_ID%; var offset = %OFFSET%; // ....

And in our class, before calling api with this code, we will replace% VAR_NAME% with the variable value.
We add our Executor class as follows

The source code of the modified class

 <source lang="php"> <?php class Vkapi_Model_Executor { // ...... public function __call( $methodName, $arguments ) { $script = $this->_getScript($methodName); if(count($arguments)){ $script = $this->_prepareParams($script, $arguments[0]); } $response = $this->_api->api('execute', array('code' => $script)); if( $error = $this->_getError($response) ){ throw new Exception($error->error_msg, $error->error_code); } return $response->response; } // ...... private function _prepareParams($script, $params) { foreach ($params as $key => $value){ $script = str_replace('%' . strtoupper($key) . '%', $value, $script); } return $script; } }

In the controller, if necessary, the transfer parameter write the following

  public function wallsAction() { //....... $api = new Vkapi_Model_Api($accessToken); $executor = new Vkapi_Model_Executor($api); $response = $executor->getWalls(array( 'group_id' => -20629724, 'offset' => 0 )); $this->view->walls = $response->response; }

That, accordingly, substitute in our script instead of% GROUP_ID% and% OFFSET% passed values.
Here is the structure of the module.

We get the number of users online

There is a restriction on calling API methods in execute . Limit 22 call (found almost). Also, on the web, I did not find information about other operators (for example addition, subtraction), there are also limitations, but they exist. Since if I ran through the array of users and counted the number online, I received an error about the exceeded number of operations, it was decided to return the complete list of users from execute, and then on my server’s side to count their number.
Due to the limitation in the number of requests to the API in the execute method, we still have to execute at least 1 request for 10,000 group members, because 2 requests are required to process 1.000.
Here is the script that turned out

 var groupId = %GROUP_ID%; var offset = %OFFSET%; // API call limit var _acl = 22; var members = API.groups.getMembers({ gid : groupId }); _acl = _acl - 1; var count = members.count; var users = []; while( _acl > 1 && offset < count){ var _members = API.groups.getMembers({ gid : groupId, offset : offset }); _acl = _acl - 1; users = users + API.users.get({ uids : members.users, fields : "online" }); _acl = _acl - 1; offset = offset + 1000; } var result = { count : count, offset : offset, users : users@.online }; return result;

I will comment my code a little. Counter _acl - to prevent errors due to exceeding the limit of operations with the API. users @ .online - return only the list of values [0,1,1,0,0,0,1,0,1] online offline.
In the controller, we invoke this script, successively increasing the offset, until we run through all the group members.

  $count = 1; $offset = 0; $nowOnline = 0; while($count > $offset){ $users = $executor->getOnline(array( 'group_id' => $groupId, 'offset' => $offset )); $count = $users->count; $offset = $users->offset; foreach ( $users->users as $online){ if($online){ $nowOnline++; } } }

So, we’ll test and see that the data received through the API almost coincides with the data from vk.com, perhaps this inaccuracy is due to caches, or for some other reason not visible from the outside.

Remarks

VKScript does not support functions, increment or decrement operators.

Total

We developed tools for working with the vk.com API through the execute method. With the help of it you can develop applications for collecting statistics, etc. and it will look very nice. Screwing the interface to this whole thing is a trivial task. In the end, I note that another social network Facebook provides access to the execution of code written in a language called FQL (Facebook Query Language, similar to SQL), which clearly has more capabilities than VKScript with all its limitations.