Sphinx indexing from a remote server using PHP

Good day, dear readers!

I want to tell you about an interesting problem that has come to me in the framework of the project and, of course, about its solution.

Initial data:
Standard set LAMP (further SS)
Yii framework (the version is not important here),
remote server (hereinafter referred to as CSS) on which the Sphinx daemon is installed, searchd.
On the user created user with root rights (but not the root itself).
The ssh2_mod module for PHP is installed on the CC.
')
Immediately make a reservation, in this article I will not describe the features of Sphinx, who are interested, can read the official manual sphinxsearch.com/docs/current.html .
I will confine myself only to general information.

So, Sphinx is a search daemon, in my case working with MySQL. The main feature is that it indexes the database for certain queries (described in the Sphinx config), and the result of the sample is saved to its files. For the information to be relevant (in MySQL it is possible to add and edit records), you need to start sphynx indexing. Then, he will re-sample and save it himself.

Task:
Run sphinx indexing on CSS.
The reason for the remote launch is that it is necessary to run cron commands with specific parameters defined in the code. Crowns start with SS.
Those. The server runs cron, the method of which performs indexing on the CSS.

The only solution that I found is using ssh2_mod for apache2 (if you are interested, you can check the installation manual on CentOS here www.stableit.ru/2010/12/ssh2-php-centos-55-pecl.html ).

I looked at the ssh2 manual (http://www.php.net/manual/en/book.ssh2.php), I found the remarkable ssh2_exec function, which accepts the current session and command as input, but as it turned out, it has a number of limitations.
For example, when trying to execute the indexer --all --rotate command for a delta index, I received an error:

WARNING: failed to open pid_file '/var/run/sphinx/searchd.pid'. WARNING: indices NOT rotated. 

This error means that my user does not have enough rights to perform rotate (and I have a user with root rights, sudo -s), although from the console directly I quietly executed this command without any errors.
Then I decided to search for more, and found that it is possible to emulate command input through the terminal (ssh2_shell function). Using the standard stream and the fwrite function, you can write commands to the “terminal” and receive the same standard output stream, i.e. the result issued by the terminal. Occurs by progressive reading from the output stream using fgets.

Everything is good, the check of the delta index was successful, I was delighted, but ...
“BUT” happened when I tried to index the main index (about 400k records, it takes a few minutes). It turned out that the output stream stops at the slightest delay in the execution of the command in the terminal. In simple language, when you enter a command, and the terminal "thinks". As a result, I still had “under-indexed” files.

I decided to google how people solve problems, ran into a piece of code, directly in ssh2 mana on php.net. The author of the solution suggested setting the start and end markers of the command (echo '[start]'; $ command; echo '[end]') and setting the max_execution_time for the script.
The code is below.

 $ip = 'ip_address'; $user = 'username'; $pass = 'password'; $connection = ssh2_connect($ip); ssh2_auth_password($connection,$user,$pass); $shell = ssh2_shell($connection,"bash"); //Trick is in the start and end echos which can be executed in both *nix and windows systems. //Do add 'cmd /C' to the start of $cmd if on a windows system. $cmd = "echo '[start]';your commands here;echo '[end]'"; $output = user_exec($shell,$cmd); fclose($shell); function user_exec($shell,$cmd) { fwrite($shell,$cmd . "\n"); $output = ""; $start = false; $start_time = time(); $max_time = 2; //time in seconds while(((time()-$start_time) < $max_time)) { $line = fgets($shell); if(!strstr($line,$cmd)) { if(preg_match('/\[start\]/',$line)) { $start = true; }elseif(preg_match('/\[end\]/',$line)) { return $output; }elseif($start){ $output[] = $line; } } } } 

It seemed to me a good decision, but ...
Here BUT was in the condition preg_match. When outputting information to $ output, everything written on the output terminal is written. The above problem with the “thinking terminal” again became relevant, since during a pause, a command to output the completion marker echo '[end]' was output to the terminal (the command itself, and not the result of the execution). Everything was decided by adding the start and end of the string limit in preg_match:
preg_match('/^\[start\]\s*$/',$line)

and checking on is_string for $ line.

It remained only to file a file, and, voila, in the project on Yii a component was created, which is a kind of layer for ssh2 functions.

 <?php class SshException extends CException {} /** * Class Ssh * It is a base class for the simplify a ssh connection management * and related commands execution * * @author Ivanenko Vladyslav */ class Ssh { const EXEC_TYPE_EXEC = 'exec'; // type for ssh2_exec() const EXEC_TYPE_SHELL = 'shell'; // type for ssh2_shell() const START_MARK = '__start__'; const FINISH_MARK = '__finish__'; const MAX_EXECUTION_TIME = 1800; // max script execution time in sec private $user; private $password; private $host; private $port; private $shellType = 'bash'; // shell type private $shell = null; //shell identificator private $ssh = null; //connection private $execType; /** * Construct * * @param null $user * @param null $password * @param null $host */ public function __construct($user = null, $password = null, $host = null, $port = null) { $config = Yii::app()->params['ssh']; $params = array('user', 'password', 'host', 'port'); foreach($params as $param) { if(isset(${$param}) && !is_null(${$param})) { $this->{$param} = ${$param}; } else { $this->{$param} = @$config[$param]; } } return true; } /** * Connect to Ssh * * @return resource * @throws SshException */ public function connect() { $this->ssh = @ssh2_connect($this->host, $this->port); if(empty($this->ssh)) { throw new SshException('Cant connect to ssh'); } if(empty($this->execType)) { $this->execType = self::EXEC_TYPE_SHELL; } return $this->ssh; } /** * Login to ssh * * @throws SshException * @return bool */ public function login() { if(!@ssh2_auth_password($this->ssh, $this->user, $this->password)) { throw new SshException('Cant login by ssh'); } return true; } /** * Exec command by ssh * * @param $cmd * @param $type * * @return string * @throws SshException */ public function exec($cmd, $type = self::EXEC_TYPE_SHELL) { if(is_null($this->ssh)) { $this->connect(); $this->login(); } $this->execType = $type; switch($this->execType) { case self::EXEC_TYPE_EXEC: $result = $this->execCommand($cmd); break; case self::EXEC_TYPE_SHELL: $result = $this->execByShell($cmd); break; default: throw new SshException('Incorrect exec type'); break; } return $result; } /** * Executes command by the direct ssh2_exec * * @param $command * * @return string * @throws SshException */ private function execCommand($command) { if (!($stream = ssh2_exec($this->ssh, $command))) { throw new SshException('Ssh command failed'); } stream_set_blocking($stream, true); $data = ""; while ($buf = fread($stream, 4096)) { $data .= $buf; } fclose($stream); return $data; } /** * Executes command within the shell opening * * @param $command * * @return string */ private function execByShell($command) { $this->openShell(); return $this->writeShell($command); } /** * opens shell * * @throws SshException */ private function openShell() { if(is_null($this->shell)) { // here is hardcoded width and height, you can change them. $this->shell = @ssh2_shell($this->ssh, $this->shellType, null, 80, 40, SSH2_TERM_UNIT_CHARS); } if( !$this->shell ) { throw new SshException('SSH shell command failed'); } } /** * * Write the command to the open shell * * @param $cmd * @param int $maxExecTime in sec * * @return string */ private function writeShell($cmd, $maxExecTime = self::MAX_EXECUTION_TIME) { // write start marker fwrite($this->shell, $this->getMarker(self::START_MARK)); // write command fwrite($this->shell, $cmd . PHP_EOL); // write end marker fwrite($this->shell, $this->getMarker(self::FINISH_MARK)); stream_set_blocking($this->shell, true); sleep(1); $output = ""; $start = false; // define the time until the script can be executed $timeUntil = time() + $maxExecTime; while(true) { if(time() > $timeUntil) { break; } $line = fgets($this->shell, 4096); // if any delay is happened while command is processing if(!is_string($line)) { sleep(1); continue; } // define the start executed command if(preg_match('/^' . self::START_MARK . '\s*$/', $line)) { $start = true; } elseif(preg_match('/^' . self::FINISH_MARK . '\s*$/', $line)) { // define the last executed command break; } elseif($start) { // add console output to the script output data $output .= $line; } } return $output; } /** * Disconnect from ssh */ public function disconnect() { $this->exec('exit'); $this->ssh = null; if(!is_null($this->shell)) { fclose($this->shell); } } /** * Disconnect in destruct */ public function __destruct() { $this->disconnect(); } /** * Returns marker command * * @param string $type * * @return string */ private function getMarker($type = self::START_MARK) { return 'echo "' . $type . '"' . PHP_EOL; } } 

P.S. This class can be expanded, because ssh2 is not limited to only two functions for executing commands, there are also functions for working with files, and other types of authorization, etc. etc.

Thank you for your attention, I hope the article will be useful.
I will be glad to hear any feedback and constructive criticism!

Author: Vladislav Ivanenko, PHP Developer Zfort Group