Syncman - deploy and synchronize projects on a remote server

Before starting your story, ask one small question: How do you deploy your projects to the server?
If you manage one small site on a free virtual hosting, then you do not have synchronization problems - you connected via ftp, copied files and that's it. And if you need to control the deployment of several hundred projects (from small business card sites to loaded applications), on which more than a dozen people work? If qualification allows, then you can use rsync, unison, etc. Especially desperate can simply update the working copy of the project (svn, git, etc.) from the repository. And what will do the dozens of artists who urgently need to fix this little picture? Indeed, for many of them, the console is a magical dark dense forest.
Under the cut description of one solution, which greatly simplifies the process of deploying applications.

The country must know its heroes

The Syncman project already has a rather large history, but, as it seems to me, it has not yet lost its relevance. Thanks pachanga , korchasa , romankitaev , wIliam , dbrain , etc., who could not find on Habré. All these are current and former employees of BIT Creative . Thanks to them, this project appeared and still lives. Working with them, I learned a lot.

Hero of the occasion

Today, under the gun, the wonderful opensource project Syncman .
Syncman sets itself the main goal to facilitate synchronization of remote projects so that even people far from programming (for example, artists, designers, managers, etc.) can independently synchronize projects with remote servers.
Syncman has excellent web and console interfaces. Below are screenshots of the Syncman web interface.
Nothing extra. List of projects, divided into groups. For each project, synchronize buttons, view changes, roll back changes. When choosing a particular project, you can see its current settings. If the project is synchronized, it is highlighted in green. If there were new commits that are not on the remote server, then the color is red.
After setting up the project, its synchronization is reduced to the following sequence of actions:

Changed the project
Made a commit to the repository
I clicked the button “Synchronize project” in the Web-interface Syncman

Over time, you get used to it a lot, for example, I synchronize something like this:

..... svn commit -m "my changes" syncman project_to_sync

All project is synchronized.

What does Syncman do during synchronization?

Synchronization takes place in 4 main stages:

The project is selected or updated from the repository to a temporary folder on the Syncman server, somewhere in the local network.
Run the command presync_cmd, specified in the project settings. At this stage, the layout is prepared for deployment. For example, presync_cmd can delete files that have nothing to do on a remote server: for example, you can delete temporary directories, unnecessary designs, unit test files.
Deploy the project. Normally, this is done using rsync over SSH, but not necessarily. For example, you can use FTP (if you cannot arrange access for rsync).
At the last stage, the postsync_cmd command is executed, specified in the project settings. Usually this command is executed via SSH, but this is completely optional. The task of this command is to remove obsolete files after deployment (for example, some paths, templates, etc. are cached in Limb), besides, migration for the database can be used.

Main characteristics

I have been using this solution for a very long time. There was enough functionality, and after moving Syncman to github, I didn’t follow the changes at all. Actually, these changes and pushed me to write this post. So Syncman:

Open source project;
Written in php;
Based on the Limb3 framework;
Easy to set up;
Flexible in settings. Most often, the standard settings are enough (see below), but through the project configuration file you can override almost all commands to satisfy the most demanding user;
Syncman code is flexible. All the same, this is the son of the project Limb3! Actually about expanding the functionality just below;
Out of the box works with the svn and git repositories;
Out of the box supports synchronization over SSH using rsync and FTP;
Easy to use. You do not need to have specific knowledge to perform synchronization.

How to set up syncman?

We analyze all the points:
1. Pick up files from repository

  cd /path/to/hosts git clone https://github.com/limb-php-framework/limb-app-syncman.git syncman git submodule init git submodule update

2. Add a new virtual host
For example, for example, if you use apache, then the config may be as follows:

Standard config

 <VirtualHost 192.168.0.xxx:80> ServerName syncman DocumentRoot /home/user/syncman/www <Directory /> Options FollowSymLinks AllowOverride All </Directory> <Directory /home/user/syncman/www> Options Indexes FollowSymLinks MultiViews AllowOverride All Order allow,deny allow from all </Directory> ErrorLog /tmp/apache_syncman_error.log LogLevel warn CustomLog /tmp/apache_syncman_access.log combined </VirtualHost>

If the web interface is not planned to be used, then this item can be skipped.

3. Setting up projects
3.1. Brief example
If you follow the description above, the downloaded files should be in the folder / path / to / hosts / syncman. Then all projects should be stored in the directory / path / to / hosts / syncman / projects.
A separate folder is created for each project. The name of this folder is the name of the project. In the project folder is a single file settings.conf.php with the project settings.

Configuration File Example1

 $conf = array( //    ,      'server' => array( 'host' => 'myhost.com', 'user' => 'syncman', 'password' => 'qwerty', 'remote_dir' => '/var/www/myhost.com', ), // ,    git 'repository' => array( //-- allowed types: git, svn 'type' => 'git', 'path' => 'myrepos/myhost.com/', 'branch' => 'project_branch', ), //  //-- allowed types: rsync, ftp 'type_sync' => 'ftp', //,      Syncman 'presync_cmd' => 'php %local_dir%/cli/pre_sync.php', //       ,     'postsync_cmd' => 'ssh -i %key% %user%@%host% \'php %remote_dir%/cli/post_sync.php\'', //  ftp   'history' => false, //  'category' => 'MyCategory', );

Sample configuration file2

 $conf = array( // ,   SSH 'server' => array( 'host' => 'myhost.com', 'user' => 'syncman', 'port' => 22, 'key' => '/home/syncman/.ssh/id_dsa', 'remote_dir' => '/var/www/myhost.com', ), // ,    Subversion 'repository' => array( //-- allowed types: git, svn 'type' => 'svn', 'path' => 'myrepos/myhost.com/trunk', ), // rsync  SSH //-- allowed types: rsync, ftp 'type_sync' => 'rsync', //.   'presync_cmd' => 'php %local_dir%/cli/pre_sync.php', 'postsync_cmd' => 'ssh -i %key% %user%@%host% \'php %remote_dir%/cli/post_sync.php\'', //   'history' => true, //  'category' => 'MyCategory', //   . //!!! ,       'ssh_get_date' => "date +%F_%R", 'ssh_mkdir' => "mkdir -p \$dir", 'ssh_ln_edit' => "rm -f \$ln_path; ln -s \$new_dir \$ln_path;", 'ssh_cp' => "cp -pRT \$dir_of/ \$dir_in/", //        &> /dev/null 'ssh_ls' => "ls -F --classify -1 \$dir", 'ssh_preg_dir' => "/(.)+\//", 'ssh_readlink' => "readlink -v \$link", );

3.2. A detailed example. Step-by-step instruction
If, after reading example 3.1. It's still not clear how to set up a project, then ...

detailed instructions

Let us analyze the second example in more detail:

We go to the Syncman server via SSH (We assume that Syncman is not located on your computer);
Go to the Syncman directory, if the projects directory is not created, then create it;
Go to the projects directory, and create a project folder myhost.com;
Go to the folder myhost.com and with the help of your favorite text editor create the file settings.conf.php (content as in example 2);
We assume that synchronization will be performed over SSH (as experience shows, this is the most frequently used option).
If required, create a key pair for SSH using ssh-keygen. We copy the public key to the remote server of the myhost.com project, authorize it. The path to the secret part is prescribed in the key section.
When writing a config, you can change any settings. Moreover, if you specify% name% when specifying the setting, then the corresponding value will be substituted for this template (see, for example, the postsync_cmd command setting).

Next, create the script cli / pre_sync.php in the repository of your project.

A typical example of pre_sync.php for Limb applications

 echo "Pre syncing...\n"; //   $remove_dirs = array($all_shared = dirname(__FILE__) . "/../www/shared"); $remove_dirs[] = dirname(__FILE__) . "/../_docs"; //  foreach($remove_dirs as $dir) `rm -rf $dir`; //     mkdir($all_shared); foreach(glob(dirname(__FILE__) . "/../lib/limb/*/shared") as $pkg_shared) { echo "Moving $pkg_shared..\n"; $pkg = basename(dirname($pkg_shared)); rename($pkg_shared, "$all_shared/$pkg"); } echo "done.\n";

Next, create a script cli / post_sync.php in the repository of your project.

Typical post_sync.php example for Limb applications

 $dir = dirname(__FILE__); echo "Post syncing...\n"; include('migrate.php');//  `rm -rf $dir/../var/compiled`; `rm -rf $dir/../var/locale`; `rm -rf $dir/../var/locators`; `rm -f $dir/../var/db_info*`; echo "done.\n";

Both of the latest files should be added to the repository and synchronized with the project.
The whole project is set up.

Naturally, points 1-2 are performed only once. Point 3 is performed 1 time for each project.
4. Deploy and sync
Using the web interface
To synchronize a project, you need to open the web address where Syncman is located in any browser. If everything was done correctly, you should see your myhost.com project in the project list.
Projects

Click on the “Syncronize” button. A new window should open in which the sync log will be displayed.
Sync

If you want to see the changes, click on the "show changes" button. A new window should open in which the sync log will be displayed.
Diff

From console
Above was mentioned the console interface. All scripts to perform synchronization from the console are in the bin folder of the Syncman project.
The sync.php script is an old version of the synchronization script. Usage example:

  sync.php <project>[,<project>,<project>...]  php sync.php <project>[,<project>,<project>...]

In addition, there is a new version of this script - syncman.php - with more features. The work is based on the taskman package.

Extending Syncman Behavior

In Syncman, everything revolves around the Project class, so to understand how the application works, you should deal with this class. At first glance it may seem that everything is complicated. But it only seems. Project inherits from the Limb framework class - lmbOblect. It is with lmbOblect and you have to start studying Project, then there will be no difficulty.
To add support for new version control systems, you need to write a class to support the corresponding version control system by analogy with the SvnRepository and GitRepository classes. They can be found in the src / model / directory. After your class is ready, you need to make changes to the RepositoryFactory class (located in the src / factory directory).
To add support for new types of transport during synchronization, you need to write the corresponding class by analogy with the RSyncProjectSync and FtpProjectSync classes. They can be found in the src / model / directory. After your class is ready, you need to make changes to the ProjectSyncFactory class (located in the src / factory directory).

Conclusion

For me, web programming is just a hobby, and I don’t want to spend time manually synchronizing my sites. This helps me Syncman. And believe it, using it has saved me more than a dozen hours of my personal precious time.
This application was born in the company BIT Creative. With it, they synchronize hundreds of projects. Imagine how much time they save.
Hope I'm interested in my reader. And thus aroused his interest in Syncman and Limb.