GitHub, website and automatic test site creation from the latest source code version

This article will talk about how to automatically get the latest version of the source code from the main branch of your repository and deploy from it a project on a virtual hosting. Just want to note that I met GitHub and Git only yesterday. Therefore, this article may seem trivial to an experienced web programmer. And I hope that will help those who are just starting their way as a web programmer.

Introduction

I have a small website on a virtual hosting. It does not have full shell access and scripts are limited in some rights. For example, I cannot use the PHP system function and the file_get_contents function. After I created the repository on GitHub, learned to work a bit with the changes and updated the source code, it was time to think about what to do next. I wanted to see my changes in action, but at the same time so that the main site continued to work.

Of the scripting languages ​​available to me, I only know PHP. The choice on what to write was made automatically. I understood that my script should somehow receive update notifications from GitHub and download the source code. I decided to make a subdomain of development.mysite.com and post the latest version of the source code there. In addition, I have a website configuration file with passwords for the database, which I did not share on GitHub. This file must be added to the downloaded sources for it to work.
')
Thus, the whole process can be divided into the following steps:

• figure out how to get notifications from GitHub;
• download source codes;
• make the necessary changes with them.

Notifications from GitHub

Everything is quite simple here. GitHub supports hooks. We register the address of our script in the Post-Receive URLs and that's it. It will be called at each change of the main repository branch. Read more about this on the GitHub site (in English). In this case, in my script, I do not process information about the last transaction (commit).

Download source

The developer has two options for downloading the code:

• use GitHub API;
• Download an archived version of the main branch.

Github API

Using the program interfaces you can get information about the last commit. It contains the Tree SHA ID. This identifier allows you to consistently get a list and contents of all project files.

An example of using the GitHub API for PHP is described in the blog of David Volsh . Take some useful features from there and add your own. Let's start writing our script. First of all the parameters

1. <?php /* static settings */ $user = '<github_username>' ; $repo = '<github_reponame>' ; $user_repo = $user . '/' . $repo; $tree_base_url = "http://github.com/api/v2/json/tree/show/" . $user_repo; // path on the server where your repository will go $stage_dir = $_SERVER[ 'DOCUMENT_ROOT' ] . dirname($_SERVER[ 'SCRIPT_NAME' ]); ?>
2. <?php /* static settings */ $user = '<github_username>' ; $repo = '<github_reponame>' ; $user_repo = $user . '/' . $repo; $tree_base_url = "http://github.com/api/v2/json/tree/show/" . $user_repo; // path on the server where your repository will go $stage_dir = $_SERVER[ 'DOCUMENT_ROOT' ] . dirname($_SERVER[ 'SCRIPT_NAME' ]); ?>
3. <?php /* static settings */ $user = '<github_username>' ; $repo = '<github_reponame>' ; $user_repo = $user . '/' . $repo; $tree_base_url = "http://github.com/api/v2/json/tree/show/" . $user_repo; // path on the server where your repository will go $stage_dir = $_SERVER[ 'DOCUMENT_ROOT' ] . dirname($_SERVER[ 'SCRIPT_NAME' ]); ?>
4. <?php /* static settings */ $user = '<github_username>' ; $repo = '<github_reponame>' ; $user_repo = $user . '/' . $repo; $tree_base_url = "http://github.com/api/v2/json/tree/show/" . $user_repo; // path on the server where your repository will go $stage_dir = $_SERVER[ 'DOCUMENT_ROOT' ] . dirname($_SERVER[ 'SCRIPT_NAME' ]); ?>
5. <?php /* static settings */ $user = '<github_username>' ; $repo = '<github_reponame>' ; $user_repo = $user . '/' . $repo; $tree_base_url = "http://github.com/api/v2/json/tree/show/" . $user_repo; // path on the server where your repository will go $stage_dir = $_SERVER[ 'DOCUMENT_ROOT' ] . dirname($_SERVER[ 'SCRIPT_NAME' ]); ?>
6. <?php /* static settings */ $user = '<github_username>' ; $repo = '<github_reponame>' ; $user_repo = $user . '/' . $repo; $tree_base_url = "http://github.com/api/v2/json/tree/show/" . $user_repo; // path on the server where your repository will go $stage_dir = $_SERVER[ 'DOCUMENT_ROOT' ] . dirname($_SERVER[ 'SCRIPT_NAME' ]); ?>
8. <?php /* static settings */ $user = '<github_username>' ; $repo = '<github_reponame>' ; $user_repo = $user . '/' . $repo; $tree_base_url = "http://github.com/api/v2/json/tree/show/" . $user_repo; // path on the server where your repository will go $stage_dir = $_SERVER[ 'DOCUMENT_ROOT' ] . dirname($_SERVER[ 'SCRIPT_NAME' ]); ?>
9. <?php /* static settings */ $user = '<github_username>' ; $repo = '<github_reponame>' ; $user_repo = $user . '/' . $repo; $tree_base_url = "http://github.com/api/v2/json/tree/show/" . $user_repo; // path on the server where your repository will go $stage_dir = $_SERVER[ 'DOCUMENT_ROOT' ] . dirname($_SERVER[ 'SCRIPT_NAME' ]); ?>
10. <?php /* static settings */ $user = '<github_username>' ; $repo = '<github_reponame>' ; $user_repo = $user . '/' . $repo; $tree_base_url = "http://github.com/api/v2/json/tree/show/" . $user_repo; // path on the server where your repository will go $stage_dir = $_SERVER[ 'DOCUMENT_ROOT' ] . dirname($_SERVER[ 'SCRIPT_NAME' ]); ?>

A copy of the source code will be created in the directory where our script is located. Next, insert the function to retrieve the data at the address, peeped from David:

1. <? php
2. / * gets url * /
3. function get_content_from_github ($ url)
4. {
5. $ ch = curl_init ();
6. curl_setopt ($ ch, CURLOPT_URL, $ url);
7. curl_setopt ($ ch, CURLOPT_RETURNTRANSFER, 1);
8. curl_setopt ($ ch, CURLOPT_CONNECTTIMEOUT, 1);
9. echo "Getting: {$ url}" ;
10. $ content = curl_exec ($ ch);
11. curl_close ($ ch);
12. return $ content;
13. }
14. ?>

Then comes the function that finds the Tree SHA and starts downloading files:

1. <? php
2. function get_repo_json ()
3. {
4. global $ user, $ repo, $ user_repo, $ tree_base_url, $ stage_dir;
6. $ json = array ();
7. $ list_commits_url = 'http://github.com/api/v2/json/commits/list/' . $ user_repo. '/ master' ;
9. echo "Master branch url: {$ list_commits_url} \ n <br>" ;
10. $ json [ 'commit' ] = json_decode (get_content_from_github ($ list_commits_url), true );
12. // get sha for the latest tree
13. $ tree_sha = $ json [ 'commit' ] [ 'commits' ] [0] [ 'tree' ];
14. echo "Tree sha: {$ tree_sha} \ n <br>" ;
15. $ cont_str = $ tree_base_url. "/ {$ tree_sha}" ;
16. $ base = json_decode (get_content_from_github ($ cont_str), true );
18. // output project structure
19. echo "<pre>" ;
20. get_repo ($ base [ 'tree' ], 0, $ stage_dir);
21. echo "</ pre>" ;
22. }
23. ?>

This function calls the get_repo function, which recursively passes through all the directories of the project.

1. <? php
2. function get_repo ($ objects, $ level = 0, $ current_dir)
3. {
4. global $ tree_base_url, $ user_repo;
6. chdir ($ current_dir);
8. foreach ($ objects as & $ object )
9. {
10. $ type = $ object [ 'type' ];
11. $ sha = $ object [ 'sha' ];
12. $ name = $ object [ 'name' ];
14. // add padding
15. echo str_pad ( "" , $ level, "\ t" );
16. echo $ name. "\ n" ;
18. if (strcmp ($ type, "tree" ) == 0)
19. {
20. mkdir ($ name);
22. $ new_dir = $ current_dir. '/' . $ name;
24. $ tree = $ tree_base_url. '/' . $ sha;
25. $ new_objects = json_decode (get_content_from_github ($ tree), true );
27. get_repo ($ new_objects [ 'tree' ], $ level + 1, $ new_dir);
29. // change current directory back
30. chdir ($ current_dir);
31. }
32. else
33. {
34. // get file content
35. $ blob_url = "http://github.com/api/v2/json/blob/show/" . $ user_repo. "/" . $ sha;
36. $ data = get_content_from_github ($ blob_url);
38. $ filename = $ current_dir. '/' . $ name;
39. file_put_contents ($ filename, $ data);
40. }
41. }
42. }
43. ?>

It is worth noting that we immediately receive the contents of the file, without any additional information. Therefore, we do not need to call the json_decode function as in the cases with calls to other API functions.

Results of getting source code via API

This script has two significant drawbacks:

• it works quite slowly;
• I never managed to download the entire project. CURL ends on timeout, the whole process stops without downloading a third of the source files.

In addition, the very idea of ​​pumping a project on individual files seems ideologically incorrect.

Archive with the main branch of the project

Having poked at the GitHub buttons, I discovered the ability to download an archived version of the source files. This approach is much better! You can download a choice of either a Zip archive or Tar. My choice fell on Zip, because it is easier to unpack it on a shared hosting. Let's look at the script.

1. <? php
2. $ download = true ;
3. $ unzip = true ;
4. $ move = true ;
6. $ stage_dir = $ _SERVER [ 'DOCUMENT_ROOT' ]. dirname ($ _ SERVER [ 'SCRIPT_NAME' ]);
7. $ filepath = $ stage_dir. '/' . 'master.zip' ;
9. echo "<pre>" ;
10. ?>

The variables download , unzip and move control the program flow and allow you to disable its parts. They can be used for debugging. For example, if the archive is already downloaded, but not unpacked, then there is no point in downloading it again.

1. <? php
2. if ($ download)
3. {
4. $ url = "http://github.com/<your_github_username>/<your_github_repo_name>/zipball/master" ;
6. $ ch = curl_init ();
7. curl_setopt ($ ch, CURLOPT_URL, $ url);
8. curl_setopt ($ ch, CURLOPT_RETURNTRANSFER, 1);
9. curl_setopt ($ ch, CURLOPT_CONNECTTIMEOUT, 1);
10. echo "Getting: {$ url} \ n" ;
11. $ content = curl_exec ($ ch);
12. echo "Got \" {$ content} \ "\ n" ;
13. curl_close ($ ch);
15. $ dom = new DOMDocument ();
16. @ $ dom-> loadHTML ($ content);
18. $ xpath = new DOMXPath ($ dom);
19. $ hrefs = $ xpath-> evaluate ( "/ html / body // a" );
21. $ href = $ hrefs-> item (0);
22. $ zipurl = $ href-> getAttribute ( 'href' );
23. echo "Zip url: {$ zipurl} \ n" ;
25. $ data = http_get_file ($ zipurl);
26. if (substr ($ data, "http: //" ))
27. {
28. $ data = http_get_file ($ data);
29. }
31. file_put_contents ($ filepath, $ data);
32. }
33. ?>

GitHub makes several redirects that for some reason are not executed with CURL. Therefore, with the help of it we find the address of the first redirect, then we try to go through it. We get another redirect address and finally get to the coveted archive. The http_get_file function used in the code above:

1. <? php
2. function http_get_file ($ url)
3. {
4. $ url_stuff = parse_url ($ url);
5. $ port = isset ($ url_stuff [ 'port' ])? $ url_stuff [ 'port' ]: 80;
7. $ path = $ url_stuff [ 'path' ];
8. $ last = $ path [strlen ($ path) -1];
9. if (strcmp ($ last, "_" ) == 0)
10. {
11. $ path = substr_replace ($ path, "" , -1);
12. }
14. $ fp = fsockopen ($ url_stuff [ 'host' ], $ port);
16. $ query = 'GET' . $ path. "HTTP / 1.0 \ n" ;
17. $ query. = 'Host:' . $ url_stuff [ 'host' ];
18. $ query. = "\ n \ n" ;
20. fwrite ($ fp, $ query);
22. while ($ line = fread ($ fp, 1024))
23. {
24. $ buffer. = $ line;
25. }
26. if (preg_match ( '/ ^ Location: (. +?) $ / m' , $ buffer, $ matches))
27. {
28. return $ matches [1];
29. }
31. preg_match ( '/ Content-Length: ([0-9] +) /' , $ buffer, $ parts);
32. return substr ($ buffer, - $ parts [1]);
33. }
34. ?>

Found a strange functionality. After the parse_url function is executed, the underscore is added to the end of the file name at the last address pointing to the real archive.

Unpack the archive:

1. <? php
2. if ($ unzip)
3. {
4. echo "Uncompressing archive ... \ n" ;
5. $ zip = new ZipArchive;
6. $ res = $ zip-> open ($ filepath);
7. if ($ res === TRUE)
8. {
9. $ zip-> extractTo ($ stage_dir);
10. $ zip-> close ();
11. echo "Done! \ n" ;
12. } else
13. {
14. echo "Failed \ n" ;
15. exit (1);
16. }
17. }
18. ?>

Inside the archive we are waiting for a folder with the name consisting of the user name, the repository name and a piece of SHA code of the commit. And inside this folder are the project files. I have a code folder. The next step is to move the code folder up one level. This is necessary in order to properly display the subdomain. The subdomain is configured, for example, to the / public_html / development folder. The archive is unpacked in / public_html / development / <user> _ <repo> _ <sha> / <files> .

1. <? php
2. if ($ move)
3. {
4. $ files = scandir ($ stage_dir);
5. $ match_array = preg_grep ( '/ <user_name> * /' , $ files);
6. if (is_array ($ match_array))
7. {
8. // remove all directory if any
9. delete_directory ( "code" );
10. $ dir_name = current ($ match_array);
11. $ rep_dir = $ dir_name. "/ code" ;
12. echo "Try to move {$ rep_dir} to code \ n" ;
13. rename ($ rep_dir, "code" );
14. rmdir ($ dir_name);
15. echo "Done moving files \ n" ;
16. }
17. }
19. function delete_directory ($ dirname)
20. {
21. if (is_dir ($ dirname))
22. $ dir_handle = opendir ($ dirname);
23. if (! $ dir_handle)
24. return false ;
25. while ($ file = readdir ($ dir_handle))
26. {
27. if ($ file! = "." && $ file! = ".." )
28. {
29. if (! is_dir ($ dirname. "/" . $ file))
30. unlink ($ dirname. "/" . $ file);
31. else
32. delete_directory ($ dirname. '/' . $ file);
33. }
34. }
35. closedir ($ dir_handle);
36. rmdir ($ dirname);
37. }
38. ?>

And finally, I copy the configuration file, which is located in the / public_html / development / folder

1. <? php
2. copy ( "config.php" , "<new_path> /core.php" );
3. echo "All jobs have been done! \ n" ;
4. echo "</ pre>" ;
5. ?>

* This source code was highlighted with Source Code Highlighter .

Script results downloading archived source code

The script does what you need! =)

Conclusion

This article reviewed approaches to creating a test software environment for a website, the source code of which is stored in the GitHub system. The proposed approaches in the future can be combined. First create a complete copy of the repository by downloading the archive, and then keep track of which files were changed in the last commit and update only them.

The solution I used to create a test software environment may not be ideal. I am very interested to know how other people do it. Share your knowledge!

PS This article was published with the support of dive 'habrauser, who sent me an invite. Thank!