How to make Jenkins make your life easier and become happy
When the development of mobile applications in the company goes to industrial rails, the question of automatic assembly invariably pops up. Continuous Integration is an integral part of the process. The result of this process is builds for testing on the company's devices for OTA distribution to customers and remote testers.
When publishing under the account of your company or under the account of the customer you have to deal with a large number of profiles and certificates, keys, properties and settings. It is not always convenient to replace a specific key on the device. Therefore, in our company, we have come up with a solution that automates this process. Automation of the distribution of artifacts on all devices used for assembly, and will be described in this article.
So, this article is about how to implement copying certificates and profiles for iOS assemblies and properties, and keys for Android assemblies to all Jenkins slaves in the framework of Continuous Integration. This automation saves time and helps to avoid mistakes. The article will be of interest to developers of mobile applications and those who administer Jenkins.
')
Go!
A few years ago, when we just set up Jenkins (continuous integration server), we had only one machine (mac-mini) on which all the builds were performed. Over time, Jenkins moved to the virtual machine, and the minik turned into a slave (a secondary Jenkins node used to build projects). iOS projects can only be built on Mac OS, and Android everywhere, so iOS projects were collected on the slave, and Android projects were collected on the wizard (where Jenkins himself is installed). Everything that is needed for the assembly of projects was installed and laid out on two machines with his hands, and everything was ok! We described how to set up Jenkins in one of our previous articles, “Automatically building iOS applications on different versions of Xcode using Jenkins” .
The number of projects grew, and we connected first one builder for iOS and Android projects, and more later. To build applications on freshly connected slaves, you must have the installed tools, the SDK, and for each project the keys and files with which the application is signed.
We decided to optimize the configuration of Jenkins slaves, namely, to make automatic copying and installation of the necessary files for signing iOS and Android applications.
There are several jenkins slaves for building iOS projects. All slaves are labeled with labels that allow to determine the version of Xcode installed on them. For load balancing, all assemblies are labeled with labels, and can be performed on any slave with the appropriate Xcode.
In order to build an iOS application, you must have:
Usually on jenkins we collect Distribution assemblies - AdHoc, Enterprise and AppStore. We just have to collect for a large number of developer accounts, respectively, and we have quite a lot of certificates with keys.
To put a certificate with a key on a remote machine, you need:
To do the same through ssh, you need to use the security command.
To install profiles, simply copy them to the desired directory: ~ / Library / MobileDevice / Provisioning \ Profiles /
In this case, it is desirable to remove the old profile, since When building Xcode, it can use the old profile instead of the updated one.
Usually, to update the profile list, just go to the Xcode settings and, having selected the account you need, click the Reload button. Xcode itself will replace the old profiles with new ones, if they have been edited on the portal. But doing it on several machines under all accounts is not effective and generally dull.
You can set up folder synchronization, for example, with rsync, and update profiles manually with only one machine. We chose a slightly different approach, because several people have access to editing and creating certificates and profiles, and everyone can work both on a virtual machine and on their own machine: we have created a git repository for certificates, keys and profiles.
There is only a little more left - by updating in gita, run a script that will decompose files into jenkins nodes and install certificates and keys. My first thought was to write git-hook, but the guitars do not have access to the jenkins nodes. But Jenkins has one. And jenkins perfectly fulfills job'y, which start when there is a push to git - for this purpose, a web hook is added to the project in the gita.
As a result, we did:
With Android projects, the situation is generally similar. Only instead of profiles and certificates, keystore files are used to sign assemblies. To use the keystore file you need to know the password and alias. For Android builds, we use Gradle and in the settings file gradle.properties we specify the following information:
In a simple situation, you can put the keystore file on the assembly machine and upload it to the project properties git. Difficulties begin when developers and assemblers work on different operating systems: it is difficult to put the keys on all the machines along the same path. Our solution for Android builds:
An assembly that decomposes files into slaves is always performed on the master, since the master has access to all the assemblers. In order to specify the correct paths for different operating systems, files are put into git with the placeholder “PATH_TO_KEYS”, i.e. In our case, the file looks like this:
The necessary paths in the files are substituted during the execution of the assembly using the sed command. To use files downloaded from the repository several times, before changing paths, they are copied into a separate folder. In our example, on the slaves, the home directory is the same, but different from the home directory of the wizard.
In the project settings for the release assembly, the path to gradle.properties is relative to the project directory, so the properties file needs to be copied from the common directory of the collector ~ / gradle_keys to the working directory.
In order not to write out the addresses of all slaves, you can use the Elastic + Axis plugin, which allows you to customize the assembly to be performed on several nodes. The script for setting up iOS collectors will look like this:
The script for setting up Android collectors will look like this:
They say that if you want to write a good article, you need to write conclusions about the strengths and weaknesses of your decision. With strengths everything is simple here:
Weak sides:
The proposed approach allowed our company to simplify the procedure for managing the numerous artifacts needed to build applications. Now changing and adding keys is fast, stable and on all the right machines. With the expansion of infrastructure due to the increase in the number of projects, adding new nodes for jenkins is just as fast and easy. We are happy to share our experience and would be happy if this article helps to solve a similar problem in those companies where a large number of developed and supported applications have accumulated.
When publishing under the account of your company or under the account of the customer you have to deal with a large number of profiles and certificates, keys, properties and settings. It is not always convenient to replace a specific key on the device. Therefore, in our company, we have come up with a solution that automates this process. Automation of the distribution of artifacts on all devices used for assembly, and will be described in this article.
So, this article is about how to implement copying certificates and profiles for iOS assemblies and properties, and keys for Android assemblies to all Jenkins slaves in the framework of Continuous Integration. This automation saves time and helps to avoid mistakes. The article will be of interest to developers of mobile applications and those who administer Jenkins.
')
Go!
Problem
A few years ago, when we just set up Jenkins (continuous integration server), we had only one machine (mac-mini) on which all the builds were performed. Over time, Jenkins moved to the virtual machine, and the minik turned into a slave (a secondary Jenkins node used to build projects). iOS projects can only be built on Mac OS, and Android everywhere, so iOS projects were collected on the slave, and Android projects were collected on the wizard (where Jenkins himself is installed). Everything that is needed for the assembly of projects was installed and laid out on two machines with his hands, and everything was ok! We described how to set up Jenkins in one of our previous articles, “Automatically building iOS applications on different versions of Xcode using Jenkins” .
The number of projects grew, and we connected first one builder for iOS and Android projects, and more later. To build applications on freshly connected slaves, you must have the installed tools, the SDK, and for each project the keys and files with which the application is signed.
We decided to optimize the configuration of Jenkins slaves, namely, to make automatic copying and installation of the necessary files for signing iOS and Android applications.
Let's start with iOS
There are several jenkins slaves for building iOS projects. All slaves are labeled with labels that allow to determine the version of Xcode installed on them. For load balancing, all assemblies are labeled with labels, and can be performed on any slave with the appropriate Xcode.
In order to build an iOS application, you must have:
- Xcode
- Developer / Distribution certificate + Private key
- Provisioning profile
Usually on jenkins we collect Distribution assemblies - AdHoc, Enterprise and AppStore. We just have to collect for a large number of developer accounts, respectively, and we have quite a lot of certificates with keys.
To put a certificate with a key on a remote machine, you need:
- export the certificate with the key on the machine on which they are installed
- copy the .p12 file to the remote machine
- go to the remote machine and open the .p12 file, enter the password to install the certificate and key in keychain,
- After installing the certificate and key, open the key information and in the Access Control tab select “Allow all applications to access this item”
To do the same through ssh, you need to use the security command.
To install profiles, simply copy them to the desired directory: ~ / Library / MobileDevice / Provisioning \ Profiles /
In this case, it is desirable to remove the old profile, since When building Xcode, it can use the old profile instead of the updated one.
Usually, to update the profile list, just go to the Xcode settings and, having selected the account you need, click the Reload button. Xcode itself will replace the old profiles with new ones, if they have been edited on the portal. But doing it on several machines under all accounts is not effective and generally dull.
You can set up folder synchronization, for example, with rsync, and update profiles manually with only one machine. We chose a slightly different approach, because several people have access to editing and creating certificates and profiles, and everyone can work both on a virtual machine and on their own machine: we have created a git repository for certificates, keys and profiles.
There is only a little more left - by updating in gita, run a script that will decompose files into jenkins nodes and install certificates and keys. My first thought was to write git-hook, but the guitars do not have access to the jenkins nodes. But Jenkins has one. And jenkins perfectly fulfills job'y, which start when there is a push to git - for this purpose, a web hook is added to the project in the gita.
As a result, we did:
- git repository, in which in the profiles and certificates folders are * .mobileprovision and * .p12
- job in jenkins, which takes files from the gita and executes the script:
#!/bin/bash slaves=( "192.168.2.10" "192.168.2.20" "192.168.2.30" ) for i in "${slaves[@]}" do ssh jenkins@"$i" rm -f "~/Library/MobileDevice/Provisioning\ Profiles/*" scp ${WORKSPACE}/profiles/* "jenkins@$i:/Users/jenkins/Library/MobileDevice/Provisioning\ Profiles/" ssh jenkins@"$i" mkdir -p "~/pp_cc" scp ${WORKSPACE}/certificates/* "jenkins@$i:/Users/jenkins/pp_cc/" ssh jenkins@"$i" 'security unlock-keychain -p jenkins /Users/jenkins/Library/Keychains/login.keychain; cd /Users/jenkins/pp_cc/; for a in *; do security import $a -k /Users/jenkins/Library/Keychains/login.keychain -P xxxXXXxxx -A; done' done
slaves - is an array of jenkins, and more precisely their ip-addresses. xxxXXXxxx - password from p12. Yes, we use the same password for all certificates that we put in git.
And the same magic for Android
With Android projects, the situation is generally similar. Only instead of profiles and certificates, keystore files are used to sign assemblies. To use the keystore file you need to know the password and alias. For Android builds, we use Gradle and in the settings file gradle.properties we specify the following information:
releaseStoreFile=/home/jenkins/gradle_keys/customer/project_name.keystore releaseStorePassword=project_pass releaseKeyAlias=project_alias releaseKeyPassword=project_pass In a simple situation, you can put the keystore file on the assembly machine and upload it to the project properties git. Difficulties begin when developers and assemblers work on different operating systems: it is difficult to put the keys on all the machines along the same path. Our solution for Android builds:
- git repository for keystore and properties
- build on jenkins, which executes the script:
# slaves cp -r keys keys_for_ubuntu cd keys_for_ubuntu PATHREPL='\/home\/jenkins\/gradle_keys' find . -name "*\.properties" | while read line; do sed -i -e 's/'PATH_TO_KEYS'/'"$PATHREPL"'/g' "$line"; done slaves=( "192.168.2.10" "192.168.2.20" "192.168.2.30" ) for i in "${slaves[@]}" do scp -r ./ jenkins@"$i":/home/jenkins/gradle_keys/ done #master cd ../keys PATHREPL='\/var\/lib\/jenkins\/gradle_keys' find . -name "*\.properties" | while read line; do sed -i -e 's/'PATH_TO_KEYS'/'"$PATHREPL"'/g' "$line"; done cp -r ./ /var/lib/jenkins/gradle_keys/
An assembly that decomposes files into slaves is always performed on the master, since the master has access to all the assemblers. In order to specify the correct paths for different operating systems, files are put into git with the placeholder “PATH_TO_KEYS”, i.e. In our case, the file looks like this:
releaseStoreFile=PATH_TO_KEYS/customer/project_name.keystore releaseStorePassword=project_pass releaseKeyAlias=project_alias releaseKeyPassword=project_pass The necessary paths in the files are substituted during the execution of the assembly using the sed command. To use files downloaded from the repository several times, before changing paths, they are copied into a separate folder. In our example, on the slaves, the home directory is the same, but different from the home directory of the wizard.
In the project settings for the release assembly, the path to gradle.properties is relative to the project directory, so the properties file needs to be copied from the common directory of the collector ~ / gradle_keys to the working directory.
cp ~/gradle_keys/customer/project_name.properties ${WORKSPACE}/ How to do better
In order not to write out the addresses of all slaves, you can use the Elastic + Axis plugin, which allows you to customize the assembly to be performed on several nodes. The script for setting up iOS collectors will look like this:
rm -f ~/Library/MobileDevice/Provisioning\ Profiles/* cp ${WORKSPACE}/profiles/* ~/Library/MobileDevice/Provisioning\ Profiles/ security unlock-keychain -p jenkins $HOME/Library/Keychains/login.keychain; cd ${WORKSPACE}/certificates/; for a in *; do security import $a -k $HOME/Library/Keychains/login.keychain -P xxxXXXxxx -A; done The script for setting up Android collectors will look like this:
cd keys KEYS_PATH="$HOME/gradle_keys" REPL_PATH=$(echo "$KEYS_PATH" | sed -e 's/[\/&]/\\&/g') find . -name "*\.properties" | while read line; do sed -i -e "s/PATH_TO_KEYS/$REPL_PATH/g" "$line"; done cp -r ./ $KEYS_PATH findings
They say that if you want to write a good article, you need to write conclusions about the strengths and weaknesses of your decision. With strengths everything is simple here:
- this solution saves time when modifying and adding files for assemblies
- this solution saves time on adding slaves while expanding the infrastructure
- This solution allows you to restrict access to important files (keys and certificates are not in the repository of the project) - we haven't had them before, but they say that some people do it
- this solution is cool because jenkins "tunes" himself
Weak sides:
- This solution is designed for the fact that one repository is used for keys from all Android projects and similarly with iOS. Accordingly, the setting of access to it is inflexible - the person either has access to everything, or there is no access to anything. If access to such files should have not a chief technical specialist in the direction, but, for example, a project manager, this solution will not work.
- The security of this approach largely depends on the security of the implementation of the idea of ​​release assemblies on jenkins in general, that is, if you use them, you still need to store keys somewhere, prescribe paths to them, etc., and security depends on settings for accessing jenkins tasks, to the machines on which jenkins and its collectors are installed, on the way the distributed applications are distributed.
Conclusion
The proposed approach allowed our company to simplify the procedure for managing the numerous artifacts needed to build applications. Now changing and adding keys is fast, stable and on all the right machines. With the expansion of infrastructure due to the increase in the number of projects, adding new nodes for jenkins is just as fast and easy. We are happy to share our experience and would be happy if this article helps to solve a similar problem in those companies where a large number of developed and supported applications have accumulated.
Source: https://habr.com/ru/post/262527/
All Articles