Deploy OpenSource Puppet 4 with multiple puppet masters. Part I. Preparatory
→ Deploying OpenSource Puppet 4 with multiple Puppet masters. Part II. Puppet Masters Setup
→ Deploying OpenSource Puppet 4 with multiple Puppet masters. Part III. Setting up puppet-db with Puppet
My experience using puppet. Before writing this article, I worked with Open Source Puppet version 3 in a stand alone configuration, and used it to manage several hundred hosts. But the time has come to grow: the number of managed hosts has gone beyond a thousand, and threatens to pass several thousand in the near future. It was decided to distribute the load and increase resiliency to deploy Open Source Puppet version 4 with multiple Puppet Master servers and a separate PuppetDB server with postgresql. And also use a git repository on a git server for storing environments with configurations of end hosts.
At first, I would like to offer a brief overview of existing articles on habrahabr.
')
Setting up a modern Puppet server from scratch
The translation of the article “Setup of modern Puppet of the server from scratch” made by grundic , the original of which I managed to find only in the Google cache. This article was taken by me as a basis for the preparation of the publication. The details described in the original article “Setup of the server from scratch”, as well as the additions of the translator in its translation, have already become a little out of date. This, as well as the desire to share a description of additional subtleties, prompted me to write my own article.
Deployment Option for Linux Systems Based on Puppet 4. Part III: Installing Puppet Server (cfpuppetserver)
The article is part of a cycle where the author andvgal describes the deployment of an entire infrastructure. In the article, the author proposes to automate the installation of puppet using the cfpuppetserver package developed by the author of the article. Very interesting, but difficult for the novice puppeteer.
Puppet under load
An interesting article from the company Badoo , containing a large number of schemes that reveal the mechanisms of Puppet 3.
How to become a puppeteer or Puppet for beginners
The article starts well, gives basic concepts about Puppet. At the end of the article, the author Aecktann promises to continue, which, unfortunately, has not appeared in 4 years.
Puppet, configuration management system. Article in two parts: Part I , Part II
The author of spanasik gives basic concepts about Puppet, and also describes the installation of stand alone puppet master without other components.
General information about Puppet. Puppet agents are installed on managed nodes, which are called nodes. At certain time intervals (the default is 30 minutes), puppet agents connect to Puppet servers, transmit data about the nodes (facts) on which they are installed; receive configuration descriptions for managed nodes, as well as the necessary data and tools for configuring configurations from servers.
Configurations for managed nodes are stored on the puppet servers as text files. Such files are called manifests. Manifestos unite in environments. Each puppet environment contains its own set of manifests. The default environment is called production. For testing, a developmement environment is usually used. Each managed node can only be in one environment at a time.
Environments containing text files with puppet manifests are usually stored in a git repository that can be hosted on a git server. Each environment will correspond to a branch (branch) in the git-repository.
To synchronize the environments on each puppet server, the R10k robot is used, which is launched using a post-receive hook in the git repository. Read more about this robot in its creator's blog .
The data transferred by puppet agents to puppet servers can be stored locally on servers as text files, or in the PuppetDB database. PuppetDB allows you to use the advanced features of puppet, and can also be used in third-party applications.
To extend the basic capabilities of Puppet, various modules are used that can be downloaded from Puppet Forge , GitHub, or another source, and also create them yourself.
puppetmaster.example.com is the name of the cluster.
puppet-master01.example.com - Puppet Master node 1, also the Certificate Authority Service will work on it.
puppet-master02.example.com - Puppet Master node 2.
Accordingly, you can continue puppet-masterN.example.com - Puppet Master node N. All settings made for puppet-master02.example.com are suitable for subsequent nodes.
puppet-db.example.com - PuppetDB server, postgresql server.
sgl-git.example.com is a git server for storing a git repository that contains environments with puppet manifests.
Ubuntu 16.04 OS is installed on all servers.
aspetrenko is an admin user with permissions to execute sudo (present on all servers).
gitolite3 is a user on whose behalf the gitolite3 git server is running (on the sgl-git server).
r10k is a user on whose behalf the r10k robot is running (on puppet-master servers).
Options for configuring DNS servers are described in the corresponding Puppet section of the documentation . In my case, the approach described in the section on Round-Robin DNS is used . Although it does not matter.
aspetrenko, aspetrenko.pub - private and public administrator key.
gitolite3, gitolite3.pub - user keys gitolite3. The public key of gitolite3.pub needs to be placed in the list of authorized keys (~ / ssh / authorized_keys) of the user r10k on each puppet-master server so that the user gitolite3 can log in and start updating the environments.
r10k, r10k.pub - user r10k keys. The public key r10k.pub must be placed in the gitolite3 administrative repository (the keydir directory in the gitolite-admin repository) so that the r10k robot can log in to the git server and pick up the environments from the git repository to update the environments on the local file system. Keys can be created on any computer with ssh-keygen. Keys must be created without password protection so that authorization can occur without human intervention. Create administrator keys, and copy the public key to sgl-git:
It will play the role of the gitolite3 admin key. Create a user for r10k on puppet-master01 and puppet-master02 servers:
And give the user r10k a password:
The password will be more convenient during the installation process. Then the password entry can be disabled, and leave authorization only by key.
Create the user r10k keys on puppet-master01, and copy the private key to the puppet-master02 server:
During the installation of the gitolite3 git server, a user with the same name gitolite3 will be automatically created. You will need to add this user's public key to the list of trusted keys of user r10k on puppet-master01 and puppet-master02 servers, so that gitolite3 can use post-receive hook in the puppet-environments repository to launch R10k robot.
More information about the automation with the use of keys can be found, for example, here . As a result, it should turn out as in the diagram below.
More information about gitolite can be read in the official documentation or on the habr .
Install git and perl:
Run the gitolite installation from the Ubuntu repository:
During the installation, we specify the absolute path to the public key of the administrator aspetrenko.pub. If you make a mistake, you can specify the path to the administrator key as follows from the system user gitolite3:
The key will be saved in the gitolite-admin administrative repository in the keydir directory as admin.pub.
Gitolite3 stores repositories in its home directory / var / lib / gitolite3 / repositories /. If you want to store repositories in another place, like me, you can transfer them as follows:
Add to the /etc/gitolite3/gitolite.rc file in the "% RC = ("
And move the repositories to a new location:
You cannot directly edit git repositories in the / var / lib / gitolite3 / repositories directory. On the aspetrenko-pc computer, from where we will work with repositories on our server, we specify the ssh authorization settings for sgl-git in the ~ / .ssh / config file:
where aspetrenko is the private admin key. We clone the administrative repository into our home directory:
If the server asks for a password, then access using ssh keys was configured incorrectly. As a result, we get:
Repository access settings are located in gitolite-admin / conf / gitolite.conf
admin is the user whose key was specified when setting up gitolite3:
Put the public key of user r10k in the gitolite3 administrative repository:
Add the r10k.pub key to git:
Create a repository for storing puppet environments. To do this, add the following lines to gitolite-admin / conf / gitolite.conf on aspetrenko-pc:
Fix the changes in git:
And send it all to sgl-git:
Gitolite3 using hooks will create a new empty sgl-git repository:
Do not forget to create keys for the user gitolite3, and add the public key to the list of trusted keys of the user r10k on each server puppet-master01 and puppet-master02:
The private key gitolite3 should appear in the /var/lib/gitolite3/.ssh directory.
Deploy OpenSource Puppet 4 with multiple puppet masters. Part II. Puppet Masters Setup
→ Deploying OpenSource Puppet 4 with multiple Puppet masters. Part III. Setting up puppet-db with Puppet
Foreword
My experience using puppet. Before writing this article, I worked with Open Source Puppet version 3 in a stand alone configuration, and used it to manage several hundred hosts. But the time has come to grow: the number of managed hosts has gone beyond a thousand, and threatens to pass several thousand in the near future. It was decided to distribute the load and increase resiliency to deploy Open Source Puppet version 4 with multiple Puppet Master servers and a separate PuppetDB server with postgresql. And also use a git repository on a git server for storing environments with configurations of end hosts.
Overview of articles on habrahabr on puppet deployment
At first, I would like to offer a brief overview of existing articles on habrahabr.
')
Setting up a modern Puppet server from scratch
The translation of the article “Setup of modern Puppet of the server from scratch” made by grundic , the original of which I managed to find only in the Google cache. This article was taken by me as a basis for the preparation of the publication. The details described in the original article “Setup of the server from scratch”, as well as the additions of the translator in its translation, have already become a little out of date. This, as well as the desire to share a description of additional subtleties, prompted me to write my own article.
Deployment Option for Linux Systems Based on Puppet 4. Part III: Installing Puppet Server (cfpuppetserver)
The article is part of a cycle where the author andvgal describes the deployment of an entire infrastructure. In the article, the author proposes to automate the installation of puppet using the cfpuppetserver package developed by the author of the article. Very interesting, but difficult for the novice puppeteer.
Puppet under load
An interesting article from the company Badoo , containing a large number of schemes that reveal the mechanisms of Puppet 3.
How to become a puppeteer or Puppet for beginners
The article starts well, gives basic concepts about Puppet. At the end of the article, the author Aecktann promises to continue, which, unfortunately, has not appeared in 4 years.
Puppet, configuration management system. Article in two parts: Part I , Part II
The author of spanasik gives basic concepts about Puppet, and also describes the installation of stand alone puppet master without other components.
Introduction
General information about Puppet. Puppet agents are installed on managed nodes, which are called nodes. At certain time intervals (the default is 30 minutes), puppet agents connect to Puppet servers, transmit data about the nodes (facts) on which they are installed; receive configuration descriptions for managed nodes, as well as the necessary data and tools for configuring configurations from servers.
Configurations for managed nodes are stored on the puppet servers as text files. Such files are called manifests. Manifestos unite in environments. Each puppet environment contains its own set of manifests. The default environment is called production. For testing, a developmement environment is usually used. Each managed node can only be in one environment at a time.
Environments containing text files with puppet manifests are usually stored in a git repository that can be hosted on a git server. Each environment will correspond to a branch (branch) in the git-repository.
To synchronize the environments on each puppet server, the R10k robot is used, which is launched using a post-receive hook in the git repository. Read more about this robot in its creator's blog .
The data transferred by puppet agents to puppet servers can be stored locally on servers as text files, or in the PuppetDB database. PuppetDB allows you to use the advanced features of puppet, and can also be used in third-party applications.
To extend the basic capabilities of Puppet, various modules are used that can be downloaded from Puppet Forge , GitHub, or another source, and also create them yourself.
Server names
puppetmaster.example.com is the name of the cluster.
puppet-master01.example.com - Puppet Master node 1, also the Certificate Authority Service will work on it.
puppet-master02.example.com - Puppet Master node 2.
Accordingly, you can continue puppet-masterN.example.com - Puppet Master node N. All settings made for puppet-master02.example.com are suitable for subsequent nodes.
puppet-db.example.com - PuppetDB server, postgresql server.
sgl-git.example.com is a git server for storing a git repository that contains environments with puppet manifests.
About OS on servers
Ubuntu 16.04 OS is installed on all servers.
Users
aspetrenko is an admin user with permissions to execute sudo (present on all servers).
gitolite3 is a user on whose behalf the gitolite3 git server is running (on the sgl-git server).
r10k is a user on whose behalf the r10k robot is running (on puppet-master servers).
DNS setup
Options for configuring DNS servers are described in the corresponding Puppet section of the documentation . In my case, the approach described in the section on Round-Robin DNS is used . Although it does not matter.
Ssh authorization using keys
aspetrenko, aspetrenko.pub - private and public administrator key.
gitolite3, gitolite3.pub - user keys gitolite3. The public key of gitolite3.pub needs to be placed in the list of authorized keys (~ / ssh / authorized_keys) of the user r10k on each puppet-master server so that the user gitolite3 can log in and start updating the environments.
r10k, r10k.pub - user r10k keys. The public key r10k.pub must be placed in the gitolite3 administrative repository (the keydir directory in the gitolite-admin repository) so that the r10k robot can log in to the git server and pick up the environments from the git repository to update the environments on the local file system. Keys can be created on any computer with ssh-keygen. Keys must be created without password protection so that authorization can occur without human intervention. Create administrator keys, and copy the public key to sgl-git:
aspetrenko@aspetrenko-pc:~$ ssh-keygen -t rsa -f ~/.ssh/aspetrenko aspetrenko@aspetrenko-pc:~$ scp ~/.ssh/aspetrenko.pub aspetrenko@sgl-git:/home/aspetrenko/.ssh/ It will play the role of the gitolite3 admin key. Create a user for r10k on puppet-master01 and puppet-master02 servers:
sudo useradd -m -s /bin/bash r10k And give the user r10k a password:
sudo passwd r10k The password will be more convenient during the installation process. Then the password entry can be disabled, and leave authorization only by key.
Create the user r10k keys on puppet-master01, and copy the private key to the puppet-master02 server:
aspetrenko@puppet-master01:~$ sudo su - r10k r10k@puppet-master01:~$ ssh-keygen -t rsa -f ~/.ssh/r10k r10k@puppet-master01:~$ scp /home/r10k/.ssh/r10k r10k@puppet-master02:/home/r10k/.ssh/ During the installation of the gitolite3 git server, a user with the same name gitolite3 will be automatically created. You will need to add this user's public key to the list of trusted keys of user r10k on puppet-master01 and puppet-master02 servers, so that gitolite3 can use post-receive hook in the puppet-environments repository to launch R10k robot.
More information about the automation with the use of keys can be found, for example, here . As a result, it should turn out as in the diagram below.
Installing and configuring the git-server
More information about gitolite can be read in the official documentation or on the habr .
Install gitolite3 on sgl-git
Install git and perl:
aspetrenko@sgl-git:~$ sudo apt install git perl Run the gitolite installation from the Ubuntu repository:
aspetrenko@sgl-git:~$ sudo apt install gitolite3 During the installation, we specify the absolute path to the public key of the administrator aspetrenko.pub. If you make a mistake, you can specify the path to the administrator key as follows from the system user gitolite3:
aspetrenko@sgl-git:~$ sudo su - gitolite3 gitolite3@sgl-git:~$ $HOME/bin/gitolite setup -pk /home/aspetrenko/.ssh/aspetrenko.pub The key will be saved in the gitolite-admin administrative repository in the keydir directory as admin.pub.
Gitolite3 stores repositories in its home directory / var / lib / gitolite3 / repositories /. If you want to store repositories in another place, like me, you can transfer them as follows:
Add to the /etc/gitolite3/gitolite.rc file in the "% RC = ("
# Custom path for repos GL_REPO_BASE => "/media/data/repositories", And move the repositories to a new location:
sudo mv /var/lib/gitolite3/repositories /media/data/ Configure gitolite3
You cannot directly edit git repositories in the / var / lib / gitolite3 / repositories directory. On the aspetrenko-pc computer, from where we will work with repositories on our server, we specify the ssh authorization settings for sgl-git in the ~ / .ssh / config file:
host sgl-git.example.com HostName sgl-git.example.com IdentityFile ~/.ssh/aspetrenko User gitolite3 where aspetrenko is the private admin key. We clone the administrative repository into our home directory:
aspetrenko@aspetrenko-pc:~$ mkdir sgl-git aspetrenko@aspetrenko-pc:~$ cd sgl-git aspetrenko@aspetrenko-pc:~/sgl-git$ git clone gitolite3@sgl-git.example.com:gitolite-admin If the server asks for a password, then access using ssh keys was configured incorrectly. As a result, we get:
aspetrenko@aspetrenko-pc:~/sgl-git/gitolite-admin$ tree . ├── conf │ └── gitolite.conf └── keydir └── admin.pub 2 directories, 2 files Repository access settings are located in gitolite-admin / conf / gitolite.conf
aspetrenko@aspetrenko-pc:~/sgl-git/gitolite-admin$ cat conf/gitolite.conf repo gitolite-admin RW+ = admin repo testing RW+ = @all admin is the user whose key was specified when setting up gitolite3:
aspetrenko@aspetrenko-pc:~/sgl-git$ cat gitolite-admin/keydir/admin.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIFC/2v1S4WvvITHAXCuVle7dLZz0zL7z4dAWXvmMcqCLepfGI1suDBby6PW04tVwHvniAW5B/5HbZ2Fr7zCoeMrhGE5Z76/DBfidhO15CZbAMPOcs3X7aP4aZFK2GfiXCt7/yunP6f3zp3i6KbsZhcSeeUmmkeQFwccJsstVJbj9ciWHrLN/UDHwT5OTBVFKBpRZGM5pT6xzvWaPNN4IRlk5AN4ClrhWf13 aspetrenko@aspetrenko-pc Put the public key of user r10k in the gitolite3 administrative repository:
r10k@puppet-master01:~$ scp /home/r10k/.ssh/r10k.pub aspetrenko@aspetrenko-pc:/home/aspetrenko/sgl-git/gitolite-admin/keydir Add the r10k.pub key to git:
aspetrenko@aspetrenko-pc:~/sgl-git/gitolite-admin$ git add keydir/r10k.pub Create a repository for storing puppet environments. To do this, add the following lines to gitolite-admin / conf / gitolite.conf on aspetrenko-pc:
repo puppet-environments RW+ = admin R = r10k Fix the changes in git:
aspetrenko@aspetrenko-pc:~/sgl-git/gitolite-admin$ git commit -a -m "Add puppet-environments repo" [master 585326d] Add puppet-environments repo 1 file changed, 3 insertions(+) And send it all to sgl-git:
aspetrenko@aspetrenko-pc:~/sgl-git/gitolite-admin$ git push Counting objects: 7, done. Delta compression using up to 2 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (4/4), 411 bytes | 0 bytes/s, done. Total 4 (delta 0), reused 0 (delta 0) remote: Initialized empty Git repository in /media/data/repositories/puppet-environments.git/ To gitolite3@sgl-git:gitolite-admin 7946dea..585326d master -> master Gitolite3 using hooks will create a new empty sgl-git repository:
aspetrenko@sgl-git:~$ sudo ls /media/data/repositories/puppet-environments.git branches config gl-conf HEAD hooks info objects refs Do not forget to create keys for the user gitolite3, and add the public key to the list of trusted keys of the user r10k on each server puppet-master01 and puppet-master02:
aspetrenko@sgl-git:~$ sudo su - gitolite3 gitolite3@sgl-git:~$ ssh-keygen -t rsa -f ~/.ssh/gitolite3 gitolite3@sgl-git:~$ ssh-copy-id -i ~/.ssh/gitolite3.pub r10k@puppet-master01 gitolite3@sgl-git:~$ ssh-copy-id -i ~/.ssh/gitolite3.pub r10k@puppet-master02 The private key gitolite3 should appear in the /var/lib/gitolite3/.ssh directory.
Deploy OpenSource Puppet 4 with multiple puppet masters. Part II. Puppet Masters Setup
Source: https://habr.com/ru/post/316400/
All Articles