In a section: the news aggregator on Android with backend. Configuration Management System (Puppet)
Introduction (with links to all articles)
In ITIL (v3), among the described processes, there are 2 particularly interesting ones: the “Configuration Management Process” and the “Change Management Process”, intended for analyzing and managing system configuration changes. To continue the narration, you need to decide what the "system" is. This concept includes a huge number of components that affect (directly or indirectly) the provision of services:
Trying to minimize the contour of the system of your project (type, backup does not relate to the functioning of the system) - it means to dig yourself a pit into which you will fall sooner or later.
This list of system components is at least for most projects deployed on more than one server (and even on one). At the same time, their number changes as the project develops and grows. In the absence of any process of their accounting, it becomes obvious what this leads to: you have a test stand (or development stand) on which everything works perfectly, but an attempt to duplicate it on production or create a new stand leads to:
As a result, the environment for the correct operation of your product is a way out of your control and the available automation and version tools.
Things seem obvious, but in most development groups their eyes are closed - many people argue that they have only one server, few settings, competence of workers (the argument “we have professionals and are not going to leave”, simply dumped me), small risks, etc. etc.
Despite the seeming complexity and redundancy of what is described in ITIL (there really is a lot of things written), first of all we need to take the requirements for automation of these processes from it. Automating the assembly, automating the testing, automating the search for a vulnerability is all implemented and is considered by all as a necessary minimum during development. Automation speeds up the process, instills confidence, provides transparency and a guaranteed result, as well as removes fear from the performers for the timing and result.
Taking into account the powerful introductory part in the previous paragraph, the advantages for a single developer will be described more easily and more briefly:
Do not try to take, from the processes described in ITIL, all the steps that are described there - the situation will be worse than before! I know how such steps are implemented in some banking systems and what this leads to (especially without automation) - the bureaucratic machine will stifle any dynamically developing project / system.
In my case, Puppet was chosen. When choosing between Chef and Ansible , the choice in favor of it was made taking into account a good documentation base , good support, a large number of modules (from developers and the community) , active project development and implementation in Ruby (which is more or less familiar to me).
Frankly, the learning curve for Puppet was anything but gentle. Due to the fact that a large number of all sorts of elements are used in the developed system, each of them can be configured differently and all this can be deployed on different stands - the study of the tool required a complete and thorough. As the tool was studied, some of its drawbacks (in most cases, “by-design”) and limitations (a good article on the Puppet philosophy explaining some architectural solutions ) became obvious. Also, as I studied the already and partial implementation of the necessary scripts, I learned a little more about Ansible, in which some problems that exist in Puppet were solved (which will not cancel the possibility of having their own problems, which are absent in Puppet). So, the subsequent story is not an advertisement for Puppet, but a description of the possibilities and experience of use.
Puppet uses its own configuration language (DSL), which was designed to be accessible to system administrators. The Puppet language is positioned as a non-formal programming knowledge and its syntax was influenced by the format of the Nagios configuration files.
The main purpose of the Puppet language is to identify resources (resources in Puppet are files, directories, services, limits, firewalls, users, etc.). All other parts of the language exist only to add flexibility and convenience in how resources are defined.
Resource groups can be organized into classes that are larger configuration units. While a resource can describe a single file / directory or package, a class can describe everything that is needed to configure a service or application (including the required number of packages, configuration files, daemons / services, and maintenance tasks). Smaller classes can be combined into large ones that describe the whole system role - “database server” or “working cluster node”.
An example of a class with resources inside:
Machines that perform different roles should, in general, get a different set of classes. The task of configuring which classes will be applied to which machines is the task of the Puppet nodes.
Examples of determining Puppet nodes:
Hiera Facts and Database. Before executing the Puppet code, information about the node is collected, the collected information is arranged in the form of predefined facts - variables that can be used anywhere in the code. Hiera is a built-in “key-value” database. By default, the YAML or JSON format files are used as the data source, although an extension is possible to use any data source. Due to its hierarchy and the possibility of changing data depending on the node - its use is an integral part of the work of most modules / classes.
Modules are self-contained blocks of code and data (classes, templates, files, etc.). These reusable, publicly accessible elements are the basic building blocks of Puppet.
Those planning to use Puppet will basically have to create classes and sometimes modules.
When implementing Puppet, 2 options are possible with and without centralized configuration storage:
My script looks like this:
In my case, of course, a decentralized system is used, since it is easier to implement (from the point of view of the organization of the infrastructure) and it greatly simplifies the build scripts for deploying a test bench, which I create and run several dozen times a day.
When running the specified script:
The list of tasks that the Puppet client performs at startup is huge (checking and doing it with your hands would be simply impossible):
When developing the Puppet script system for my stand, I developed my own modules that are copied to the machine / container on which the Puppet code was executed. The modules contain data (in most configuration files) and Puppet code for configuration. I carried out the basic settings from the scripts in Hiera - as a result, the scripts turned out to be quite universal and independent of the nodes on which they are executed.
I will give some examples of code and settings.
Setting up ngnix (not from the packages, but from the native repository) (hiding the spoiler in the block due to the size. But for those interested - required to view A lot of the nuances are visible when studying it)
As you can see in each comments before the definition of the resource indicated - "if necessary." Puppet will never perform any operations if the state of the resource already matches its definition.
In this case, you can see how using the code
Hiera configuration file:
In this case, the hierarchy is visible (the “hierarchy key”) source, where each source at a higher level overrides the values of the keys at a lower level. This allows you to have a key, for example, “www.server.port” with a value of “80” in “common.yaml” and with a value of “81” in “nodes / webserver1.yaml” - in the end we will get the value of this key when executing Puppet code: "81" on the host named "webserver1" and "80" on all others.
Hiera's common.yaml
site.pp (Puppet site definition file)
If someone is interested in a specific implementation of any of the tasks: write - write off the implementation in the comments or add to “Tips”.
Puppet is actively developing: there is a clear improvement in the syntax and unification of the behavior of classes depending on the context of use (however, there are still some specific features of the resolution of variables in different contexts, which sometimes leads to confusion).
Thanks for attention!
In ITIL (v3), among the described processes, there are 2 particularly interesting ones: the “Configuration Management Process” and the “Change Management Process”, intended for analyzing and managing system configuration changes. To continue the narration, you need to decide what the "system" is. This concept includes a huge number of components that affect (directly or indirectly) the provision of services:
- servers
')- security settings (users, groups, rights, firewalls);
- installed applications and libraries;
- application settings (limits on descriptors, memory, CPU time, etc.);
- backup;
- monitoring systems for the application and system software;
- configuration files of the product itself, its components, auxiliary system and application applications
- ...
Trying to minimize the contour of the system of your project (type, backup does not relate to the functioning of the system) - it means to dig yourself a pit into which you will fall sooner or later.
This list of system components is at least for most projects deployed on more than one server (and even on one). At the same time, their number changes as the project develops and grows. In the absence of any process of their accounting, it becomes obvious what this leads to: you have a test stand (or development stand) on which everything works perfectly, but an attempt to duplicate it on production or create a new stand leads to:
- a very long process of deployment / duplication (everyone remembers what and how is set up as failures occur and often compares the settings with the existing stand);
- Suddenly, there are problems in the operation of the deployed product (they forget to make any adjustments that they made at the last moment on the test bench, but did not transfer to the product);
- the testing process is slowed down (each publication of a new build, even for internal testers, is a feat in both time and complexity);
- and finally - there is fear among developers before the release of the new version and, as a result, delaying the calculation of the new release (no one will say a real serious problem, but its effect will be felt constantly).
As a result, the environment for the correct operation of your product is a way out of your control and the available automation and version tools.
Motivation
Things seem obvious, but in most development groups their eyes are closed - many people argue that they have only one server, few settings, competence of workers (the argument “we have professionals and are not going to leave”, simply dumped me), small risks, etc. etc.
Despite the seeming complexity and redundancy of what is described in ITIL (there really is a lot of things written), first of all we need to take the requirements for automation of these processes from it. Automating the assembly, automating the testing, automating the search for a vulnerability is all implemented and is considered by all as a necessary minimum during development. Automation speeds up the process, instills confidence, provides transparency and a guaranteed result, as well as removes fear from the performers for the timing and result.
Motivation for a lone developer
Taking into account the powerful introductory part in the previous paragraph, the advantages for a single developer will be described more easily and more briefly:
- because development is not carried out daily, as a result, some features of the deployment disappear smoothly after a couple of days;
- automation for a loner - the most reliable partner;
- The speed of work performed by automation allows us not to lose the ignition and the desire to develop the project (this is a very important specific moment).
Important disclaimer about ITIL
Do not try to take, from the processes described in ITIL, all the steps that are described there - the situation will be worse than before! I know how such steps are implemented in some banking systems and what this leads to (especially without automation) - the bureaucratic machine will stifle any dynamically developing project / system.
Puppet
In my case, Puppet was chosen. When choosing between Chef and Ansible , the choice in favor of it was made taking into account a good documentation base , good support, a large number of modules (from developers and the community) , active project development and implementation in Ruby (which is more or less familiar to me).
Frankly, the learning curve for Puppet was anything but gentle. Due to the fact that a large number of all sorts of elements are used in the developed system, each of them can be configured differently and all this can be deployed on different stands - the study of the tool required a complete and thorough. As the tool was studied, some of its drawbacks (in most cases, “by-design”) and limitations (a good article on the Puppet philosophy explaining some architectural solutions ) became obvious. Also, as I studied the already and partial implementation of the necessary scripts, I learned a little more about Ansible, in which some problems that exist in Puppet were solved (which will not cancel the possibility of having their own problems, which are absent in Puppet). So, the subsequent story is not an advertisement for Puppet, but a description of the possibilities and experience of use.
Little bit about puppet
Puppet uses its own configuration language (DSL), which was designed to be accessible to system administrators. The Puppet language is positioned as a non-formal programming knowledge and its syntax was influenced by the format of the Nagios configuration files.
The main purpose of the Puppet language is to identify resources (resources in Puppet are files, directories, services, limits, firewalls, users, etc.). All other parts of the language exist only to add flexibility and convenience in how resources are defined.
Resource groups can be organized into classes that are larger configuration units. While a resource can describe a single file / directory or package, a class can describe everything that is needed to configure a service or application (including the required number of packages, configuration files, daemons / services, and maintenance tasks). Smaller classes can be combined into large ones that describe the whole system role - “database server” or “working cluster node”.
An example of a class with resources inside:
class apache (String $version = 'latest') { package {'httpd': ensure => $version, # Using the class parameter from above before => File['/etc/httpd.conf'], } file {'/etc/httpd.conf': ensure => file, owner => 'httpd', content => template('apache/httpd.conf.erb'), # Template from a module } service {'httpd': ensure => running, enable => true, subscribe => File['/etc/httpd.conf'], } } Machines that perform different roles should, in general, get a different set of classes. The task of configuring which classes will be applied to which machines is the task of the Puppet nodes.
Examples of determining Puppet nodes:
node 'www1.example.com', 'www2.example.com', 'www3.example.com' { include common include apache, squid } node /^(foo|bar)\.example\.com$/ { include common } Hiera Facts and Database. Before executing the Puppet code, information about the node is collected, the collected information is arranged in the form of predefined facts - variables that can be used anywhere in the code. Hiera is a built-in “key-value” database. By default, the YAML or JSON format files are used as the data source, although an extension is possible to use any data source. Due to its hierarchy and the possibility of changing data depending on the node - its use is an integral part of the work of most modules / classes.
Modules are self-contained blocks of code and data (classes, templates, files, etc.). These reusable, publicly accessible elements are the basic building blocks of Puppet.
Those planning to use Puppet will basically have to create classes and sometimes modules.
Deployment Options
When implementing Puppet, 2 options are possible with and without centralized configuration storage:
- Centralized storage configuration: the advantages are clearly seen in the presence of a large number of servers. In this case, information relating only to the machine itself is transmitted to client machines, which also provides some level of security and minimizes traffic.
- Decentralized configuration storage: justified with a small number of servers, while the machines must have a complete set of configuration scripts and files and when they start up the agents will be compiled and executed on the part related to this machine. It is implemented by the usual cron-task, launched every 15 minutes.
My script looks like this:
#!/bin/sh PUPPET_BIN='/opt/puppetlabs/bin/puppet' # apt-get update && apt-get -y install git mc htop apt-transport-https nano wget lsb-release apt-utils curl python # `puppet-agent` if [ ! -d /etc/puppetlabs ]; then rm *.deb.* *.deb # possible trash wget https://apt.puppetlabs.com/puppetlabs-release-pc1-xenial.deb && dpkg -i puppetlabs-release-pc1-xenial.deb apt-get update && apt-get -y install puppet-agent fi # `environment` /opt/puppetlabs/bin/puppet config set environment $PUPPET_ENV if [ ! -d /etc/puppetlabs/code/environments/$PUPPET_ENV ]; then cp -r /etc/puppetlabs/code/environments/production /etc/puppetlabs/code/environments/$PUPPET_ENV fi # Install puppet modules $PUPPET_BIN module install puppetlabs-ntp $PUPPET_BIN module install aco-oracle_java $PUPPET_BIN module install puppetlabs-firewall $PUPPET_BIN module install saz-ssh $PUPPET_BIN module install saz-sudo $PUPPET_BIN module install saz-limits $PUPPET_BIN module install thias-sysctl $PUPPET_BIN module install yo61-logrotate $PUPPET_BIN module install puppetlabs-apt $PUPPET_BIN module install puppet-archive # git pull "deployment" project and go in it only if POVISION_NO_GIT_CLONE set to "true" if [ ${POVISION_NO_GIT_CLONE:-"false"} = "true" ]; then echo "do nothing" else LOCAL_REV="" if [ -f local_latest.sha1 ]; then LOCAL_REV=`cat local_latest.sha1` fi REMOTE_REV=`git ls-remote --tags | grep "latest" | awk '{print $1}'` if [ $LOCAL_REV = $REMOTE_REV ]; then exit 0 fi git fetch --all --tags --prune git checkout -f tags/latest fi # replace puppet configs cp puppet_config/hiera.yaml /etc/puppetlabs/code/environments/$PUPPET_ENV/ # replace hiera db rm /etc/puppetlabs/code/environments/$PUPPET_ENV/hieradata/* cp -r $PUPPET_ENV/hieradata/* /etc/puppetlabs/code/environments/$PUPPET_ENV/hieradata # replace storyline_* modules rm -r /etc/puppetlabs/code/environments/$PUPPET_ENV/modules/storyline_* cp -r modules/* /etc/puppetlabs/code/environments/$PUPPET_ENV/modules # copy site.pp cp $PUPPET_ENV/site.pp /etc/puppetlabs/code/environments/$PUPPET_ENV/manifests/site.pp #echo "hostname:" #hostname $PUPPET_BIN apply /etc/puppetlabs/code/environments/$PUPPET_ENV/manifests/site.pp echo $REMOTE_REV > local_latest.sha1 In my case, of course, a decentralized system is used, since it is easier to implement (from the point of view of the organization of the infrastructure) and it greatly simplifies the build scripts for deploying a test bench, which I create and run several dozen times a day.
When running the specified script:
- the Puppet client itself and the necessary packages are installed
- the necessary Puppet modules are installed
- checking the change of the commit number for the “latest” label (done with successful integration testing of the new version)
- replaces the configuration (hiera.yaml) of the Hiera Puppet database in the current environment ($ PUPPET_ENV variable);
- replaces the YAML data files for the Hiera Puppet database;
- copied with the replacement of the description of my modules;
- the configuration with nodes (servers of my system) is copied;
- invokes all the settings that were copied / set ($ PUPPET_BIN apply ....)
The list of tasks that the Puppet client performs at startup is huge (checking and doing it with your hands would be simply impossible):
- limits are set for open files, the amount of memory involved, swapping and the number of connections;
- log rotation is configured (both for the system, and for my application and the services it needs);
- create the necessary user accounts for administration with the necessary groups and powers;
- the NTP server is installed and configured;
- SSH server is installed and configured;
- installs Oracle JDK;
- firewall is configured;
- A large number of components necessary for the operation of a project or its component on this particular node are installed and configured.
Life examples
When developing the Puppet script system for my stand, I developed my own modules that are copied to the machine / container on which the Puppet code was executed. The modules contain data (in most configuration files) and Puppet code for configuration. I carried out the basic settings from the scripts in Hiera - as a result, the scripts turned out to be quite universal and independent of the nodes on which they are executed.
I will give some examples of code and settings.
Setting up ngnix (not from the packages, but from the native repository) (hiding the spoiler in the block due to the size. But for those interested - required to view A lot of the nuances are visible when studying it)
The nginx class from the storyline_infra module
class storyline_infra::nginx () { $params = lookup({"name" => "storyline_infra.nginx", "merge" => {"strategy" => "deep"}}) $reverse_port = $params['reverse_port'] $reverse_url = $params['reverse_url'] $pid_file = $params['pid_file'] $init_script = $params['init_script'] $dir_data = $params['dir_data'] $dir_logs = $params['dir_logs'] $version = $params['version'] $enabled_startup = $params['enabled_startup'] $enabled_running = $params['enabled_running'] # topology_configuration $enabled_topology_configuration = $params['enabled_topology_configuration'] $topology_configuration_port = $params['topology_configuration_port'] # ( ) user { 'nginx': ensure => "present", managehome => true, } # ( ) exec { "nginx-mkdir": command => "/bin/mkdir -p /data/db && /bin/mkdir -p /data/logs", cwd => "/", unless => '/usr/bin/test -d /data/db -a -d /data/logs', } -> # working dir file { [ $dir_logs, $dir_data] : ensure => "directory", recurse => "true", owner => "nginx", group=> "nginx", require => Exec['nginx-mkdir'], } # ( ) # see by "gpg --verify keyfile" apt::key { 'nginx-key': id => '573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62', source => 'http://nginx.org/keys/nginx_signing.key', } -> # ( ) # deb http://nginx.org/packages/ubuntu/ xenial nginx apt::source { 'nginx-repo': comment => 'nginx repo', location => "http://nginx.org/packages/ubuntu/", release => "xenial", repos => "nginx", include => { 'deb' => true, 'deb-src' => true, }, } -> # ( ) package { 'nginx': ensure => $version, # notify => Exec['disable_nginx'], } → # ( ). , . !!! file { "/etc/nginx/nginx.conf": replace => true, content => epp('storyline_infra/nginx.epp'), owner => "nginx", group=> "nginx", notify => Service['nginx'], }-> # ( ) file { "/etc/nginx/conf.d/default.conf": replace => true, content => epp('storyline_infra/nginx_default.epp'), owner => "nginx", group=> "nginx", notify => Service['nginx'], }→ # ( ) file { $init_script: replace => true, content => epp('storyline_infra/nginx_startup.epp'), mode=>"ug=rwx,o=r", notify => Service['nginx'], }→ # / ( ) service { 'nginx': ensure => $enabled_running, enable => $enabled_startup, start => "${init_script} start", stop => "${init_script} stop", status => "${init_script} status", restart => "${init_script} restart", hasrestart => true, hasstatus => true, } # nginx ( ) if $enabled_topology_configuration { file { "/etc/nginx/conf.d/topology.conf": replace => true, content => epp('storyline_infra/nginx_topology.epp'), mode=>"ug=rwx,o=r", notify => Service['nginx'], } } # ( ) if $enabled_startup != true { exec { "disable_nginx": require => Package['nginx'], command => "/bin/systemctl disable nginx", cwd => "/", } } } As you can see in each comments before the definition of the resource indicated - "if necessary." Puppet will never perform any operations if the state of the resource already matches its definition.
In this case, you can see how using the code
«$params = lookup({"name" => "storyline_infra.nginx", "merge" => {"strategy" => "deep"}})» get data from Hiera (examples I will give her data later), which are further used to fill all variables.Hiera configuration file:
--- version: 5 defaults: datadir: "hieradata" data_hash: yaml_data hierarchy: - name: "1" path: "nodes/%{trusted.certname}.yaml" - name: "2" path: "version.yaml" - name: "3" path: "common.yaml" In this case, the hierarchy is visible (the “hierarchy key”) source, where each source at a higher level overrides the values of the keys at a lower level. This allows you to have a key, for example, “www.server.port” with a value of “80” in “common.yaml” and with a value of “81” in “nodes / webserver1.yaml” - in the end we will get the value of this key when executing Puppet code: "81" on the host named "webserver1" and "80" on all others.
Hiera's common.yaml
--- limits::entries: '*/nofile': both: 1048576 '*/memlock': both: unlimited logrotate::config: su_user: root su_group: syslog compress: true # sysctl sysctl::base::purge: false sysctl::base::values: net.core.somaxconn: value: '65536' vm.swappiness: ensure: absent fs.file-max: value: '500000' vm.max_map_count: value: '262144' storyline_base: oracle_java: version: "8u92" storyline_infra: collectd: server_address: "XXX.nlp-project.ru" pid_file: '/data/logs/collectd/collectd.pid' init_script: '/etc/init.d/collectd' dir_data: '/data/db/collectd' dir_logs: '/data/logs/collectd' version: "1.2.0-1" enabled_mongodb: false mongodb_user: "collectd" mongodb_password: "######" enabled_storm: false enabled_elasticsearch: false elasticsearch_port: "####" elasticsearch_cluster: "elastic_storyline" enabled_startup: false enabled_running: true influxdb: port_http: "####" port_rpc: "####" pid_file: '/data/logs/influxdb/influxdb.pid' init_script: '/etc/init.d/influxdb' dir_data: '/data/db/influxdb' dir_logs: '/data/logs/influxdb' version: "present" enabled_auth: true enabled_startup: false enabled_running: true …. site.pp (Puppet site definition file)
node "XXX.nlp-project.ru" { include ::limits include ::sysctl::base include ::logrotate include storyline_base::ntp include storyline_base::srv_oper include storyline_base::ssh include storyline_base::oracle_java ….. include storyline_infra::monit include storyline_base::firewall } node "YYYY.nlp-project.ru" { include ::limits include ::sysctl::base include ::logrotate include storyline_base::ntp include storyline_base::srv_oper include storyline_base::ssh include storyline_base::oracle_java …. include storyline_infra::zookeeper include storyline_components::server_storm include storyline_infra::monit include storyline_base::firewall } If someone is interested in a specific implementation of any of the tasks: write - write off the implementation in the comments or add to “Tips”.
Puppet is actively developing: there is a clear improvement in the syntax and unification of the behavior of classes depending on the context of use (however, there are still some specific features of the resolution of variables in different contexts, which sometimes leads to confusion).
Tips
- When developing a module, remember to write code not only to add a function, but also to disable it. In the absence of such functionality, when transferring a component to another server, you will have 2 of them: in the new and in the old place — in the old one you will need to be removed by hands, which contradicts the main task of automating configuration management;
- Good books on Puppet for beginners - Learning Puppet and Puppet 4 Essentials ;
- A good module for getting artifacts from nexus sonatype (https://github.com/cescoffier/puppet-nexus);
- Take the maximum number of parameters into the Hiera data files for ease of node configuration and achieving the universality of the code for the modules themselves.
Thanks for attention!
Source: https://habr.com/ru/post/335608/
All Articles