Programming example in Puppet version 3.8 using Hiera and R10K
We Need To Go Deeper.
Inception.
Here I want to describe programming examples / techniques. The reason is the same: there should be more docks on the Internet. In my last article I talked about installing and configuring Puppet version 3.8 using the example of Centos 6.5. There would be the simplest example to test the client-server bundle. Now let's see how this can be complicated and why it is necessary.
')
Chain of transmission of our text in the parameters:
class ($ text1 $ text2) -> file (content ($ {text1} - $ {text2})
So the next level: we have many classes and we want to make an analogue of the library of variables. Parameters can be stored in one recipe and then used throughout the module. For example, we want to collect some system variables and add something of our own.
A little bit of the basics:
We go to the Master server in / etc / puppet / modules and generate the skeleton:
We press 8 times the input, because all this can then be redone. Get the directory myname-test1, which we will rename to test1. You need the full name if you want to knock out and upload your module to the public Puppet forge .
Create the file /etc/puppet/modules/test1/manifests/init.pp
Create the file /etc/puppet/modules/test1/manifests/params.pp
Explanations:
test1 - Module name - The root name of the project in the puppet value system.
init.pp, params.pp - manifests - One manifest stores one class. File name and class must match.
class test1 - the initial class, the code of which works by default with a simple call after the class via the simple include test1
If desired, you can leave empty and create separate noun classes (see below).
class test1 :: params is a name class. The name params is chosen for convenience and can be any.
You can check the syntax in advance in 2 ways:
- Initially available via type command:
- Put a more advanced spell checker ppuppet-lint through gem install puppet-lint and then check the files (at the same time you can slightly brush the syntax with the key --fix):
But do not rely on them, you can easily miss an error like the wrong name of the manifest that will generate an error already when running on the client like:
Again, I want to warn you that I noticed that in the 3.8.4 branch now having changed the module code, I have to restart the puppetmaster or httpd service so that the changes in the modules are applied immediately, before it did not ignore it. Perhaps in other versions it will not appear. I suspect that the module cache is stuck somewhere.
Add a call to our test1 module in the /etc/puppet/manifests/site.pp file for the test stage.test.net node.
Check on client:
You can configure the service and wait for XX (usually 30) minutes until it works and then look at the logs. And you can start everything manually:
As you can see the file was successfully created. If anyone wants to see how parameters can be reached, here is an example of a module for apache.
The chain of transmission of our text ptestint:
manifests / site.pp -> modules / test1 / init.pp ($ test1 :: params :: text1) -> file (content ($ {text1})
Now, how can you change the default variables in /etc/puppet/manifests/site.pp, since they have higher priority.
Check on client:
...
-production - test: ptestint
\ No newline at end of file
+ production - test: newparam
\ No newline at end of file
...
As you can see the update was successful.
The chain of transmission of our text newparam:
manifests / site.pp ($ text1) -> modules / test1 / init.pp (text1) -> file (content ($ {text1})
Such storage of parameters is also convenient if we do not create all manifests in one directory, but have made another level in the form /test1/manifests/check/time.pp
And then use this class anywhere through the form call:
The minus variation from the last example is that there is only one line in content Therefore, it is better to use templates for generating large files.
Add to the test:
- File /etc/puppet/modules/test1/manifests/usetmpl.pp - new class for working with the template
Changes in the replacement of the content of the text to the call of the template puppetenv.erb plus we brought everything into a separate class, although we could add the creation of the second file in init.pp.
- The file /etc/puppet/modules/test1/templates/puppetenv.erb - Our generator template.
The variables $ {text1} are here passed in Ruby format as <% = @ text1%> . $ envname is taken from params.pp, $ text1 I again redefined in site.pp and at the same time added a “system” variable (they are called facts in puppet) <% = hostname %>
Check on client:
Total:
The chain of our text param_tmpl:
manifests / site.pp ($ text1) -> modules / test1 / init.pp :: usetmpl ($ text1) -> file (content (puppetenv.erb (<% = @ text1%>)))
If you need to work with 1-10 servers, then the usual modules will be enough, but if there are more plus, the separation into subclusters went, where each module is configured in its own way, then you can get lost in the swollen parameters of the site.pp modules, or in modules of the same name and their versions. We go deeper and tune Hiera.
Hiera is a Ruby library; it is included by default in Puppet and helps organize data for all modules in a single directory.
To work our storage you need:
- Create a file /etc/puppet/hiera.yaml of the following form:
Here, we are waiting for a backdoor from the system as a priority of the native file /etc/hiera.yaml
T. h. You need to replace it with the symlink /etc/hiera.yaml -> /etc/puppet/hiera.yaml
- Create a folder / etc / puppet / hieradata (you can give your name and specify it in: datadir)
Files in this folder must have the extension .yaml and data format YAML.
- Create file /etc/puppet/hiera/common.yaml
For example, here we can write the second test parameter available to all nodes
Since we set the directory as the storage point for the node parameters , “nodes /% {:: fqdn}” , then for our test node we create the file /etc/puppet/hiera/nodes/stage.test.net.yaml . In it, we can now set our third test parameter and a small array in which there will be a parameter and one more array
Checking the availability of parameters from the command line in debug and simple mode:
Now we need to save them in the parameters and use in the template.
/etc/puppet/modules/test1/manifests/params.pp
/etc/puppet/modules/test1/manifests/usetmpl.pp
/etc/puppet/modules/test1/templates/hieratst.erb
Check on client:
Actually if the account of the servers went to tens of hundreds, then it becomes safer to divide them into environments. You can do this with handles, but it is better to use R10K, which allows you to run a separate configuration using modules, which is stored in its personal settings. That is. In essence, it replaces a single giant site.pp with its configs tree.
I will not do the test, just give the conditional algorithm for such a setting on the example of my working configuration.
- Servers are divided into settings groups that are stored in separate directories in / etc / puppet / environments
For example, we will test our recipe on the server group test_devops
Hyer's tree
- stored on the gita / beatback
- tends to update the application of changes to the Master server
- in hiera.yaml added to: hierarchy:
- Our parameters will be conditionally stored in files
/ etc / puppet / hieradata / test_devops / test_devops.yaml - for all nodes via additional tag for R10K
/ etc / puppet / hieradata / test_devops / stage.test.net.yaml for server plus need label for R10K
Configuring module startup for test_devops via R10K
- On the stage.test.net site, add the line to the / etc / puppet / puppet.conf file
- / etc / puppet / environments / test_devops / Puppetfile - all used modules are stored here.
Recording examples
They are then downloaded / updated in the console via a command like
/ etc / puppet / environments / test_devops / modules / test1 - Where our module went down
- / etc / puppet / environments / test_devops / dist / profiles / manifests / - module launch manifest tree. File names must not be the same as module names.
Create a file of type runtest1.pp here
As you can see the nodes are no longer indicated. If other nodes need other parameters, you can create runtest2.pp, etc. Additional levels are supported. For example, you can create the file / etc / puppet / environments / test_devops / dist / profiles / manifests / ver2 / runtest3.pp
- Now you need to bind the launch manifest modules to the nodes:
/etc/puppet/environments/test_devops/dist/roles/manifests/init.pp
- Actually there was a dump of the totals of the settings
And then it will be applied on the node by autorun or you can test it manually via puppet agent --test
This is actually all. Thank you for reading to here.
Additions or options for other versions of those who wish can be included in this article.
I will not compare it with the experience of using chef client-server / chef + berkshelf / chef + AWS Opswork, since there is a completely different algorithm for organizing kukbok- “modules” and cleaner Ruby in recipes- “manifestos”.
Additional docks:
1. According to micro samples of Puppet code.
2. By introduction to Hiera
3. A bit on R10K
Inception.
Here I want to describe programming examples / techniques. The reason is the same: there should be more docks on the Internet. In my last article I talked about installing and configuring Puppet version 3.8 using the example of Centos 6.5. There would be the simplest example to test the client-server bundle. Now let's see how this can be complicated and why it is necessary.
Example â„–1: Using parameters in module manifests
')
class test1 { # $text1 = “our” # $text2 = “text” # file { 'puppetenv file': # , «puppetenv file» – # file. . path => '/tmp/puppetenv', # ensure => file, # content => "test: ${text1} - ${text2}" # } } Chain of transmission of our text in the parameters:
class ($ text1 $ text2) -> file (content ($ {text1} - $ {text2})
Example 2: Uniform storage of parameters in the module manifest
So the next level: we have many classes and we want to make an analogue of the library of variables. Parameters can be stored in one recipe and then used throughout the module. For example, we want to collect some system variables and add something of our own.
A little bit of the basics:
We go to the Master server in / etc / puppet / modules and generate the skeleton:
puppet module generate myname-test1 We press 8 times the input, because all this can then be redone. Get the directory myname-test1, which we will rename to test1. You need the full name if you want to knock out and upload your module to the public Puppet forge .
Create the file /etc/puppet/modules/test1/manifests/init.pp
class test1 ( $envname = $test1::params::env, # params $text1 = $test1::params::text1 ) inherits test1::params { # inherits file { 'puppetenv file': path => '/tmp/puppetenv', ensure => file, content => "${env} - test: ${text1}" } } Create the file /etc/puppet/modules/test1/manifests/params.pp
class test1::params { $env = $settings::environment # puppet $text1 = 'ptestint' # } Explanations:
test1 - Module name - The root name of the project in the puppet value system.
init.pp, params.pp - manifests - One manifest stores one class. File name and class must match.
class test1 - the initial class, the code of which works by default with a simple call after the class via the simple include test1
If desired, you can leave empty and create separate noun classes (see below).
class test1 :: params is a name class. The name params is chosen for convenience and can be any.
You can check the syntax in advance in 2 ways:
- Initially available via type command:
puppet parser validate /etc/puppet/modules/test1/manifests/* - Put a more advanced spell checker ppuppet-lint through gem install puppet-lint and then check the files (at the same time you can slightly brush the syntax with the key --fix):
puppet-lint --fix /etc/puppet/modules/test1/manifests/* But do not rely on them, you can easily miss an error like the wrong name of the manifest that will generate an error already when running on the client like:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find parent resource type 'test::params' of type hostclass in production at /etc/puppet/modules/test1/manifests/init.pp:1 on node stage.test.net Again, I want to warn you that I noticed that in the 3.8.4 branch now having changed the module code, I have to restart the puppetmaster or httpd service so that the changes in the modules are applied immediately, before it did not ignore it. Perhaps in other versions it will not appear. I suspect that the module cache is stuck somewhere.
Add a call to our test1 module in the /etc/puppet/manifests/site.pp file for the test stage.test.net node.
node default { } node 'stage.test.net' { include test1 } Check on client:
You can configure the service and wait for XX (usually 30) minutes until it works and then look at the logs. And you can start everything manually:
[root@stage ~]# puppet agent --test Info: Retrieving pluginfacts Info: Retrieving plugin Info: Caching catalog for stage.test.net Info: Applying configuration version '1459348679' Notice: /Stage[main]/Test1/File[puppetenv file]/ensure: defined content as '{md5}af1b5424181346c5f4827af742b07abf' Notice: Finished catalog run in 0.12 seconds [root@stage ~]# cat /tmp/puppetenv production - test: ptestint As you can see the file was successfully created. If anyone wants to see how parameters can be reached, here is an example of a module for apache.
The chain of transmission of our text ptestint:
manifests / site.pp -> modules / test1 / init.pp ($ test1 :: params :: text1) -> file (content ($ {text1})
Now, how can you change the default variables in /etc/puppet/manifests/site.pp, since they have higher priority.
node 'stage.test.net' { # include test1 class { 'test1': text1 => 'newparam', } } Check on client:
...
-production - test: ptestint
\ No newline at end of file
+ production - test: newparam
\ No newline at end of file
...
As you can see the update was successful.
The chain of transmission of our text newparam:
manifests / site.pp ($ text1) -> modules / test1 / init.pp (text1) -> file (content ($ {text1})
Such storage of parameters is also convenient if we do not create all manifests in one directory, but have made another level in the form /test1/manifests/check/time.pp
And then use this class anywhere through the form call:
class { '::test1::check::time': } Example 3: Add a template.
The minus variation from the last example is that there is only one line in content Therefore, it is better to use templates for generating large files.
Add to the test:
- File /etc/puppet/modules/test1/manifests/usetmpl.pp - new class for working with the template
class test1::usetmpl ( $envname = $test1::params::env, #1 $text1 = $test1::params::text1 #2 ) inherits test1::params { file { 'puppetenv file from tmpl': path => '/tmp/puppetenv', ensure => file, content => template('test1/puppetenv.erb'), } } Changes in the replacement of the content of the text to the call of the template puppetenv.erb plus we brought everything into a separate class, although we could add the creation of the second file in init.pp.
- The file /etc/puppet/modules/test1/templates/puppetenv.erb - Our generator template.
# Created by puppet Our text: <%= @text1 %> Env: <%= @envname %> Host: <%= @hostname %> The variables $ {text1} are here passed in Ruby format as <% = @ text1%> . $ envname is taken from params.pp, $ text1 I again redefined in site.pp and at the same time added a “system” variable (they are called facts in puppet) <% = hostname %>
Check on client:
[root@stage ~]# puppet agent --test ... -production - test: newparam \ No newline at end of file +# Created by puppet +Our text: param_tmpl +Env: production +Host: stage ... Total:
[root@stage ~]# cat /tmp/puppetenv # Created by puppet Our text: param_tmpl Env: production Host: stage The chain of our text param_tmpl:
manifests / site.pp ($ text1) -> modules / test1 / init.pp :: usetmpl ($ text1) -> file (content (puppetenv.erb (<% = @ text1%>)))
Example number 4: Hiera or even more centralization
If you need to work with 1-10 servers, then the usual modules will be enough, but if there are more plus, the separation into subclusters went, where each module is configured in its own way, then you can get lost in the swollen parameters of the site.pp modules, or in modules of the same name and their versions. We go deeper and tune Hiera.
Hiera is a Ruby library; it is included by default in Puppet and helps organize data for all modules in a single directory.
To work our storage you need:
- Create a file /etc/puppet/hiera.yaml of the following form:
:hierarchy: - "%{::clientcert}" - "%{::custom_location}" - "nodes/%{::fqdn}" - "nodes/%{::environment}" - "virtual/%{::virtual}" - common - "%{::environment}" :backends: - yaml :yaml: :datadir: "/etc/puppet/hieradata" Here, we are waiting for a backdoor from the system as a priority of the native file /etc/hiera.yaml
T. h. You need to replace it with the symlink /etc/hiera.yaml -> /etc/puppet/hiera.yaml
- Create a folder / etc / puppet / hieradata (you can give your name and specify it in: datadir)
Files in this folder must have the extension .yaml and data format YAML.
- Create file /etc/puppet/hiera/common.yaml
For example, here we can write the second test parameter available to all nodes
test::text2: common-hiera Since we set the directory as the storage point for the node parameters , “nodes /% {:: fqdn}” , then for our test node we create the file /etc/puppet/hiera/nodes/stage.test.net.yaml . In it, we can now set our third test parameter and a small array in which there will be a parameter and one more array
testparam::text3: 'node stage hiera' arrexmpl::colors: bw: "B&W" rgb: - red - blue - green Checking the availability of parameters from the command line in debug and simple mode:
[root@pmaster /etc]# hiera -d test::text2 DEBUG: Wed Mar 30 13:06:13 -0400 2016: Hiera YAML backend starting DEBUG: Wed Mar 30 13:06:13 -0400 2016: Looking up test::text2 in YAML backend DEBUG: Wed Mar 30 13:06:13 -0400 2016: Looking for data source common DEBUG: Wed Mar 30 13:06:13 -0400 2016: Found test::text2 in common common-hiera [root@pmaster /etc]# hiera testparam::text3 ::fqdn=stage.test.net node stage hiera hiera arrexmpl::colors ::fqdn=stage.test.net {"rgb"=>["red", "blue", "green"], "bw"=>"B&W"} Now we need to save them in the parameters and use in the template.
/etc/puppet/modules/test1/manifests/params.pp
class test1::params { # sss $env = $settings::environment $text1 = 'ptestint' $text2 = hiera('test::text2', 'ptestint2') # 'test::text2'. 'ptestint2' $text3 = hiera('testparam::text3', 'ptestint3') $colors = hiera('arrexmpl::colors', 'nohiera') # arrexmpl::colors if $colors != 'nohiera' { # $c1 = $colors['bw'] $c2 = $colors['rgb'] } else { $c1 = "lost" } } /etc/puppet/modules/test1/manifests/usetmpl.pp
class test1::usetmpl ( $envname = $test1::params::env, $text1 = $test1::params::text1, $text2 = $test1::params::text2, # $text3 = $test1::params::text3, $c1 = $test1::params::c1, $c2 = $test1::params::c2 ) inherits test1::params { file { 'puppetenv file': path => '/tmp/puppetenv', ensure => file, content => template('test1/puppetenv.erb'), } file { 'hiera test': path => '/tmp/phiera', ensure => file, content => template('test1/hieratst.erb'), } } /etc/puppet/modules/test1/templates/hieratst.erb
# Hiera test # Created by puppet Our text1: <%= @text1 %> Our text2: <%= @text2 %> Our text3: <%= @text3 %> Colors: BW = <%= @c1 %> <% if c1 == "lost" %> # ! Hiera fail ! <% end -%> RGB = <%- c2.each do |colors| -%> # arrexmpl::colors::rgb: - <%= colors %> <%- end -%> Check on client:
[root@stage ~]# puppet agent --test ... [root@stage /etc/puppet]# cat /tmp/phiera # Hiera test # Created by puppet Our text1: paramtmpl Our text2: common-hiera Our text3: node stage hiera Colors: BW = B&W RGB = - red - blue - green Example number 5: R10K or even more centralization
Actually if the account of the servers went to tens of hundreds, then it becomes safer to divide them into environments. You can do this with handles, but it is better to use R10K, which allows you to run a separate configuration using modules, which is stored in its personal settings. That is. In essence, it replaces a single giant site.pp with its configs tree.
I will not do the test, just give the conditional algorithm for such a setting on the example of my working configuration.
- Servers are divided into settings groups that are stored in separate directories in / etc / puppet / environments
For example, we will test our recipe on the server group test_devops
Hyer's tree
- stored on the gita / beatback
- tends to update the application of changes to the Master server
- in hiera.yaml added to: hierarchy:
- %{environment}/%{role}/%{calling_module} - %{environment}/%{role} - %{role} - %{environment}/%{environment} - Our parameters will be conditionally stored in files
/ etc / puppet / hieradata / test_devops / test_devops.yaml - for all nodes via additional tag for R10K
classes: - roles::base / etc / puppet / hieradata / test_devops / stage.test.net.yaml for server plus need label for R10K
classes: - roles::stagesrv::test Configuring module startup for test_devops via R10K
- On the stage.test.net site, add the line to the / etc / puppet / puppet.conf file
environment = stage_nbc210 - / etc / puppet / environments / test_devops / Puppetfile - all used modules are stored here.
Recording examples
mod 'saz/sudo', '3.0.1' mod 'stdlib', :git => 'git://github.com/puppetlabs/puppetlabs-stdlib.git', :ref => '4.1.0' mod 'test1', :git => 'git://github.com/fake_link/test1.git', :ref => 'master' They are then downloaded / updated in the console via a command like
sudo r10k deploy module test1 -e test_devops / etc / puppet / environments / test_devops / modules / test1 - Where our module went down
- / etc / puppet / environments / test_devops / dist / profiles / manifests / - module launch manifest tree. File names must not be the same as module names.
Create a file of type runtest1.pp here
class profiles::runtest1{ class { 'test1': text1 => 'newparam', } As you can see the nodes are no longer indicated. If other nodes need other parameters, you can create runtest2.pp, etc. Additional levels are supported. For example, you can create the file / etc / puppet / environments / test_devops / dist / profiles / manifests / ver2 / runtest3.pp
class profiles::ver2::runtest3 class { 'test1': text1 => 'newparam v2', } } - Now you need to bind the launch manifest modules to the nodes:
/etc/puppet/environments/test_devops/dist/roles/manifests/init.pp
class roles { } class roles::base inherits roles { include profiles::base } class roles::private inherits roles { include profiles::private } class roles::public inherits roles { include profiles::public } class roles::stagesrv::test { # <- Hiera ? include profiles::runtest1 } # , . # , runtest3 # runtest1, roles::ver2::runtest3, . . . class roles::ver2::runtest3 inherits roles::stagesrv::test { include profiles::ver2::runtest3 } - Actually there was a dump of the totals of the settings
sudo r10k deploy environment test_devops And then it will be applied on the node by autorun or you can test it manually via puppet agent --test
This is actually all. Thank you for reading to here.
Additions or options for other versions of those who wish can be included in this article.
I will not compare it with the experience of using chef client-server / chef + berkshelf / chef + AWS Opswork, since there is a completely different algorithm for organizing kukbok- “modules” and cleaner Ruby in recipes- “manifestos”.
Additional docks:
1. According to micro samples of Puppet code.
2. By introduction to Hiera
3. A bit on R10K
Source: https://habr.com/ru/post/280632/
All Articles