About a month ago, I had a choice: whether to write a module for puppet "in the table" (that is, for internal infrastructure) or make it universal, open the source and publish it on puppet forge . Of course, it would be faster and easier to sketch out 2-3 classes under you quickly and calm down, but the experience gained during the module publishing process is valuable and I want to share it. In Runet, there is no information on the use of the puppet development kit (hereinafter PDK ), so this can be considered a kind of tutorial.

What is the article about

In the process of developing a module (or rather, two), I discovered PDK, which greatly facilitates both the development and maintenance of modules. Namely:

• Automatic metadata.json formatting when updating the latest
• Configuration generation for various CI systems that can do the following:
  
  • Check ruby ​​code with rubocop linter
  • Run unit tests
  • Under certain conditions - automatic filling of the working code on puppet forge
• Tag-based documentation generation in comments using yard
• Dish [PDK] for a module on puppet forge. Trifle, but nice!

All interested I ask under the cat!

As examples

If you want to look and feel in the process of reading what is meant, you can open one of the two (or both) mentioned modules: clickhouse and xmlsimple . Both were developed using PDK and other tools described in the article.

Content

• What is the article about
  
  • As examples
• What is PDK?
  
  • Installation
  • PDK content
  • Create a module
  • new class
  • new defined_type
  • new provider & task
  • Documentation generation with puppet-strings
  • Customize templates
  • Run various CI
  • Bonus: write unit tests for classes, types and functions
• Instead of output

What is PDK?

From official documentation:

Create a complete module. PDK provides a complete module structure, defined types of tasks, and a testing infrastructure. You can validate your application.

In my free translation:

Allows you to create a complete module with classes, types, tasks and tests to verify the operation of the module. PDK provides complete structure and templates for all of the above. With this tool, you can test the operation of the module with different versions of puppet, as well as in different operating systems.

Sounds good? Well, so it really is. Until the moment when I started working on a module that was decided to write immediately for open source, I had no idea about this tool, and now I intend to transfer all internal infrastructure to PDK.

I will describe how to put it, and what tools and commands it contains.

Installation

The official installation page . Under this link you are almost guaranteed to find the right way to install the PDK on your host. If for some reason you are not lucky and your OS is not there, there is always a detour in the form:

 gem install pdk 

In fact, the PDK is just a heme, and it is put that way.

PDK content

In general, a PDK is nothing more than a set of gems to facilitate the development of modules. It contains the following tools:

Create a module

PDK is installed, now you can play. The simplest pdk help command pdk help display the available commands. Suppose that we are in the folder where you have all the other modules. Then let's create a new one:

 $ pdk new module --template-url=https://github.com/puppetlabs/pdk-templates.git *** We need to create the metadata.json file for this module, so we're going to ask you 5 questions. *** [Q 1/5] If you have a name for your module, add it here. --> dummy [Q 2/5] If you have a Puppet Forge username, add it here. --> felixoid [Q 3/5] Who wrote this module? --> Mikhail f. Shiryaev [Q 4/5] What license does this module code fall under? --> MIT [Q 5/5] What operating systems does this module support? --> RedHat based Linux, Debian based Linux, Windows Metadata will be generated based on this information, continue? Yes pdk (INFO): Module 'dummy' generated at path '/tmp/dummy', from template 'https://github.com/puppetlabs/pdk-templates.git'. 

The utility asks questions to fill in the metadata.json file, and at the output we have exactly what is indicated: the module and the auxiliary files, compiled according to templates from the gita.

A small remark - Templates change quite often, including some critical bugs fixed recently. Therefore, it is better to use not the defaults from the installed PDK, but the latest version. True, there is a downside: when using the --template-url PDK argument, this parameter is added to the ~.pdk/cache/answers.json and, judging by the delays in the further execution of any of the pdk commands, it tries to download them. So either remove this parameter from answers.json , or do not use it when creating a module and change it in metadata.json .

Let's go through the next steps that can be performed using the PDK.

new class

 $ pdk new class dummy::class pdk (INFO): Creating '/tmp/dummy/manifests/class.pp' from template. pdk (INFO): Creating '/tmp/dummy/spec/classes/class_spec.rb' from template. $ cat manifests/class.pp # A description of what this class does # # @summary A short summary of the purpose of this class # # @example # include dummy::class class dummy::class { } $ cat spec/classes/class_spec.rb require 'spec_helper' describe 'dummy::class' do on_supported_os.each do |os, os_facts| context "on #{os}" do let(:facts) { os_facts } it { is_expected.to compile } end end end 

This command creates 2 files: directly the manifest for the class and the spec-file for its testing. On the tags for documentation, I’ll talk about it a bit later.

new defined_type

 $ pdk new defined_type type pdk (INFO): Creating '/tmp/dummy/manifests/type.pp' from template. pdk (INFO): Creating '/tmp/dummy/spec/defines/type_spec.rb' from template. 

All the same: manifest for resource type and spec-file.

new provider & task

PDK can also create a new provider or a tusk, but I didn’t work closely with them, so I’ll honestly say that, if necessary, it’s better to learn more about this topic yourself.

Documentation generation with puppet-strings

I don’t really understand why puppet strings not part of the PDK toolkit, however, ce la vie. If during development you correctly set the tags for the yard, then there are 2 main ways to provide documentation to the user:

• Generate it as HTML / Markdown / JSON and put it next to the code. This is done with the command puppet string generate [--format FORMAT] , where the format can be omitted or take the value json / markdown .
  
  • The standard for documentation is to have REFERENCE.md in the root of the repository, which is generated by the command puppet strings generate --format markdown .
• Publish to the repository with code (provided that it is on github) github-pages. This is quite simple, you need 3 commands:
  

   #  Gemfile.lock,    PDK rm -f Gemfile.lock #     Gemfile   bundle bundle install --path vendor/bundle #   gh-pages   rake-task bundle exec rake strings:gh_pages:update 

It seems to be no magic, but at the exit we have a module with instructions. The advantage is that even if you do not describe, say, each of the parameters using the @param tag, the output will still be a class / type / function with a minimum description of the parameters with the type and default value. In my humble opinion, even this is better than nothing, and will make the module more attractive for use.

Of course, all this can be automated and added as a stage CI. It would be perfect. My hands have not reached me yet, but it is gathering dust in the backlog. If suddenly someone has something to say on this topic - I will be grateful. As a thought: at least add a check to see if REFERENCE.md changes after running puppet-strings. And if so, consider the tests failed.

Customize templates

Template documentation is located in the pdk-templates repository. In short, everything is configured using the .sync.yml file in the root directory of the module, and the changes are applied using the pdk update command. Each of the parameters of this file is the name of a different file in the module directory that needs to be changed in one way or another. Most of the parameters for each of the templates I had to pick up "by touch", looking at the source code, often - by trial and error. Documentation here sometimes lags far behind. Unfortunately, there is almost nothing more to say, except to give a link to an example from our own repository.

Very fluently I will describe several parameters that I changed using the .sync.yml from the example above:

• Gemfile : added two Gemfile as dependencies in different groups: pdk in the development group; xml-simple in the dependencies group. When running tests, the system_tests group is not installed, so I add the dependency to another group.
• spec/spec_helper.rb : spec/spec_helper.rb method was changed, a minimum test coverage threshold was added, below which tests are considered failed.
• .travis.yml : this file has been polished for a long time, as it is used to check the code base and load the finished module to puppet-forge. Changes:
  
  • User and encrypted password to fill the module on puppet-forge. You can read more about the warmth in puppet-forge with Travis.
  • Created the order tests → deploy with the launch of the latter only with successful tests.
  • Added the deployment stage of the module to puppet-forge, provided that CI is launched from a tag that starts with the character "v".
• Rakefile : added some exceptions for the linter.

Run various CI

Everything is quite simple here. Immediately after the module is generated using PDK, validation can be launched in appveyor, travis and gitlab-ci. To start the tests, everything is ready right out of the box, while the same .sync.yml used for tuning. I have no special preferences, so I will not recommend anything. Just use what is more convenient.

Bonus: write unit tests for classes, types and functions

This item is quite a bit beyond the basic material, which I planned to describe, but it seems to me very useful.

So, we have a module with manifests and a library, which, in turn, contain classes, types, and functions (also do not forget about Tasks and Providers, but in this part I have no expertise). Since any code exists for the purpose of changing, it would be nice, obviously, to impose tests on it to make sure of 2 things:

• Changes do not break current behavior (or behavior changes with tests)
• Your manifestos do exactly what is expected and use all the resources as expected.

Puppetlabs provides an extension for the rspec framework called puppet-rspec . Links to documentation on testing classes , types and functions . Do not be lazy to look more closely, there are other sections.

It's easy enough to start using it without even knowing ruby. If classes or types were created, as shown above, with the help of pdk new <thing> , then the *_spec.rb file also already exists. So, suppose we have a class dummy::class . To test it, the file spec/classes/class_spec.rb should be created with the following content:

 require 'spec_helper' describe 'dummy::class' do on_supported_os.each do |os, os_facts| context "on #{os}" do let(:facts) { os_facts } it { is_expected.to compile } end end end 

You can check it by running the pdk test unit from the root directory of the module.

This is almost all we need. Now it remains to supplement the class_spec.rb necessary is_expected with the appropriate conditions. For example, to verify that the class contains the resource file {'/file/path': } with certain parameters, you can do this:

 it do is_expected.to contain_file('/file/path').with( 'ensure' => 'file', 'mode' => '0644' ) end 

You can set class parameters using let(:params) { {'param1' => 'value'} } , it is possible to conduct tests under various input conditions, placing each it inside the selected sections of the context 'some description' {} . You can check both dependencies between resources and between classes: if it is implied, for example, that the class declaration contains inherits , then you can add the is_expected.to contain_class('parent_class_name') . Need to test behavior in different OS? It is also possible: we simply indicate the necessary facts in a separate context:

 context 'with Debian' do let(:facts) do { os: { architecture: 'amd64', distro: { codename: 'stretch', id: 'Debian', release: { full: '9.6', major: '9', minor: '6', }, }, family: 'Debian', name: 'Debian', release: { full: '9.6', major: '9', minor: '6', }, selinux: { enabled: false, }, }, osfamily: 'Debian', } end it { is_expected.to something } end 

In general, as far as I could notice in the process of writing tests, the framework allows you to check almost everything that may be needed. And the presence of tests once helped me out when some parameters were moved from the child classes to the top class of the module: they showed that the refactoring did not break anything and the behavior of the whole module did not change.

Instead of output

As it could already be understood from the general intonation of the article, I am greatly inspired by how much Puppet made it easier to work with modules and manifests thanks to the PDK. Routine actions are automated, wherever possible, templates are used, configs are available from the box for popular CIs. It may seem like some kind of overhead, and using it may not bring the expected fruits, but it is definitely worth it. If we compare how to develop modules without and with PDK, then for me it looks like this:

Development without beards PDK	PDK Development

Try, put, facilitate yourself and colleagues life. I will be happy to answer potential questions.

May the automatization be with us!