Automatic monitoring using Nagios and Puppet

Good day.
I want to write about the problem I encountered about a year ago. For our project, allocate an account on AWS and it was decided to transfer the development process to the cloud. Everything is convenient, virtual servers are deployed and configured smartly, but the further we moved into production, the more acutely the issue of monitoring was emphasized. New servers were added every day, and autoscaling was planned in production.

Just in case, short description:

Nagios is a program for monitoring open source computer systems and networks. It is intended for monitoring, monitoring the status of computing nodes and services, notifies the administrator in the event that some of the services stop (or resume) their work.
Puppet is a cross-platform client-server application that allows you to centrally manage the configuration of operating systems and programs installed on multiple computers. Puppet is written in the Ruby programming language.

Most puppet manifests for project components were already written then, and the current way of adding new nodes was a bottleneck in this system. And to my great surprise, I discovered that puppet began to support such resources as nagios_host, nagios_contact, etc.
puppet type reference
')
As a result, there are puppet-server, nagios-servers (dev / stage / production), and a bunch of nodes. It was necessary to say something like nagios server that we have a new node.
The algorithm is as follows:
Bootstrap new node => Launch Puppet agent on node => Launch Puppet agent on node with nagios (every 30 minutes by default).
Puppet can perfectly store the exported node resources in the database (puppetdb / mysql / postgress / etc?). Exported resources help to extract variables from the facter node, such as fqdn, ip_address, etc ... This was what I needed.

Let's proceed to the puppetdb installation,
In fact, everything is simple
Add a repository Puppetlabs Puppet Labs Package Repositories

This method works for me

wget apt.puppetlabs.com/puppetlabs-release-precise.deb
sudo dpkg -i puppetlabs-release-precise.deb
sudo apt-get update
sudo puppet resource package puppetdb ensure = latest
sudo puppet resource service puppetdb ensure = running enable = true

/etc/puppet/puppet.conf

[master]
storeconfigs = true
storeconfigs_backend = puppetdb

/etc/puppet/puppetdb.conf

[main]
server = puppet # dns name
port = 8081

How to install mysql or postgresql instead of puppetdb described in detail here

Example

For an example of using exported resources. Take 2 classes

class test {

file {"/ tmp / 1":
ensure => present,
content => "$ :: ipaddress",
}

}

class test {

@@file {"/ tmp / 1":
ensure => present,
content => "$ :: ipaddress",
}

}

In the first case, when the node applies the test manifest, the contents of the $ :: ipaddress variable from the facter are copied to the / tmp / 1 file. In the second case, the file on the node is not created, and the @@file resource is stored in puppetdb for later recall.
You can call it using constructs

class export_test {

File << | | >> {
}

}

it is declared in class and says: Give me all the export resources in the File view.

site.pp

node 'firstnode' {
include test
}
node 'secondnode' {
include exporttest
}

As a result, resources from fistnode will be copied to the secondnode.
We do the same with nagios resources.

Config from nagios

define host {
address 23.253.222.185
alias magnetodb-1
host_name magnetodb-1
use linux-server
hostgroups dev
}

define service {
service_description ssh
use local-service
check_command check_ssh
servicegroups GENERIC_GROUP
host_name magnetodb-1
}

define service {
service_description PING
use nagios-graph-service
check_command check_ping! 100.0.20%! 500.0.60%
servicegroups GENERIC_GROUP
host_name magnetodb-1
}

We are writing a class for a client that will add a host and 2 checks to puppetdb

class nagios :: host :: generic {

@@nagios_host {"$ nagios_hostname":
ensure => present,
alias => $ nagios_hostname,
host_name => "$ nagios_hostname",
address => $ ipaddress,
hostgroups => $ env,
use => 'linux-server',
target => "$ nagios :: params :: nagios_base / hosts / $ {env} _ $ {nagios_hostname} .cfg", # the location of the resource at the node where it will be exported
tag => $ :: deployment_id,
notify => Service ["nagios"],
require => File [$ nagios :: params :: nagios_dirs],
}

@@nagios_service {"ssh $ ipaddress":
ensure => present,
check_command => 'check_ssh',
host_name => $ nagios_hostname,
servicegroups => 'GENERIC_GROUP',
service_description => 'ssh',
use => 'local-service',
target => "$ nagios :: params :: nagios_base / hosts / services / $ {env} _ $ {nagios_hostname} .cfg",
tag => $ :: deployment_id,
notify => Service ["nagios"],
require => File [$ nagios :: params :: nagios_dirs]
}

@@nagios_service {"ping $ ipaddress":
ensure => present,
check_command => 'check_ping! 100.0.20%! 500.0.60%',
host_name => $ nagios_hostname,
servicegroups => 'GENERIC_GROUP',
service_description => 'PING',
use => 'nagios-graph-service',
target => "$ nagios :: params :: nagios_base / hosts / services / $ {env} _ $ {nagios_hostname} .cfg",
tag => $ :: deployment_id,
notify => Service ["nagios"],
}

}

The headers of the exported resources must be unique for each node, otherwise we get an error about the duplicate parameter in exported resources. To do this, add a unique $ ipaddress or $ fqdn.

Class for server

class nagios_server {
Nagios_host << | tag == $ :: deployment_id | >> {
}
Nagios_service << | tag == $ :: deployment_id | >> {
}

#tag == $ :: deployment_id means that we select all resources with a specific tag from the database, this is convenient when we have several nagios that must be checked by different hosts.
# variable $ deployment_id will need to be pre-declared in site.pp

}

site.pp

$ deployment_id = "dev"
$ env = "dev"

node "nagios-1" {
$ nagios_hostname = "$ {hostname}"
class {'nagios :: server':
}
}

node "nagios-client-1" {
$ nagios_hostname = "$ hostname_ $ ipaddress"
class {'nagios :: hosts :: generic':}

}

In order to clean all the resources of the node from puppetdb

puppet node clean "node_certname"

At one time, this method helped me save a lot of time in servicing 100 nodes in AWS.
I hope this article will help someone. Thanks for attention

References:

docs.puppetlabs.com/puppetdb/1/connect_puppet_master.html
projects.puppetlabs.com/projects/1/wiki/using_stored_configuration
docs.puppetlabs.com/guides/exported_resources.html