Geekly Articles each Day
📜 ⬆️ ⬇️

Autoconfiguration with Puppet and AWS Cloud Formation

image So the day came when it was necessary to put aside cookbooks, recipes, a chef's knife and a little work with puppets.
To begin with, the formulation of the problem is rather trivial - to arrange for developers the ability to quickly and easily deploy the environment. Mandatory requirement - for autoconfiguration to use Puppet Enterprise
So, in more detail about the necessary environment. It will consist of two components, the first one is FrontEnd, the functions of which are performed by the IIS server, the second one is BackEnd, which will contain some actually developed Worker service and the MongoDB database. Both components, as already understood, will be implemented on Windows Server. The sources for the FrontEnd content and the Worker service will be taken from AWS S3, where they are regularly added every night by Jenkins.

Create Cloud Fromation Template

Implementing a Cloud Formation template that will start two Windows servers is absolutely not difficult. It is much more interesting to figure out how to tell Puppet what configuration to apply to these servers.
In the official documentation, Puppet proposes applying regular expressions to the client’s hostname, which in our case is not convenient to use, since the AWN Amazon hostname is issued automatically and can change after the instance's stop-start, that is, I would be forced to invent a post-start script that should change the hostname of the machine and only then start the puppet agent.
Having rummaged still in documentation, I found that it is necessary - Custom External Facts . For those who work with Chef Server, the facts are the equivalent of attributes.
To add your facts to the Windows machine, you need to create a bat or ps1 file of the following content and put it in " C: \ ProgramData \ PuppetLabs \ facter \ facts.d \" .
@echo off echo node_role=frontend echo app_version=Build1.2.0

Where serverRole is, as the name implies, the role that will be assigned to the server, and buildNumber is the version of the application that will be downloaded from S3 AWS.
This file will be created by the Cloud Formation template.
DevEnv.tmpl
 { "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "Developers Stack", "Parameters" : { "KeyName" : { "Description" : "Key-pair name", "Type" : "String" }, "SuffixName" : { "Description" : "Suffix for all created resources", "Type" : "String" }, "FrontEndInstanceType" : { "Type" : "String", "Default" : "m1.small", "AllowedValues" : [ "m1.small", "m1.medium", "m1.large", "m1.xlarge"], "Description" : "EC2 instance type" }, "BackEndInstanceType" : { "Type" : "String", "Default" : "m1.small", "AllowedValues" : [ "m1.small", "m1.medium", "m1.large", "m1.xlarge"], "Description" : "EC2 instance type" }, "PuppetServer": { "Description" : "Puppet Server URL", "Type" : "String", "Default" : "ec2-231-231-123-123.us-west-2.compute.amazonaws.com" }, "Zone" : { "Type" : "CommaDelimitedList", "Description" : "The Availability Zone ", "Default" : "us-west-2c" }, "BuildVersion" : { "Type" : "String", "Description" : "Version of application build" }, "RoleName" : { "Type" : "String", "Description" : "Instance IAM role", "Default" : "WebInstance" }, "SecurityGroup" : { "Type" : "String", "Description" : "Default security group for stack", "Default" : "taws-security-group" } }, "Mappings" : { "WindowsInstanceType" : { "t1.micro" : { "Arch" : "64" }, "m1.small" : { "Arch" : "64" }, "m1.medium" : { "Arch" : "64" }, "m1.large" : { "Arch" : "64" }, "m1.xlarge" : { "Arch" : "64" } }, "WindowsRegionMap" : { "us-east-1" : { "AMI" : "ami-e55a7e8c" }, "us-west-2" : { "AMI" : "ami-1e53c82e" }, "us-west-1" : { "AMI" : "ami-b687b1f3" }, "eu-west-1" : { "AMI" : "ami-5f3ad728" }, "ap-southeast-1" : { "AMI" : "ami-96cd98c4" }, "ap-southeast-2" : { "AMI" : "ami-ab4a2daa" }, "ap-northeast-1" : { "AMI" : "ami-133fa329" }, "sa-east-1" : { "AMI" : "ami-bd3d9ba0" } } }, "Resources" : { "FrontEnd" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyName" }, "ImageId" : { "Fn::FindInMap" : [ "WindowsRegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}, "InstanceType" : { "Ref" : "FrontEndInstanceType" }, "IamInstanceProfile" : { "Ref" : "RoleName" }, "SecurityGroups" : [{ "Ref" : "SecurityGroup" }], "Tags" : [ {"Key" : "Name", "Value" : { "Fn::Join" : ["",[{"Ref" : "SuffixName"},"-DEV-FrontEnd"]]}} ], "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ "<powershell>\n", "$MsiUrl = \"https://s3-us-west-2.amazonaws.com/mybucket/puppet.msi\"\n", "$downloadPath = \"c:\\puppet.msi\"\n", "$webClient = New-Object System.Net.WebClient\n", "$webClient.DownloadFile($MsiUrl, $downloadPath)\n", "$process = Start-Process -File $downloadPath -arg \"/qn /norestart\" -PassThru |wait-process\n", "$PublicHostName = Invoke-RestMethod -Uri http://169.254.169.254/latest/meta-data/public-hostname -Method Get\n", "Clear-Content 'C:\\ProgramData\\PuppetLabs\\puppet\\etc\\puppet.conf'\n", "Add-Content 'C:\\ProgramData\\PuppetLabs\\puppet\\etc\\puppet.conf' \"[main]\", \"runinterval=300\", \"certname=$PublicHostName\", \"server=",{ "Ref" : "PuppetServer" },"\", \"environment=",{ "Ref" : "PuppetEnvironment" },"\"\n", "Add-Content 'C:\\ProgramData\\PuppetLabs\\facter\\facts.d\\facts.bat' \"@echo off\", \"echo node_role=frontend\", \"echo app_version=",{ "Ref" : "BuildVersion" },"\"\n", "Restart-Service pe-puppet\n", "$MsiUrl = \"https://s3-us-west-2.amazonaws.com/mybucket/7zip.msi\"\n", "$downloadPath = \"c:\\7zip.msi\"\n", "$webClient = New-Object System.Net.WebClient\n", "$webClient.DownloadFile($MsiUrl, $downloadPath)\n", "$process = Start-Process -File $downloadPath -arg \"/qn \" -PassThru |wait-process\n", "</powershell>\n" ]]}} } }, "BackEnd" : { "Type" : "AWS::EC2::Instance", "Properties" : { "KeyName" : { "Ref" : "KeyName" }, "ImageId" : { "Fn::FindInMap" : [ "WindowsRegionMap", { "Ref" : "AWS::Region" }, "AMI" ]}, "InstanceType" : { "Ref" : "BackEndInstanceType" }, "IamInstanceProfile" : { "Ref" : "RoleName" }, "SecurityGroups" : [{ "Ref" : "SecurityGroup" }], "Tags" : [ {"Key" : "Name", "Value" : { "Fn::Join" : ["",[{"Ref" : "SuffixName"},"-DEV-BackEnd"]]}} ], "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ "<powershell>\n", "$MsiUrl = \"https://s3-us-west-2.amazonaws.com/mybucket/puppet.msi\"\n", "$downloadPath = \"c:\\puppet.msi\"\n", "$webClient = New-Object System.Net.WebClient\n", "$webClient.DownloadFile($MsiUrl, $downloadPath)\n", "$process = Start-Process -File $downloadPath -arg \"/qn /norestart\" -PassThru |wait-process\n", "$PublicHostName = Invoke-RestMethod -Uri http://169.254.169.254/latest/meta-data/public-hostname -Method Get\n", "Clear-Content 'C:\\ProgramData\\PuppetLabs\\puppet\\etc\\puppet.conf'\n", "Add-Content 'C:\\ProgramData\\PuppetLabs\\puppet\\etc\\puppet.conf' \"[main]\", \"runinterval=300\", \"certname=$PublicHostName\", \"server=",{ "Ref" : "PuppetServer" },"\", \"environment=",{ "Ref" : "PuppetEnvironment" },"\"\n", "Add-Content 'C:\\ProgramData\\PuppetLabs\\facter\\facts.d\\facts.bat' \"@echo off\", \"echo node_role=backend\", \"echo app_version=",{ "Ref" : "BuildVersion" },"\"\n", "Restart-Service pe-puppet\n", "$MsiUrl = \"https://s3-us-west-2.amazonaws.com/mybucket/7zip.msi\"\n", "$downloadPath = \"c:\\7zip.msi\"\n", "$webClient = New-Object System.Net.WebClient\n", "$webClient.DownloadFile($MsiUrl, $downloadPath)\n", "$process = Start-Process -File $downloadPath -arg \"/qn \" -PassThru |wait-process\n", "</powershell>\n" ]]}} } } }, "Outputs" : { "FrontEndPublicDnsName" : { "Description" : "Public IP address of FrontEnd", "Value" : { "Fn::Join" : ["",[{ "Fn::GetAtt" : [ "FrontEnd", "PublicDnsName" ] }]]} }, "BackEndPublicDnsName" : { "Description" : "Public IP address of BackEnd", "Value" : { "Fn::Join" : ["",[{ "Fn::GetAtt" : [ "BackEnd", "PublicDnsName" ]}]]} } } }


Parameters that are used in the template:

IAM Role and Security Group can be created by the same template, it will be even more correct. In my example, this is not done in order to simplify understanding.
In the UserData section, the puppet agent, 7zip is downloaded and installed, and puppet.conf and facts.bat are generated.
With Cloud Formation finished, it's time to move on to setting up Puppet.

Configure Puppet Server Enterprise

To install Puppet Server Enterprise, you only need to download the installer archive , unpack and run puppet-server-installer. To enable automatic client registration on the server, you need to create the /etc/puppetlabs/puppet/autosign.conf file with the following content:
 *

Create the necessary modules. Modules are some kind of kukbuk in Chef. They are located in the / etc / puppetlabs / puppet / modules folder.
Simplified module structure:

First, add the necessary modules from PuppetLabs for IIS installation and management.
 puppet module install dism puppet module install opentable-iis

Now we need to tweak the manifest for opentable-iis a bit
/etc/puppetlabs/puppet/modules/nodes/manifests/init.pp
class iis {
iis :: manage_app_pool {"$ {fqdn}":
enable_32_bit => true
managed_runtime_version => 'v4.0',
} ->
')
iis :: manage_site {"$ {fqdn}":
site_path => 'C: \ MyAppPath',
port => '80',
ip_address => '*',
host_header => "$ {fqdn}",
app_pool => "$ {fqdn}"
}

}

I got seven modules (maybe the number will increase further).
  1. nodes - the module that will, based on the value of node_role , connect the next necessary module
    /etc/puppetlabs/puppet/modules/nodes/manifests/init.pp
    class nodes {
    if "$ {node_role}" == "backend" {
    include backend
    }
    if "$ {node_role}" == "frontend" {
    include frontend
    }
    }

  2. getbuild - this module is needed for downloading and unpacking the application archive from AWS S3.
    /etc/puppetlabs/puppet/modules/getbuild/manifests/init.pp
    class getbuild {
    file {'c: \ config':
    ensure => 'directory'
    } ->
    file {'c: \ Build':
    ensure => 'directory'
    } ->
    exec {'download_build':
    creates => "c: \\ config \\ $ {app_version}",
    path => $ :: path,
    command => "powershell.exe -executionpolicy unrestricted start-bitstransfer -source s3-us-west-2.amazonaws.com/mybucket $ {app_version} -Destination 'c: \\ config \\'",
    } ->
    exec {'app_install':
    creates => "c: \\ Build \ CustomBackendService.exe.config",
    command => "\" c: \\ Program Files \\ 7-Zip \\ 7z.exe \ "xc: \\ config \\ $ {app_version} -oC: \\ Build",
    }

    }

  3. mongodb - a module for installing MongoDB
    /etc/puppetlabs/puppet/modules/mongodb/manifests/init.pp
    class mongodb {
    file {'c: / config':
    ensure => directory,
    } ->
    file {'c: /config/mongodb.zip':
    ensure => file,
    mode => '0777',
    source => 'puppet: ///modules/mongodb/mongodb-win32-x86_64-v2.4-latest.zip',
    } ->
    file {'c: / MongoDB':
    ensure => directory,
    } ->
    file {'c: / MongoDB / bin':
    ensure => directory,
    } ->
    file {'c: / MongoDB / Data':
    ensure => directory,
    } ->
    file {'c: / MongoDB / logs':
    ensure => directory,
    } ->
    exec {'mongodb-unzip':
    creates => 'c: /MongoDB/bin/mongod.exe',
    command => "" c: \\ Program Files \\ 7-Zip \\ 7z.exe "ec: \\ config \ mongodb.zip -oC: \\ MongoDB \\ bin ',
    } ->
    exec {'mongodb-install':
    creates => 'c: /MongoDB/logs/mongodb.log',
    command => "" c: \\ MongoDB \\ mongod.exe "--dbpath = c: \\ MongoDB \\ Data --port 27017 - logpath = c: \\ MongoDB \ logs \\ mongodb.log - install --serviceName mongodb --serviceDisplayName "MongoDB Server" --serviceDescription "MongoDB Server" ',
    } ->
    exec {'mongodb-run':
    path => $ :: path,
    command => 'powershell.exe start-service mongodb'
    }
    }

  4. api - module for installing an application on FrontEnd
    /etc/puppetlabs/puppet/modules/api/manifests/init.pp
    class api {

    include getbuild

    dism {'IIS-WebServerRole':
    ensure => present,
    } ->

    dism {'IIS-WebServer':
    ensure => present,
    require => Dism ['IIS-WebServerRole'],
    }

    }

  5. worker - module for installing the application on BackEnd
    /etc/puppetlabs/puppet/modules/worker/manifests/init.pp
    class worker {
    include getbuild
    exec {'service_install':
    creates => "c: \\ Build \\ Custom.AWS.BackendService.InstallLog",
    command => "c: \\ Build \\ Custom.AWS.BackendService.exe -install",
    } ->
    exec {'service-run':
    path => $ :: path,
    command => 'powershell.exe start-service Custom.AWS.Backend'
    }
    }

  6. frontend - a module that connects all the necessary modules for FrontEnd to work
    /etc/puppetlabs/puppet/modules/frontend/manifests/init.pp
    class frontend {
    include api
    include iis
    }

  7. backend - a module that connects all the necessary modules for running BackEnd
    /etc/puppetlabs/puppet/modules/backend/manifests/init.pp
    class backend {
    include mongodb
    include worker
    }



In my manifestos, I used the exec resource almost everywhere. With the properly selected creates option, this resource works flawlessly.
More detail on one of the examples:
 exec { 'mongodb-unzip': creates => 'c:/MongoDB/bin/mongod.exe', command => '"c:\\Program Files\\7-Zip\\7z.exe" ec:\\config\mongodb.zip -oC:\\MongoDB\\bin', }

If the executable file c: /MongoDB/bin/mongod.exe is missing, the archive will be unpacked.

Now, for convenience, you can create a task in your favorite CI system, for example Jenkins , place a script there to launch the Cloud Formation template and developers will be able to expand the environment in one click.

That's all. I hope this guide will be useful ...
If among those who have read this article, there will be experts on the use of Puppet, I will hear your opinion with great gratitude.

Source: https://habr.com/ru/post/208710/

More articles:


All Articles