Chef for 21 days. Part Three Chef and AWS
Hello, habrauzer. So the third part of my article arrived, which will summarize the cycle ( part 1 and part 2 ) of articles for beginners. This part will focus on a specific example of how to use Chef in the Amazon cloud. As I already mentioned, this is quite a popular scenario. 
For ease of understanding, we will consider the case of two ec2-instance (Amazon virtual servers), one of which will act as a Chef server, and the second as a node.
Immediately I will clarify that we will run the instance using AWS CloudFormation . One could, of course, start and manage them manually, but what is the point of such automation?
CloudFormation can be divided into 2 concepts :
- template , which is a json-file, which describes all the resources we need to run the instance;
- stack , which is the AWS resources themselves, described in the template.
For those who are starting on AWS, Amazon provides a ready-made sample template that covers most aspects that are needed when working with AWS. The link to the templates will be given at the end of the article.
Consider what the template is . In the base case, it consists of 4 blocks: Parameters, Mappings, Resources, Outputs .
The Parameters block describes variables and their values that will be passed to the stack at the time it is created. Parameter values can be entered when creating resources, or can be set using the default field in the parameter description. Parameter can be any information, starting from a password and ending with a network port or path to a directory. To get the parameter value, the Ref function is used in the template.
The Mappings block contains a set of keys with the appropriate parameters and their values. Most often, you can see how mapping is used to define AWS regions and their corresponding virtual images (instance). To get the value of a mapping, use the Fn :: FindInMap function, which indicates the key and parameters by which a particular value is searched for.
The Resources block describes our ec2-instance or other AWS resources. It is in this section that the images for the Chef server and client node are declared. In the description, you must specify the type of resource (for example, AWS :: EC2 :: Instance), it is also possible to specify metadata, in which you can specify the description of our node or pre-install procedure directives (for example, if you need to install any package when running the image ). The main part of the block is the Properties block, which describes in detail the image being launched. In this block, you can specify the type of image that will be launched (for example, Amazon Linux 32-bit), the identity of the image being launched to this or that Security Group (in fact, this is a firewall , with defined traffic rules, in which the default action is deny ). However, the most important part of the Properties block is User Data . This is where we describe the script that will turn our faceless instance into a Chef server or Chef client.
The template I use is shown below under the cat, consider it and I will comment on it.
')
From the template you can see that we have five parameters declared. Two of them have default values - the type of the created instance (in this case, m1.small) and the subnet of IP addresses from which SSH access to the host will be allowed. When creating a stack, you will need to specify 3 parameters - a key for SSH access to the nodes (created separately in the AWS Console), an access key and a secret key (both are generated when registering an AWS access account).
In the mapping , two types of instance are described, both with a 64-bit architecture. And also, AWSRegionArch2AMI describes virtual image IDs that correspond to instances from Amazon Linux OS (ID data can be obtained from AWS Console).
Next, we describe the resources of the Chef server and the Chef client. In both cases, through the Metadata section, before running commands from the User Data section, wget is installed (just in case, in fact, Amazon Linux images must contain such packages). The resources created are determined by the ImageId and InstanceType variables (in this case, these are the previously specified parameters, in this case Amazon Linux, m1.small, and 64-bit architecture). Next comes the main body of the resource - User Data . It is the body of the configuration bash script, which is executed step by step after our instance is initialized.
In short, for the node that will be the Chef server , the following actions are taken:
I understand that this may seem like a confusing and incomprehensible explanation, but a detailed examination of each part of the script would take a lot of space. Therefore - if you have questions - feel free to write me in the LAN or in the comments.
Go back to our template. For a node that will be a Chef-client , the following actions are taken:
It is worth noting that thanks to such an option as json_attribs , we can create a label for the node that will determine its role in the infrastructure. This is done in the case when there may be nodes that take on various infrastructure roles among Chef clients.
The following resources, WaitHandle and WaitCondition , describe the conditions under which the stack creation process can be suspended. If WaitHandle receives a signal that the process has completed successfully during the timeout specified in WaitCondition, then the stack creation process continues / ends.
The next advertised resource is the Security Group - a firewall for our sites. The group describes the ports forwarded and the source address of the packets.
The last block, Outputs , is used to successfully start stack and instance to get any variables that we are interested in. For example, a domain name in order to gain access to the instance.
As a result, we get a universal template and the ability to "expand" our modest infrastructure (if you are interested in a larger number of instance - use the Auto-Scaling Group) by executing one command in the AWS management console. The result of the launch can be viewed in the CloudFormation section.
What's next? Then you get the ability to control nodes by knife, cookbooks and roles. You can use the community cookbook , write your own, write wrappers to other cookbooks . There are many possibilities and everything depends on the final task.
In this series of articles, I tried, albeit superficially, to describe the process of automating the management of the PC fleet and interaction with AWS cloud resources. Hopefully for beginners DevOps these articles will be interesting.
If you have any questions and suggestions - feel free to write in the dialogues or comments on my articles. Thanks to everyone who took the time to read the articles.
See you soon!
References :
For ease of understanding, we will consider the case of two ec2-instance (Amazon virtual servers), one of which will act as a Chef server, and the second as a node.AWS and Chef
Immediately I will clarify that we will run the instance using AWS CloudFormation . One could, of course, start and manage them manually, but what is the point of such automation?
CloudFormation can be divided into 2 concepts :
- template , which is a json-file, which describes all the resources we need to run the instance;
- stack , which is the AWS resources themselves, described in the template.
For those who are starting on AWS, Amazon provides a ready-made sample template that covers most aspects that are needed when working with AWS. The link to the templates will be given at the end of the article.
Consider what the template is . In the base case, it consists of 4 blocks: Parameters, Mappings, Resources, Outputs .
The Parameters block describes variables and their values that will be passed to the stack at the time it is created. Parameter values can be entered when creating resources, or can be set using the default field in the parameter description. Parameter can be any information, starting from a password and ending with a network port or path to a directory. To get the parameter value, the Ref function is used in the template.
The Mappings block contains a set of keys with the appropriate parameters and their values. Most often, you can see how mapping is used to define AWS regions and their corresponding virtual images (instance). To get the value of a mapping, use the Fn :: FindInMap function, which indicates the key and parameters by which a particular value is searched for.
The Resources block describes our ec2-instance or other AWS resources. It is in this section that the images for the Chef server and client node are declared. In the description, you must specify the type of resource (for example, AWS :: EC2 :: Instance), it is also possible to specify metadata, in which you can specify the description of our node or pre-install procedure directives (for example, if you need to install any package when running the image ). The main part of the block is the Properties block, which describes in detail the image being launched. In this block, you can specify the type of image that will be launched (for example, Amazon Linux 32-bit), the identity of the image being launched to this or that Security Group (in fact, this is a firewall , with defined traffic rules, in which the default action is deny ). However, the most important part of the Properties block is User Data . This is where we describe the script that will turn our faceless instance into a Chef server or Chef client.
The template I use is shown below under the cat, consider it and I will comment on it.
Template
{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "Template for stack", "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances", "Type" : "String", "MinLength" : "1", "MaxLength" : "255", "AllowedPattern" : "[\\x20-\\x7E]*", "ConstraintDescription" : "can contain only ASCII characters." }, "HostKeys" : { "Description" : "Public Key", "Type" : "String" }, "SecretAccessKey" : { "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances", "Type" : "String" }, "InstanceType" : { "Description" : "Chef Server EC2 instance type", "Type" : "String", "Default" : "m1.small", "AllowedValues" : [ "t1.micro","m1.small"], "ConstraintDescription" : "must be a valid EC2 instance type." }, "SSHLocation" : { "Description" : " The IP address range that can be used to SSH to the EC2 instances", "Type": "String", "MinLength": "9", "MaxLength": "18", "Default": "0.0.0.0/0", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})", "ConstraintDescription": "must be a valid IP CIDR range of the form xxxx/x." } }, "Mappings" : { "AWSInstanceType2Arch" : { "t1.micro" : { "Arch" : "64" }, "m1.small" : { "Arch" : "64" } }, "AWSRegionArch2AMI" : { "us-east-1" : { "32" : "ami-d7a18dbe", "64" : "ami-bba18dd2", "64HVM" : "ami-0da96764" }, "us-west-2" : { "32" : "ami-def297ee", "64" : "ami-ccf297fc", "64HVM" : "NOT_YET_SUPPORTED" }, "us-west-1" : { "32" : "ami-923909d7", "64" : "ami-a43909e1", "64HVM" : "NOT_YET_SUPPORTED" } } }, "Resources" : { ChefClient" : { "Type" : "AWS::EC2::Instance", "Metadata" : { "Description" : "Chef Client", "AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "git" : [] } } } } }, "Properties": { "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] }, "InstanceType" : { "Ref" : "InstanceType" }, "SecurityGroups" : [ {"Ref" : "WebServerSecurityGroup"} ], "KeyName" : { "Ref" : "KeyName" }, "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bash -v\n", "yum update -y aws-cfn-bootstrap\n", "function error_exit\n", "{\n", " cfn-signal -e 1 -r \"$1\" '", { "Ref" : "WaitHandle" }, "'\n", " exit 1\n", "}\n", "yum update -y\n", "yum install git -y\n", "/sbin/service iptables stop\n", "/sbin/service ip6tables stop\n", "/sbin/chkconfig iptables off\n", "/sbin/chkconfig iptables off\n", "yum install git -y\n", "/usr/bin/curl -L https://www.opscode.com/chef/install.sh | bash\n", "cd /root/\n", "/usr/bin/git git://github.com/opscode/chef-repo.git\n", "/bin/mkdir -p /root/chef-repo/.chef\n", "/bin/mkdir -p /etc/chef\n", "/bin/mkdir /root/.aws\n", "/bin/touch /root/.aws/config\n", "/bin/echo '[default]' >> /root/.aws/config\n", "/bin/echo 'region = ", {"Ref" : "AWS::Region" }, "' >> /root/.aws/config\n", "/bin/echo 'aws_access_key_id = ", { "Ref" : "HostKeys" }, "' >> /root/.aws/config\n", "/bin/echo 'aws_secret_access_key = ", { "Ref" : "SecretAccessKey" }, "' >> /root/.aws/config\n", "/usr/bin/aws s3 cp s3://storage/admin.pem /root/chef-repo/.chef\n", "/usr/bin/aws s3 cp s3://storage/chef-validator.pem /root/chef-repo/.chef\n", "/usr/bin/aws s3 cp s3://storage/knife.rb /root/chef-repo/.chef\n", "/usr/bin/aws s3 cp s3://storage/client.rb /etc/chef\n", "/usr/bin/aws s3 cp s3://storage/json_attribs.json /etc/chef\n", "/bin/cp -p /root/chef-repo/.chef/chef-validator.pem /etc/chef/validation.pem\n", "/usr/sbin/ntpdate -q 0.europe.pool.ntp.org\n", "/bin/echo '\nchef_server_url \"", { "Ref" : "ChefServerURL" }, "\"' >> /etc/chef/client.rb\n", "/bin/echo '\nchef_server_url \"", { "Ref" : "ChefServerURL" }, "\"' >> /root/chef-repo/.chef/knife.rb\n", "/usr/bin/chef-client\n", "/opt/aws/bin/cfn-signal -e 0 -r \"ChefClient setup complete\" '", { "Ref" : "WaitHandle" }, "'\n" ]]}} } }, "WaitHandle" : { "Type" : "AWS::CloudFormation::WaitConditionHandle" }, "WaitCondition" : { "Type" : "AWS::CloudFormation::WaitCondition", "DependsOn" : "ChefClient", "Properties" : { "Handle" : {"Ref" : "WaitHandle"}, "Timeout" : "1200" } }, "ChefServer" : { "Type" : "AWS::EC2::Instance", "Metadata" : { "Description" : "Bootstrap ChefServer", "AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "wget" : [] } } } } }, "Properties": { "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] }, "InstanceType" : { "Ref" : "InstanceType" }, "SecurityGroups" : [ {"Ref" : "WebServerSecurityGroup"} ], "KeyName" : { "Ref" : "KeyName" }, "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bash\n", "cfn-init --region ", { "Ref" : "AWS::Region" }, " -s ", { "Ref" : "AWS::StackId" }, " -r ChefServer ", " -c orderby ", " --access-key ", { "Ref" : "HostKeys" }, " --secret-key ", {"Ref" : "SecretAccessKey"}, " || error_exit 'Failed to run cfn-init'\n", "yum update -y aws-cfn-bootstrap\n", "function error_exit\n", "{\n", " cfn-signal -e 1 -r \"$1\" '", { "Ref" : "WaitHandle" }, "'\n", " exit 1\n", "}\n", "yum update -y\n", "/sbin/service iptables stop\n", "/sbin/service ip6tables stop\n", "/sbin/chkconfig iptables off\n", "/sbin/chkconfig ip6tables off\n", "#Install ChefServer package\n", "cd /root/\n", "/usr/bin/wget https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.10-1.el6.x86_64.rpm\n", "/bin/rpm -ivh /root/chef-server-11.0.10-1.el6.x86_64.rpm\n", "/usr/bin/wget https://s3.amazonaws.com/storage/default.rb\n", "/bin/cp -f default.rb /opt/chef-server/embedded/cookbooks/runit/recipes/default.rb\n", "#Configure ChefServer\n", "su - -c '/usr/bin/chef-server-ctl reconfigure'\n", "su - -c '/usr/bin/chef-server-ctl restart'\n", "#AWS creds installation\n", "/bin/mkdir /root/.aws\n", "/bin/touch /root/.aws/config\n", "/bin/echo '[default]' >> /root/.aws/config\n", "/bin/echo 'region = ", {"Ref" : "AWS::Region" }, "' >> /root/.aws/config\n", "/bin/echo 'aws_access_key_id = ", { "Ref" : "HostKeys" }, "' >> /root/.aws/config\n", "/bin/echo 'aws_secret_access_key = ", { "Ref" : "SecretAccessKey" }, "' >> /root/.aws/config\n", "#Upload files for client\n", "/usr/bin/aws s3 cp /etc/chef-server/admin.pem s3://storage/\n", "/usr/bin/aws s3 cp /etc/chef-server/chef-validator.pem s3://storage/\n", "#Chef client and dirs for it\n", "/usr/bin/curl -L https://www.opscode.com/chef/install.sh | /bin/bash\n", "/bin/mkdir /root/.chef\n", "/bin/mkdir /etc/chef\n", "/bin/mkdir /etc/chef/cookbooks\n", "/bin/mkdir /etc/chef/roles\n", "#Knife client config files from S3\n", "/bin/cp /etc/chef-server/admin.pem /etc/chef/client.pem\n", "/usr/bin/aws s3 cp s3://storage/knife_admin.rb /root/.chef/knife.rb\n", "#Roles and cookbooks from S3\n", "/usr/bin/aws s3 cp s3://storage/roles/ /etc/chef/roles/ --recursive\n", "/usr/bin/aws s3 cp s3://storage/cookbooks/ /etc/chef/cookbooks/ --recursive\n", "#Cookbooks from community\n", "/usr/bin/knife cookbook site download cron\n", "/usr/bin/knife cookbook site download jenkins\n", "/usr/bin/knife cookbook site download ntp\n", "/usr/sbin/ntpdate -q 0.europe.pool.ntp.org\n", "yum remove ruby -y\n", "yum install ruby19 -y\n", "#Unpack and move cookbooks\n", "/bin/mv /root/*.tar.gz /etc/chef/cookbooks\n", "for i in `/bin/ls /etc/chef/cookbooks/*.tar.gz`; do /bin/tar zxf $i -C /etc/chef/cookbooks/; /bin/rm -f $i; done\n", "for i in `/bin/ls /etc/chef/cookbooks`; do /usr/bin/knife cookbook upload $i; done\n", "#Upload cookbooks and roles\n", "/usr/bin/knife cookbook upload * -c '/root/.chef/knife.rb'\n", "/usr/bin/knife role from file /etc/chef/roles/*.rb\n", "/bin/echo -e \"*/5 * * * * root /usr/bin/knife exec -E 'nodes.find(\\\"!roles:BaseRole\\\") { |n| puts n.run_list.add(\\\"role[BaseRole]\\\"); n.save}' -c '/root/.chef/knife.rb'\" >> /etc/crontab\n", "/bin/echo -e \"*/5 * * * * root /usr/bin/knife exec -E 'nodes.find(\\\"env_role:master AND !roles:master\\\") { |n| puts n.run_list.add(\\\"role[master]\\\"); n.save}' -c '/root/.chef/knife.rb'\" >> /etc/crontab\n", "/bin/echo -e \"*/5 * * * * root /usr/bin/knife exec -E 'nodes.find(\\\"env_role:slave AND !roles:slave\\\") { |n| puts n.run_list.add(\\\"role[slave]\\\"); n.save}' -c '/root/.chef/knife.rb'\" >> /etc/crontab\n", "/opt/aws/bin/cfn-signal -e 0 -r \"ChefServer setup complete\" '", { "Ref" : "WaitHandle" }, "'\n" ]]}} } }, "WaitHandle" : { "Type" : "AWS::CloudFormation::WaitConditionHandle" }, "WaitCondition" : { "Type" : "AWS::CloudFormation::WaitCondition", "DependsOn" : "ChefServer", "Properties" : { "Handle" : {"Ref" : "WaitHandle"}, "Timeout" : "1200" } }, "WebServerSecurityGroup" : { "Type" : "AWS::EC2::SecurityGroup", "Properties" : { "GroupDescription" : "Enable HTTP access via port 80 and SSH access", "SecurityGroupIngress" : [ {"IpProtocol" : "tcp", "FromPort" : "80", "ToPort" : "80", "CidrIp" : "0.0.0.0/0"}, {"IpProtocol" : "tcp", "FromPort" : "8080", "ToPort" : "8080", "CidrIp" : "0.0.0.0/0"}, {"IpProtocol" : "tcp", "FromPort" : "443", "ToPort" : "443", "CidrIp" : "0.0.0.0/0"}, {"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : { "Ref" : "SSHLocation"}} ] } }, } ')
From the template you can see that we have five parameters declared. Two of them have default values - the type of the created instance (in this case, m1.small) and the subnet of IP addresses from which SSH access to the host will be allowed. When creating a stack, you will need to specify 3 parameters - a key for SSH access to the nodes (created separately in the AWS Console), an access key and a secret key (both are generated when registering an AWS access account).
In the mapping , two types of instance are described, both with a 64-bit architecture. And also, AWSRegionArch2AMI describes virtual image IDs that correspond to instances from Amazon Linux OS (ID data can be obtained from AWS Console).
Next, we describe the resources of the Chef server and the Chef client. In both cases, through the Metadata section, before running commands from the User Data section, wget is installed (just in case, in fact, Amazon Linux images must contain such packages). The resources created are determined by the ImageId and InstanceType variables (in this case, these are the previously specified parameters, in this case Amazon Linux, m1.small, and 64-bit architecture). Next comes the main body of the resource - User Data . It is the body of the configuration bash script, which is executed step by step after our instance is initialized.
In short, for the node that will be the Chef server , the following actions are taken:
- disabling iptables (so that there are no problems with drop-ohm packages);
- Installing the Open Source Chef Server;
- the substitution of default.rb (yes, the bug of Amazon Linux images is that they are presented not as redhat, which is in essence, but as amazon, and the Chef-server service cannot fully work);
- auto configuration and server restart;
- creating a configuration file for the AWS console;
- uploading admin.pem and chef-validator.pem files to the repository (we will need them on the client);
- installing chef-client on the node (yes, the Chef server does not have its own knife);
- getting the client.pem and knife.rb files for the knife running on the server (such a starter-kit from the first part of the article);
- Recreation of the chef-repo directory structure in which our cookbooks, role files, etc. are stored;
- download and install cookbooks to the server (our cookbooks are in the storage, and some are taken from the community);
- Download and install roles on the server;
- adding a periodic task that will poll every node every 5 minutes and establish a base role for each new one.
I understand that this may seem like a confusing and incomprehensible explanation, but a detailed examination of each part of the script would take a lot of space. Therefore - if you have questions - feel free to write me in the LAN or in the comments.
Go back to our template. For a node that will be a Chef-client , the following actions are taken:
- disabling iptables (so that there are no problems with drop-ohm packages);
- Install Chef Client;
- creating a configuration file for the AWS console;
- Downloading client configuration files from the storage;
- adding the address of the Chef server to the client’s configuration (it may change even when the images are restarted, or when the stack is started again);
- adding a periodic task to run the Chef client.
It is worth noting that thanks to such an option as json_attribs , we can create a label for the node that will determine its role in the infrastructure. This is done in the case when there may be nodes that take on various infrastructure roles among Chef clients.
The following resources, WaitHandle and WaitCondition , describe the conditions under which the stack creation process can be suspended. If WaitHandle receives a signal that the process has completed successfully during the timeout specified in WaitCondition, then the stack creation process continues / ends.
The next advertised resource is the Security Group - a firewall for our sites. The group describes the ports forwarded and the source address of the packets.
The last block, Outputs , is used to successfully start stack and instance to get any variables that we are interested in. For example, a domain name in order to gain access to the instance.
As a result, we get a universal template and the ability to "expand" our modest infrastructure (if you are interested in a larger number of instance - use the Auto-Scaling Group) by executing one command in the AWS management console. The result of the launch can be viewed in the CloudFormation section.
What's next? Then you get the ability to control nodes by knife, cookbooks and roles. You can use the community cookbook , write your own, write wrappers to other cookbooks . There are many possibilities and everything depends on the final task.
In this series of articles, I tried, albeit superficially, to describe the process of automating the management of the PC fleet and interaction with AWS cloud resources. Hopefully for beginners DevOps these articles will be interesting.
If you have any questions and suggestions - feel free to write in the dialogues or comments on my articles. Thanks to everyone who took the time to read the articles.
See you soon!
References :
- AWS Documentation - aws.amazon.com/documentation
- AWS CloudFormation - aws.amazon.com/documentation/cloudformation
- AWS EC2 - aws.amazon.com/documentation/ec2
- AWS Sample Templates - aws.amazon.com/cloudformation/aws-cloudformation-templates
- AWS Console - docs.aws.amazon.com/awsconsolehelpdocs/latest/gsg/getting-started.html
Source: https://habr.com/ru/post/211050/
All Articles