If you are seriously using AWS (Amazon Web Services), then you probably know about the ability to describe the infrastructure using JSON templates. In AWS, this service is called CloudFormation . In essence, this solution allows you to describe the desired state of any resources available in AWS (instances, opsworks layers, ELB, security groups, etc.). A collection of resources is called a stack. After loading the CloudFormation template, the system will either create the necessary resources on the stack itself, if they are not already there, or try to update the existing ones to the desired state.

This works well if you have a small amount of resources, but as the infrastructure grows, problems arise:

• In JSON, it is not possible to use cycles and for similar resources you have to repeat the same parameters even in case of a change (not DRY)
• For escaping cloud-init , double escaping is needed
• There are no comments in JSON and it has poor human readability.

In order to avoid such problems, engineers from Heavy Water wrote on ruby ​​DSL and CLI for generating and working with these templates called SparkleFormation ( github ).

DRY

When I came to my current project we had a CloudFormation template containing about 1500 lines of resources description and about 0 lines of comments. After using SparkleFormation, the template began to occupy 300 lines, many of which are comments. How did we do it? First, let's look at how CloudFormation works, a typical resource description looks like this:

Since SparkleFormation allows you to use normal ruby ​​code inside DSL, you can rewrite it like this:

As you can see, we are no longer repeated in the description of each port and adding a new one will take us only one line. Moreover, if we need to create many almost of the same type resources, but differing in 1-2 parameters, SparkleFormation provides such an entity as dynamics, where you can describe an abstract resource, to which parameters are passed:

Thus, we can reuse the resources we need and, if necessary, change everything in one place.
')

Cloud init

We often take the opportunity to transfer an instance when loading a cloud-init config as a yaml file and use it to install packages, configure CoreOS, individual services and other settings. The problem is that yaml should pass the instance to the user-data in the CloudFormation template and it looked like this:

As you can see, this is absolutely unreadable, ugly and poorly supported, not to mention the fact that syntax highlighting can be forgotten. Due to the fact that ruby ​​code can be used inside DSL, then the entire yaml can be put into a separate file and simply called:

 user_data Base64.encode64(IO.read('files/cloud-init.yml')) 

As you can see, this is much nicer than editing it inside JSON. Instead of using IO.read, you can use an HTTP call for any parameters, if you need it.

CLI

In our project, we use our own wrapper for managing templates, but the same command provides the CLI (Command Line Interface) for managing templates, called sfn . With it, you can load, delete and update CloudFormation stacks, with the sfn create, sfn destroy and sfn update commands. There is also implemented integration with knife.

In general, after 4 months of using SparkleFormation, I am satisfied with it and I hope no longer return to plain JSON to describe the infrastructure. Plans to try the entire workflow, including sfn, offered by the Heavy Water team.