Berkshelf and Chef cookbook dependencies

Hi, Hub users!
I continue my immersion in the piquancy of automation and configuration management , in parallel trying to share my experience with the community .

It will be discussed again about the automation tool for resolving dependencies of the Chef cookbooks that our company uses, namely, Berkshelf .

And what have Berkshelf?

Chef has a fairly large and growing community, which constantly makes its contribution to the creation and editing of cookbooks . All of them are stored on the Community site, many of them are very often used by us.
')
However, there is one “but” in our company associated with editing the community cookbooks . The correct way to make any changes to those related to the specifics of the company's infrastructure is to write wrapper- containing changes (for example, reassigning attributes, changing recipes, etc.). In short, the process of writing a wrapper is described in this article .

But, having the right path does not mean that everyone will follow it. That is why in the repository of the cookbooks of our company there turned out to be a lot of community cookbooks with unfair edits. And it's time to clean the repository :)

The plan was as follows:

- select the community cookbook and in our repository;
- assess the presence and number of local edits ( diff between identical versions of cookbooks );
- make changes to the wrapper and include a call to the community cookbook or recipe from it;
- remove the community cookbook and add it to the wrapper dependencies.

And what to do with the "cleaned" community cookbooks ? How after removal - will they appear “clean” on the Chef Server ?

Just here Berkshelf comes to the rescue .

What is Berkshelf and what is it eaten with?

Berkshelf is a dependency manager for Chef cookbooks . It is written in Ruby and has methods and APIs for interacting with the Chef Server .
The Berkshelf installation comes from Ruby Gems either through the use of Chef DK .
We are using Berkshelf version 2.0 , however, version 3.0 was recently released, which brought some changes and “buns”.
If you decide to use Ruby Gem - I advise you to install it within the framework of Ruby installed by Chef Server (the executable files are on the path - / opt / chef / embedded / bin / ).
To interact with Chef Server, you must configure Berkshelf, indicating to it the address of our server and certificates for authorization on it. To configure, you must run the following command:

berks configure 

The configuration files are in one of the following paths:

 $PWD/.berkshelf/config.json $PWD/berkshelf/config.json $PWD/berkshelf-config.json $PWD/config.json ~/.berkshelf/config.json 

After you have correctly installed Berkshelf , you can proceed to resolving dependencies for cookbooks .
For this, Berkshelf first copies the cookbook and their dependencies to their shelf (local storage / repository Berkshelf , the default is ~ / .berkshelf / cookbooks / ), and then uploads everything to the Chef Server .
Immediately there are questions: " How does Berkshelf know about dependencies? ", " What does he do to resolve them? "
Berkshelf instructions are contained in the Berksfile , located in the root directory of the cookbook . This file is created using the command

 berks init 

in the root directory.
The contents of this file describe the dependencies and the source for resolving them. An example of such a file:

 site :opscode metadata cookbook 'my-cookbook', (:path | :git | :github) cookbook 'my-book-2', ('> 1.0.0') 

This file is prescribed as follows:
- take into account the dependencies from the metadata.rb file of our cookbook and download them from the Opscode Community site;
- my-cookbook will be loaded along the path specified in brackets (this can be either a local path or a link to Git );
- The my-book-2 of the latest version, higher than 1.0.0, will be downloaded from the Opscode Community site.

After that, you can run

 berks install 

and wait for the successful copying of dependencies to the Berkshelf local storage. As a result, the storage directory should contain all the cookbooks, and those mentioned in the depends field of the metadata.rb file, their dependencies (dependencies), and the two cookbooks mentioned in the Berksfile . You can verify the result with the command

 berks shelf list 

The next step is to launch

 berks upload 

which will download everything from local storage to the Chef Server . As a result (provided that the Chef Server is available and we can log in using the certificate files) - all dependencies will be resolved. The result of the download can be checked with the command

 knife cookbook list 

the output of which should contain new cookbooks -and

Essentially, this is the whole process of basic use of Berkshelf . Of course, this is not all the functionality, because Berkshelf interaction with Vagrant , Chef Solo , Chef Client , and also some innovations of version 3.0 are not mentioned. However, I think this is enough for most.

Our history of using Berkshelf

Everything would be fine if again not one “but” - Berkshelf does not know how to work cyclically. I am very surprised, because I did not find an option in which he could " natively " take into account the nesting of directories with a cookbook , they are a kind of nested mode. What I mean?
For example, imagine a typical situation, there is a chef-repo directory in which the cookbooks directory is embedded, which contains our cookbooks ( ./chef-repo/cookbooks/ ).
In the current version of Berkshelf - it is necessary to go to each of the cookbook directories and execute commands related to Berkshelf in it . That is - a bash script that would do it, not with your own hands, in fact ... This is a solution, but alas - not the best, frankly speaking - “a crutch”.
On the Internet, an article was found in which using Ruby-code, this issue was solved.
I will give the code of our " root " Berksfile, located in the directory ./chef-repo/ :

 site :opscode metadata def dependencies(path) berks = "#{path}/Berksfile" instance_eval(File.read(berks)) if File.exists?(berks) end Dir.glob('./cookbooks/*').each do |path| dependencies path cookbook File.basename(path), :path => path end cookbook 'gecode', '= 2.1.0' 

In fact, this piece of code is going to the contents of all the directories of cookbooks (their metadata.rb and Berksfile ) in the “ root ” Berksfile.

We successfully applied the proposed solution and voila - all the cookbooks were on our Chef Server .
However, in the mentioned article one big feature was not mentioned - the Berksfile format of our cookbooks .
Experimentally, it was found that they should look like this:

 group :name_of_cookbook do cookbook 'my-cookbook', (:path | :git | :github) cookbook 'my-book-2', ('> 1.0.0') end 

That is, a list of cookbooks that must meet certain conditions — for example, a version or location that is enclosed in a group (so that the same dependencies of different cookbooks do not cause multiple entry conflicts). All other dependencies are taken from the file metadata.rb . Under the cut - a specific example, so that it was clearer:

As a result, for the cookbook database, 4 cookbooks will be downloaded that meet certain conditions, as well as all the other dependencies mentioned in metadata.rb , as well as the dependencies mentioned (for excuse the tautology).
All that remains to be done after editing the Berksfiles of our cookbooks is to run the commands in the ./chef-repo directory .

 berks install && berks upload 

Here is a solution, perhaps not very reliable or convenient - but a solution. Worked with our repositories several times successfully.
If someone came across this issue and can share experience - write comments or PM.

Thank you all for your attention, to new articles.

PS A colleague says that Berkshelf v.3.0 is broken, so we do not recommend it!