Geekly Articles each Day
📜 ⬆️ ⬇️

CodingFuture + Puppet. Part IV: Centralized Management (cftotalcontrol)

In short:


  1. cftotalcontrol is a module for creating a special account environment with SSH access to other infrastructure sites and performing mass routine tasks in parallel.
  2. Abstracted from the implementation of the theory of the organization of safe and controlled access to infrastructure.
  3. Describes the practical application of a specific solution.


Thematic cycle:



The concept of organizing access to infrastructure


Of course, this article is not an exhaustive guide. However, for clarity, we consider a classic example of admin access to servers, in which each admin has a private key and direct access to any server controlled by him:


Classic example of access organization


This approach has certain disadvantages:


  1. The need to prescribe the public key of each admin on each server.
    • Writing new keys through Puppet is not a problem, but the actual configuration change of all available systems is required.
  2. Necessity to organize through access to each system from any location of the administrator.
    • Often, SSH Agent Forwarding is dangerous for this.
    • Sometimes you can find a completely non-flexible solution through DNAT.
    • Prescribing all admin IPs and / or opening ports on the knock on each system is also a somewhat dubious decision.
  3. Each admin has to manually contain a shared or own SSH config and a set of scripts for routine tasks, and also to update them in time. This is guaranteed to lead to errors.
  4. With the growing number of systems, the audit of the organization of access in the infrastructure becomes a non-trivial task.
  5. Directly centralized management (conducting) is not solved.
    • Of course, there are various solutions based on the event model, but their safety raises some doubts. Puppet itself has MCollective for this.

For comparison, consider access with a conditional infrastructure administrator in the form of a domain group:


Access via domain group


This approach is called differently, but somehow it has long existed in integrated commercial solutions. Usually this is implemented through a certain domain group of privileged users, which has access to all or part of infrastructure systems. This approach has some (2, 3 and 5) drawbacks of the previous one, plus an obvious nuance - the need for the constant availability of authentication and authorization services (of course, not without caching). By the way, it is not too difficult to organize something similar with the help of PAM and LDAP.


So, we quickly got to the core of the implemented access concept in cftotalcontrol :


Access scheme via cftotalcontrol


In essence, this approach combines the two previous ones:


  1. For access is used exclusively SSH without any additional services. Puppet is used only when authorization keys are distributed.
  2. A management user is introduced - a special account on highly secure infrastructure nodes to which admins connect. Abstractly, such an account is a group of admins.
  3. The system itself configures the intermediate nodes and the SSH client of the managing user. The systems have a minimum number of authorization keys, which scales well with the growth of both the number of nodes and the number of administrators.
  4. At the workplace admin does not require any configuration, except for connecting to the control account.
  5. Fully controlled account allows you to easily set up an audit of admin actions with logging. Especially useful for low privileged groups.
  6. There are ready-made solutions for conducting a large number of systems:
    • You can run the command on all available nodes.
    • You can define nodes in groups statically or dynamically upon request in PuppetDB.
    • You can run commands sequentially interactively.
    • It is possible to run commands in parallel with the limitation of the maximum number of parallel connections and receive the output of the command upon completion.
  7. For routine tasks, template commands are created for each node, for a group of nodes, and for all nodes. The list of standard commands includes:
    • force puppet deployment
    • updating meta-information from package repositories.
    • system update.
    • remove old unused packages.
    • any arbitrary commands specified in the configuration.
  8. It is possible to create low-privileged control users ( Scoped Control Users ) instead of privileged "total" controls ( Total Control Users ).
    • such users will have access only to those systems that are explicitly marked as belonging to this area ( Control Scope ).

Installation


This implies that the infrastructure is built on the previously considered cfnetwork , cfauth and cfpuppetserver , but cftotalcontrol will live perfectly without conflicts and in a mixed environment.


First of all, in order to sleep peacefully, it is better to have at least one SSH key, which will be registered on all systems through the cfauth module, but which will not be used for "domestic" needs. Of course, you should not forget about access to the hardware via IPMI and to virtual machines via the serial / GUI console for an emergency.


Secondly, just look at the example and:


  1. add cftotalcontrol module to Puppetfile
     mod 'codingfuture/cftotalcontrol'
  2. to the classes list of all nodes ( common.yaml file)
     classes: #  ,   ""  - cftotalcontrol::auth
  3. to the classes list of the node configuration with the control user (hereinafter referred to as the "control node")
     classes: #    - cftotalcontrol
  4. you need to run the Puppet deployment ( /opt/puppetlabs/bin/puppet agent --test ) in the following order:
    1. Deploy Puppet on a management node — create a management user and its environment.
    2. Deploy Puppet on all other nodes — this will register the system in PuppetDB to automatically generate the configuration of the controlling user.
    3. Log in as a manager user and generate a private key (it will be offered automatically). After generation, the deployment of Puppet will be automatically triggered to export the public key to PuppetDB.
    4. Deploy Puppet on all systems once again - the public key of the managing user will be written everywhere, and the user will create a configuration with access to all nodes.
    5. This sequence of actions is required only during the initial installation. Further, the key update is automated when generating a new one.

After these simple manipulations, a cftotalcontrol will be created on the nodes with the cftotalcontrol class with a specially sharpened Bash environment and SSH client configuration. It is strongly recommended to change the username to a non-standard user through the parameter cftotalcontrol::control_user .


All standard admin keys from cfauth will be cfauth to this user, but in order to organize ideologically correct access, admin keys must be registered only in cftotalcontrol::ssh_auth_keys . Of course, you should first check that everything works and there is access to all servers.


SSH Proxy Access


In a large infrastructure, almost always only a small part of the systems looks to the outside world and through them it is required to organize access to the other systems.


In an SSH client, this is easily implemented through the ProxyCommand option with the value ssh -W target:port proxyhost . This approach is more secure than using SSH Agent Forwarding, which allows you to steal the admin's private key on a remote system. In principle, knowledge of these technical details is not necessary to use the cftotalcontrol module, since all necessary configuration is generated based on the cftotalcontrol::pool_proxy .


Of course, the enumeration of the proxy node for each target node is not only inefficient, but also necessarily leads to frequent errors when adding new systems. Instead, the cf_location and cf_location_pool facts described in the previous sections are used for "navigation". Usually these facts are set upon initial installation by the client initialization script cf_gen_puppet_client_init , but can be changed through the parameters of the cfsystem module.


When generating the SSH client config, the proxy nodes are searched in the following order:



Of course, the whole thing is integrated with cfnetwork and without unnecessary gestures adds everything that is needed in the network filter configuration.


Node groups


Typically, specific tasks of the same type need to be run only on nodes with certain attributes (OS, set of services, role, etc.). In order not to fence your garden by segregation of nodes, this is easily done through the parameter cftotalcontrol::host_groups , which is an associative array, where the key is the name of the group, and the value can be a hard-coded list of node or string in the form of a Puppet DB query There is an opportunity to read in the description of the puppetdbquery module.


Example:


 cftotalcontrol::host_groups: puppetserver: "Package['puppetserver']" infra: "cf_location_pool = 'infra'" custom: - 'web.example.com' - 'db.example.com'

Standard Routine Commands


It should be clear without further explanation:


 cftotalcontrol::standard_commands: helloworld: 'echo "Hello world!"' gethostname: 'hostname --fqdn'

Familiarity with the centralized management environment


Here begins the virtual post admin. When you first log in, the system will kindly offer to generate a private key, and on subsequent runs, launch the SSH Agent and add a private key to it, asking for its password along the way.


But so that life does not seem honey, the system will remind you to update your private key if it is older than cftotalcontrol::ssh_old_key_days = 180 days. If you don’t even log in, cron will do the same, angrily sending out daily letters. You can generate a new key using the cftc_gen_key command, which also automatically installs it on all monitored nodes, without requiring other actions. If something goes wrong, the old key can always be found in the ~/.ssh/ with the UNIX timestamp extension in seconds at the time the new key is generated. This may occur due to a connection error, Puppet configuration, or a banal offline-controlled system.


In general, the following is happening under the hood:



Utility bash environment commands:



Directly work with nodes:



Significant environment variables:



The standard commands are the same as those added to cfauth for calling through sudo without a password:



Creating Admins with Restricted Access


Sooner or later, certain admin powers need to be given to people. whose area of ​​responsibility is not the administration of the entire infrastructure. These include DevOps, DBA, Release Manager, and others. Some clustering tools also rely on SSH access to fellow nodes, which is conveniently configured by the same module, including automatic generation of a private key without a password.


In short, the standard way to create unprivileged admins is through the parameter cftotalcontrol::extra_users , which correctly creates resources like cftotalcontrol::admin . Specific differences from the user of "total" control:



There remains only the question of how to designate the target node. This is done via the cftotalcontrol::auth::control_scope . You can hang several spheres on the same node at once. Example:


 cftotalcontrol::auth::control_scope: - web - devops

cftotalcontrol module cftotalcontrol


cftotalcontrol class



class cftotalcontrol::auth



type cftotalcontrol::admin


Usually initialized via cftotalcontrol::extra_users . All parameters are the same as for cftotalcontrol , except for extra_users itself.



')

Source: https://habr.com/ru/post/279375/

More articles:


All Articles