Geekly Articles each Day
📜 ⬆️ ⬇️

Understanding the xCAT server deployment and maintenance system

In continuation of the theme of our previous article , today we will talk about the tool that we use every day.

If your activity is somehow connected with the installation and configuration of large quantities of equipment, then the material, in our opinion, will be useful and interesting for you.

Any engineer who manages Badoo's fleet of servers, the most popular dating network on the Internet, does not even want to remember that you had to manually edit a DHCP server configuration, collect images for network PXE loading ... We want to tell you where and how Our company has successfully used the xCAT software solution.
')
First of all, let us designate the initial conditions within which we will describe the situation:
Returning to our first article, which we also recommend to read , let's stop at the place where we need to prepare the configuration of the DHCP server, or rather, add information about new servers to it. From this moment we begin to actively use xCAT.

xCAT is a server park management system that has the following features:
The biggest advantage of the xCAT system is that it is not a completely new product, but in some ways a synthesis of all previously known, i.e. during operation, the engineer does not have to learn the new syntax, which undoubtedly affects the speed of mastering the program.

Briefly describe the process of initial configuration of the product .
  1. We go to the project site, select the appropriate distribution, we get a description of the installation process.
  2. Install the required packages.
  3. Install directly xCAT-server.
Then proceed to configure .

The main commands to control (add, change, check input): tabedit and tabdump

Rule the main system configuration file:
tabedit site

Key points to pay attention to:

"installdir","/install" is the directory where xCAT will add all system images for installation, and various post scripts are located here. In the case of an http installation of the system, the directory will be accessible via the web interface. Also there will be templates for automatic installation of the system.
"ipmiretries","3" - attempts to connect to the server management interface.
"ipmitimeout","2" - no comments.
"master","master_server_name" is the name of the master server that should be resolved, and you should also have it in the / etc / hosts file .
"tftpdir","/tftpboot" is the path to the directory where the files required for PXE boot will be located, also the TFTP server should have access to the directory, because it will be the one that will distribute the pxe-boot images.
"xcatconfdir","/etc/xcat" - path to the directory with xCAT configuration files.
"timezone","GMT" - time zone.
"useNmapfromMN","yes" - use nmap on the master server to display the status of hosts (including nodestat hostname)
"dhcpinterfaces","ethN,!remote!" - interfaces on which the DHCP server responds to DHCP requests. (The first is the physical interface, the second is for the VLAN, where our server is registered as the issuing address)
"nameservers","1.1.1.1" - addresses of DNS servers, where after issuing a lease our server will try to send information about the PTR record for the host.

This configuration is quite enough for familiarization, as well as for a test run of the xCAT server.

Now we try to add access to an unprivileged user.

For this you need:
- presence of the user in the system;
- issue and sign certificates on xCAT-server
/opt/xcat/share/xcat/scripts/setup-local-client.sh username

We get the $ HOME / .xcat directory , with the following content:
ca.pem
client-cert.pem
client-cred.pem
client-key.pem
client-req.pem

- add user information to service tables xCAT. In this case, we give the user maximum authority, for this we will execute
tabedit policy

and add a line like this:
username,allow

After the actions we have done, the user gets access to manage the xCAT server.

Next, to create an installation repository, we need to have images of the operating systems that we are going to use.

XCAT includes the copycds utility, which works in the following scenario:
copycds [{-n|--name|--osver}=distroname] [{-a|--arch}=architecture] 1st.iso [2nd.iso …]

Thus, if we have an installation DVD image, then simply execute
copycds PATH_TO_ISO SLES-11-DVD-x86_64-GM-DVD1.iso

xCAT will automatically detect the version of the operating system (if not, --osver ), then copy everything to a local disk (in our case, / install / sles11 )

xCat also has a set of configurations for a standard installation, which are located in the / opt / xcat / share / xcat / install / directory , but are of no interest to us, so we create our installation profile and locate it along the way
/install/custom/install/DISTRNAME/Templatename.tmpl .

Note: in the case of SuSe, Linux is an xml file that YaST will help us to do (although it is easier to write it yourself, having familiarized yourself with the structure and composition of the parameters on the manufacturer's website).

As soon as the typical installation template is ready, it remains only to add new machines and start installing the OS , which will be discussed below.

For the correct installation procedure, we will need information:
1) about the servers on which we are going to install the OS,
2) the hostname of each server,
3) about the subnets that we plan to use,
4) compliance with hostame-ip,
5) about the details of access to the server management interface.

Returning to our previous article , we remember that all the necessary information we have, it remains only to add it to xCAT, which we demonstrate:

1. Add server groups to the system:
nodeadd depl[1-200] groups=depl

2. Add information about server management interfaces (suppose they are named like this: depl [1-200] ipmi ):
nodeadd depl[1-200]ipmi groups=depl_ipmi

3. We talked about the fact that we have a subnet (VLAN) in which all the new equipment is located. We describe subnets as follows:

#netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers,logservers,dynamicrange,nodehostname,comments,disable

tabedit networks

"depl_vlan","2.2.2.0","255.255.255.0","!remote!","2.2.2.1",,"1.1.1.1","5.5.5.5","6.6.6.6",,"2.2.2.200-2.2.2.254",,,
"depl_vlan_ipmi","3.3.3.0","255.255.255.0","!remote!","3.3.3.1",,"1.1.1.1","5.5.5.5","6.6.6.6",,"3.3.3.200-3.3.3.3.254",,,

Note: because xCat configures the DHCP server, and stores information about fixed addresses in a file with leases , it is advisable to use fixed addresses of hosts that do not fall within the dynamic range in order to avoid possible problems with the loss of servers in the subnets.

4. Add information about the MAC addresses of the main server interfaces, as well as management interfaces:

#node,interface,mac,comments,disable

tabedit mac

"depl1","eth0","MAC1",,
"depl2","eth0","MAC2",,
"depl3","eth0","MAC3",,
"depl1ipmi","eth0","ipmi_MAC1",,
"depl2ipmi","eth0","ipmi_MAC2”,,
"depl3ipmi","eth0","ipmi_MAC3",,
...

Hint: in the last article we talked about the nodes file, with which we work closely, we will add information about MAC addresses to it, then it will be enough to execute

cat nodes | awk {'print "\""$1"\",\"eth0\",\""$15"\""'}

and get the output records of the form

"depl1","eth0","MAC1",,

The same for management interfaces:

cat nodes | awk {'print "\""$1"manage\",\"eth0\",\""$10"\""'}

"depl1ipmi","eth0","ipmi_MAC1",,

5. Add information about hostame - ip compliance:

#node,ip,hostnames,otherinterfaces,comments,disable

tabedit hosts

"depl1","2.2.2.2","depl1","depl1ipmi:3.3.3.2",,
"depl2","2.2.2.3","depl2","depl2ipmi:3.3.3.3",,
"depl3","2.2.2.4","depl3","depl3ipmi:3.3.3.4",,
...

We can generate these settings using a simple command:
for i in `seq 1 200`; do echo '"depl$i'","2.2.2.'$[$i+1]'","depl'$i'","depl'$i'manage:3.3.3.'$[$i+1]'",,'; done

6. Edit the options for virtual consoles and their speeds (here we can use all new machines to belong to the same group: in the end we only need one line instead of two hundred.):

#node,power,mgt,cons,termserver,termport,conserver,serialport,serialspeed,serialflow,getmac,comments,disable

tabedit nodehm

"testgroup","ipmi","ipmi",,,,"1","1","115200",,,,

7. We rule the noderes table:

#node,servicenode,netboot,tftpserver,nfsserver,monserver,nfsdir,installnic,primarynic,discoverynics,cmdinterface,xcatmaster,current_osimage,next_osimage,nimserver,comments,disable

tabedit noderes

"depl",,"pxe","1.1.1.1","1.1.1.1",,,,,,,,,,,,

8. Add information about the type of our servers, as well as the method of installing the OS:

#node,os,arch,profile,provmethod,supportedarchs,nodetype,comments,disable

tabedit nodetype

"depl","sles11.1","x86_64","Templatename","install",,"osi",,

In this case, we use the install method — system installation; you can use netboot — for network booting, as well as for performing service procedures.

9. Add information about the compliance of the main server interface and its management interface:

#node,bmc,bmcport,username,password,comments,disable

tabedit ipmi

"depl","/\z/ipmi/","0",”ipmi_username”,"ipmi_password",,

Here we also cheat and we will not indicate separately the correspondence for each server, but use the group. Considering that the naming of the management interface is nothing more than the addition of ipmi to the hostname of the server, this is exactly what we need. And we take into account that the username & password for IPMI is the same for all servers, which is also indicated in the table.

10. Now we generate the configuration of the / etc / hosts file :
makehosts

11. Create a configuration for the DHCP server:
makedhcp -a -n

You can view information on any node:
lsdef nodename

Now the server is ready to start the installation process.

12. We carry out:
nodeset depl[1-10] install

Instead of depl [1-10] it is allowed to use the group name or comma-separated list.
At this stage, we get the configuration for PXE-loading on each host in /tftpboot/pxelinux.cfg/ and a copy of the auto-installation file in / install / autoinst / .

13. We give the server a command to boot from the network interface at the next boot.
rsetboot depl[1-10] net

14. We send the server to reboot.
rpower depl[1-10] boot

Then we can observe the installation process in server consoles, or configure the conserver (this feature is not considered in this article).

Last of all, we collect an image for network boot . In Badoo, images are used to diagnose, restore and “revitalize” problem equipment, prepare non-standard hardware, and conduct performance tests.

To do this, execute the genimage command — it will launch an uncomplicated wizard, by answering whose questions we will get an image of the system in
/install/netboot/$DISTR/$ARCH/profile_name

The resulting image can be changed and edited at its discretion, after which you should run
packimage -o $DISTR -p profile_name -a $ARCH

So we get a ready-made netboot image to which we send our servers:

nodeset depl[11-200] netboot=$DISTR-$ARCH-profile_name

rsetboot depl[11-200] net

rpower depl[11-200] boot

It is advisable to put public keys into the netboot image, but you can use the access details from the passwd table ( tabedit passwd )

After all the preparatory procedures on the servers are completed, we start the OS installation on them - we will get the result in 15 minutes.

It remains only to place all our servers in the required subnets, after which they pass through Puppet , which installs the necessary software individually, in groups, and also sets the necessary system parameters.

In conclusion, I would like to note that in the presence of a system of this kind, it is enough for an engineer to give the necessary commands, making a minimum of effort to configure the servers.

Of course, today we told far from all the advantages of the xCAT system and the intricacies of setting up, such moments as the management of hypervisors, as well as virtual servers on them through xCAT, were left out of the article; installation and management of the xCAT server via a web interface, access to server consoles from the management server.

We will be happy to continue the cycle of articles on the above topics, if they cause your interest, which you can always tell us in the comments.

Source: https://habr.com/ru/post/134228/

More articles:


All Articles