Geekly Articles each Day
📜 ⬆️ ⬇️

FreeBSD auto configuration

Good time, Habrovchane!



I want to share my experience in automating the installation and configuration of FreeBSD with sh (bash). Here is how it was:
One day, the company needed to raise several servers on FreeBSD. Putting one, followed by the second and third axis, my colleague and I (there are only two IT specialists in the staff) thought towards automating this process by writing a script that performs setting up a freshly installed OS. The task of writing fell on my shoulders. A colleague took up the issue of an automatic installation, which I will discuss in another post. So let's get started!

At once I’ll clarify: the script was written exclusively for myself, therefore some code is not designed properly or simply is “not perfect”. But, nevertheless, it works!
The script consists of 3 files: a loader file, a main executable file and a file with libraries. I'll start in order.

loader.sh

#!/bin/sh lib1='parser.sh' #CORE FILE lib2='parserlib.sh' #LIB FILE #Defining variables --begin-- FTPServer='ftp://10.10.10.50' #FTP server address FileName='tarball.tar.gz' #Default tarball name LongVersion=`uname -r` #Long name of current OS version ShortVersion=${LongVersion%%-*} #Calculatig short version of OS, like "8.3" (DON'T TOUCH THIS LINE, please!) LocalDirectory='/tmp/script/' #Temporary directory. You can manualy clean it later. If it's not exist - just relax, script will create it for you :-) DataDirectory='/var/parser-md5-list/' #Directory with datafiles LockFile='EditLocker.p' #Lock file. Needed to prevent changes in files by script after manual editing. EditLock="$DataDirectory$LockFile" #Absolute LockFile path forcer='-no-force' #Variable must be not empty in case if -f key was not specified #Defining variables --end-- usage (){ echo "Only acceptable options: -f to force rebuild without check -s to force execution without restart -d to delete locker file (needed if files was edited manyally or by system, but now you want to rebuild it) -v <version> to manually specify FreeBSD version. Please be sure to specify it correctly! Example: ./loader.sh -v 9.1 " } #Defining options --begin-- if [ $# -ge 1 ] then while getopts fsdv: opt; do case "$opt" in f) forcer='-f';; s) skip='-s';; d) DL='-d';; v) ShortVersion="$OPTARG";; [?]) usage; exit 1;; esac done fi #Defining options --end-- echo "FreeBSD configuration tool V 1.0" echo "Detected OS version: $LongVersion" echo "Applying tarball for version $ShortVersion" echo; echo "Downloading files..." if [ "$DL" == "-d" ] then touch $EditLock #If file does not exists - this line will create it (needed to avoid error message from rm) rm $EditLock #Remove Lock file fi if ! [ "$skip" == "-s" ] then if ! [ -e run.sh ] then fetch "$FTPServer/FreeBSD/loader.sh" touch run.sh echo "#!/bin/sh clear echo \"*********************************************** * Loader file was updated. Process restarted. * *********************************************** \" " >> run.sh echo "./loader.sh $@" >> run.sh chmod +x run.sh ./run.sh exit 1 else rm run.sh fi fi fetch "$FTPServer/FreeBSD/$lib1" fetch "$FTPServer/FreeBSD/$lib2" chmod +x $lib1 chmod +x $lib2 . $lib1 #Attaching core file to process if [ $? -ne 0 ] #Checking for errors then echo "ERROR! Library $lib1 not found!" #Core file does not exist. exit 1 fi LetItGo $FTPServer $FileName $ShortVersion $LocalDirectory $forcer $DataDirectory $EditLock

')
parser.sh

 #!/bin/sh LetItGo() #Body of script { lib1='parserlib.sh' #Lib file . $lib1 #Attaching lib file to process if [ $? -ne 0 ] #Checking for errors then echo "ERROR! Library $lib1 not found!" #Lib file does not exist. kill $$ exit 1 fi #Defining variables --begin-- server=$1 #FTP server address file=$2 #Default tarball name ver=$3 #Version of OS, like "8.3" LocalDir=$4 #Temporary directory. You can manually clean it later. If it's not exist - just relax, script will create it for you :-) DataDirectory=$6 #Directory with data files EditLock=$7 #This file needed to prevent overriding for manually edited files. #Defining variables --end-- cdOrCreate $DataDirectory #Creating data directory if not exists cdOrCreate $LocalDir #Enter temporary directory for file downloads dirchek=`pwd`/ if [ "$dirchek" == "$LocalDir" ] #Checking current directory then rm -rf * #If script successfully entered the temp directory - it will be cleaned else echo "$LocalDir is not accesible! Please check permissions!" kill $$ exit 1 fi fetch "$server/FreeBSD/$ver/$file" #Download tarball hshchk=$(md5 $file) HashCheck ${hshchk##* } $LocalDir $file $5 $DataDirectory cd $LocalDir echo "Extracting files" tar -zxf $file #Unpack it echo "DONE! " rm $file #Remove tarball echo "Tarball was removed from local server." touch $EditLock echo "Lockfile created: $EditLock" for f in $( find $LocalDir ); do #Proceed all files one by one if [ -f $f ] then #check file for manual changes NEWFILE=${f#$LocalDir} NEWFILE=/${NEWFILE#*/} HCK=$(md5 $f) HCK=${HCK##* } #TIMEEDIT="$NEWFILE `stat -f %Sm -t %Y%m%d%H%M%S "$NEWFILE"`" EDITC="$NEWFILE $HCK" CHECK=`grep "$EDITC" "$EditLock"` SIMPLECHECK=`grep "$NEWFILE" "$EditLock"` #You may add your own subtree in additional elif below (for example: immediately script execution, assigning permissions, etc) if [ "`expr "$f" : '.*\(/Merge/\)'`" == "/Merge/" ] #If file should be merged then TempPath=${f##*/Merge} #Cut filepath. Original location will remain echo; echo "Merge: $f --> $TempPath" if ! [ -f $TempPath ] #If original file exist then MoveToLocal $f Merge #Then just replace it by new one else MergeFiles $f $TempPath #Else - merge new file to the old one line by line sort -u $TempPath > $TempPath.tmp #Delete repeating lines if exists mv -f $TempPath.tmp $TempPath #Rewriting merged file by filtered unique data CleanEOL $TempPath #Cleaning empty lines fi echo "DONE!" elif [ "`expr "$f" : '.*\(/Replace/\)'`" == "/Replace/" ] #If file should be replaced then if [ "$EDITC" == "$CHECK" ] || [ "$SIMPLECHECK" = '' ] then echo; echo "Replace: $f --> ${f##*/Replace}" MoveToLocal $f Replace #Then just replace it echo "DONE!" echo "$EDITC" >> $EditLock sort -u $EditLock > $EditLock.tmp #Delete repeating lines if exists mv -f $EditLock.tmp $EditLock else echo; echo "File $NEWFILE was edited manually. Skipped. Use -d key to ignore it." fi elif [ "`expr "$f" : '.*\(/Scripts/\)'`" == "/Scripts/" ] #If tarball contains a scripts, which should have +x permissions then echo; echo "Replace script: $f --> ${f##*/Scripts}" MoveToLocal $f Scripts #Then replace it (scripts cannot be merged) chmod +x ${f##*/Scripts} #And give eXecution permissions echo "DONE!" else echo; echo "DON'T match. Cannot proceed $f. Skipping." #This message means there is another subtree in tarball. It should be removed or described here fi fi done echo; echo "====================================================================" echo; echo "Cleaning temporary files" cd $LocalDir dirchek=`pwd`/ if [ "$dirchek" == "$LocalDir" ] #Checking current directory then rm -rf * echo "Succesfully cleaned" else echo "Temporary files was NOT deleted!" fi echo "DONE!" echo " Tarball was successfully applied." echo "To re-apply it again - use force key (-f)." #Finished }


And, actually, the functions:
parserlib.sh

 #!/bin/sh cdOrCreate() #Enter the directory. Create if it's not exist, then enter. Arguments: 1) Path to directory (alternate or absolute). { if ! [ -d $1 ] #If directory does not exists then mkdir -p "$1" #Then create it fi cd "$1" #Enter the directory } MoveToLocal() #Create path and move file there (or replace existing file). Arguments: 1) full filename with full filepath 2) Folder identifyer, without slashes. { TempPath=${1##*/$2} #Deleting folder identifier from path AbsolutePath=${TempPath%/*} #Completing absolute path cdOrCreate $AbsolutePath #See cdOrCreate() description cd ${1%/*}"/" #Entering directory with file for move mv ${1##*/} $AbsolutePath"/"${1##*/} #Move file to new (absolute path) location } MergeFiles() #Using for check each file from "Merge" subtree and replace lines, or add line to end of file if not exist (?). Files MUST BE in conf syntax. { cat $1 | while read line do lineName=${line%=*} #Calculating key name lineName="$lineName=" lineHashedName=${lineName##\#} #Calculating name if commented sed -i -e 's/^'$lineHashedName'.*/'$line'/g' $2 #Replace line with key (uncommented) sed -i -e 's/^#'$lineHashedName'.*/'$line'/g' $2 #Replace line with key (commented by one hash) echo "$line" >> $2 #Append key to the end of file (dublicates will be sorted). done } CleanEOL() #This function needed for delete ^M from end of replaced lines and delete every empty line. Arguments: 1) Filename with path. { mv $1 tempconfig.conf cat tempconfig.conf | tr -d '\r' > tempconfig.conf.1 #Deleting ^M grep '.' tempconfig.conf.1 > $1 #Deleting empty lines and move file to original location rm tempconfig.conf* #Deleting temporary files } HashCheck() #Checks MD5 of tarball. Arguments: 1) Filename 2) Path 3) Tarball name 4) Flag (force rebuild existing installation) 5) Data directory { cdOrCreate "/var/parser-md5-list/" #See cdOrCreate description fpath=$5 #Location of currently downloaded tarball pointcheck=$4 #Force flag. If equal to "-f" then check will be skipped if ! [ -f $fpath ] #If checkfile does not exists then touch $fpath #Then create it echo $(date) >> $fpath #And write date and time into it elif ! [ "$pointcheck" == '-f' ] #If file exists and force flag was not specified then cat $1 | while read line #Then read date and time from existing file do #Show message echo " ===========================================================" echo " This tarball was applied at $line " echo " Use -f (force) to ignore this warning and rebuild anyway " echo "=========================================================== " cd "$2" #Enter directory which contains currently downloaded tarball rm "$3" #And delete tarball kill $$ #Kill parent process and exit exit 1 done esle #If file exists and -f was specified rm $fpath #Delete existing file touch $fpath #And create a new one fi }


The FTP server runs a script for creating archives with settings:

 #!/bin/sh cdOrCreate() #Enter the directory. Create if it's not exist, then enter. Arguments: 1) Path to directory (alternate or absolute). { if ! [ -d $1 ] #If directory does not exists then mkdir "$1" #Then create it fi cd "$1" #Enter directory } cd /data/ftproot/FreeBSD/ cat list.txt | while read line do if [ "$1" == '-v' ] then echo " == Processing subtree for version $line ==" fi cdOrCreate $line rm tarball.tar.gz if [ "$1" == '-v' ] then tar -zcvf tarball.tar.gz * cd .. else tar -zcf tarball.tar.gz * cd .. fi done echo " DONE!" if ! [ "$1" == '-v' ] then echo "Use -v for detailed output." fi


The archive includes all files in the subdirectories that are listed in the list.txt list, i.e. The file contains the names of the parent directories corresponding to the version number, one per line.
After unpacking the archive, the script checks the Merge and Replace branches. For the first one, the parameters are added or replaced in configuration files, if necessary, the lines are commented out or uncommented. For the second, the usual file replacement is done. For each modified file, its MD5 is saved in the $ DataDirectory $ LockFile list and, in the case of repeated runs of the script, files with inappropriate MD5 will not be changed. This was done to prevent rollback of changes made by the administrator manually.
Also, in case of preventing erroneous changes in the script, a restart function is made through the run.sh file which is created, restarts the script and is deleted. In principle, this function is easy to cut.
The script accepts the following keys (in any order):
-f skip checking archive reuse
-s skip script restart
-d removes lockfile. It is necessary to roll back manual changes.
-v VER enforces FreeBSD version
Any other key will call the usage function and the script will exit.
You can also add your own options for processing subdirectories in the archive. To do this, describe them in the parser.sh file in the elif branch below the corresponding comment.
The structure of one of my archives is as follows:

Merge / boot / loader.conf
Merge / etc / rc.conf
Replace / usr / local / etc
Replace / usr / local / etc / svnup.conf
Replace / usr / share / skel
Replace / usr / share / skel / dot.cshrc
Replace / etc / ntp.conf
Replace / etc / adduser.conf
Replace / etc / portsnap.conf
Replace / root / .cshrc

Where after the directory name (I have this Merge and Replace) the original path to the file is saved. The name of the directory and everything that is before it is removed, then the file is processed by the code in the corresponding if branch.
The script is written using only native functions, i.e. will work on any freshly installed stuff.
To start, you need to spatchite the loader.sh file, give it permissions to execute (chmod + x loared.sh) and, of course, run it.

I will be glad to constructive criticism, comments and suggestions, because I understand perfectly well that the solution is not ideal, and I’m happy to finalize it.

PS: I am very sorry for the unfinished post on Friday. Accidentally published a copy and did not notice this.

Source: https://habr.com/ru/post/198148/

More articles:


All Articles