How to quickly run voluntary distributed computing on a hundred machines
While working in the IT department, I constantly observe computers that are idle for various organizational reasons and are very bored. The golden days of mining bitcoins on the CPU have passed, and in search of a new useful business, I came to voluntary distributed computing, in particular, to World Community Grid . First of all, the search for a cure for cancer was puzzled by the server from the cold standby and the virtual machine with a low priority on the virtualization cluster. More difficult with workstations, they constantly come and go, install, configure, and then delete BOINC for a long time and unevenly on each.
It was decided to assemble a live distribution kit with a built-in BOINC and distribute it over the network. Turned on the computer, pressed F12, chose the desired item - and you are already benefiting humanity!
The platform was chosen by Debian, which a) has been familiar for a long time and b) has a wonderful manual on the desired topic. Nevertheless, it was not without a rake, and in this case almost every new rake means a rather long reassembly of the image. I hope this post will save some amount of admin time, and at the same time remind you of the existence of such a wonderful project as WCG.
Notice, everything was done in a very closed environment, and it took a little attention to security. Perhaps in your case, you will need additional work on security.
')
The system consists of the following:
1. Network boot server. Everything was ready for me, the configured TFTP and DHCP remained with me from the project on thin clients. If you do not, then raise a new one is easy. In a nutshell, we install and run tftpd-hpa, and in DHCP we indicate the parameters 66 and 67. Just do not let anyone boot via the network (in my case these are cadets), this can be dangerous. In addition to the BIOS, you can password protect a portion of the TFTP server boot menu.
2. NFS server. Firstly, BOINC should be able to save its data in the course of work. It is assumed that the local hard disk cannot be touched, therefore, using NFS, we are allowed to write to the directory, for example,
In
after which we restart the service (for some reason, the recommended
3. Assembly station. This is just a virtual machine with normal Debian Wheezy. The
We leave for the assembly station.
Create a basic config for our distribution, specifying the address of the NFS server:
Some directory tree is formed, having in them different content you can customize your build. We will add the following:
1.
2.
3.
After adding the necessary settings, run the actual assembly:
The value for us is the resulting
It will be unpacked in
The contents of the
Now everything is in its place.
If after the build you want to check the contents of the root file system, then it is not necessary to boot with it, you can simply mount it as a loop device:
If you need to rebuild the distribution kit with new parameters, then before that you can do either
Everything, in a few minutes, new tasks will appear in the “Ready to launch” and “Running” state.
This procedure needs to be performed only once for each computer (more precisely, for each MAC address). Even returning to you after a long period of operation elsewhere, the computer at the MAC address will find its data on the NFS server and continue to work immediately after switching on (only some tasks will have already expired, but this is a trifle, it will receive new ones).
What remains unresolved:
It was decided to assemble a live distribution kit with a built-in BOINC and distribute it over the network. Turned on the computer, pressed F12, chose the desired item - and you are already benefiting humanity!
The platform was chosen by Debian, which a) has been familiar for a long time and b) has a wonderful manual on the desired topic. Nevertheless, it was not without a rake, and in this case almost every new rake means a rather long reassembly of the image. I hope this post will save some amount of admin time, and at the same time remind you of the existence of such a wonderful project as WCG.
Notice, everything was done in a very closed environment, and it took a little attention to security. Perhaps in your case, you will need additional work on security.
')
Training
The system consists of the following:
- Network boot server.
- NFS server
- Assembly station
1. Network boot server. Everything was ready for me, the configured TFTP and DHCP remained with me from the project on thin clients. If you do not, then raise a new one is easy. In a nutshell, we install and run tftpd-hpa, and in DHCP we indicate the parameters 66 and 67. Just do not let anyone boot via the network (in my case these are cadets), this can be dangerous. In addition to the BIOS, you can password protect a portion of the TFTP server boot menu.
2. NFS server. Firstly, BOINC should be able to save its data in the course of work. It is assumed that the local hard disk cannot be touched, therefore, using NFS, we are allowed to write to the directory, for example,
/srv/boinc-nfs . Here, each computer will create a subdirectory with the same name as its MAC address. Secondly, in the /srv/debian-live directory there will be a root file system for network boot. So: mkdir /srv/debian-live mkdir /srv/boinc-nfs chown nobody:nogroup /srv/boinc-nfs chmod 755 /srv/boinc-nfs In
/etc/exports add: /srv/boinc-nfs *(rw,sync,no_root_squash,no_subtree_check) /srv/debian-live *(ro,async,no_root_squash,no_subtree_check) after which we restart the service (for some reason, the recommended
exportfs -rv did not give me any results): /etc/init.d/nfs-kernel-server restart 3. Assembly station. This is just a virtual machine with normal Debian Wheezy. The
live-build package is installed, which will do the main work. There should be internet.Assembly process
We leave for the assembly station.
mkdir /srv/live-default && cd /srv/live-default Create a basic config for our distribution, specifying the address of the NFS server:
lb config -b netboot --net-root-path "/srv/debian-live" --net-root-server "192.168.15.20" Some directory tree is formed, having in them different content you can customize your build. We will add the following:
1.
config/package-lists/boinc.list - list of packages that will be needed in our assembly. We write to it: boinc-client nfs-common 2.
config/includes.chroot/etc/init.d/boinc-preps - the init script that will install NFS, configure BOINC and change the hostname ( perhaps identical hostnames prevent WCG from identifying the computer, many of my tasks went to them detached state). In this script you need to insert the address of your NFS and the addresses of the hosts from which password-free management will be allowed. Script content: #!/bin/bash ### BEGIN INIT INFO # Provides: boinc-preps # Required-Start: nfs-common # Required-Stop: # Should-Start: # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Various stuff for BOINC # Description: Various stuff for BOINC ### END INIT INFO PATH=/sbin:/usr/sbin:/bin:/usr/bin . /lib/init/vars.sh do_start () { MYMAC=`ifconfig eth0 | grep -o -E '([[:xdigit:]]{1,2}:){5}[[:xdigit:]]{1,2}' | sed s/://g` ancien=`hostname` nouveau=DYNWCG-$MYMAC mkdir -p /mnt/boinc-nfs mount 192.168.15.20:/srv/boinc-nfs /mnt/boinc-nfs && mkdir -p /mnt/boinc-nfs/$MYMAC service boinc stop sed -i "s/^BOINC_DIR=.*/BOINC_DIR=\/mnt\/boinc-nfs\/$MYMAC/;s/^BOINC_USER=.*/BOINC_USER=\"root\"/" /etc/default/boinc-client echo "192.168.10.60" > /mnt/boinc-nfs/$MYMAC/remote_hosts.cfg echo "192.168.10.61" >> /mnt/boinc-nfs/$MYMAC/remote_hosts.cfg echo "" >> /mnt/boinc-nfs/$MYMAC/gui_rpc_auth.cfg for file in \ /etc/hostname \ /etc/hosts # #/etc/ssh/ssh_host_rsa_key.pub \ #/etc/ssh/ssh_host_dsa_key.pub \ # SSH do [ -f $file ] && sed -i.old -e "s:$ancien:$nouveau:g" $file done invoke-rc.d hostname.sh start invoke-rc.d networking force-reload service boinc start } case "$1" in start|"") do_start ;; restart|reload|force-reload|status) echo "Error: argument '$1' not supported" >&2 exit 3 ;; stop) # NOP exit 3 ;; *) echo "Usage: ... [start|stop]" >&2 exit 3 ;; esac : 3.
config/hooks/boinc-preps-init.chroot - a script from one command that will be executed when building and will add boinc-preps from the previous item to autoload: #!/bin/sh update-rc.d boinc-preps defaults After adding the necessary settings, run the actual assembly:
lb build The value for us is the resulting
binary.netboot.tar file. Extract it to /srv : cd /srv && tar -xvf live-default/binary.netboot.tar It will be unpacked in
/srv/debian-live (root file system for network boot) and /srv/tftpboot (files for the TFTP server). In my case, the assembly station and the NFS server are one computer, therefore /srv/debian-live already in place.The contents of the
tftpboot is a ready boot menu, it needs to be placed on the TFTP server. I did not check his work, since I have a working TFTP server with my menu, and from here I need only a part of the data. First, I copied all the files from the tftpboot/live/ to the TFTP server in images/debian-live/ (relative to the root directory of the TFTP server). Secondly, I borrowed text from tftpboot/live.cfg , which adds a new menu item, while changing it to this state (here you also need to specify the address of your NFS server): label live-686-pae menu label BOINC-live (686-pae) linux images/debian-live/vmlinuz1 initrd images/debian-live/initrd1.img append boot=live config nosplash root=/dev/nfs nfsroot=192.168.15.20:/srv/debian-live Now everything is in its place.
If after the build you want to check the contents of the root file system, then it is not necessary to boot with it, you can simply mount it as a loop device:
mount -o loop,ro /srv/debian-live/live/filesystem.squashfs /mnt/squash/ If you need to rebuild the distribution kit with new parameters, then before that you can do either
lb clean --binary or lb clean .Using
- We turn on the computer and select the boot from the network (usually it’s enough to press F12).
- Depending on the boot menu, either select the “BOINC-live” item, or just wait for it to load by timeout.
- In the command line that appears (if everything went as it should), we write
sudo ifconfig(no password will be required) and write down the IP address. - On the management computer (one of those that we specified in
config/includes.chroot/etc/init.d/boinc-preps), launch boinc-manager, click "Advanced - Change computer" (this button is only in "Full view" ). However, BOINC should not request any passwords from the user. - After connecting, a wizard will appear in which you will need to select a project (in my case it is World Community Grid) and enter your login / password.
Everything, in a few minutes, new tasks will appear in the “Ready to launch” and “Running” state.
This procedure needs to be performed only once for each computer (more precisely, for each MAC address). Even returning to you after a long period of operation elsewhere, the computer at the MAC address will find its data on the NFS server and continue to work immediately after switching on (only some tasks will have already expired, but this is a trifle, it will receive new ones).
Results
What remains unresolved:
- Autologin in WCG. Probably, you can sew a project config into the image in order to not introduce anything at all with your hands, but in a reasonable time the solution did not google.
- Mail notifications. In order not to enter "sudo ifconfig", you can automatically send the address of the running computer to the admin. However, I did not do this, because the crutches, and it is better to implement the previous point instead.
- The IP of the NFS server is indicated twice, perhaps one of them can be removed.
- To create a computer directory, the eth0 interface MAC address is always used. Good or not - I can not say with certainty.
Don't let computers get bored! And write comments, I will gladly answer or add to the article.
Source: https://habr.com/ru/post/234697/
All Articles