ToFoIn v 1. Backup Gateways and Switch Between External Channels on FreeBSD

annotation

The last article addressed the issue of organizing redundancy for local area network gateways. As a solution, a script was proposed, which at that time solved the problem, but had several disadvantages. After some time, it turned out to eliminate these shortcomings, partially rewrite the code and get something acceptable at the output. Now we can say that the scripts are sufficiently tested to be called stable. To simplify the understanding of the entire system, the main points on setting up secondary services (in terms of the topic of the article) will be partially duplicated below. The reason is simple - during this time, the ipfw rules were also reworked, the dns went live in AD on Samba4 with bind-frontend and safely updating records from isc-dhcpd using kerberos, as well as secondary dns-servers in the form of bind on gateways, was CARP is tuned ... In general, it became much more interesting, but in more detail about how and how it works - below. All that can be given links to the source, will be designed in this way, so as not to produce the essence. What has been taken from any other places, but which is no longer available, will be presented here with appropriate comments.

Introduction

So, to increase the noise immunity of the communication channel with the outside world by the consumer, there are two ways: reservation of the gateway and reservation of the connection point. In other words, in the first case, a second gateway is put in place, in case the first one fails, in the second case, a backup Internet channel is organized in case of any problems with the main one, and the more they intersect, the better it is. If the first task for FreeBSD is solved by CARP , then the second, after the organization of the second external channel, can again be solved in several ways. At a minimum, traffic balancing or channel switching can be arranged. Due to the large difference in bandwidth of external channels, the first option did not fit me, so the main culprit of the publication was written: ToFoIn is a set of bash scripts that is aimed at solving problems of diagnostics and switching to a working external channel. After completion, it can be used for n gateways and m channels. The situation is poorly represented, where n and m are more than 2, but the following scripts should work up to large values ​​of n and m, since logical limit is not set. In general, I suspect that using these scripts you can solve a fairly wide range of tasks depending on the connection status, limited, perhaps, only by your imagination.


Approximately this network topology assumes in its simplest form the use of a set of ToFoIn scripts. Of course, the scripts must also work in the case of a single router, but in this case, the Daemon module will have to be greatly modified to remove the dependence of the sequence of actions on the state of CARP, which will simply be absent in the system. Further reservation of these and other nodes depends only on the degree of importance of the respective services.

Targets and goals

The goal of the project, as before, is to create a universal and easily scalable software package focused on identifying problems with external and internal connections and automatically switching to workable connections. In general, the logic is as follows:
')

• There are n “routers” with m external channels on each. In this case, all n "routers" are in a strict hierarchy and are connected to each other with the help of CARP on all the necessary interfaces.
• On all machines, an agent works independently of each other, whose task, based on the current CARP status of his car:
  
  • If backup - identify and configure the machine on the router, which is the master at the moment;
  • If master - check the status of connections at the current time and, if necessary, switch between external channels.

Decision

CARP operates in the internal network, which provides backup gateways and allows you to not change the settings of other network devices of the internal network when switching channels.

Dhcpd operates in the primary - secondary mode and, in general, it doesn’t matter what other roles its car plays - the connection between dhcpd occurs in the internal network, which routers always look at.

Master bind is removed in AD, which is hidden in the local network, while secondary bind servers operate on gateway routers on an equal basis.

The ipfw rules differ depending on which channel is considered primary at a given moment and are restarted by the Daemon module when the role changes.

Finally, about the scripts themselves. Now the files are located in the appropriate directories, work from their user and have a startup script in rc.d. Tasks requiring root access are handled by sudo. There is an installation script that takes into account the possible presence of an installed version, as well as a fairly detailed configuration file. The modules are the same with minor changes, some almost did not change in functionality:

Daemon - as the name implies, is the main process that starts the testing and switching modules by timer and also tracks CARP.
Tester - tests the availability of communication through external channels still with the help of the ping command. (if it is running, it considers that the car has CARP in Master state)
Judge - based on the test results, determines which external channel is working and whether switching is necessary, performs switching (if it is running, it considers that the machine has CARP in the Master state).
Scout is a new module. Runs when CARP is in Backup status. It is needed to determine which of the remaining routers is currently the primary one.
Logger - is responsible for logging events. It is necessary so that information about events is not duplicated and the magazine is easier to read.
Watchdog - runs on a schedule from the crontab. It determines the "freezing" of all modules and (if possible) tries to solve the problems that have arisen. Those. nail everyone, to put it simply.

In addition to the scripts themselves, it is worth considering some more important files:

Tofoin.conf - a single configuration file.
Tofoin.log is a single event log file.
Result_ <internal channel number> is a work file, test results are added here, created in / tmp next to .pid and other work files.

I am happy to answer questions regarding the work of the modules, explaining the decisions in the comments.

Technical part

Equipment

Compared to the previous time, the gateways moved to P4, received 1536 Mb of RAM and three 40 Gb HDDs each (mirror + spare). Network cards are still PCI, BP are normal, naturally in the presence of a UPS.
The increase in capacity is associated with the released iron and unnecessarily tedious update from the source, but mostly - the first. OS FreeBSD 11.1, FS zfs.

System Component Settings

Third Party Software

Little about ToFoIn

All the text of the project along with the installation script is available on gitlab .

Total

It turned out to be quite a workable and reliable set of scripts that copes well with the task of switching to the working channel in the case of 2 routers with 2 external communication channels.

Plans

My plans for this project are, perhaps, rewriting from bash to pure sh in order to get rid of the extra software on the server. On the other hand, now everything works amazingly and I don’t really want to interfere in this process, besides the transition to sh is fraught with more terrible language constructions necessary to achieve the same result.
For the rest, probably, it would be worthwhile to think about the best implementation of test modules.

References:

← Previous article
→ ToFoIn project page on gitlab