Scheduling DHCP server
At work, there is often a question - does our DHCP server work well? I work in an internet provider, DHCP ensures the work of the client network. The following work pattern has historically taken shape: DHCP servers are two, the configuration is generated on the billing server and pushed to the server using rsync. We use Centos as a working system and ISC DHCP as a server. No failover configured or configured - no need. It is quite enough that the servers work in the identical configuration. The subscriber's binding to the poppy address is used; in case of changing the equipment, the subscriber has the opportunity to enter the personal account and specify a new poppy. Config generation is performed every 5 minutes, if the new config has the md5 amount changed - then the service restarts. The scheme works for several years, there are no problems.
Periodically there were problems that the config was generated syntactically incorrect - and after the restart the service fell. Added a syntax check (dhcpd -t) to the restart script, the crashes stopped. Well, from the side of billing it is hung with checks - for the presence of an address, poppy, etc.
A couple of times we encountered a situation where requests come from the network, and the answers do not reach the network. Turned to blame aggregator, treated. All this time there was a lack of simple graphical analysis - how a DHCP server works. As the experience of clashes with problems has shown - by banal viewing of logs and estimating how many messages are received and sent by the server, one can already approximately estimate the existence of a problem in the network. Well, in principle, the schedule can be shown to the night operator with instructions - if there is a dramatic change in performance - call the admins.
So, we formulate a task. According to the standard, the following types of messages exist in the DHCP protocol:
')
DHCPDISCOVER - client's request for addresses
DHCPOFFER - server offer to receive address
DHCPREQUEST - client's request for an address (suggested by the server in DHCPOFFER)
DHCPACK - server confirmation of the issuance of the address
DHCPDECLINE - client’s refusal to receive proposed address
DHCPNAK - server failure to issue the requested address
DHCPRELEASE - client notification of the release address
DHCPINFORM - client request for additional parameters
We want to build a schedule of all types of messages.
As a source of information, you can use the server log file. I did not find another source, and it can’t be special - if we assume that dhcp knows how to accumulate this information somewhere - firstly there would be means of extracting such statistics, And secondly - permanent restarts of the process reduce its usefulness to zero .
In our network, the dhcp server is a separate virtual machine, the machine that builds graphics (called mrtg) is also a separate virtual machine. Those. need a way to transfer information between machines.
As a result, I implemented just such a scheme. On the dhcp server we add to the config:
In syslog (these virtual machines still work on centos 5, uptime of machines a year, it would be more if in our city a year ago they didn’t turn the lights off for a month) we add the following:
Those. we prohibit local6 from being output to the general / var / log / messages file, and output it to two files at the same time.
The next step is to teach mrtg to go to dhcp by key without a password. Well, here's a mrtg script:
I'll tell you how this script works.
For each dhcp server, a separate directory is created on the web server. Then, we use scp to retrieve the current dhcpd-stat.log from the server, and use the following command to clear it. For the next time to pick up the log file for the past time. Since the script is called from the crown every 5 minutes, our statistics processes logs in 5 minutes of work. I am more than sure that in the time that elapses between the copy and clear commands, some data that I am losing gets into the log. But I think that a few lines of weather do not.
At the same time I will answer the question - why dhcp logs are output in two files. The first is the usual log, it rotates with the standard logrotate, does not correspond constantly, it is intended for a person - if you have to read it. The second is for the script, it is cleared every 5 minutes.
Well, then in the script, in principle, everything should be clear: we initialize the rrd database, if there is one, we use awk to count the number of each message type, enter the rrd base, then build a standard 4-picture - daily chart, weekly chart, monthly chart annual chart. Well and we create index.shtml if it is not present.
Skriptik began to work, the graphics are drawn. Now you can see immediately - dhcp works, or not ...
Here is the graph:
Periodically there were problems that the config was generated syntactically incorrect - and after the restart the service fell. Added a syntax check (dhcpd -t) to the restart script, the crashes stopped. Well, from the side of billing it is hung with checks - for the presence of an address, poppy, etc.
A couple of times we encountered a situation where requests come from the network, and the answers do not reach the network. Turned to blame aggregator, treated. All this time there was a lack of simple graphical analysis - how a DHCP server works. As the experience of clashes with problems has shown - by banal viewing of logs and estimating how many messages are received and sent by the server, one can already approximately estimate the existence of a problem in the network. Well, in principle, the schedule can be shown to the night operator with instructions - if there is a dramatic change in performance - call the admins.
So, we formulate a task. According to the standard, the following types of messages exist in the DHCP protocol:
')
DHCPDISCOVER - client's request for addresses
DHCPOFFER - server offer to receive address
DHCPREQUEST - client's request for an address (suggested by the server in DHCPOFFER)
DHCPACK - server confirmation of the issuance of the address
DHCPDECLINE - client’s refusal to receive proposed address
DHCPNAK - server failure to issue the requested address
DHCPRELEASE - client notification of the release address
DHCPINFORM - client request for additional parameters
We want to build a schedule of all types of messages.
As a source of information, you can use the server log file. I did not find another source, and it can’t be special - if we assume that dhcp knows how to accumulate this information somewhere - firstly there would be means of extracting such statistics, And secondly - permanent restarts of the process reduce its usefulness to zero .
In our network, the dhcp server is a separate virtual machine, the machine that builds graphics (called mrtg) is also a separate virtual machine. Those. need a way to transfer information between machines.
As a result, I implemented just such a scheme. On the dhcp server we add to the config:
log-facility local6
In syslog (these virtual machines still work on centos 5, uptime of machines a year, it would be more if in our city a year ago they didn’t turn the lights off for a month) we add the following:
*.info;mail.none;authpriv.none;cron.none;local6.none /var/log/messages local6.* /var/log/dhcpd.log local6.* /var/log/dhcpd-stat.log Those. we prohibit local6 from being output to the general / var / log / messages file, and output it to two files at the same time.
The next step is to teach mrtg to go to dhcp by key without a password. Well, here's a mrtg script:
statistics update script
#!/bin/bash for HOST in dhcp1 dhcp2 do TARGET=/srv/www/noc.lds.net.ua/dhcp/$HOST mkdir -p $TARGET cd $TARGET scp $HOST:/var/log/dhcpd-stat.log ./ ssh $HOST "echo -n > /var/log/dhcpd-stat.log" if [ ! -f dhcpqueries.rrd ] then /usr/bin/rrdtool create dhcpqueries.rrd \ DS:request:GAUGE:600:0:10000000 \ DS:ack:GAUGE:600:0:10000000 \ DS:decline:GAUGE:600:0:10000000 \ DS:discover:GAUGE:600:0:10000000 \ DS:release:GAUGE:600:0:10000000 \ DS:nak:GAUGE:600:0:10000000 \ DS:info:GAUGE:600:0:10000000 \ DS:offer:GAUGE:600:0:10000000 \ RRA:AVERAGE:0.5:1:800 \ RRA:AVERAGE:0.5:6:800 \ RRA:AVERAGE:0.5:24:800 \ RRA:AVERAGE:0.5:288:800 \ RRA:MAX:0.5:1:800 \ RRA:MAX:0.5:6:800 \ RRA:MAX:0.5:24:800 \ RRA:MAX:0.5:288:800 fi out=$(awk ' BEGIN {request=0;ack=0;decline=0;discover=0;release=0;nak=0;info=0;offer=0} { if($6 == "DHCPREQUEST") {request = request + 1} if($6 == "DHCPACK") {ack = ack + 1} if($6 == "DHCPDECLINE") {decline = decline + 1} if($6 == "DHCPDISCOVER") {discover = discover + 1} if($6 == "DHCPRELEASE") {release = release + 1} if($6 == "DHCPNAK") {nak = nak + 1} if($6 == "DHCPINFORM") {info = info + 1} if($6 == "DHCPOFFER") {offer = offer + 1} } END {print request ":" ack ":" decline ":" discover ":" release ":" nak ":" info ":" offer} ' dhcpd-stat.log) /usr/bin/rrdtool update dhcpqueries.rrd --template \ request:ack:decline:discover:release:nak:info:offer \ N:$out /usr/bin/rrdtool graph dhcprequest.png \ -v "Requests/Minute" \ --rigid \ -l 0 \ --start -86400 \ DEF:a=dhcpqueries.rrd:request:AVERAGE \ DEF:b=dhcpqueries.rrd:ack:AVERAGE \ DEF:c=dhcpqueries.rrd:decline:AVERAGE \ DEF:d=dhcpqueries.rrd:discover:AVERAGE \ DEF:e=dhcpqueries.rrd:release:AVERAGE \ DEF:f=dhcpqueries.rrd:nak:AVERAGE \ DEF:g=dhcpqueries.rrd:info:AVERAGE \ DEF:h=dhcpqueries.rrd:offer:AVERAGE \ CDEF:cdefa=a,5,/ \ CDEF:cdefb=b,5,/ \ CDEF:cdefc=c,5,/ \ CDEF:cdefd=d,5,/ \ CDEF:cdefe=e,5,/ \ CDEF:cdeff=f,5,/ \ CDEF:cdefg=g,5,/ \ CDEF:cdefh=h,5,/ \ LINE1:cdefa#9C7BBD:DHCPREQUEST \ GPRINT:cdefa:LAST:" Cur\:%8.1lf" \ GPRINT:cdefa:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefa:MAX:"Max\:%8.1lf\n" \ LINE1:cdefb#3152A5:DHCPACK \ GPRINT:cdefb:LAST:" Cur\:%8.1lf" \ GPRINT:cdefb:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefb:MAX:"Max\:%8.1lf\n" \ LINE1:cdefc#750F7D:DHCPDECLINE \ GPRINT:cdefc:LAST:" Cur\:%8.1lf" \ GPRINT:cdefc:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefc:MAX:"Max\:%8.1lf\n" \ LINE1:cdefd#157419:DHCPDISCOVER \ GPRINT:cdefd:LAST:" Cur\:%8.1lf" \ GPRINT:cdefd:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefd:MAX:"Max\:%8.1lf\n" \ LINE1:cdefe#6DC8FE:DHCPRELEASE \ GPRINT:cdefe:LAST:" Cur\:%8.1lf" \ GPRINT:cdefe:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefe:MAX:"Max\:%8.1lf\n" \ LINE1:cdeff#FFAB00:DHCPNAK \ GPRINT:cdeff:LAST:" Cur\:%8.1lf" \ GPRINT:cdeff:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdeff:MAX:"Max\:%8.1lf\n" \ LINE1:cdefg#FF0000:DHCPINFORM \ GPRINT:cdefg:LAST:" Cur\:%8.1lf" \ GPRINT:cdefg:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefg:MAX:"Max\:%8.1lf\n" \ LINE1:cdefh#00FF00:DHCPOFFER \ GPRINT:cdefh:LAST:" Cur\:%8.1lf" \ GPRINT:cdefh:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefh:MAX:"Max\:%8.1lf\n" \ /usr/bin/rrdtool graph dhcprequest-weekly.png \ -v "Requests/Minute" \ --rigid \ -l 0 \ --start -604800 \ DEF:a=dhcpqueries.rrd:request:AVERAGE \ DEF:b=dhcpqueries.rrd:ack:AVERAGE \ DEF:c=dhcpqueries.rrd:decline:AVERAGE \ DEF:d=dhcpqueries.rrd:discover:AVERAGE \ DEF:e=dhcpqueries.rrd:release:AVERAGE \ DEF:f=dhcpqueries.rrd:nak:AVERAGE \ DEF:g=dhcpqueries.rrd:info:AVERAGE \ DEF:h=dhcpqueries.rrd:offer:AVERAGE \ CDEF:cdefa=a,5,/ \ CDEF:cdefb=b,5,/ \ CDEF:cdefc=c,5,/ \ CDEF:cdefd=d,5,/ \ CDEF:cdefe=e,5,/ \ CDEF:cdeff=f,5,/ \ CDEF:cdefg=g,5,/ \ CDEF:cdefh=h,5,/ \ LINE1:cdefa#9C7BBD:DHCPREQUEST \ GPRINT:cdefa:LAST:" Cur\:%8.1lf" \ GPRINT:cdefa:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefa:MAX:"Max\:%8.1lf\n" \ LINE1:cdefb#3152A5:DHCPACK \ GPRINT:cdefb:LAST:" Cur\:%8.1lf" \ GPRINT:cdefb:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefb:MAX:"Max\:%8.1lf\n" \ LINE1:cdefc#750F7D:DHCPDECLINE \ GPRINT:cdefc:LAST:" Cur\:%8.1lf" \ GPRINT:cdefc:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefc:MAX:"Max\:%8.1lf\n" \ LINE1:cdefd#157419:DHCPDISCOVER \ GPRINT:cdefd:LAST:" Cur\:%8.1lf" \ GPRINT:cdefd:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefd:MAX:"Max\:%8.1lf\n" \ LINE1:cdefe#6DC8FE:DHCPRELEASE \ GPRINT:cdefe:LAST:" Cur\:%8.1lf" \ GPRINT:cdefe:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefe:MAX:"Max\:%8.1lf\n" \ LINE1:cdeff#FFAB00:DHCPNAK \ GPRINT:cdeff:LAST:" Cur\:%8.1lf" \ GPRINT:cdeff:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdeff:MAX:"Max\:%8.1lf\n" \ LINE1:cdefg#FF0000:DHCPINFORM \ GPRINT:cdefg:LAST:" Cur\:%8.1lf" \ GPRINT:cdefg:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefg:MAX:"Max\:%8.1lf\n" \ LINE1:cdefh#00FF00:DHCPOFFER \ GPRINT:cdefh:LAST:" Cur\:%8.1lf" \ GPRINT:cdefh:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefh:MAX:"Max\:%8.1lf\n" \ /usr/bin/rrdtool graph dhcprequest-monthly.png \ -v "Requests/Minute" \ --rigid \ -l 0 \ --start -2592000 \ DEF:a=dhcpqueries.rrd:request:AVERAGE \ DEF:b=dhcpqueries.rrd:ack:AVERAGE \ DEF:c=dhcpqueries.rrd:decline:AVERAGE \ DEF:d=dhcpqueries.rrd:discover:AVERAGE \ DEF:e=dhcpqueries.rrd:release:AVERAGE \ DEF:f=dhcpqueries.rrd:nak:AVERAGE \ DEF:g=dhcpqueries.rrd:info:AVERAGE \ DEF:h=dhcpqueries.rrd:offer:AVERAGE \ CDEF:cdefa=a,5,/ \ CDEF:cdefb=b,5,/ \ CDEF:cdefc=c,5,/ \ CDEF:cdefd=d,5,/ \ CDEF:cdefe=e,5,/ \ CDEF:cdeff=f,5,/ \ CDEF:cdefg=g,5,/ \ CDEF:cdefh=h,5,/ \ LINE1:cdefa#9C7BBD:DHCPREQUEST \ GPRINT:cdefa:LAST:" Cur\:%8.1lf" \ GPRINT:cdefa:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefa:MAX:"Max\:%8.1lf\n" \ LINE1:cdefb#3152A5:DHCPACK \ GPRINT:cdefb:LAST:" Cur\:%8.1lf" \ GPRINT:cdefb:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefb:MAX:"Max\:%8.1lf\n" \ LINE1:cdefc#750F7D:DHCPDECLINE \ GPRINT:cdefc:LAST:" Cur\:%8.1lf" \ GPRINT:cdefc:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefc:MAX:"Max\:%8.1lf\n" \ LINE1:cdefd#157419:DHCPDISCOVER \ GPRINT:cdefd:LAST:" Cur\:%8.1lf" \ GPRINT:cdefd:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefd:MAX:"Max\:%8.1lf\n" \ LINE1:cdefe#6DC8FE:DHCPRELEASE \ GPRINT:cdefe:LAST:" Cur\:%8.1lf" \ GPRINT:cdefe:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefe:MAX:"Max\:%8.1lf\n" \ LINE1:cdeff#FFAB00:DHCPNAK \ GPRINT:cdeff:LAST:" Cur\:%8.1lf" \ GPRINT:cdeff:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdeff:MAX:"Max\:%8.1lf\n" \ LINE1:cdefg#FF0000:DHCPINFORM \ GPRINT:cdefg:LAST:" Cur\:%8.1lf" \ GPRINT:cdefg:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefg:MAX:"Max\:%8.1lf\n" \ LINE1:cdefh#00FF00:DHCPOFFER \ GPRINT:cdefh:LAST:" Cur\:%8.1lf" \ GPRINT:cdefh:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefh:MAX:"Max\:%8.1lf\n" \ /usr/bin/rrdtool graph dhcprequest-yearly.png \ -v "Requests/Minute" \ --rigid \ -l 0 \ --start -31536000 \ DEF:a=dhcpqueries.rrd:request:AVERAGE \ DEF:b=dhcpqueries.rrd:ack:AVERAGE \ DEF:c=dhcpqueries.rrd:decline:AVERAGE \ DEF:d=dhcpqueries.rrd:discover:AVERAGE \ DEF:e=dhcpqueries.rrd:release:AVERAGE \ DEF:f=dhcpqueries.rrd:nak:AVERAGE \ DEF:g=dhcpqueries.rrd:info:AVERAGE \ DEF:h=dhcpqueries.rrd:offer:AVERAGE \ CDEF:cdefa=a,5,/ \ CDEF:cdefb=b,5,/ \ CDEF:cdefc=c,5,/ \ CDEF:cdefd=d,5,/ \ CDEF:cdefe=e,5,/ \ CDEF:cdeff=f,5,/ \ CDEF:cdefg=g,5,/ \ CDEF:cdefh=h,5,/ \ LINE1:cdefa#9C7BBD:DHCPREQUEST \ GPRINT:cdefa:LAST:" Cur\:%8.1lf" \ GPRINT:cdefa:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefa:MAX:"Max\:%8.1lf\n" \ LINE1:cdefb#3152A5:DHCPACK \ GPRINT:cdefb:LAST:" Cur\:%8.1lf" \ GPRINT:cdefb:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefb:MAX:"Max\:%8.1lf\n" \ LINE1:cdefc#750F7D:DHCPDECLINE \ GPRINT:cdefc:LAST:" Cur\:%8.1lf" \ GPRINT:cdefc:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefc:MAX:"Max\:%8.1lf\n" \ LINE1:cdefd#157419:DHCPDISCOVER \ GPRINT:cdefd:LAST:" Cur\:%8.1lf" \ GPRINT:cdefd:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefd:MAX:"Max\:%8.1lf\n" \ LINE1:cdefe#6DC8FE:DHCPRELEASE \ GPRINT:cdefe:LAST:" Cur\:%8.1lf" \ GPRINT:cdefe:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefe:MAX:"Max\:%8.1lf\n" \ LINE1:cdeff#FFAB00:DHCPNAK \ GPRINT:cdeff:LAST:" Cur\:%8.1lf" \ GPRINT:cdeff:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdeff:MAX:"Max\:%8.1lf\n" \ LINE1:cdefg#FF0000:DHCPINFORM \ GPRINT:cdefg:LAST:" Cur\:%8.1lf" \ GPRINT:cdefg:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefg:MAX:"Max\:%8.1lf\n" \ LINE1:cdefh#00FF00:DHCPOFFER \ GPRINT:cdefh:LAST:" Cur\:%8.1lf" \ GPRINT:cdefh:AVERAGE:"Ave\:%8.1lf" \ GPRINT:cdefh:MAX:"Max\:%8.1lf\n" \ if [ ! -f index.shtml ] then cat > ./index.shtml << END <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <HTML> <HEAD> <TITLE>$HOST</TITLE> <META HTTP-EQUIV="Refresh" CONTENT="300"> <META HTTP-EQUIV="Cache-Control" content="no-cache"> <META HTTP-EQUIV="Pragma" CONTENT="no-cache"> <META HTTP-EQUIV="Expires" CONTENT="Mon, 04 Feb 2008 16:28:46 GMT"> <style type="text/css"> </style> </HEAD> <BODY bgcolor="#ffffff" text="#000000" link="#000000" vlink="#000000" alink="#000000"> <!--#include virtual="/menu.shtml" --> <CENTER><H1><B>$HOST</B></H1></CENTER> <TABLE BORDER=0 CELLPADDING=0 CELLSPACING=0 align="center"> <tbody> <tr><td align="center"><b><h2>Query types</h2></b></td></tr> <tr> <td align="center"> <b>daily</B><br> <IMG BORDER=0 SRC="dhcprequest.png"><br> </td> </tr> <tr> <td align="center"> <b>weekly</B><br> <IMG BORDER=0 SRC="dhcprequest-weekly.png"><br> </td> </tr> <tr> <td align="center"> <b>monthly</B><br> <IMG BORDER=0 SRC="dhcprequest-monthly.png"><br> </td> </tr> <tr> <td align="center"> <b>yearly</B><br> <IMG BORDER=0 SRC="dhcprequest-yearly.png"><br> </td> </tr> </tbody> </table> </BODY> </HTML> END fi done I'll tell you how this script works.
For each dhcp server, a separate directory is created on the web server. Then, we use scp to retrieve the current dhcpd-stat.log from the server, and use the following command to clear it. For the next time to pick up the log file for the past time. Since the script is called from the crown every 5 minutes, our statistics processes logs in 5 minutes of work. I am more than sure that in the time that elapses between the copy and clear commands, some data that I am losing gets into the log. But I think that a few lines of weather do not.
At the same time I will answer the question - why dhcp logs are output in two files. The first is the usual log, it rotates with the standard logrotate, does not correspond constantly, it is intended for a person - if you have to read it. The second is for the script, it is cleared every 5 minutes.
Well, then in the script, in principle, everything should be clear: we initialize the rrd database, if there is one, we use awk to count the number of each message type, enter the rrd base, then build a standard 4-picture - daily chart, weekly chart, monthly chart annual chart. Well and we create index.shtml if it is not present.
Skriptik began to work, the graphics are drawn. Now you can see immediately - dhcp works, or not ...
Here is the graph:
Source: https://habr.com/ru/post/267343/
All Articles