Why Python Networker
“The network administrator must be able to program” - this phrase often causes objections to many networkers.
- What for? Hands it safer.
- But you can automate typical operations.
- And put a bunch of gadgets if something goes wrong?
- You can put a bunch of devices with your hands.
You have listened to a summary of typical discussions on this issue. Most admins stop editing previously copied config pieces in the text editor and copying them to the console. Or preparation of standard configuration files, but adding them to the equipment by hand through the console.
')
If you look towards the manufacturers of network equipment,
it turns out that the same Cisco has long been offering various options for automating work with network equipment: from TCL on IOS to Python on NX-OS and IOS-XR . All this is called network automation or network programmability, and Cisco has courses in this area.
And Cisco is not alone here: Juniper with PyEZ , HP, Huawei, and so on.
Many tools - Netconf, Restconf, Ansible, Puppet and Python, Python, Python. The analysis of specific instruments will be postponed until later, we turn to a specific example.
The second question, which sometimes causes heated debates, usually leads to a complete misunderstanding of each other: “Do you need network devices in DNS?”.
Let's leave a detailed analysis of the participants' positions for later, formulating the task that led to Python and SNMP. It all started with traceroute.
Despite the presence of various monitoring systems that watch and see a lot, MPLS-TE, which turns traffic in a bizarre way, the right ICMP and traceroute and ping utilities in many cases are able to provide the necessary information quickly and now. But outputting traceroute only in the form of IP addresses in a large network will require additional efforts to understand where the packets came from. For example, we see that forward and reverse traffic from a user goes through different routers, but for which ones? The solution is obviously to add the addresses of the routers to the DNS. And for corporate networks, where unnumbered is rarely used, putting separate addresses on connectors, if the addresses of interfaces are entered into DNS, you can quickly understand which interface the ICMP packet came from the router.
However, to manually maintain the DNS database on a large network requires a lot of hard work is not the most difficult work. But the domain name of the interface will consist of the name of the interface, the description of the interface, the hostname of the router and the domain name. All this router carries in its configuration. The main thing is to collect and properly glue and bind to the correct address.
So this task should be automated.
The first thought, the analysis of configurations, quickly faded away, the network is large, multi-vendor, and even the equipment from different generations, so the idea of parsing configs quickly became unpopular.
The second thought is to use what gives the necessary answers to universal requests to the equipment of different vendors. The answer was obvious - SNMP. It, with all its features, is implemented in the software of any vendor.
We put Python.
We will need modules to work with SNMP, IP addresses, over time. But to install them you need to put pip. True, now it comes with python.
And now we put the modules.
Let's try to get its hostname from the router. SNMP uses an OID to query the host. On the OID, the host will return information corresponding to that OID. We want to get the hostname - you need to request 1.3.6.1.2.1.1.5.0.
And so the first script that requests only the hostname.
Run and get:
Let's analyze the script in more detail:
First we import the necessary modules:
1. pysnmp - provides the script with the host via SNMP
2. ipaddress - provides work with addresses. Checking addresses for correctness, checking for occurrences of the address in the network address, and so on.
3. datetime- get current time. In this task, you need to organize logs.
Then we get four variables:
1. community
2. host address
3. SNMP port
4. OID value
Two functions:
1. snmp_getcmd
2. snmp_get_next
The first function sends a GET request to the specified host, on the specified port, with the specified comminity and OID.
The second function is the snmp_getcmd generator. Probably break into two functions was not quite right, but it happened :)
This script lacks some things:
1. In the script you need to load the ip address of the hosts. For example, from a text file. When downloading, you need to check the download address for correctness, otherwise pysnmp can be very surprised and the script will stop with a traceback. It is unprincipled where you will get the addresses from the file, from the database, but you must be sure that the addresses you received are correct. And so, the source of addresses is a text file, one line - one address in decimal form.
2. Network equipment may be turned off at the time of the survey, may be configured incorrectly, as a result, pysnmp will give out in this case absolutely not what we are waiting for and with further processing of the received information we will get the script to stop with traceback. We need an error handler for our SNMP interaction.
3. You need a log file in which the processed errors will be recorded.
Enter the variable for the file name.
We write the check_ip function to check if the address is correct.
We write the get_from_file function of loading addresses, which checks each address for correctness and if this is not the case, writes a message about it to the log.
We implement data loading in the list.
Create a file ip.txt
The second address in this list does not respond to snmp. Run the script and verify that an error handler is needed for SNMP.
From the contents of the traceback is impossible to understand that the cause of the failure was an inaccessible host. Let's try to intercept possible reasons for stopping the script and record all the information in the log.
The snmp_get_next function already has error error output, errorStatus, errorIndex, varBinds. In varBinds, the resulting data is uploaded, error information is uploaded to variables beginning with error. It only needs to be properly processed. Since in the future there will be several more functions for working with snmp in the script, it makes sense to bring the error handling into a separate function.
And now we add error handling and writing to the log file to the snmp_get_next function. The function should now return not only data, but also a message about whether there were any errors.
Now it is necessary to rewrite the code section a bit, taking into account the fact that now there are messages about the success of the request. In addition, add a few checks:
1. Sysname is less than three characters. We write to the log file, so that later look at the closer.
2. We will find out that some Huawei and Catos give only the hostname for the request. Since we don’t want to look for the OID separately for them (it’s not a fact that it exists at all, maybe it’s a software bug), we’ll add such domain hosts manually.
3. We find that hosts with the wrong comminity behave differently, most initiate the error handler, and some for some reason respond that the script perceives as a normal situation.
4. Add a different logging level at the time of debugging, in order not to pick out unnecessary messages throughout the script.
Check this script on the same ip.txt file.
Everything worked normally, we caught all the errors, the script missed the hosts with errors. Now this script can collect the hostname from all devices responding to snmp.
I hide the full text of the script under the spoiler.
Now it remains to collect the interface names, interface descriptions, interface addresses and correctly expand into the bind configuration files. But about this in the second part.
PS: I note that in a good way the messages in the log file should be formed in a different way.
For example: time special character code_ error, special character description_ error, special character, additional_information. This will help you later set up automatic log processing.
UPD: error correction.
- What for? Hands it safer.
- But you can automate typical operations.
- And put a bunch of gadgets if something goes wrong?
- You can put a bunch of devices with your hands.
You have listened to a summary of typical discussions on this issue. Most admins stop editing previously copied config pieces in the text editor and copying them to the console. Or preparation of standard configuration files, but adding them to the equipment by hand through the console.
')
If you look towards the manufacturers of network equipment,
it turns out that the same Cisco has long been offering various options for automating work with network equipment: from TCL on IOS to Python on NX-OS and IOS-XR . All this is called network automation or network programmability, and Cisco has courses in this area.
And Cisco is not alone here: Juniper with PyEZ , HP, Huawei, and so on.
Many tools - Netconf, Restconf, Ansible, Puppet and Python, Python, Python. The analysis of specific instruments will be postponed until later, we turn to a specific example.
The second question, which sometimes causes heated debates, usually leads to a complete misunderstanding of each other: “Do you need network devices in DNS?”.
Let's leave a detailed analysis of the participants' positions for later, formulating the task that led to Python and SNMP. It all started with traceroute.
Despite the presence of various monitoring systems that watch and see a lot, MPLS-TE, which turns traffic in a bizarre way, the right ICMP and traceroute and ping utilities in many cases are able to provide the necessary information quickly and now. But outputting traceroute only in the form of IP addresses in a large network will require additional efforts to understand where the packets came from. For example, we see that forward and reverse traffic from a user goes through different routers, but for which ones? The solution is obviously to add the addresses of the routers to the DNS. And for corporate networks, where unnumbered is rarely used, putting separate addresses on connectors, if the addresses of interfaces are entered into DNS, you can quickly understand which interface the ICMP packet came from the router.
However, to manually maintain the DNS database on a large network requires a lot of hard work is not the most difficult work. But the domain name of the interface will consist of the name of the interface, the description of the interface, the hostname of the router and the domain name. All this router carries in its configuration. The main thing is to collect and properly glue and bind to the correct address.
So this task should be automated.
The first thought, the analysis of configurations, quickly faded away, the network is large, multi-vendor, and even the equipment from different generations, so the idea of parsing configs quickly became unpopular.
The second thought is to use what gives the necessary answers to universal requests to the equipment of different vendors. The answer was obvious - SNMP. It, with all its features, is implemented in the software of any vendor.
So, let's begin
We put Python.
sudo apt-get install python3
We will need modules to work with SNMP, IP addresses, over time. But to install them you need to put pip. True, now it comes with python.
sudo apt install python3-pip
And now we put the modules.
pip3 install pysnmp
pip3 install datetime
pip3 install ipaddress
Let's try to get its hostname from the router. SNMP uses an OID to query the host. On the OID, the host will return information corresponding to that OID. We want to get the hostname - you need to request 1.3.6.1.2.1.1.5.0.
And so the first script that requests only the hostname.
# import section from pysnmp.hlapi import * from ipaddress import * from datetime import datetime # var section #snmp community_string = 'derfnutfo' # From file ip_address_host = '192.168.88.1' # From file port_snmp = 161 OID_sysName = '1.3.6.1.2.1.1.5.0' # From SNMPv2-MIB hostname/sysname # function section def snmp_getcmd(community, ip, port, OID): return (getCmd(SnmpEngine(), CommunityData(community), UdpTransportTarget((ip, port)), ContextData(), ObjectType(ObjectIdentity(OID)))) def snmp_get_next(community, ip, port, OID): errorIndication, errorStatus, errorIndex, varBinds = next(snmp_getcmd(community, ip, port, OID)) for name, val in varBinds: return (val.prettyPrint()) #code section sysname = (snmp_get_next(community_string, ip_address_host, port_snmp, OID_sysName)) print('hostname= ' + sysname) Run and get:
hostname = MikroTik
Let's analyze the script in more detail:
First we import the necessary modules:
1. pysnmp - provides the script with the host via SNMP
2. ipaddress - provides work with addresses. Checking addresses for correctness, checking for occurrences of the address in the network address, and so on.
3. datetime- get current time. In this task, you need to organize logs.
Then we get four variables:
1. community
2. host address
3. SNMP port
4. OID value
Two functions:
1. snmp_getcmd
2. snmp_get_next
The first function sends a GET request to the specified host, on the specified port, with the specified comminity and OID.
The second function is the snmp_getcmd generator. Probably break into two functions was not quite right, but it happened :)
This script lacks some things:
1. In the script you need to load the ip address of the hosts. For example, from a text file. When downloading, you need to check the download address for correctness, otherwise pysnmp can be very surprised and the script will stop with a traceback. It is unprincipled where you will get the addresses from the file, from the database, but you must be sure that the addresses you received are correct. And so, the source of addresses is a text file, one line - one address in decimal form.
2. Network equipment may be turned off at the time of the survey, may be configured incorrectly, as a result, pysnmp will give out in this case absolutely not what we are waiting for and with further processing of the received information we will get the script to stop with traceback. We need an error handler for our SNMP interaction.
3. You need a log file in which the processed errors will be recorded.
Download addresses and create a log file
Enter the variable for the file name.
We write the check_ip function to check if the address is correct.
We write the get_from_file function of loading addresses, which checks each address for correctness and if this is not the case, writes a message about it to the log.
We implement data loading in the list.
filename_of_ip = 'ip.txt' # Ip #log filename_log = 'zone_gen.log' # def check_ip(ip): # ip try: ip_address(ip) except ValueError: return False else: return True def get_from_file(file, filelog): # ip . - fd = open(file,'r') list_ip = [] for line in fd: line=line.rstrip('\n') if check_ip(line): list_ip.append(line) else: filed.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ': Error ip ' + line) print('Error ip ' + line) fd.close() return list_ip #code section # filed = open(filename_log,'w') # filed.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + '\n') ip_from_file = get_from_file(filename_of_ip, filed) for ip_address_host in ip_from_file: sysname = (snmp_get_next(community_string, ip_address_host, port_snmp, OID_sysName)) print('hostname= ' + sysname) filed.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + '\n') filed.close() Create a file ip.txt
192.168.88.1
172.1.1.1
12.43.dsds.f4
192.168.88.1
The second address in this list does not respond to snmp. Run the script and verify that an error handler is needed for SNMP.
Error ip 12.43.dsds.f4
hostname = MikroTik
Traceback (most recent call last):
File "/snmp/snmp_read3.py", line 77, in print ('hostname =' + sysname)
TypeError: Can't convert 'NoneType' object to str implicitly
Process finished with exit code 1
From the contents of the traceback is impossible to understand that the cause of the failure was an inaccessible host. Let's try to intercept possible reasons for stopping the script and record all the information in the log.
Create an error handler for pysnmp
The snmp_get_next function already has error error output, errorStatus, errorIndex, varBinds. In varBinds, the resulting data is uploaded, error information is uploaded to variables beginning with error. It only needs to be properly processed. Since in the future there will be several more functions for working with snmp in the script, it makes sense to bring the error handling into a separate function.
def errors(errorIndication, errorStatus, errorIndex, ip, file): # False if errorIndication: print(errorIndication, 'ip address ', ip) file.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : ' + str(errorIndication) + ' = ip address = ' + ip + '\n') return False elif errorStatus: print(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : ' + '%s at %s' % (errorStatus.prettyPrint(), errorIndex and varBinds[int(errorIndex) - 1][0] or '?')) file.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : ' + '%s at %s' % (errorStatus.prettyPrint(), errorIndex and varBinds[int(errorIndex) - 1][0] or '?' + '\n')) return False else: return True And now we add error handling and writing to the log file to the snmp_get_next function. The function should now return not only data, but also a message about whether there were any errors.
def snmp_get_next(community, ip, port, OID, file): errorIndication, errorStatus, errorIndex, varBinds = next(snmp_getcmd(community, ip, port, OID)) if errors(errorIndication, errorStatus, errorIndex, ip, file): for name, val in varBinds: return (val.prettyPrint(), True) else: file.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : Error snmp_get_next ip = ' + ip + ' OID = ' + OID + '\n') return ('Error', False) Now it is necessary to rewrite the code section a bit, taking into account the fact that now there are messages about the success of the request. In addition, add a few checks:
1. Sysname is less than three characters. We write to the log file, so that later look at the closer.
2. We will find out that some Huawei and Catos give only the hostname for the request. Since we don’t want to look for the OID separately for them (it’s not a fact that it exists at all, maybe it’s a software bug), we’ll add such domain hosts manually.
3. We find that hosts with the wrong comminity behave differently, most initiate the error handler, and some for some reason respond that the script perceives as a normal situation.
4. Add a different logging level at the time of debugging, in order not to pick out unnecessary messages throughout the script.
for ip_address_host in ip_from_file: # sysname hostname+domainname, sysname, flag_snmp_get = (snmp_get_next(community_string, ip_address_host, port_snmp, OID_sysName, filed)) if flag_snmp_get: # , snmp if sysname == 'No Such Object currently exists at this OID': # community . , traceback. , community, hostname, print('ERROR community', sysname, ' ', ip_address_host) filed.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : ' + 'ERROR community sysname = ' + sysname + ' ip = ' + ip_address_host + '\n') else: if log_level == 'debug': filed.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : ' + ' sysname ' + sysname + ' type ' + str(type(sysname)) + ' len ' + str(len(sysname)) + ' ip ' + ip_address_host + '\n') if len(sysname) < 3 if log_level == 'debug' or log_level == 'normal': filed.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : ' + 'Error sysname 3 = ' + sysname + ' ip = ' + ip_address_host + '\n') if sysname.find(domain) == -1: # - hostname , Huawei Catos sysname = sysname + '.' + domain if log_level == 'debug' or log_level == 'normal': filed.write("check domain : " + sysname + " " + ip_address_host + " " + "\n") print('hostname= ' + sysname) Check this script on the same ip.txt file.
Error Garbage in the source of ip addresses 12.43.dsds.f4
hostname = MikroTik.mydomain.ru
No SNMP response received before timeout ip address 172.1.1.1
hostname = MikroTik.mydomain.ru
Everything worked normally, we caught all the errors, the script missed the hosts with errors. Now this script can collect the hostname from all devices responding to snmp.
I hide the full text of the script under the spoiler.
Script
# import section from pysnmp.hlapi import * from ipaddress import * from datetime import datetime # var section #snmp community_string = 'derfnutfo' ip_address_host = '192.168.88.1' port_snmp = 161 OID_sysName = '1.3.6.1.2.1.1.5.0' # From SNMPv2-MIB hostname/sysname filename_of_ip = 'ip.txt' # Ip #log filename_log = 'zone_gen.log' # log_level = 'debug' domain='mydomain.ru' # function section def snmp_getcmd(community, ip, port, OID): # type class 'generator' errorIndication, errorStatus, errorIndex, result[3] - # get SNMP OID return (getCmd(SnmpEngine(), CommunityData(community), UdpTransportTarget((ip, port)), ContextData(), ObjectType(ObjectIdentity(OID)))) def snmp_get_next(community, ip, port, OID, file): # class generator def snmp_get # errors, class 'pysnmp.smi.rfc1902.ObjectType' OID ( name) ( val) # errorIndication, errorStatus, errorIndex, varBinds = next(snmp_getcmd(community, ip, port, OID)) if errors(errorIndication, errorStatus, errorIndex, ip, file): for name, val in varBinds: return (val.prettyPrint(), True) else: file.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : Error snmp_get_next ip = ' + ip + ' OID = ' + OID + '\n') return ('Error', False) def get_from_file(file, filelog): # ip file, filelog fd = open(file, 'r') list_ip = [] for line in fd: line=line.rstrip('\n') if check_ip(line): list_ip.append(line) else: filed.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ': Error ip ' + line) print('Error ip ' + line) fd.close() return list_ip def check_ip(ip): # ip . False . try: ip_address(ip) except ValueError: return False else: return True def errors(errorIndication, errorStatus, errorIndex, ip, file): # False file if errorIndication: print(errorIndication, 'ip address ', ip) file.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : ' + str( errorIndication) + ' = ip address = ' + ip + '\n') return False elif errorStatus: print(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : ' + '%s at %s' % ( errorStatus.prettyPrint(), errorIndex and varBinds[int(errorIndex) - 1][0] or '?' )) file.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : ' + '%s at %s' % ( errorStatus.prettyPrint(), errorIndex and varBinds[int(errorIndex) - 1][0] or '?' + '\n')) return False else: return True #code section # filed = open(filename_log,'w') # filed.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + '\n') ip_from_file = get_from_file(filename_of_ip, filed) for ip_address_host in ip_from_file: # sysname hostname+domainname, sysname, flag_snmp_get = (snmp_get_next(community_string, ip_address_host, port_snmp, OID_sysName, filed)) if flag_snmp_get: # , snmp if sysname == 'No Such Object currently exists at this OID': # community . , traceback. , community, hostname, print('ERROR community', sysname, ' ', ip_address_host) filed.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : ' + 'ERROR community sysname = ' + sysname + ' ip = ' + ip_address_host + '\n') else: if log_level == 'debug': filed.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : ' + ' sysname ' + sysname + ' type ' + str( type(sysname)) + ' len ' + str(len(sysname)) + ' ip ' + ip_address_host + '\n') if len(sysname) < 3: sysname = 'None_sysname' if log_level == 'debug' or log_level == 'normal': filed.write(datetime.strftime(datetime.now(), "%Y.%m.%d %H:%M:%S") + ' : ' + 'Error sysname 3 = ' + sysname + ' ip = ' + ip_address_host + '\n') if sysname.find(domain) == -1: # - hostname , Huawei Catos sysname = sysname + '.' + domain if log_level == 'debug' or log_level == 'normal': filed.write("check domain : " + sysname + " " + ip_address_host + " " + "\n") print('hostname= ' + sysname) filed.close() Now it remains to collect the interface names, interface descriptions, interface addresses and correctly expand into the bind configuration files. But about this in the second part.
PS: I note that in a good way the messages in the log file should be formed in a different way.
For example: time special character code_ error, special character description_ error, special character, additional_information. This will help you later set up automatic log processing.
UPD: error correction.
Source: https://habr.com/ru/post/333614/
All Articles