Automating the ip-network using the tools at hand (Python)
This article is suitable for network professionals who are looking for examples of possible automation of an ip network with the help of available tools.
As one of the options for automation, this interaction software environment with the CLI (Command Line Interface) equipment, the so-called 'Screen Scraping'. Actually, this option will be discussed.
As a software environment, the Python version 3.3 programming language will be used. For those who doubt the need to learn a programming language, it should be noted that basic Python programming skills are fairly easy to learn and are sufficient for solving the problems described below. In the future, with the improvement of skills, the code and level of products will be improved. For remote interaction with the equipment, the SSH protocol will be mainly used, therefore, as an option for working with SSH, to facilitate tasks, an additional module for Python is chosen - Paramiko. As a rule, consideration of solving specific tasks can contribute to better mastering the material, so without delaying the process, selective examples of tasks of increasing degree of complexity will be further considered and solved using the above described tools (it is important to note that all ip addresses, logins, passwords, names and specific parameter values ​​from network devices are fictional, any coincidence is random).
It is necessary to periodically analyze the network routing table, with the determination of the number of prefixes received from the Aplink, Peering, IX junctions and client inclusions, divided by the number of AS BGPs to the final resource. This analysis in a certain period of time, can show the dynamics of improving indicators of connectivity, not only on the basis of the client cone
')
Solution: In most networks, the division of routing information among the joints can be determined on the basis of the value of the BGP (LP) attribute Local Preference. Accordingly, determining which request in the router CLI allows you to display the LP and AS_PATH values ​​for the active routes, and then processing the output, you can get the desired statistics. Suppose Juniper routers are used on the studied network, respectively, one of these commands may be:
The result of a request for such a command is the following output:
One of the possible implementations may be the following code (comments are written directly in the code after #):
To run the script on your own network, you need on a device that has direct access to the Juniper network router, install Python3 + Paramiko, copy the above code into a file with the .py extension, substituting your own LP and ip values, login, password and tcp port for ssh. Run the resulting script (for example, on FreeBsd using the command python3.3 route_scan.py Enter). The output of the program will be as follows:
Based on the obtained results, it is possible to assess the degree of network connectivity, for networks with poorly developed external connections, the meter will be mostly at Uplink joints. For networks with an actively pursued peering policy, the counter will increase in favor of IX and peering, and ultimately with the ability to correctly present the connectivity of the network, in the direction of client connections.
It is necessary to implement automatic network configuration changes in response to unwanted events. As an option, there are external junctions with good network quality parameters, but insufficient bandwidth and periodic spontaneous utilization of traffic. There are also other external interfaces with the availability of the same external resources, sufficient capacity, but less attractive parameters. Thus, it is necessary to automatically redirect traffic when certain junction utilization thresholds are reached. The script is desirable to track through logging.
Solution: Suppose the network uses Cisco equipment (IOS). Traffic at the junction is dominated by outgoing, resulting in the generation of content within the network. Depending on the BGP Community assigned at the junction, in another segment of its own network, the priority is assigned to the prefix and the choice of the best route. Accordingly, the task of the software is to track the specified threshold for disposal and change the assigned BGP Community at the junction.
One possible implementation could be the following code (all comments are written directly in the code after #):
To periodically run the script, you can use the standard cron utility, which is in each UNIX system (the cycle is not more than once every 2 minutes). The results will be recorded in a separate file with the date and time of the changes.
With the help of this construction, it is possible to change other parameters, as well as to influence incoming traffic, using the BGP community policies selected by the interacting operator to control traffic.
It is necessary to implement in the network an automatic update system for BGP filters at external interfaces. At the same time, the system operator must choose independently which filters are updated and which are not. Interaction with the system should be carried out through the Web interface. To minimize errors, the candidate configuration of the candidate configuration must be implemented before use.
Solution: Suppose the network used equipment Juniper (Junos). Filters are based on a regular AS_PATH expression, taking into account AS Origin. As settings on the router, the as-path-group is used as part of the policy-statement used in the BGP import policies. Accordingly, the systems should, once a day, in the presence of changes in the Radb database, update the AS_Path filters on the external network junctions. The following system of interrelated scripts can be used as an implementation:
The system is built using several program files:
Since there is some trend towards personalization of software, let's call our system Fibber.
Consider each block separately:
The main web page is used to enter the as-set and ip addresses of the router, as well as to work with other modules. Below is one of the simplest implementations based on HTML markup:
To simplify the operation of the system, specialized database management systems are not used. The database is a set of text files and a variable python list.
So, after clicking the “add to queue” button on the main page, the system accesses the Python script, through which the entered values ​​are added to the list. Below is the construction of the Script (explained in the code):
The module is a list of data on all routers and As-set participating in auto-update, enclosed in a function for use as a Python library. This file is completely overwritten when adding and deleting data, this can be seen from the example above. The design below:
This module is used to display the list of ip routers and the names of the changed as-set. It also contains an HTML form for deleting list items. The construction below (explanation after #):
The module works in the same way as the module for adding data, with the difference that it is not adding, but deleting data from the list. The design below:
The module retrieves the AS BGP list from the RADB database for AS-SET, comparing the AS BGP list with the list obtained in the previous period. Creation of AS lists for changing equipment configuration. The construction below (explanation after #):
The module for working with equipment serves to change the configuration, based on the generated data for the configuration in the section of the candidate. During execution, the module performs not only the configuration, but also a series of checks to minimize the probability of errors in the configuration. The design of the module is presented below (explanation in the code after #):
, error_log.html, :
, , :
— AS-SET (Python) – 8.00 18.00
— (Python) – 8.00 18.00
— RADB – 18.00
— – 00.00
, , , . , (, , , ..). , , , , ..
, , unix Crontab, :
, Python .
, , . , . , , , Netmiko.
Paramiko. NetConf, , .
blog.dzinko.org/2011/03/python.html - Detailed description of regular expressions with examples.
stackoverflow.com - Site question answer for programmers. Most likely the answer to the unresolved problem is there.
habrahabr.ru/post/115436 - a detailed description of regular expressions.
pythonworld.ru - a lot of accessible clear information on python with examples.
www.radb.net
github.com/paramiko/paramiko - Paramiko library.
github.com/ktbyers/netmiko is a ready-made Python library for working with network devices.
As one of the options for automation, this interaction software environment with the CLI (Command Line Interface) equipment, the so-called 'Screen Scraping'. Actually, this option will be discussed.
As a software environment, the Python version 3.3 programming language will be used. For those who doubt the need to learn a programming language, it should be noted that basic Python programming skills are fairly easy to learn and are sufficient for solving the problems described below. In the future, with the improvement of skills, the code and level of products will be improved. For remote interaction with the equipment, the SSH protocol will be mainly used, therefore, as an option for working with SSH, to facilitate tasks, an additional module for Python is chosen - Paramiko. As a rule, consideration of solving specific tasks can contribute to better mastering the material, so without delaying the process, selective examples of tasks of increasing degree of complexity will be further considered and solved using the above described tools (it is important to note that all ip addresses, logins, passwords, names and specific parameter values ​​from network devices are fictional, any coincidence is random).
1. Task: Analysis of network performance
It is necessary to periodically analyze the network routing table, with the determination of the number of prefixes received from the Aplink, Peering, IX junctions and client inclusions, divided by the number of AS BGPs to the final resource. This analysis in a certain period of time, can show the dynamics of improving indicators of connectivity, not only on the basis of the client cone
')
Solution: In most networks, the division of routing information among the joints can be determined on the basis of the value of the BGP (LP) attribute Local Preference. Accordingly, determining which request in the router CLI allows you to display the LP and AS_PATH values ​​for the active routes, and then processing the output, you can get the desired statistics. Suppose Juniper routers are used on the studied network, respectively, one of these commands may be:
# show route protocol bgp table inet.0 active-path | no-more.
The result of a request for such a command is the following output:
1.1.1.0/24 * [BGP / 170] 2w3d 23:44:20, MED 0, localpref 150
AS path: 3356 6453 4755 45528 I, validation state: unverified
> to 2.1.1.1 via ae0.0
1.2.1.0/24 * [BGP / 170] 1d 20:20:51, MED 0, localpref 170, from 10.0.0.1
AS path: 9498 45528 I, validation state: unverified
> to 2.1.1.5 via ae10.0
...
One of the possible implementations may be the following code (comments are written directly in the code after #):
Python code
#!/usr/local/bin/python3.3 ## -*- coding: koi8-r -*- ### import paramiko import time import datetime import sys import re import os import socket import base64 ### user = 'user' secret= pas = base64.b64decode(b'cGFzc3dvcmQ=').decode('ascii') # , . # base64.b64encode('password'.encode('ascii')) port = por = 22 host='10.10.10.10' ### paramiko : remote_conn_pre = paramiko.SSHClient() remote_conn_pre.set_missing_host_key_policy(paramiko.AutoAddPolicy()) ### SHA remote_conn_pre.connect(hostname=host, username=user, password=pas, port=por, timeout=90) ### remote_conn = remote_conn_pre.invoke_shell() ### , remote_conn.settimeout(20) ### 20 sec remote_conn.send ('\n') ### 'Enter' time.sleep(1) ### 1 check=remote_conn.recv(2048) ### 2048 print(check.decode('ascii')) ### . .decode('ascii') . , # ( #) remote_conn.send ('show route protocol bgp table inet.0 active-path | no-more' + '\n') ### , 'Enter' def while_not_end_plus_recive(): ### def - , buff = b'' ### , . resp1=b'' while not buff.endswith(b'0> '): ### while 0> resp = remote_conn.recv(12002048) buff += resp ### buff resp print ('!', end='', flush=True) ### , , return buff ### , While check=str(while_not_end_plus_recive()) ### check, # str() print('\n') ### remote_conn_pre.close() ### SSH . ### , : timestr = time.strftime("%d%m%Y") ### , , log_out=open('/usr/SCRIPTS_FOR_PYTHON/route_tables/route_table_'+timestr+'.txt', 'w') ### log_out.write(check) ### log_out.close() ### , log_out=open('/usr/SCRIPTS_FOR_PYTHON/route_tables/route_table_'+timestr+'.txt', 'r') ### . check. data=log_out.read() ### . log_out.close() ### ip . , data # . 'BGP' comp=re.compile('BGP') ### re, comp. split_out=comp.split(data)### ### , : prefixes_1_as_uplink=0 prefixes_2_as_uplink=0 prefixes_3_as_uplink=0 prefixes_4_as_uplink=0 prefixes_more_then_4_as_uplink=0 prefixes_1_as_IX=0 prefixes_2_as_IX=0 prefixes_3_as_IX=0 prefixes_4_as_IX=0 prefixes_more_then_4_as_IX=0 prefixes_1_as_peer=0 prefixes_2_as_peer=0 prefixes_3_as_peer=0 prefixes_4_as_peer=0 prefixes_more_then_4_as_peer=0 prefixes_1_as_client=0 prefixes_2_as_client=0 prefixes_3_as_client=0 prefixes_4_as_client=0 prefixes_more_then_4_as_client=0 own_as_prefixes=0 other_prefixes=0 ### for, i split_out: for i in range(len(split_out)): if len(re.findall('localpref 150',str(split_out[i])))==1: ### if i, for, # LP Uplink count_as_from_uplink_pre=re.findall(r'AS path:([^]]*), val', split_out[i]) ### Uplink, AS BGP count_as_from_uplink=re.findall(r'[\d]+', str(count_as_from_uplink_pre)) ### AS BGP if len(set(count_as_from_uplink))==1: ### AS BGP, , prefixes_1_as_uplink = prefixes_1_as_uplink + 1 if len(set(count_as_from_uplink))==2: prefixes_2_as_uplink = prefixes_2_as_uplink + 1 if len(set(count_as_from_uplink))==3: prefixes_3_as_uplink = prefixes_3_as_uplink + 1 if len(set(count_as_from_uplink))==4: prefixes_4_as_uplink = prefixes_4_as_uplink + 1 if len(set(count_as_from_uplink))>4: prefixes_more_then_4_as_uplink = prefixes_more_then_4_as_uplink + 1 else: pass elif len(re.findall('localpref 170|localpref 175',str(split_out[i])))==1: ### if i, for, # LP IX. , LP 1 . count_as_from_IX_pre=re.findall(r'AS path:([^]]*), val', split_out[i]) count_as_from_IX=re.findall(r'[\d]+', str(count_as_from_IX_pre)) if len(set(count_as_from_IX))==1: prefixes_1_as_IX = prefixes_1_as_IX + 1 if len(set(count_as_from_IX))==2: prefixes_2_as_IX = prefixes_2_as_IX + 1 if len(set(count_as_from_IX))==3: prefixes_3_as_IX = prefixes_3_as_IX + 1 if len(set(count_as_from_IX))==4: prefixes_4_as_IX = prefixes_4_as_IX + 1 if len(set(count_as_from_IX))>4: prefixes_more_then_4_as_IX = prefixes_more_then_4_as_IX + 1 else: pass elif len(re.findall('localpref 180',str(split_out[i])))==1: ### if i, for, # LP Peer. count_as_from_peer_pre=re.findall(r'AS path:([^]]*), val', split_out[i]) count_as_from_peer=re.findall(r'[\d]+', str(count_as_from_peer_pre)) if len(set(count_as_from_peer))==1: prefixes_1_as_peer = prefixes_1_as_peer + 1 if len(set(count_as_from_peer))==2: prefixes_2_as_peer = prefixes_2_as_peer + 1 if len(set(count_as_from_peer))==3: prefixes_3_as_peer = prefixes_3_as_peer + 1 if len(set(count_as_from_peer))==4: prefixes_4_as_peer = prefixes_4_as_peer + 1 if len(set(count_as_from_peer))>4: prefixes_more_then_4_as_peer = prefixes_more_then_4_as_peer + 1 else: pass elif len(re.findall('localpref 200|localpref 190',str(split_out[i])))==1: ### if i, for, # LP , Clients. count_as_from_client_pre=re.findall(r'AS path:([^]]*), val', split_out[i]) count_as_from_client=re.findall(r'[\d]+', str(count_as_from_client_pre)) if len(set(count_as_from_client))==1: prefixes_1_as_client = prefixes_1_as_client + 1 if len(set(count_as_from_client))==2: prefixes_2_as_client = prefixes_2_as_client + 1 if len(set(count_as_from_client))==3: prefixes_3_as_client = prefixes_3_as_client + 1 if len(set(count_as_from_client))==4: prefixes_4_as_client = prefixes_4_as_client + 1 if len(set(count_as_from_client))>4: prefixes_more_then_4_as_client = prefixes_more_then_4_as_client + 1 if count_as_from_client==[]: own_as_prefixes= own_as_prefixes + 1 else: pass else: other_prefixes=other_prefixes + 1 ### : print('prefixes_1_as_uplink: '+str(prefixes_1_as_uplink)) print('prefixes_2_as_uplink: '+str(prefixes_2_as_uplink)) print('prefixes_3_as_uplink: '+str(prefixes_3_as_uplink)) print('prefixes_4_as_uplink: '+str(prefixes_4_as_uplink)) print('prefixes_more_then_4_as_uplink: '+str(prefixes_more_then_4_as_uplink)) print('all_uplink_prefixes: '+str(prefixes_1_as_uplink+prefixes_2_as_uplink+prefixes_3_as_uplink+prefixes_4_as_uplink+prefixes_more_then_4_as_uplink)+'\n') print('prefixes_1_as_IX: '+str(prefixes_1_as_IX)) print('prefixes_2_as_IX: '+str(prefixes_2_as_IX)) print('prefixes_3_as_IX: '+str(prefixes_3_as_IX)) print('prefixes_4_as_IX: '+str(prefixes_4_as_IX)) print('prefixes_more_then_4_as_IX: '+str(prefixes_more_then_4_as_IX)) print('all_IX_prefixes: '+str(prefixes_1_as_IX+prefixes_2_as_IX+prefixes_3_as_IX+prefixes_4_as_IX+prefixes_more_then_4_as_IX)+'\n') print('prefixes_1_as_peer: '+str(prefixes_1_as_peer)) print('prefixes_2_as_peer: '+str(prefixes_2_as_peer)) print('prefixes_3_as_peer: '+str(prefixes_3_as_peer)) print('prefixes_4_as_peer: '+str(prefixes_4_as_peer)) print('prefixes_more_then_4_as_peer: '+str(prefixes_more_then_4_as_peer)) print('all_peer_prefixes: '+str(prefixes_1_as_peer+prefixes_2_as_peer+prefixes_3_as_peer+prefixes_4_as_peer+prefixes_more_then_4_as_peer)+'\n') print('prefixes_1_as_client: '+str(prefixes_1_as_client)) print('prefixes_2_as_client: '+str(prefixes_2_as_client)) print('prefixes_3_as_client: '+str(prefixes_3_as_client)) print('prefixes_4_as_client: '+str(prefixes_4_as_client)) print('prefixes_more_then_4_as_client: '+str(prefixes_more_then_4_as_client)) print('all_client_prefixes: '+str(prefixes_1_as_client+prefixes_2_as_client+prefixes_3_as_client+prefixes_4_as_client+prefixes_more_then_4_as_client)+'\n') print('own_as_prefixes: '+str(own_as_prefixes)) print('other_prefixes: '+str(other_prefixes)) To run the script on your own network, you need on a device that has direct access to the Juniper network router, install Python3 + Paramiko, copy the above code into a file with the .py extension, substituting your own LP and ip values, login, password and tcp port for ssh. Run the resulting script (for example, on FreeBsd using the command python3.3 route_scan.py Enter). The output of the program will be as follows:
prefixes_1_as_uplink: 2684 prefixes_2_as_uplink: 90048 prefixes_3_as_uplink: 132173 prefixes_4_as_uplink: 61119 prefixes_more_then_4_as_uplink: 15472 all_uplink_prefixes: 301496 prefixes_1_as_IX: 21876 prefixes_2_as_IX: 72699 prefixes_3_as_IX: 38738 prefixes_4_as_IX: 13233 prefixes_more_then_4_as_IX: 2960 all_IX_prefixes: 149506 prefixes_1_as_peer: 8990 prefixes_2_as_peer: 18772 prefixes_3_as_peer: 17150 prefixes_4_as_peer: 3236 prefixes_more_then_4_as_peer: 1372 all_peer_prefixes: 49520 prefixes_1_as_client: 14348 prefixes_2_as_client: 13166 prefixes_3_as_client: 981 prefixes_4_as_client: 175 prefixes_more_then_4_as_client: 13 all_client_prefixes: 28683 own_as_prefixes: 103 other_prefixes: 21911
Based on the obtained results, it is possible to assess the degree of network connectivity, for networks with poorly developed external connections, the meter will be mostly at Uplink joints. For networks with an actively pursued peering policy, the counter will increase in favor of IX and peering, and ultimately with the ability to correctly present the connectivity of the network, in the direction of client connections.
2. Task: System reaction, in response to undesirable events occurring.
It is necessary to implement automatic network configuration changes in response to unwanted events. As an option, there are external junctions with good network quality parameters, but insufficient bandwidth and periodic spontaneous utilization of traffic. There are also other external interfaces with the availability of the same external resources, sufficient capacity, but less attractive parameters. Thus, it is necessary to automatically redirect traffic when certain junction utilization thresholds are reached. The script is desirable to track through logging.
Solution: Suppose the network uses Cisco equipment (IOS). Traffic at the junction is dominated by outgoing, resulting in the generation of content within the network. Depending on the BGP Community assigned at the junction, in another segment of its own network, the priority is assigned to the prefix and the choice of the best route. Accordingly, the task of the software is to track the specified threshold for disposal and change the assigned BGP Community at the junction.
One possible implementation could be the following code (all comments are written directly in the code after #):
Python code
#!/usr/local/bin/python3.3 ## -*- coding: koi8-r -*- import paramiko import time import datetime import sys import re import os import socket import subprocess import random import cgi import cgitb import base64 host='10.10.10.10' user='user' pas= base64.b64decode(b'cGFzc3dvcmQ=').decode('ascii') por=22 def log( message): ### , log_out1=open('/usr/SCRIPTS_FOR_PYTHON/speed_log.txt', 'a') ### log_out1.write(str(datetime.datetime.now()) + ':'+ str(message) + '\n') ### log_out1.close() pass def while_not_end_plus_recive(): ### buff = b'' resp1=b'' try: ### , While while not buff.endswith(b'#'): resp = remote_conn.recv(12002048) buff += resp except socket.timeout: ### , LOG log('Device did not respond') sys.exit() return buff ### , . : #1. SNMP . #2. , 'sh int ge-1/1/1 | include output' ( ) # , , . #3. , . ### SNMP. SNMP NET_SNMP, - Python in_rate=open('/usr/SCRIPTS_FOR_PYTHON/test_rate.txt', 'w') ### ### snmp OID ifHCOutOctets ifName subprocess.call(['snmpwalk -v2c -c TEST 10.222.0.177 1.3.6.1.2.1.31.1.1.1.10.10201'], bufsize=0, shell=True, stdout=(in_rate)) in_rate.close() f=open('/usr/SCRIPTS_FOR_PYTHON/test_rate.txt', 'r') inrate1_pre=f.read() ### f.close() time.sleep(60) ### in_rate=open('/usr/SCRIPTS_FOR_PYTHON/test_rate.txt', 'w') # subprocess.call(['snmpwalk -v2c -c TEST 10.222.0.177 1.3.6.1.2.1.31.1.1.1.10.10201'], bufsize=0, shell=True, stdout=(in_rate)) in_rate.close() f=open('/usr/SCRIPTS_FOR_PYTHON/test_rate.txt', 'r') inrate2_pre=f.read()### f.close() inrate1=re.findall('[\d]+', str(re.findall(': [\d]+', inrate1_pre))) ### inrate2=re.findall('[\d]+', str(re.findall(': [\d]+', inrate2_pre))) rate=(int(inrate2[0])-int(inrate1[0]))*8/60 ### ### (). . # #input_rate_recv=str(remote_conn.send('show interface ge-1/1/1/ | include input\n')) #input_rate_recv_out=while_not_end_plus_recive() #output_rate_recv=str(remote_conn.send('show interface ge-1/1/1/ | include output\n')) #output_rate_recv_out=while_not_end_plus_recive() #input_rate_search_pre=str(re.findall('[\d]+' + bps, input_rate_recv_out)) #input_rate_search=re.findall('[\d]+', input_rate_search_pre) #input_rate=''.join(input_rate_search) #output_rate_search_pre=str(re.findall('[\d]+' + bps, output_rate_recv_out)) #output_rate_search=re.findall('[\d]+', output_rate_search_pre) #output_rate=''.join(output_rate_search) if rate>9000000000: # , remote_conn_pre = paramiko.SSHClient()### , SSH remote_conn_pre.set_missing_host_key_policy(paramiko.AutoAddPolicy()) remote_conn_pre.connect(hostname=host, username=user, password=pas, port=por, timeout=90) remote_conn = remote_conn_pre.invoke_shell() remote_conn.settimeout(90) remote_conn.send('sh running-config partition route-map | section include FROM-TEST-POLICY'+'\n'+'\n') ### , check=while_not_end_plus_recive()### print (check) check_policy=re.findall('65000:1', str(check)) ### community print (str(check_policy)) if check_policy==['65000:1']: ### BGP community , . remote_conn_pre.close() log('Policy already_changed') ### sys.exit() else: ### , pass remote_conn.send('conf t'+'\n') ### while_not_end_plus_recive() remote_conn.send('route-map FROM-TEST-POLICY permit 10'+'\n') while_not_end_plus_recive() remote_conn.send('no set community'+'\n') while_not_end_plus_recive() remote_conn.send('set community 65000:1 additive'+'\n') while_not_end_plus_recive() remote_conn.send('exit'+'\n') while_not_end_plus_recive() remote_conn.send('exit'+'\n') while_not_end_plus_recive() remote_conn.send('exit'+'\n') log('Policy has been changed') ### remote_conn_pre.close() time.sleep(3600) ### . # , ( ) ### , remote_conn_pre.connect(hostname=host, username=user, password=pas, port=por, timeout=90) remote_conn = remote_conn_pre.invoke_shell() remote_conn.settimeout(780) remote_conn.send('sh running-config partition route-map | section include FROM-TEST-POLICY'+'\n'+'\n') ### check=while_not_end_plus_recive() check_policy=re.findall('65000:2', str(check)) if check_policy==['65000:2']: remote_conn_pre.close() log('Policy already rewert') ### sys.exit() else: pass remote_conn.send('conf t'+'\n'+'\n') while_not_end_plus_recive() remote_conn.send('route-map FROM-TEST-POLICY permit 10'+'\n') while_not_end_plus_recive() remote_conn.send('no set community'+'\n') while_not_end_plus_recive() remote_conn.send('set community 65000:2 additive'+'\n'+'\n') while_not_end_plus_recive() remote_conn.send('exit'+'\n') while_not_end_plus_recive() remote_conn.send('exit'+'\n') while_not_end_plus_recive() remote_conn.send('exit'+'\n') log('Policy has been rewerted') ### remote_conn_pre.close() time.sleep(2) else: pass To periodically run the script, you can use the standard cron utility, which is in each UNIX system (the cycle is not more than once every 2 minutes). The results will be recorded in a separate file with the date and time of the changes.
With the help of this construction, it is possible to change other parameters, as well as to influence incoming traffic, using the BGP community policies selected by the interacting operator to control traffic.
3. Task: Planned changes on the network
It is necessary to implement in the network an automatic update system for BGP filters at external interfaces. At the same time, the system operator must choose independently which filters are updated and which are not. Interaction with the system should be carried out through the Web interface. To minimize errors, the candidate configuration of the candidate configuration must be implemented before use.
Solution: Suppose the network used equipment Juniper (Junos). Filters are based on a regular AS_PATH expression, taking into account AS Origin. As settings on the router, the as-path-group is used as part of the policy-statement used in the BGP import policies. Accordingly, the systems should, once a day, in the presence of changes in the Radb database, update the AS_Path filters on the external network junctions. The following system of interrelated scripts can be used as an implementation:
The system is built using several program files:
- Home web page (html).
- Module add data AS-SET and router in the database (Python).
- Module for storing a list of data (Python).
- Module for viewing the database with a form for deleting positions (Python).
- Module for removing data from the data list (Python).
- The module works with the database RADB.
- The module works with network equipment.
Since there is some trend towards personalization of software, let's call our system Fibber.
Consider each block separately:
Home web page (html)
The main web page is used to enter the as-set and ip addresses of the router, as well as to work with other modules. Below is one of the simplest implementations based on HTML markup:
HTML code
<meta charset="koi8-r"> <form name=f1 method="get"> <div id="parentId"> <div> <p><b>IP Loopback:</b><br> <!-- IP ( )--> <nobr><input name="name1" required pattern="\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}" type="text" style="width:300px;" placeholder="Enter ip Loopback" /> <p><b>AS-SET name:</b><br> <!-- AS-SET, RADB --> <input name="url1" required type="text" style="width:300px;" placeholder="Enter AS-SET name" /> </div> </div> <!-- --> <input type="submit" value=" " onclick="f1.action='/cgi-bin/add_to_db_info.py'" style="background-color:#26c809; color:#fdfafa;"/> </form> <br> <!-- , --> <p><b> :</b><br> <p> AS-SETs AS-PATH:<br> <a href="/cgi-bin/bd_host_and_aspath_print.py"> AS-PATH</a> <br> <br> <p> :<br> <a href="/sdn/error_log.html"> </a> <br><br><br> <a> <b>Fibber</b> - </a> Module for adding AS-SET data and a router to the database (path /cgi-bin/add_to_db_info.py)
To simplify the operation of the system, specialized database management systems are not used. The database is a set of text files and a variable python list.
So, after clicking the “add to queue” button on the main page, the system accesses the Python script, through which the entered values ​​are added to the list. Below is the construction of the Script (explained in the code):
Python code
#!/usr/local/bin/python3.3 # -*- coding: koi8-r -*- import paramiko import time import datetime import sys import re import socket ### - import cgi import cgitb cgitb.enable() ### from bd_host_and_aspath import all_data ### , , ip AS-SET. # bd_host_and_aspath form = cgi.FieldStorage() ### ip1 = form.getfirst('name1', 'EMPTY') ### , 'EMPTY' as_path1= form.getfirst('url1', 'EMPTY') host_and_aspath=[ip1, as_path1] ### data_check=all_data() ### , data_check.append(host_and_aspath) ### , . add_data=open('/usr/local/www/apache22/cgi-bin/bd_host_and_aspath.py', 'w') ### add_data.write("""#!/usr/local/bin/python3.3 # -*- coding: koi8-r -*- def all_data(): all_data="""+str(data_check)+""" return all_data""") add_data.close() print("Content-type:text/html\r\n\r\n") ### print ('For router Lo '+str(ip1)+' AS-SET '+str(as_path1)+' added to queue, configuration will be update at night.<br>') print ('Please check <a href="/cgi-bin/bd_host_and_aspath_print.py">Config File</a> if needed, or return to <a href="/sdn">start filter update page</a>') Data list storage module (path /cgi-bin/bd_host_and_aspath.py)
The module is a list of data on all routers and As-set participating in auto-update, enclosed in a function for use as a Python library. This file is completely overwritten when adding and deleting data, this can be seen from the example above. The design below:
Python code
#!/usr/local/bin/python3.3 # -*- coding: koi8-r -*- def all_data(): all_data=[['10.10.10.1', 'AS-TEST1'], ['10.10.10.10.2', 'AS-TEST2']] return all_data Database viewing module with the ability to delete positions (path /cgi-bin/bd_host_and_aspath_print.py)
This module is used to display the list of ip routers and the names of the changed as-set. It also contains an HTML form for deleting list items. The construction below (explanation after #):
Python code
#!/usr/local/bin/python3.3 # -*- coding: koi8-r -*- import cgi import bd_host_and_aspath ### as-set data=bd_host_and_aspath.all_data() ### print ("Content-type:text/html\r\n\r\n") ### Web print ('<a href="/sdn/filters1.html">Back to main page</a><br><br>') ### print ('Num.-- [ROUTER LOOPBACK, AS-SET]', end='<br><br>') for k in data: ### . print(str(data.index(k))+'--') ### print(k, end='<br>') print ('<form action="/cgi-bin/remove_data_from_db.py" method="get">') ### , remove . print ('<p><b>------REMOVE DATA--------<br>') print ('<p><b>Enter Number of position for removing data:</b><br><nobr><input name="remove1" required pattern="^[0-9]+$" type="text" style="width:300px;" placeholder="Enter Number" />') print ('<p><input type="submit" value="Remove" /></form><br>') Module to remove data from the data list (Python /cgi-bin/remove_data_from_db.py)
The module works in the same way as the module for adding data, with the difference that it is not adding, but deleting data from the list. The design below:
Python code
#!/usr/local/bin/python3.3 # -*- coding: koi8-r -*- import cgi from bd_host_and_aspath import all_data import cgitb cgitb.enable() form = cgi.FieldStorage() num_of_el = int(form.getfirst('remove1', 'EMPTY')) data_check=all_data() data_check.pop(num_of_el) add_data=open('/usr/local/www/apache22/cgi-bin/bd_host_and_aspath.py', 'w') add_data.write("""#!/usr/local/bin/python3.3 # -*- coding: koi8-r -*- def all_data(): all_data="""+str(data_check)+""" return all_data""") add_data.close() print("Content-type:text/html\r\n\r\n") print ('data has been removed for Prefix filters.<br>') print ('Please check <a href="/cgi-bin/bd_host_and_aspath_print.py">Config File</a> if needed, or return to <a href="/sdn/filters1.html">start filter update page</a>') The module for working with the RADB database (Python /usr/SCRIPTS_FOR_PYTHON/radb_v3.py)
The module retrieves the AS BGP list from the RADB database for AS-SET, comparing the AS BGP list with the list obtained in the previous period. Creation of AS lists for changing equipment configuration. The construction below (explanation after #):
Python code
#!/usr/local/bin/python3.3 ## -*- coding: koi8-r -*- import time import datetime import sys import re import os import socket import check_OS import subprocess import bd_host_and_aspath ### RADB : # 1 AS AS-SET # 2 whois AS AS-SET RADB # 3 AS RADB, ( RADB ), existing candidate, candidate . # candidate . # 4 AS existing. 3, # candidate, . , candidate def asset(host, asset): radb=open('/usr/SCRIPTS_FOR_PYTHON/data_from_radb/temp/'+asset+'.txt', 'w') # subprocess.call(['/usr/bin/whois -h whois.radb.net -p 43 \!i'+asset+',1/n/'], bufsize=0, shell=True, stdout=(radb)) # whois radb.close() f=open('/usr/SCRIPTS_FOR_PYTHON/data_from_radb/temp/'+asset+'.txt', 'r') data=f.read() f.close() filter_data_pre_temp=str(re.findall('AS[\d]+', data)) try: with open('/usr/SCRIPTS_FOR_PYTHON/data_from_radb/existing/'+host+','+asset+'.txt', 'r'): pass except IOError: f=open('/usr/SCRIPTS_FOR_PYTHON/data_from_radb/existing/'+host+','+asset+'.txt', 'w') f.close() s=open('/usr/SCRIPTS_FOR_PYTHON/data_from_radb/candidate/'+host+','+asset+'.txt', 'w') ## ip as-set, s.write(str(filter_data_pre_temp)) s.close() f_ext=open('/usr/SCRIPTS_FOR_PYTHON/data_from_radb/existing/'+host+','+asset+'.txt', 'r') data_ext=f_ext.read() f_ext.close() filter_data_pre_ext=str(re.findall('AS[\d]+', data_ext)) if filter_data_pre_ext>filter_data_pre_temp or filter_data_pre_ext<filter_data_pre_temp and len(filter_data_pre_temp)>3: update=open('/usr/SCRIPTS_FOR_PYTHON/data_from_radb/existing/'+host+','+asset+'.txt', 'w') update.write(filter_data_pre_temp) update.close() update=open('/usr/SCRIPTS_FOR_PYTHON/data_from_radb/candidate/'+host+','+asset+'.txt', 'w') update.write(filter_data_pre_temp) update.close() elif filter_data_pre_ext==filter_data_pre_temp or len(filter_data_pre_temp)<3: as_path='not_updated' try: os.remove('/usr/SCRIPTS_FOR_PYTHON/data_from_radb/candidate/'+host+','+asset+'.txt') pass except IOError: pass else: pass host_and_asset=bd_host_and_aspath.all_data() for i in range(len(host_and_asset)): # asset ip AS-SET. asset(str(host_and_asset[i][0]), str(host_and_asset[i][1])) Network equipment module (path /usr/SCRIPTS_FOR_PYTHON/ssh_stend_v5.py)
The module for working with equipment serves to change the configuration, based on the generated data for the configuration in the section of the candidate. During execution, the module performs not only the configuration, but also a series of checks to minimize the probability of errors in the configuration. The design of the module is presented below (explanation in the code after #):
Python code
#!/usr/local/bin/python3.3 ## -*- coding: koi8-r -*- ### import paramiko import time import datetime import sys import re import os import socket import subprocess import random import bd_host_and_aspath import base64 user = 'Fibber' secret = base64.b64decode(b'cGFzc3dvcmQ=').decode('ascii') port = 22 #### def chunks_in_filter(data_from_radb, n): # n , as-path-group return [data_from_radb[i:i + n] for i in range(0, len(data_from_radb), n)] def asset(host, asset): # AS_PATH 15 AS, candidate, try: f=open('/usr/SCRIPTS_FOR_PYTHON/data_from_radb/candidate/'+host+','+asset+'.txt', 'r') pass data=f.read() f.close() filter_data_pre=str(re.findall('AS[\d]+', data)) filter_data=re.findall('[\d]+', filter_data_pre) as_path_pre='|'.join(filter_data) as_path=chunks_in_filter(filter_data, 15) except IOError: as_path='ERROR' return as_path ### , , - # def log(logout, message): log_out=open('/usr/SCRIPTS_FOR_PYTHON/python_log.txt', 'a') log_out.write(str(datetime.datetime.now()) +'!!!'+str(message)+'<----------------------------'+'\n'+':'+ str(logout) + '\n') log_out.close() log_out1=open('/usr/SCRIPTS_FOR_PYTHON/error_log.html', 'a') log_out1.write(str(datetime.datetime.now()) + ':'+ str(message) + '\n') log_out1.close() pass # def enter_conf(device_type): # device_type device=check_OS.check_OS(check) if device_type=='IOS_XR' or device_type=='IOS' or device_type=='JUNOS': syntax='configure' else: #device_type=='HUAWEI': syntax='system-view' return syntax ### def config(host, user, pas, por, asset1, number_as_set='500'): def while_not_end(): ### , buff = b'' try: while not buff.endswith(b'# '): # , buff.endswith , resp = remote_conn.recv(2048) buff += resp except socket.timeout: ### , HTML , log(' ', '->'+host+'->'+asset1+'-> <font color="red" >ERROR:</font> Device did not respond, please check candidate conf and do rollback if needed<br>') buff=b'no data' return buff as_path=asset(host, asset1) ### - ( 15 ) if as_path=='ERROR': ### - ( candidate) log(' ', '->'+host+'->'+asset1+'->EROROR: Fibber did not find data in local base module asset in ssh_stend<br>') else: pass remote_conn_pre = paramiko.SSHClient() ### paramiko remote_conn_pre.set_missing_host_key_policy(paramiko.AutoAddPolicy()) try: ### , remote_conn_pre.connect(hostname=host, username=user, password=pas, port=por, timeout=90) ### pass except: log(' ', '->'+host+'->'+asset1+'-> WARNING: Device is unreachable') remote_conn = remote_conn_pre.invoke_shell() ### remote_conn.settimeout(5*60) remote_conn.send ('\n') time.sleep(2) ### check=str(remote_conn.recv(2048)) ### CLI, while_not_end() ( ) #device=str(check_OS(check)) ### , , Juniper, device='JUNOS' # . remote_conn.send(enter_conf(device)+'\n'+'\n') ### , , check_edit=str(while_not_end()) ### CLI if device=='JUNOS': if re.findall('Users currently editing the configuration|The configuration has been changed but not committed', check_edit) == []: ### # remote_conn.send('run set cli screen-length 10000' +'\n') ### , , no-more while_not_end() remote_conn.send('show policy-options as-path-group ?') ### AS-PATH, a # time.sleep(1) remote_conn.send('as-test'+'\n') check=while_not_end() check=str(check) check_find=str(re.findall(asset1, check)) if check_find=='['"""'"""+asset1+"""'"""']': ### AS-PATH-GROUP , RADB remote_conn.send('delete policy-options as-path-group '+str(asset1)+'\n') while_not_end() asset_out=str(asset1) for z in as_path: ### remote_conn.sendall('set policy-options as-path-group '+asset_out+' as-path '+asset_out+'-'+str(as_path.index(z))+' ".*('+'|'.join(z)+')$"'+'\n') while_not_end() ### , # as-path-gruop RADB remote_conn.send('show policy-options as-path-group '+asset1+' | no-more'+'\n') check_candidate_conf=str(while_not_end()) comp=re.compile('no-more') # CLI split_out=comp.split(check_candidate_conf) try: split_one_index_pre=split_out[1] pass except: split_one_index_pre=['no data'] split_one_index_pre2=re.findall(r'\((.*?)\)', str(split_one_index_pre)) ### split_one_index=re.findall('[\d]+', str(split_one_index_pre2))### as_path_find_as=re.findall('[\d]+', str(as_path)) ### RADB check_candidate_set=set(split_one_index) ### , CLI RADB as_path_find_as_set=set(as_path_find_as) remote_conn.send('sh | compare | no-more' + '\n') ### , as-path-group check_candidate_rollback=str(while_not_end()) split_rollback=comp.split(check_candidate_rollback) # CLI try: split_rollback_one_index=split_rollback[1] pass except: split_rollback_one_index=['no data'] check_rollback_headers=re.findall(r'\[([^]]*)\]', split_rollback_one_index) ### time.sleep(1) ### , : # - # - as-path-group # - AS BGP ( ) RADB if re.findall('edit (?!policy)|edit policy-options [^a]|edit policy-options as-path[^-]|edit policy-options as-path-group (?!'+asset1+')', str(check_rollback_headers))==[]: pass else: log(check_candidate_rollback, '->'+host+'->'+asset1+'-> <font color="gold" >WARNING:</font> Fibber decided do not commit, reason have another changes candidate conf which Fibber did not make<br>') if re.findall('edit policy-options as-path-group '+asset1, str(check_rollback_headers))!=[]: pass else: log(check_candidate_rollback, '->'+host+'->'+asset1+'-> <font color="gold" >WARNING:</font> Fibber did not see config to commit, reason as-set have no changes on device<br>') if check_candidate_set==as_path_find_as_set: pass else: log(check_candidate_rollback, '->'+host+'->'+asset1+'-> <font color="gold" >WARNING:</font> Fibber decided do not commit, reason candidate as_path and radb as_path is different<br>') ### , . if check_candidate_set==as_path_find_as_set and re.findall('edit (?!policy)|edit policy-options [^a]|edit policy-options as-path[^-]|edit policy-options as-path-group (?!'+asset1+')', str(check_rollback_headers))==[] and re.findall('edit policy-options as-path-group '+asset1, str(check_rollback_headers))!=[]: remote_conn.send('commit'+'\n'+'\n') ### , while_not_end() time.sleep(10) remote_conn.send('exit'+'\n'+'\n') time.sleep(1) remote_conn.send('exit'+'\n'+'\n') log(' ', '->'+host+'->'+asset1+'-> <font color="green" >SCCESSFUL:</font>: Filter has been updated via Fibber<br>') else: ### , time.sleep(1) remote_conn.send('rollback 0'+'\n') while_not_end() remote_conn.send('exit'+'\n') time.sleep(1) remote_conn.send('exit'+'\n') else: ### AS-PATH-GROUP, remote_conn.send('sh | compare roll 0 | no-more'+'\n') while_not_end() remote_conn.send('exit'+'\n') time.sleep(1) remote_conn.send('exit'+'\n') log(' ', '->'+ host+'->'+asset1+'-> <font color="red" >ERROR:</font> Fibber did not find configured AS-SET on device, please configure AS-SET first via your hand<br>') else: # , , , remote_conn.send('sh | compare roll 0 | no-more'+'\n') while_not_end() remote_conn.send('exit'+'\n') time.sleep(1) remote_conn.send('exit'+'\n') log(' ', '->'+host+'->'+asset1+'-> <font color="red" >ERROR:</font> Fibber detected, someone in edit mode or configuration has been changed but not commited<br>') else: ### Juniper , else elif . log(' ', '->'+host+'->'+asset1+'-> <font color="red" >ERROR:</font> Fibber detected - device is not a JUNIPER<br>') pass remote_conn_pre.close() time.sleep(2) pass ### , , candidate radb_v3.py host_and_asset = os.listdir("/usr/SCRIPTS_FOR_PYTHON/data_from_radb/candidate") # , ip AS-SET for i in range(len(host_and_asset)): # .txt host_and_asset[i]=(host_and_asset[i][0:len(host_and_asset[i])-4]) for k in range(len(host_and_asset)): # ip as-set host_and_asset[k]=(host_and_asset[k].split(",")) ### host_and_asset for i in range(len(host_and_asset)): config(str(host_and_asset[i][0]), str(user), str(secret), int(port), str(host_and_asset[i][1])) , error_log.html, :
LOG FILE 2015-06-15 13:48:57.828503:->10.10.10.1->as-test-> WARNING: Fibber did not see config to commit, reason as-set have no changes on device 2015-06-15 13:51:52.512611:->10.10.10.12->as-test-> ERROR: Fibber detected, someone in edit mode or configuration has been changed but not commited 2015-06-15 14:54:06.267404:->10.10.10.13->as-test-> WARNING: Fibber did not see config to commit, reason as-set have no changes on device 2015-06-15 14:55:26.954442:->10.10.10.17->as-test1-> SCCESSFUL: Filter has been updated via Fibber 2015-06-21 11:47:45.127530:->10.10.10.17->as-test-> SCCESSFUL: Filter has been updated via Fibber 2015-06-21 11:48:41.204475:->10.10.10.1->as-test-> SCCESSFUL: Filter has been updated via Fibber 2015-06-21 11:49:09.487539:->10.10.10.1->as-test3-> SCCESSFUL: Filter has been updated via Fibber 2015-06-21 11:50:14.816424:->10.10.10.1->as-test-> SCCESSFUL: Filter has been updated via Fibber 2015-06-21 11:50:41.835588:->10.10.10.5->as-test5-> SCCESSFUL: Filter has been updated via Fibber 2015-06-21 11:51:09.127567:->10.10.10.1->as-test-> SCCESSFUL: Filter has been updated via Fibber 2015-06-21 11:52:10.606458:->10.10.10.1->as-test-> SCCESSFUL: Filter has been updated via Fibber 2015-06-22 00:00:04.385457:->10.10.10.1->as-test-> ERROR: Fibber detected, someone in edit mode or configuration has been changed but not commited 2015-06-22 00:00:13.438379:->10.10.10.1->as-test1-> ERROR: Device did not respond, please check candidate conf and do rollback if needed 2015-07-26 19:43:06.316584:->10.10.10.13->as-test7-> SCCESSFUL: Filter has been updated via Fibber 2015-07-26 19:44:36.849450:->10.10.10.1->as-test-> WARNING: Fibber did not see config to commit, reason as-set have no changes on device
, , :
— AS-SET (Python) – 8.00 18.00
— (Python) – 8.00 18.00
— RADB – 18.00
— – 00.00
, , , . , (, , , ..). , , , , ..
, , unix Crontab, :
[lost@servertest SCRIPTS_FOR_PYTHON ]# crontab -l 0 0 * * * /usr/local/bin/python3.3 /usr/SCRIPTS_FOR_PYTHON/ssh_stend_v5.py >> /usr/SCRIPTS_FOR_PYTHON/cron_python.log 0 17 * * * /usr/local/bin/python3.3 /usr/SCRIPTS_FOR_PYTHON/radb_v3.py >> /usr/SCRIPTS_FOR_PYTHON/cron_python.log
, Python .
, , . , . , , , Netmiko.
Paramiko. NetConf, , .
-:
blog.dzinko.org/2011/03/python.html - Detailed description of regular expressions with examples.
stackoverflow.com - Site question answer for programmers. Most likely the answer to the unresolved problem is there.
habrahabr.ru/post/115436 - a detailed description of regular expressions.
pythonworld.ru - a lot of accessible clear information on python with examples.
www.radb.net
github.com/paramiko/paramiko - Paramiko library.
github.com/ktbyers/netmiko is a ready-made Python library for working with network devices.
Source: https://habr.com/ru/post/263859/
All Articles