Automation again: Python crawled to routers

The speed of technology development today is amazing. The jump in scientific and technological progress in recent years can only be compared with the pace of development of the space industry in the period from the late 50s to the mid 70s of the twentieth century. As then, the presence of man in space became a reality, and now the widespread replacement of people with machines no longer seems transcendental.

Automation of processes has become a full-fledged "trend" of our time and continues to expand its influence in almost all areas of activity: from agriculture to "smart homes" or artificial intelligence.
')
This trend dictates its own rules of the game in the field of business. Market players who do not invest enough in the optimization of their business and production processes, in their cheapening and acceleration through automation, will very soon be “overboard”.

Production automation is hygiene, if you will. And we are talking here about both the production of cars (with the replacement of manual assembly with a conveyor), and the production of software (with the replacement of manual testing with autotests) or the provision of communication services (with the replacement of manual labor of telephonists at first with switching equipment, and then completely with new communication technology).

Understanding the importance of automation in modern realities, we at Huawei consider it important and useful to share our experience in automating the work of Huawei network equipment with users of our products and present to your attention the cycle of posts devoted to automation.

The “first series” will deal with event management tools implemented in Huawei equipment.

Event management

Any network protocol contains a description of the reaction to certain events. For example, if an interface included in an OSPF or RIP process crashes, the router that detects this event sends notifications to its neighbors. But the multitude of reactions to events, implemented in any standard protocol, does not exhaust the variety of tasks that the specialist serving the system may encounter. Consider an example:

In this case, as we see, both data and voice are transmitted over each external channel: QoS is configured, allocating 50% of the bandwidth of each channel for voice traffic. But if one of the channels drops, it is necessary to change the remaining settings so that the priority queue for voice traffic takes not 50%, but 70%.

This task can certainly be solved using an NMS or SDN controller (i.e., an external control system). You can also probably solve it with standard tools on the router, but this solution is not very obvious.

Especially in more complex (like our life :)) situations where the speeds of the channels are different and they are received on different routers:


This is where the “high point” of the event management system comes, which was implemented by our company in the Huawei network OS.

The Open Programming System (OPS) event management system allows you to automatically perform a specific set of actions (change the configuration, save a file, or something else) when certain events occur.

Let's go to the details.

Equipment supporting OPS

OPS is currently supported on CE series switches and on NE40 and AR series routers. Each of these models has its own OPS support features, which we will definitely touch on.

Programming

Rules for reacting to events in general can be created using the CLI (command line) or writing in Python (programming language). To describe the rules themselves, you can also use both CLI and Python. In this case, you can use both tools on the CE switches, on the AR routers, only Python can be used, and on the NE40 routers, only the command line.

Let's take a look at an example of describing an event and reacting to it using the CLI.

Example # 1:

[~ CE] ops
[~ CE-ops] assistant test1
[* CE-ops-assistant-test1] condition syslog pattern ". * Loopback [3-9]. * Own”
[* CE-ops-assistant-test1] execute 1.1 command sys
[* CE-ops-assistant-test1] execute 1.2 command int gi0 / 0/0
[* CE-ops-assistant-test1] execute 1.3 command undo shutdown
[* CE-ops-assistant-test1] execute 1.4 command commit
[* CE-ops-assistant-test1] commit

// Decrypt: if any of the interfaces Loopback3, Loopback 41, etc. transitions to the down state, the Gigabit Ethernet 0/0/0 interface will exit administratively down.

In another example, the event description is set using the CLI, and the reaction is described in Python (in the backupconfig.py file).

Example # 2:

[~ CE-ops] assistant Name / * create assistant * /
[* CE-ops-assistant-Name] condition cfg_file_change condition event feature feature
[* SwitchA-ops-assistant-backup_config] execute 1 python backupconfig.py
[* SwitchA-ops-assistant-backup_config] commit

// Decrypt: when the configuration file changes, the backupconfig script is run.

Types of events

Using CLI, you can customize the response to the following types of events:

• write to syslog;
• trapbuffer entry;
• send or receive SNMP;
• the occurrence of an accident (alarm);
• timer value.

Thanks to Python, you can additionally track:

• entering a command;
• rerouting;
• interface state change;
• the result of the NQA (SLA).

Reaction to events

The most commonly used is device configuration change. In the simplest case, using CLI, a sequence of commands is set (as shown in example # 1), the second parameter determines the execution sequence (in alphanumeric order).

In the first example, the commands are entered sequentially:

1. Entering the configuration mode (system-view);
2. Log in to the sub-configuration mode of the Gigabit 0/0/0 interface;
3. Raising the interface;
4. Saving changes.

OPS allows you to specify up to 10 consecutive commands. If you need to execute more than 10 commands, you can use the bat-file and run it using the command execute 1 batch-file myfile.bat

Obviously, with the use of Python, you can use more complex structures than the linear execution of sequential configuration commands.

In addition to simply executing commands, OPS allows you to:

• save and analyze saved variables;
• send notifications to the syslog server;
• notify the user, accept from him and analyze input from the terminal;
• perform login to remote devices using telnet, ssh, etc. protocols and perform any actions there.

A bit of magic on the "Python" :)

Consider an example of a simple shell.

1. Let's write in Python the simplest shell that will read user input and execute Python commands. In this example, consider the functions of input / output to the terminal, the execution of commands and analysis of the result of execution.
   
   

   import os,sys def run(c): b=c.split() if b[0]== «ls»: if b.__len__() == 1: b.append(«/»); res = os.listdir(b[1]) print print(res) else: try: exec(c) return 1 except: print(«Something wrong») return 0 def get_command(ops,VTY): a, b = ops.terminal.write(«>>>«,vty = VTY) a, b = ops.terminal.read(maxLen = 200,timeout = 60,vty=VTY) if a==None: a='q'; return a #     sh def ops_condition (ops): value, err_str = ops.cli.subscribe(«cli1», «^sh$», enter=True, sync=True, sync_wait=500) return 0 # ,  shell def ops_execute (ops): key, value = ops.environment.get(«_cli_vty») while 2<3: command=get_command(ops,key) if command == «q»: break; if command <> '': run(command) continue print print(«exit\n») return 0 

   
   In this case, we see four functions:
   
   
   • run (command) - executes the command, the return value is not analyzed;
   • get_command (OPS, VTY) - the built-in OPS object and the terminal number are input, returning the entered command;
   • Ops_condition trap (OPS) ;
   • Execution of ops_execute (OPS) - in case of triggered "trap".
   
   The ops_condition (OPS) and ops_execute (OPS) functions are required.
   
   run (command)
   The function executes the Python command. If the ls command is entered, which is not in Python, it is replaced by the os.listdir (path) command in Python.
   
   If path is not specified, the path = ”/” substitution is made. In this way, various alias can be created, i.e. abbreviations that will be replaced by expanded expressions.
   
   get_command (OPS, VTY)
   Reads and returns user input. If there was no input within 60 seconds, returns the value of the ending “q” - quit.
   
   ops_condition (OPS)
   "Trap". In our case, we set a trigger to enter the sh command:
   ops.cli.subscribe ("cli1", "^ sh $", enter = True, sync = True, sync_wait = 500)
   “ Cli1 ” is the name of a “trap”; it is necessary to handle a complex event consisting of several “traps”;
   ^ sh $ - regexp describing a string consisting of only two characters, namely
   sh . For example, the shutdown command would fall under regexp ^ sh , and traffic-shaping would fall under sh .
   
   ops_execute (OPS)
   It first reads the terminal name into the key variable, and then in an infinite loop (using plain while 2 <3) reads the user’s input into the command variable and executes it (run (command)) or stops the execution if command == ”q”.
   
   
2. Actually, let's see what happened. User input will be indicated by a bolt , and the response from the router will be in blue .
   
   The script should be copied to the flash router, and then it must be compiled:
   
   < AR2 > ops install file <scriptname.py>
   Then you need to create an assistant:
   < AR2 > system-view
   Enter system view, return user view with Ctrl + Z.
   [AR2] ops
   [AR2-ops] script-assistant python <scriptname.py>
   
   Checking:
   
   < AR2 > sh
   >>>
   Received an invitation to enter. It should be noted here that the execution context in Python is stored only inside exec () . We did not use the exec form (code, global, local) , so we assign the assignment of a value to a variable and print it on one line, that is, inside the same exec .
   
   >>> a = 2 + 3; print ("\ n \ r"); print a
   five
   
   Of course, you could use the following form:
   
   >>> global a
   >>> globals () ["a"] = 2 + 3
   >>> print a
   
   The result would be the same, and in fact it does not matter, because in real programs we will almost never use exec, and this simple shell is written only to explore the capabilities of Python on Huawei routers.

Work with files

>>> a = dir (os); print a ['EX_CANTCREAT', ... 'O_APPEND', 'O_ASYNC', 'O_CREAT' , ..., 'O_RDWR' , ...]

As you can see, there are constants for creating files, and now you can try to create a file.

We will work with the sd1: or / mnt / sd1 directory - it’s the same thing.

>>> ls / mnt / sd1
['AR2220-V200R003C01SPC900.cc', ..., 'V600R008C10SPC300_RM.mod', 'AR2220-V200R007C00SPC600.cc', 'shelldir', 'python', ...] - we see that the output contains files from sd1 :. Compare:
< AR2 > dir

 
  Directory of sd1: / 
    Idx Attr Size (Byte) Date Time (LMT) FileName 
      0 -rw- 94,689,536 Apr 23 2014 17:38:30 AR2220-V200R003C01SPC900.cc 
  ... 
     22 -rw- 1,113,612 Dec 16 2015 14:38:44 V600R008C10SPC300_RM.mod 
     23 -rw- 123,975,040 Mar 15 2016 13:18:56 AR2220-V200R007C00SPC600.cc 
     24 drw- - Apr 01 2016 10:16:28 shelldir 
     25 drw- - Apr 06 2016 14:45:26 python 

The easiest way to create a file:

>>> os.system ('echo qqq> /mnt/sd1/s.s')
< AR2 > dir s *

  Directory of sd1: / 
    Idx Attr Size (Byte) Date Time (LMT) FileName 
      9 -rw- 4 Oct 09 2016 17:24:15 ss 
  1,961,192 KB total available (1,402,760 KB free) 

< AR2 > more ss

  Qqq 

Actually, here is our file. Now the question arises: how to read from it? In this case, we will use the tools that the os module provides.

>>> f = os.open ('/ mnt / sd1 / s.s', os.O_CREAT | os.O_RDWR); print f
47
>>> str = '222'; os.write (47, str)
>>> str = '\ n \ r111'; os.write (47, str)
>>> os.close (47)
< AR2 > more ss
222
111

By the way, a similar result can be achieved using os.system:

>>> os.system ('echo 222> /mnt/sd1/d.d')
>>> os.system ('echo 111 >> /mnt/sd1/d.d')
< AR2 > more dd
222
111

Now read from the file:

>>> f = os.open ('/ mnt / sd1 / s.s', os.O_RDONLY); print f
49
>>> s = os.read (49,8); print s
222
111

As you can see, working with files in the OPS shell is quite simple, while you can write quite complex event handlers using the described functionality.

The functionality provided by OPS allows solving the same tasks that are solved by a similar system of event management EEM (Embedded Event Management) from Cisco Systems, with the exception of sending e-mail messages.

But, first, this is only the first implementation of Python, and in the next versions additional functionality is expected. Secondly, the e-mail can still be sent via open relay, using telnet to port 25. The functions for the remaining handlers and the execution of actions are completely identical to the Cisco EEM.

End of the first series

We tried to demonstrate some of the possibilities of using CLI and Python to manage events on Huawei routers. How it turned out useful, judge you. We will be glad to see your comments, questions and feedback.

We, for our part, will continue to share our experience in the field of automation of communication equipment.

In the meantime, we will complete the first series with the words of young Sergei Syroezhkin, the hero of the film “The Adventures of Electronics”:

What progress has reached: physical labor has disappeared,
Yes, and mental replace mechanical process.

Music, titles, to be continued ...