Bash scripts, part 11: expect and automate interactive utilities

Bash scripts: start
Bash scripts, part 2: loops
Bash scripts, part 3: command line options and keys
Bash scripts, part 4: input and output
Bash Scripts, Part 5: Signals, Background Tasks, Script Management
Bash scripts, part 6: functions and library development
Bash scripts, part 7: sed and word processing
Bash scripts, part 8: awk data processing language
Bash scripts, part 9: regular expressions
Bash scripts, part 10: practical examples
Bash scripts, part 11: expect and automate interactive utilities

Last time we talked about the development of bash scripts. If we summarize everything that we have sorted out in the previous ten materials, then you, if you started reading them without knowing anything about bash, now you can do quite a lot of useful things.



Today's topic, the final one in this series of materials, is devoted to automating work with interactive utilities, for example, with scripts that, in the process of execution, interact with the user. In this case, we will help expect - a tool based on the Tcl language.

Expect allows you to create programs that await questions from other programs and give them answers. Expect can be compared with a robot that can replace a user when interacting with command line scripts.
')

Basics of expect

If expect is not installed on your system, fix it, for example, in Ubuntu, you can do this:

$ apt-get install expect 

In something like CentOs, the installation is done with the following command:

 $ yum install expect 

Expect provides a set of commands that allow you to interact with command line utilities. Here are his main commands:

• spawn — starting a process or program. For example, it could be a command shell, FTP , Telnet, ssh, scp, and so on.
  
  
• expect — waiting for data output by the program. When writing a script, you can specify what kind of output it is waiting for and how to respond to it.
  
  
• send — send reply. Expect-script with the help of this command can send input data to an automated program. It looks like the familiar echo command in ordinary bash scripts.
  
  
• interact — allows you to switch to the "manual" mode of program control.
  

Automate bash script

We write a script that interacts with the user and automate it with expect. Here is the code for the bash script questions :

 #!/bin/bash echo "Hello, who are you?" read $REPLY echo "Can I ask you some questions?" read $REPLY echo "What is your favorite topic?" read $REPLY 

Now we will write an expect-script that will run the questions script and will answer its questions:

 #!/usr/bin/expect -f set timeout -1 spawn ./questions expect "Hello, who are you?\r" send -- "Im Adam\r" expect "Can I ask you some questions?\r" send -- "Sure\r" expect "What is your favorite topic?\r" send -- "Technology\r" expect eof 

Save the script by giving it the name answerbot .

At the beginning of the script is the identification string, which, in this case, contains the path to expect, since the script will be interpreted exactly by expect.
In the second line, we disable the timeout by setting the expect timeout variable to -1. The rest of the code is the automation of working with the bash script.

First, using the spawn command, we run a bash script. Naturally, any other command line utility can be invoked here. Next, a sequence of questions from the bash-script, and the answers given to them by expect. After receiving a question from the subprocess, expect gives him the specified answer and expects the next question.

In the last command, expect expects a sign of the end of the file, the script, having reached this command, ends.

Now it's time to try it all out. Make the answerbot executable:

 $ chmod +x ./answerbot 

And call it:

 $./answerbot 

Expect script answers bash script questions

As you can see, the expect-script answered the questions of the bash-script correctly. If at this stage you encounter an error caused by the incorrect location of the expect, you can find out its address as follows:

 $ which expect 

Please note that after running the answerbot script answerbot everything happens in fully automatic mode. The same can be done for any command line utility. Here it should be noted that our bash-script is very simple, we know exactly what data it displays, so it is easy to write an expect-script to interact with it. The task is complicated when working with programs that are written by other developers. However, a tool for automated creation of expect-scripts comes to the rescue.

Autoexpect - automated creation of expect scripts

Autoexpect allows you to run programs that need to be automated, and then writes down what they output and what the user enters, answering their questions. Call autoexpect, passing the name of our script to this utility:

 $ autoexpect ./questions 

In this mode, the interaction with the bash script is no different from the usual: we ourselves enter the answers to his questions.


Running a bash script with autoexpect

After completing work with the bash-script, autoexpect will report that the collected data is recorded in the file script.exp . Take a look at this file.


Script.exp file

In general, with the exception of some details, we have the same script that we wrote on our own. If you run this script, the result will be the same.


Run an automatically generated expect script

When recording interaction sessions with some programs, such as FTP clients, you may encounter that they use information on the time of the operation in the output data, or output data reflecting the process of performing some long-term actions. In general, we are talking about the fact that the output of the program at each launch of it, correctly perceived by a person and causing input of the same answers, will, under the same conditions, look new to expect.

If in the expect-script, the lines expected from such a program will be rigidly fixed, such a script will not be able to work normally. You can cope with this either by deleting data from the expect-script that looks new every time the program is started, or by using templates, using which expect can correctly understand what the program wants from it.

As you can see, autoexpect is a very useful tool, but it is also not without flaws, which can only be corrected manually. Therefore we will continue to master the language of expect-scripts.

Work with variables and command line parameters

To declare variables in expect scripts, use the set command. For example, in order to assign the value 5 to the variable VAR1 , the following construction is used:

 set VAR1 5 

To access the value of a variable, you must add a dollar sign, $, in front of its name. In our case, it will look like $VAR1 .

In order to gain access to the command line arguments that the expect script is called with, you can do this:

 set VAR [lindex $argv 0] 

Here we declare the VAR variable and write a pointer to it to the first command-line argument, $argv 0 .

For the purposes of the updated expect script, we are going to write the value of the first argument, which is the user name that will be used in the program, to the variable my_name . The second argument, symbolizing what the user likes, will fall into the variable my_favorite . As a result, the variable declaration will look like this:

 set my_name [lindex $argv 0] set my_favorite [lindex $argv 1] 

Edit the answerbot script, bringing it to this view:

 #!/usr/bin/expect -f set my_name [lindex $argv 0] set my_favorite [lindex $argv 1] set timeout -1 spawn ./questions expect "Hello, who are you?\r" send -- "Im $my_name\r" expect "Can I ask you some questions?\r" send -- "Sure\r" expect "What is your favorite topic?\r" send -- "$my_favorite\r" expect eof 

Run it, passing SomeName as the first parameter, Programming as the second parameter:

 $ ./answerbot SomeName Programming 

Expect script using variables and command line parameters

As you can see, everything works as expected. Now the expect-script answers the questions of the bash-script, using the command line parameters passed to it.

Answers to various questions that may appear in the same place.

If an automated program can, in one situation, produce one line, and in another, in the same place, another, in expect, you can use blocks enclosed in braces and containing script response options to different data received from the program. It looks like this:

 expect {   "something" { send -- "send this\r" }   "*another" { send -- "send another\r" } } 

Here, if the expect script sees the string “something”, it will send the reply “send this”. If it is a certain line ending in "another", it will send the answer "send another".

Let's write a new script by writing it to the questions file, which randomly sets different questions in the same place:

 #!/bin/bash let number=$RANDOM if [ $number -gt 25000 ] then echo "What is your favorite topic?" else echo "What is your favorite movie?" fi read $REPLY 

Here we generate a random number each time the script is run, and, after analyzing it, we derive one of two questions.

To automate such a script, we can use the above construction:

 #!/usr/bin/expect -f set timeout -1 spawn ./questions expect {   "*topic?" { send -- "Programming\r" }   "*movie?" { send -- "Star wars\r" } } 

Answers to different questions appearing in the same place.

As you can see, when an automated script prints a string ending in a “topic?”, The expect script sends it the string “Programming”. Having received in the same place, with another launch of the program, the question ending in “movie?”, The expect-script answers: “Star wars”. This is a very useful technique.

Conditional operator

Expect supports the if-else conditional statement and other control constructs. Here is an example of using a conditional statement:

 #!/usr/bin/expect -f set TOTAL 1 if { $TOTAL < 5 } { puts "\nTOTAL is less than 5\n" } elseif { $TOTAL > 5 } { puts "\nTOTAL greater than 5\n" } else { puts "\nTOTAL is equal to 5\n" } expect eof 

Conditional operator in expect

Here we assign a certain number to the variable TOTAL , after which we check it and display the text depending on the result of the check.

Note the configuration of the curly braces. The next opening bracket must be located on the same line as the previous constructions.

While loop

The while loops in expect are very similar to those used in regular bash scripts, but, again, curly braces are used here:

 #!/usr/bin/expect -f set COUNT 0 while { $COUNT <= 5 } { puts "\nCOUNT is currently at $COUNT" set COUNT [ expr $COUNT + 1 ] } puts "" 

While loop in expect

Cycle for

The for loop in expect is arranged in a special way. At the beginning of the cycle, in independent pairs of curly brackets, it is necessary to specify the variable counter, the condition for terminating the cycle and the rule for modifying the counter. Then, again in curly braces, the body of the loop goes:

 #!/usr/bin/expect -f for {set COUNT 0} {$COUNT <= 5} {incr COUNT} { puts "\nCOUNT is at $COUNT" } puts "" 

For loop in expect

Declaration and use of functions

Expect allows a programmer to declare functions using the proc keyword:

 proc myfunc { MY_COUNT } { set MY_COUNT [expr $MY_COUNT + 1] return "$MY_COUNT" } 

This is what an expect script looks like that uses the same function declared in it:

 #!/usr/bin/expect -f proc myfunc { MY_COUNT } { set MY_COUNT [expr $MY_COUNT + 1] return "$MY_COUNT" } set COUNT 0 while {$COUNT <= 5} { puts "\nCOUNT is currently at $COUNT" set COUNT [myfunc $COUNT] } puts "" 

Functions in expect

Interact team

It happens that expect-automated programs require the input of sensitive data, such as passwords that you would not want to store in plain text in the script code. In this situation, you can use the interact command, which will allow you, after automating some part of the interaction with the program, to independently enter, say, a password, and then again transfer control to the expectation.

When this command is executed, the expect-script switches to reading the answer to the program's question from the keyboard, instead of transmitting to it the data previously recorded in it.

Here is a bash script, in general, exactly the same as we considered earlier, but now waiting to enter a password in response to one of its questions:

 #!/bin/bash echo "Hello, who are you?" read $REPLY echo "What is you password?" read $REPLY echo "What is your favorite topic?" read $REPLY 

Let's write an expect-script, which, when offered to provide a password, transfers control to us:

 #!/usr/bin/expect -f set timeout -1 spawn ./questions expect "Hello, who are you?\r" send -- "Hi Im Adam\r" expect "*password?\r" interact ++ return send "\r" expect "*topic?\r" send -- "Technology\r" expect eof 

The interact command in the expect script

Upon encountering the interact command, the expect-script will stop, giving us the opportunity to enter a password. After entering the password, you must enter “++” and the expect-script will continue to work, once again having control.

Results

Opportunities expect can be used in programs written in different programming languages ​​thanks to the appropriate libraries. Among these languages ​​are C #, Java, Perl, Python, Ruby, and others. The fact that expect is available for different development environments is far from coincidence. The thing is that this is a really important and useful tool that is used to solve a variety of tasks. Here you can check software quality and perform various network administration tasks, automate file transfer, automatically install updates and much more.

Having mastered this material, you familiarized yourself with the basic concepts of expect and learned how to use the autoexpect tool to automatically generate scripts. Now you can continue to study expect using additional sources. Here is a collection of educational and reference materials. Here is a noteworthy series of three articles ( 1 , 2 , 3 ). And here is the official expect page , where you can find links to the source code of the program and the list of publications.

This concludes the series of materials on bash-scripts. Hopefully, its eleven parts, as well as countless comments to them, helped those who wanted to learn how to write scripts on the command line to achieve the goal.



Dear readers! We thank everyone who was with us. We wish you successful automation in Linux! And, if you have experience with expect, we are waiting for your stories.