Soprocess: -What, -how, -why?

Many Bash users are aware of the existence of co-processes, which appeared in the 4th version of Bash . Somewhat fewer knows that the coprocesses in Bash are not some new feature, but the ancient KornShell functionality that appeared in the implementation of ksh88 in 1988. An even smaller number of shell users who know how to coprocess know the syntax and remember how to do it. Probably, I belong to the fourth group - those who are aware of coprocesses, who occasionally know how to use them, but who still don’t understand “why?”. I say “periodically”, because sometimes I refresh their syntax in my head, but by the time when it seems to me that “this is the case when you can apply co-proc”, I completely forget about how to do it.

With this note, I want to bring together the syntax for different shells so that in case I can figure out why I need them, if I don’t remember how to do it, I will at least know where it is written.

In the title of the article we have 3 questions. Let's go in order.
')

What?

What is co-process? Co-processing is the simultaneous execution of two procedures, one of which reads the output of the other. To implement it, you must first run a background process that performs channel functionality. When the background process starts, its stdin and stdout are assigned to the channels associated with the user processes. Accordingly, one channel for recording, the second for reading. It is easier to explain this with examples, so we will immediately move on to the second question.

How?

Realizations of co-processes in shells vary. I will focus on 3 known implementations in ksh , zsh and bash . Consider them in chronological order. Although it is not directly related to the issues of the article, I note that all the examples below are made on

$ uname -opr FreeBSD 10.1-STABLE amd64 

Ksh

 $ `echo $0` --version version sh (AT&T Research) 93u+ 2012-08-01 

Syntax

 cmd |& 

It seems to me the most logical. Here, to execute the cmd command in the background, we use a special operation | & , expressing accordingly:

- "&" - background process;
- "|" - channels.

Run the background process:

 $ tr -uab |& [2] 6053 

Make sure he is alive:

 $ ps afx | grep [6]053 6053 4 IN 0:00.00 tr -uab 

Now we can communicate with our background process. We write:

 $ print -p abrakadabra1 $ print -p abrakadabra2 $ print -p abrakadabra3 

and read:

 $ read -p var; echo $var bbrbkbdbbrb1 $ read -p var; echo $var bbrbkbdbbrb2 $ read -p var; echo $var bbrbkbdbbrb3 

or so:

 $ print abrakadabra1 >&p $ print abrakadabra2 >&p $ print abrakadabra3 >&p $ while read -p var; do echo $var; done bbrbkbdbbrb1 bbrbkbdbbrb2 bbrbkbdbbrb3 

Close the "end" of the pipe for recording:

 $ exec 3>&p 3>&- 

and for reading:

 $ exec 3<&p 3<&- 

Zsh

 $ `echo $0` --version zsh 5.2 (amd64-portbld-freebsd10.1) 

The syntax of the co-processes in zsh is not too different from ksh , which is not surprising, since his man says "zsh most closely resembles ksh" .

The main difference is the use of the coproc keyword instead of the | & operator. Otherwise, everything is very similar:

 $ coproc tr -uab [1] 22810 $ print -p abrakadabra1 $ print abrakadabra2 >&p $ print -p abrakadabra3 $ read -ep bbrbkbdbbrb1 $ while read -p var; do echo $var; done bbrbkbdbbrb2 bbrbkbdbbrb3 

To close the read / write channels, you can use the exit idiom:

 $ coproc exit [1] 23240 $ [2] - done tr -uab $ [1] + done exit 

At the same time, a new background process was launched, which immediately ended. This is another difference from ksh - we can not close the existing coprocess, but immediately initiate a new one:

 $ coproc tr -uab [1] 24981 $ print -p aaaaa $ read -ep bbbbb $ coproc tr -uad [2] 24982 $ [1] - done tr -uab $ print -p aaaaa $ read -ep ddddd $ 

In ksh, we would just get:

 $ tr -uab |& [1] 25072 $ tr -uad |& ksh93: process already exists 

Despite this possibility, it is recommended to always kill the background process explicitly, especially when using “setopt NO_HUP” .

It is worth mentioning here that sometimes we can get unexpected results related to output buffering, which is why in the examples above we use the tr with the -u option.

 $ man tr | col | grep "\-u" -u Guarantee that any output is unbuffered. 

Although it does not have to be applied solely to co-processes, I will demonstrate this behavior by example:

 $ coproc tr ab [1] 26257 $ print -pa $ read -ep ^C $ [1] + broken pipe tr ab 

The buffer is not full and we get nothing from our pipe. Fill it up to the top:

 $ coproc tr ab [1] 26140 $ for ((a=1; a <= 4096 ; a++)) do print -p 'a'; done $ read -ep b 

Of course, if this behavior does not suit us, it can be changed, for example using stdbuf

 $ coproc stdbuf -oL -i0 tr ab [1] 30001 $ print -pa $ read -ep b 

Bash

 $ `echo $0` --version GNU bash, version 4.3.42(1)-release (amd64-portbld-freebsd10.1) 

To start the co-process in bash, just like in zsh , the reserved word coproc is used , but unlike the above shells, access to the coprocess is not done using > & p and <& p , but through the $ COPROC array :

- $ {COPROC [0]} to write;
- $ {COPROC [1]} to read.

Accordingly, the write / read procedure will look something like this:

 $ coproc tr -uab [1] 30131 $ echo abrakadabra1 >&${COPROC[1]} $ echo abrakadabra2 >&${COPROC[1]} $ echo abrakadabra3 >&${COPROC[1]} $ while read -u ${COPROC[0]}; do printf "%s\n" "$REPLY"; done bbrbkbdbbrb1 bbrbkbdbbrb2 bbrbkbdbbrb3 

and closing the handles:

 $ exec {COPROC[1]}>&- $ cat <&"${COPROC[0]}" [1]+ Done coproc COPROC tr -uab 

If the name COPROC for some reason does not suit you, you can specify your own:

 $ coproc MYNAME (tr -uab) [1] 30528 $ echo abrakadabra1 >&${MYNAME[1]} $ read -u ${MYNAME[0]} ; echo $REPLY bbrbkbdbbrb1 $ exec {MYNAME[1]}>&- ; cat <&"${MYNAME[0]}" [1]+ Done coproc MYNAME ( tr -uab ) 

What for?

Before we try to answer why we need coprocesses, we will consider whether it is possible to implement their functionality in shells that do not have coproc out of the box. For example in this:

 $ man sh | col -b | grep -A 4 DESCRIPTION DESCRIPTION The sh utility is the standard command interpreter for the system. The current version of sh is close to the IEEE Std 1003.1 (``POSIX.1'') spec- ification for the shell. It only supports features designated by POSIX, plus a few Berkeley extensions. $ man sh | col -b | grep -A 1 -B 3 AUTHORS This version of sh was rewritten in 1989 under the BSD license after the Bourne shell from AT&T System V Release 4 UNIX. AUTHORS This version of sh was originally written by Kenneth Almquist. 

No one has canceled named pipes:

 $ mkfifo in out $ tr -uab <in >out & $ exec 3> in 4< out $ echo abrakadabra1 >&3 $ echo abrakadabra2 >&3 $ echo abrakadabra3 >&3 $ read var <&4 ; echo $var bbrbkbdbbrb1 $ read var <&4 ; echo $var bbrbkbdbbrb2 $ read var <&4 ; echo $var bbrbkbdbbrb3 

Perhaps, with some stretch, we can say that this is realizable with the help of pseudo-terminals, but I will not develop this topic.

Well, why are coprocesses necessary? I will quote an excerpt from the translation of the Mitch Frazier article :

While I can not think of any <...> tasks for co-processes, at least not being contrived.

And in fact, I was only once able to apply co-processes in my scripts with relative advantage. The idea was to implement a kind of “persistent connect” to access MySQL.
It looked like this:

 $ coproc stdbuf -oL -i0 mysql -pPASS [1] 19743 $ printf '%s;\n' 'select NOW()' >&${COPROC[1]} $ while read -u ${COPROC[0]}; do printf "%s\n" "$REPLY"; done NOW() 2016-04-06 13:29:57 

Otherwise, all my attempts to use coproc were really far-fetched.