Progress indicator with stack

In my work, it often happens that I implement long processes, where I cannot do without a progress indicator. The problems started when the processes became too complicated, but at the same time I wanted to have one continuous progress indicator for the whole process. For example, a process may consist of calls to the Asub, Bsub, and Csub functions, each of which takes quite a long time (say, approximately 10%, 20%, and 70% of the total time). Let Asub contains two consecutive loops, Bsub several nested loops, and Csub one loop, but in the middle of this loop Asub is called. Solving the problem in the forehead, you can bring the code to such a state that a third of all lines will calculate the current percentage and determine whether it is time to update it in the UI, and the Asub function to take additional parameters to determine what range of percentages to display (from 0 to 10 if called from the main process or some other if called from inside Csub). As a result, the code loses readability, and it becomes more difficult to maintain it. And pleasant minutes are waiting for us when we want to reuse Bsub in another place, but not in the middle, but at the end of the overall process, so that the percentages from 10% to 30% that will be displayed will be out of place. I came to the conclusion that something needs to be done with this.

I set the following requirements. Adding to the existing progress display code should not:

1. Change prototypes of existing functions and methods;
2. Add new variables inside functions;
3. To contain somehow non-trivial calculations of the current progress (say, 100 * $i / $n already considered to be non-trivial);
4. To torment a timer or count down iterations in order to understand whether it is necessary to update the progress indicator or not to waste time on this expensive operation.

We will not talk about displaying the progress indicator: it can be a widget or control in your favorite window system, transfer to a web frontend via your favorite WebSockets, or simply output a 12% line in STDOUT. Suppose we have a renderer - an output function that accepts the current progress as a percentage and optionally a text message describing the process or its stage.

Splitting process into subprocesses

A simple example might look like this:

init_progress ;
#
do_first_half ;
update_progress 50 ;
#
do_last_half ;
update_progress 100 ;


Now suppose that each half is a challenge to a long function that can output its information about progress. However, she does not know in what context she was called and what range of the overall progress indicator was set aside for its implementation. A natural implementation would be something like this:

sub do_first_half ( ) {
#
update_progress 33 ;
#
update_progress 66 ;
#
update_progress 100 ;
}


That is, we report information about our progress, and let someone else map it to the desired range (in our case, 0-50%). Here I came up with an analogy with the OpenGL matrix stack, where any affine transformations of three-dimensional coordinates are described by a 4 × 4 matrix and the sequence of transformations is placed on the stack, and when it comes to specifying the vertices of a particular object, we specify specific numbers without any calculations. OpenGL itself converts the coordinates, multiplying by a specific matrix. Here, in fact, we also have coordinates on the progress indicator, only one-dimensional. Affine transformation is described by two numbers: transfer and scaling. We will add the transformations to the stack, and the update_progress function update_progress perform the necessary conversion and pass the already converted coordinates to the renderer:

# [, ]
my @stack = ( [ 1 , 0 ] ) ;
sub update_progress ( $ ) {
my $percent = shift ;
$percent = $stack [ - 1 ] [ 0 ] * $percent + $stack [ - 1 ] [ 1 ] ;
renderer ( $percent ) ;
}


Now add the functions push_progress and pop_progress . For ease of use, we will not push_progress and transfer to push_progress , but the range to which subsequent percentages should be mapped. Of course, if some transformation is already in effect, then the parameters of push_progress also need to be converted:

sub push_progress ( $$ ) {
#
my ( $s , $e ) = @_ ;
#
( $s , $e ) = map { $stack [ - 1 ] [ 0 ] * $_ + $stack [ - 1 ] [ 1 ] } ( $s , $e ) ;
#
push @stack , [ ( $e - $s ) / 100 , $s ] ;
}

sub pop_progress ( ) {
pop @stack ;
}

Now it remains to wrap the calls of the do_first_half and do_last_half in brackets of push_progress/pop_progress :

push_progress 0 , 50 ;
do_first_half ;
pop_progress ;
push_progress 50 , 100 ;
do_last_half ;
pop_progress ;


Already not bad. Unfortunately, you have to make sure that each push_progress matching pop_progress . However, we can wrap the code fragment between push_progress and pop_progress into a block and transfer it to the sub_progress function of the sub_progress form:

sub sub_progress ( & $$ ) {
my ( $code , $s , $e ) = @_ ;
push_progress $s , $e ;
my @retval = & { $code } ( ) ;
update_progress 100 ;
pop_progress ;
return @retval ;
}


The main code is then simplified:

sub_progress { do_first_half } 0 , 50 ;
sub_progress { do_last_half } 50 , 100 ;


Notice that before pop_progress I called update_progress(100) just in case the unit forgot to do it. Now it becomes clear that the $s parameter is not needed: you can use the last displayed value of the progress indicator instead.

Cycles

Now let's see what can be done with cycles. Suppose that all iterations of the loop are approximately the same time and the number of iterations is known. It does not work with cycles like for ( $i = 1 ; $i < = 1024 ; $i*= 2 ) , but it will work with any cycles like foreach (by the way, the given cycle is easily converted to foreach : for ( map { 2 **$_ } 0. .10 ) ). Our for_progress will perform such a chain of actions for each iteration: put the range [ $i / $n * 100 , ( $i + 1 ) / $n * 100 ] for_progress stack, where $ i is the iteration number and $ n is the number of elements list, load the current element in $ _, execute a code block, call update_progress(100) , retrieve the last element from the stack. Then in the existing cycles, it is enough to replace for with for_progress , drag the list to the end (as in the map ) and assign $ _ to your variable if you used another variable. I will note that next and last continue to work (albeit with the varning), since inside for_progress normal for . The simplest test looks like this:

init_progress ;
for_progress { sleep ( 1 ) } 1. .10 ;


Since update_progress is called at the end of the block by an automaton, it is possible not to call it at all in a loop. However, if each iteration is long, you can use it, indicating the percentages of the current iteration. Of course, nested loops work, use sub_progress inside for_progress and vice versa. Here is a simple example:

sub A {
for_progress {
sleep ( 1 ) ;
} 1. .4 ;
}

sub B {
sleep ( 1 ) ;
update_progress 10 ;
sub_progress { A } 50 ;
sleep ( 1 ) ;
update_progress 60 ;
sleep ( 2 ) ;
update_progress 80 ;
sleep ( 2 ) ;
}

init_progress ;
sub_progress { A } 25 ;
sub_progress { A } 50 ;
sub_progress { B } 100 ;


Modern programming is difficult to imagine without the words map and reduce . For them, the map_progress and reduce_progress wrappers are also written:

init_progress ;
print " \n Sum of cubes from 1 to 1000000 = " .
reduce_progress { $a + $b * $b * $b } 1. .1000000 ;


Here, of course, there is a question of productivity: the iteration is too short, and the call to update the progress indicator will slow down the process by several orders of magnitude each time. update_progress takes this into account and causes the renderer not every time, but only when it deems it necessary: ​​if the percentages have reached 100, they have changed sufficiently or enough time has passed since the last update (everything is set up with the init_progress parameters). In addition, additional optimizations have been made, as a result of which my example with reduce_progress is performed “only” 4.5 times slower than with List::Util::reduce . For very short iterations, use caution.
')

Where to get

The first version of the module Progress::Stack I put in the CPAN. So far the application for the namespace has not been approved, but the package can be downloaded from the CPAN site . In addition to the features described here, there is something else, including the object interface (although it is not really needed) and the file_progress function for processing a text file by analogy with while ( <FH> ) { } . The documentation has a detailed description and examples.

Comments and suggestions are welcome :-)