Monitoring Task Execution in IPython Notebook

I would like to share a simple but useful tool. When you work a lot with data, there are often primitive, but long operations, for example: “download 10,000 URLs”, “read a 2GB file, and do something with each line”, “parse 10,000 html files and get headers” . Looking at the hung terminal for a long time is alarming, so for a long time I used the following ingenious code:

def log_progress(sequence, every=10): for index, item in enumerate(sequence): if index % every == 0: print >>sys.stderr, index, yield item 

This function is beautiful; for more than a year, it moved from task to task. But recently I noticed the IntProgress widget in the standard Jupyter delivery and I realized that it was time to change something:


There are three minor problems with logging to stderr:

1. It's not beautiful. Obviously.
2. Sometimes it blows up the buffer.
   
3. Sometimes someone else writes to stderr or stdout.
   

Like many people who work with data, I'm a fan of Jupyter. Most of the time I spend there. Therefore, I can afford the following solution incompatible with other environments:

 def log_progress(sequence, every=10): from ipywidgets import IntProgress from IPython.display import display progress = IntProgress(min=0, max=len(sequence), value=0) display(progress) for index, record in enumerate(sequence): if index % every == 0: progress.value = index yield record 

')
All the same, only the counter is displayed not in stderr, but in a special widget. Very simple and convenient. For those who are also hooked on Jupyter, I posted a slightly improved version on GitHub github.com/alexanderkuk/log-progress . The module is distributed copy-ramp Use on health.

The improved version displays, in addition to the strip, a counter. And changes color depending on whether the operation was successfully completed or not:



Supports iterators:


Naturally, in one cell there can be several progress bars:



And they can even work from different threads:


In short, once again link to the github.com/alexanderkuk/log-progress code.