Lazy computing in everyday life

And although people who use python scripts to write a shopping list or compile rent data, recount them on their heads, but if it so happens that you use scripts to solve routine tasks and sometimes the scripts work unacceptably long, then perhaps the idea of ​​using lazy calculations to everything that moves, you will have to taste.

In the previous chapters of my letter, I gave a free description of the work of the evalcache library.
Link: Disk Caching of Lazy Computing Trees

In order not to bore you with the need to study that material before reading this, a brief summary of the last part:

evalcache wraps data, functions, and methods into "lazy objects." Each lazy object has a special hash key characterized by the way it is built. Over lazy objects, operations can be performed leading to the generation of new lazy objects. These operations look exactly like operations on ordinary data, but in reality no calculations are made, instead, a tree of lazy objects referring to each other is built, remembering their operations and their arguments. If it is necessary to obtain data, an operation of opening a lazy object is performed, which either activates a chain of calculations, or pulls the result from the cache, if an object with such a key was calculated earlier.

About some changes

Since writing the last article, evalcache has received several additional mechanics.

Uncached Execution Mechanics

It turns out that the hash of a lazy object is so useful that I want to use it in a situation where caching the object itself is impossible and unnecessary.

For this purpose, a special syntax is introduced:

lazyhash = evalcache.LazyHash() #: #evalcache.Lazy(self, cache=None, fastdo=True) @lazyhash def foo(): ... 

In this embodiment, the object is calculated immediately at the time of creation, but the lazy file is returned anyway. This allows you to build computation trees without having to cache some objects.

Implicit disclosure mechanics

Implicit disclosure is the expected behavior of the memoizer. Although evalcache was originally designed not to memoize, but to work with calculation trees, implicit disclosure based on evalcache algorithms can be achieved. To do this, two new options are onplace and onuse . onplace leads to the disclosure of a lazy object immediately after creation, and onuse when trying to use it in some of the operations allowed for a lazy object.

 import evalcache lazy = evalcache.Lazy(cache={}, onuse=True) #lazy = evalcache.Lazy(cache={}, onplace=True) ###   . #lazy = evalcache.Memoize() ###   . #lazy = evalcache.Memoize(onplace=True) ###   . @lazy def fib(n): if n < 2: return n return fib(n - 1) + fib(n - 2) for i in range(0,100): print(fib(i)) 

But this is not about this unnecessary addition, designed to make the library a little more similar to the other lenifiers. And about lazy files:

Lazy files

Evalcache contains an add-on for the lenification of file-generating functions. Initially, this functionality was supposed to be used for lenification of screenshots generation. As it turned out later, with the help of the mechanics of lazy files, you can do other interesting things.

 import evalcache import evalcache.lazyfile lazyfile = evalcache.lazyfile.LazyFile(cache = evalcache.DirCache(".evalfile")) @lazyfile(field="path") def foo(data, path): f = open(path, "w") f.write(data) f.close() foo("HelloWorld","data.dat") 

How it works…
In general, the logic of work is the same as that of all lazy objects.
foo("HelloWorld","data.dat") Begins constructing a lazy object whose hash key is tied to the arguments passed to it. After that, the mechanics of implicit disclosure are applied, leading to an instant start of the calculation.

But then the procedure changes.
@lazyfile(field="path") lazyfile decorator parses the parameter with the name specified in the field. The decorator expects that after the function is executed, a file will be created along this path. evalcache takes this file and creates a hard link to it in the hash directory ".evalfile" . The hard link file name corresponds to the hash of a lazy object. In the future, if you find a file with the same name in the cache, when evalcache opens an object instead of a function call, you simply create a hard link to the file in the required place in the required place.

It is useful that a lazy file is a regular lazy object, and when it is generated, other lazy objects can be used.

 import evalcache import evalcache.lazyfile lazy = evalcache.lazy.Lazy(cache = evalcache.DirCache(".evalcache")) lazyfile = evalcache.lazyfile.LazyFile(cache = evalcache.DirCache(".evalfile")) @lazyfile(field="path") def foo(data, path): f = open(path, "w") f.write(data) f.close() @lazy def datagenerator(): return "HelloWorld" foo(datagenerator(),"data.dat") 

Applying lenification to editing video through the moviepy tool.

As you know, any problem can be solved using a python script. The set of Python libraries is so extensive that it is very difficult to find a problem not covered by Python modules.

In particular, the moviepy library and two hours of studying the documentation for it give us a simple and functional video editor. Installation - please. Sound impose - please. Special effects - please.

However, as always, there is a drawback to working with scripts. Each time the script runs, all artifacts are rebuilt. When installing a watch video, the work of such a script can last for a very long time.

The use of the evalcache library helped to make adjustments to this situation.

 #!/usr/bin/env python3 #coding:utf-8 import sys import types from moviepy.editor import * import evalcache.lazyfile lazyhash = evalcache.LazyHash() lazyfile = evalcache.lazyfile.LazyFile() LazyVideoClip = lazyhash(VideoClip) VideoFileClip = lazyhash(VideoFileClip) AudioFileClip = lazyhash(AudioFileClip) CompositeVideoClip = lazyhash(CompositeVideoClip) concatenate_videoclips = lazyhash(concatenate_videoclips) @lazyfile("path") def lazy_write_videofile(path, clip): clip.write_videofile(path) source = VideoFileClip("source.mp4") music0 = AudioFileClip("music0.mp3") music1 = AudioFileClip("music1.mp3") music = music0 dur = 3 s0 = 1 s1 = 8 s2 = 16 part0 = source.subclip(s0, s0 + dur) part1 = source.subclip(s1, s1 + dur * 2).fl_time(lambda t: t * 2).set_duration(2) part2 = source.subclip(s2, s2 + dur * 4).fl_time(lambda t: t * 5).set_duration(2) clip = concatenate_videoclips([part0, part1, part2]) clip = clip.set_audio(music.set_duration(clip.duration)) lazy_write_videofile("part0.mp4", part0) lazy_write_videofile("part1.mp4", part1) lazy_write_videofile("part2.mp4", part2) lazy_write_videofile("part0_mus.mp4", part0.set_audio(music.set_duration(part0.duration))) lazy_write_videofile("part1_mus.mp4", part1.set_audio(music.set_duration(part1.duration))) lazy_write_videofile("part2_mus.mp4", part2.set_audio(music.set_duration(part2.duration))) if len(sys.argv) > 1 and sys.argv[1] == "compile": clip.lazy_write_videofile("clip.mp4", clip) 

What is there.

We use the mechanics of uncacheable execution, since there is no desire and no need to deal with caching moviepy objects. At the same time, we get all the advantages of lazy objects for tracking changes in the execution tree.

Wrapping library calls into lenificators:

 LazyVideoClip = lazyhash(VideoClip) VideoFileClip = lazyhash(VideoFileClip) AudioFileClip = lazyhash(AudioFileClip) CompositeVideoClip = lazyhash(CompositeVideoClip) concatenate_videoclips = lazyhash(concatenate_videoclips) 

With this construct, we will generate the files:

 @lazyfile("path") def lazy_write_videofile(path, clip): clip.write_videofile(path) 

Having performed the necessary operations on the video sequence, we record parts of our clip in separate files. With and without music:

 lazy_write_videofile("part0.mp4", part0) lazy_write_videofile("part1.mp4", part1) lazy_write_videofile("part2.mp4", part2) lazy_write_videofile("part0_mus.mp4", part0.set_audio(music.set_duration(part0.duration))) lazy_write_videofile("part1_mus.mp4", part1.set_audio(music.set_duration(part1.duration))) lazy_write_videofile("part2_mus.mp4", part2.set_audio(music.set_duration(part2.duration))) 

These files will be recompiled only when the corresponding branches of execution are changed.

When the result is complete, we assemble the parts into a large file using the 'compile' option:

 if len(sys.argv) > 1 and sys.argv[1] == "compile": clip.lazy_write_videofile("clip.mp4", clip) 

Instead of concluding:

This text shows how to use the evalcache library to lazify an algorithm for generating files.
Such an approach makes it possible to reduce dependence on specialized software or to avoid writing complex logic of an electoral assembly.

References:

Project on github
Project on pypi