Pipe, the pythonic way

Some Pythonists love the code to be read, others prefer concise. Unfortunately, the balance between the first and second — solutions that are truly elegant — rarely happens in practice. More often lines like

my_function(sum(filter(lambda x: x % 3 == 1, [x for x in range(100)]))) 

Or quatrains a la

 xs = [x for x in range(100)] xs_filtered = filter(lambda x: x % 3 == 1, xs) xs_sum = sum(xs_filtered) result = my_function(xs_sum) 

Idealists would like to write something like this

 result = [x for x in range(100)] \ | where(lambda x: x % 3 == 1)) \ | sum \ | my_function 

Not in Python?

A simple implementation of such chains was recently proposed by a certain Julien Palard in his library Pipe .

Let's start right away with an example:

 from pipe import * [1,2,3,4] | where(lambda x: x<=2) #<generator object <genexpr> at 0x88231e4> 

Oops, the intuitive rush is not rolled. The pipe returns a generator, the values ​​from which are yet to be extracted.

 [1,2,3,4] | where(lambda x: x<=2) | as_list #[1, 2] 

It would be possible to pull the values ​​out of the generator with the built-in cast function of the type list (), but the author of the tool was consistent in his research and offered us the function as_list.

As you can see, the data source for the payp in the example was a simple list. Generally speaking, you can use any iterable entities of Python. Let's say “pairs” (tuples) or, which is more interesting, the same generators:

 def fib(): u"""    """ a, b = 0, 1 while 1: yield a a, b = b, a + b fib() | take_while(lambda x: x<10) | as_list #0 #1 #1 #2 #3 #5 #8 

From here you can learn a few lessons:

1. in pipes you can use lists, “pairs”, generators - any iterables.
2. the result of combining generators into chains will be a generator.
3. without an explicit requirement (type conversion or special pipe), the piping is “lazy” in the sense that the chain is a generator and can serve as an infinite source of data.

Of course, the joy would be incomplete if we did not have an easy opportunity to create our own pipes. Example:

 @Pipe def custom_add(x): return sum(x) [1,2,3,4] | custom_add #10 

Arguments? Easy:

 @Pipe def sum_head(x, number_to_sum=2): acc = 0 return sum(x[:number_to_sum]) [1,2,3,4] | sum_head(3) #6 

The author has kindly provided a lot of the prepared pipe. Some of them:

• count - recalculate the number of elements of the incoming iterable
• take (n) - extracts the first n elements from the input iterable.
• tail (n) - extracts the last n elements.
• skip (n) - skip the first n elements.
• all (pred) - returns True if all iterable elements satisfy the predicate pred.
• any (pred) - returns True if at least one iterable element satisfies the predicate pred.
• as_list / as_dist - leads iterable to the list / dictionary, if such a conversion is possible.
• permutations (r = None) - makes up all possible combinations of r elements of the input iterable. If r is not defined, then r is taken as len (iterable).
• stdout - output the iterable as a whole to a standard stream after a cast to a string.
• tee - output an iterable element to the standard stream and pass it for further processing.
• select (selector) - pass the elements iterable for further processing, after applying the function selector to them.
• where (predicate) - pass an iterable that satisfies the predicate to predicate.

But these are more interesting:

• netcat (host, port) - for each iterable element, open a socket, pass the element itself (of course, string), and pass the host response for further processing.
• netwrite (host, port) - the same thing, just do not read from the socket after sending data.

These and other pipes for sorting, traversing and processing data flow are included by default in the module itself, the benefit of which is really easy to create.

Under the hood of the Pipe Decorator

Frankly, it was amazing to see how concise the base code of the module! Judge for yourself:

 class Pipe: def __init__(self, function): self.function = function def __ror__(self, other): return self.function(other) def __call__(self, *args, **kwargs): return Pipe(lambda x: self.function(x, *args, **kwargs)) 

That's all, actually. Ordinary decorator class.
')
In the constructor, the decorator saves the function being decorated, turning it into an object of the class Pipe.

If the pipe is called by the __call__ method, a new pipe function is returned with the specified arguments.

The main subtlety is the __ror__ method. This is an inverted operator, an analogue of the operator “or” (__or__), which is called on the right operand with the left operand as an argument.

It turns out that the calculation of the chain starts from left to right. The first element is passed as an argument to the second; the result of calculating the second - the third, and so on. Painlessly pass through the chain and generators.

In my opinion, very, very elegant.

Afterword

The syntax for this kind of pipe is really simple and convenient, I would like to see something similar in popular frameworks; let's say for processing data streams; or - in a declarative form - alignment in the chain of callbacks.

The only drawback of the implementation is the rather vague error traces.

On the development of ideas and alternative implementation - in the following articles.