Efficient data conversion using transducers

Converting large data arrays can be quite resource intensive, especially when you use higher-order functions such as map and filter.

In this article, we will show the capabilities of converters for creating an effective data transformation function that does not create temporary arrays. Temporary collections are created for the map and filter functions connected to each other. This is because these functions, returning to the new collection, produce the result of the following function.

Imagine that you have a base of 1,000,000 people and you need to create a subset of “names of women over 18 years old who live in the Netherlands”. There are various ways to solve this problem, but I'll start with the chains.

const ageAbove18 = (person) => person.age > 18; const isFemale = (person) => person.gender === 'female'; const livesInTheNetherlands = (person) => person.country === 'NL'; const pickFullName = (person) => person.fullName; const output = bigCollectionOfData .filter(livesInTheNetherlands) .filter(isFemale) .filter(ageAbove18) .map(pickFullName); 

Below is an example of solving a problem using a chain approach that creates temporary arrays. Imagine the cost of triple looping in more than 1,000,000 records!

Of course, filtered collections will be somewhat reduced, but this is still quite expensive.

However, the main point is that map and filter can be determined using reduce. Let's try to implement the above code in an abbreviated format.

 const mapReducer = (mapper) => (result, input) => { return result.concat(mapper(input)); }; const filterReducer = (predicate) => (result, input) => { return predicate(input) ? result.concat(input) : result; }; const personRequirements = (person) => ageAbove18(person) && isFemale(person) && livesInTheNetherlands(person); const output = bigCollectionOfData .reduce(filterReducer(personRequirements), []) .reduce(mapReducer(pickFullName), []); 

And what's more, We can further simplify the reduction (filterReducer) with the help of composition functions.

 filterReducer(compose(ageAbove18, isFemale, livesInTheNetherlands)); 

Using this approach, we reduce (haha!) The number of temporary arrays. Below is an example of a transformation using a reduction approach.

Charming, isn't it? But we talked about transducers. Where are our transducers?
It turns out that the filterReducer and mapReducer that we created reduce the function. This can be expressed as:

 reducing-function :: result, input -> result 

Transducers are functions that take shrinking functions and return shrinking functions. This can be expressed as follows:

 transducer :: (result, input -> result) -> (result, input -> result) 

The most interesting part is that transducers are approximately symmetrical in their type signatures. They take one pruning function and return another.
Consequently, we can compose any number of reduers using a combination of functions.

Build your own transducers.

I hope all this has now become clearer. Let's now write down our own transducer functions for map and filter.

 const mapTransducer = (mapper) => (reducingFunction) => { return (result, input) => reducingFunction(result, mapper(input)); } const filterTransducer = (predicate) => (reducingFunction) => { return (result, input) => predicate(input) ? reducingFunction(result, input) : result; } 

Using the reducers we created above, let's convert some numbers. We will use the compose function from the RamdaJS library. RamdaJS is a library that contains practical, functional methods and is specifically designed for functional programming styles.

 const concatReducer = (result, input) => result.concat(input); const lowerThan6 = filterTransducer((value) => value < 6); const double = mapTransducer((value) => value * 2); const numbers = [1, 2, 3]; // Using Ramda's compose here const xform = R.compose(double, lowerThan6); const output = numbers.reduce(xform(concatReducer), []); // [2, 4] 

The function “concatReducer” is called an iterator function . It will be called at each iteration and will be responsible for converting the output function of the transducer.

In this example, we simply concatenate the result. Since each transducer only accepts the abbreviation function, we cannot use value.concat.

When we compose several converters into one function, this is often called the xform transducer. So when you see it somewhere, you will know what it means.

Compilation of several converters.

We used the usual composition of functions in the previous example, and you may be wondering what the order of evaluation of this is. Although the composition of the function is applied from right to left, the transformations will be evaluated from left to right at runtime, which will be more intuitive for those of us who read from left to right.

It will take a bit of thinking to understand why this is the case: given that our double transducer returns the redirection function, and lowerThan6 also returns the redirection function, when combined, the double value will be passed to lowerThan6 and as a result we will get the lowerThan6 function. Thus, double is the result of composition and the order of evaluation is applied from left to right.

I created an example , so you can take a look at it yourself.

Use RamdaJS to optimize readability.

Since transducers are a great example of a functional programming style, let's see how a set of Ramda methods can be useful.

 const lowerThan6 = R.filter((value) => value < 6); const double = R.map((value) => value * 2); const numbers = [1, 2, 3]; const xform = R.compose(double, lowerThan6); const output = R.into([], xform, numbers); // [2,4] 

Ramda makes it possible to use its maps and filters. This is because Ramda's internal reduction method uses the built-in Transducer Protocol .

“The purpose of this protocol is that in all JavaScript implementations of the transducer, the interaction occurs regardless of the surface API level. It calls converters regardless of the context of their input and output sources, and indicates only the essence of the transformation from the point of view of a single element.
Since transducers are separated from their input and output sources, they can be used in many different processes — collections, streams, channels, browsers, etc. Transformers are assembled directly, without reference to input or the creation of intermediate units. ”

Conclusion

Transducers are a powerful and variable way to build quality transformations that can be used in many contexts. A transducer in your hands is a lot of possibilities for transformation.

And they are especially productive when converting large amounts of data, but you can also use the same converter for working with a single record.