How can you simplify and speed up the calculation of the neural network of direct propagation

Hello, dear readers. A lot has been written and said about neural networks, mainly about how and for what they can be applied. At the same time, there are somehow not very much attention paid to two important issues: a) how to simplify the neural network and quickly compute (one computation of the exponent is implemented by the library functions of programming languages, usually not less than 15-20 processor instructions), b) what at least in part, the logic of the constructed network — in fact, the huge matrices of weights and offsets obtained after training the network somehow do not really help to understand the patterns that this network has found (they remain hidden and their task is to determine willows tion of - sometimes very important). I will talk about one of my approaches to solving these issues for ordinary neural networks of direct propagation, while I will try to do with a minimum of mathematics.

Some theory

A network of direct propagation, from a mathematical point of view, is a very large function, which includes the values ​​of network inputs, weights and displacements of neurons. In each layer of the neuron, the layer input values ​​(vector X) are multiplied by the neuron weights (vector $$$W_i$), folded with offset $$$B_i$

$$

$$s_i = W_iX + B_i$$

and enter activation functions $$$A (s_i)$forming the outputs of the neuron layer.

Activation functions may not be very simple to calculate, for example, they often contain exponents (exponential sigmoid, hyperbolic tangent). If you look at the assembler code that implements the exponents, you can find, firstly, a lot of different checks that are not always needed, and secondly, the exponential calculation itself is usually performed in at least two operations:

$$

$$exp (v) = 2 ^ {v * log_2 (e)}$$

Therefore, if we want to speed up the calculation of the network, then the first task will be to simplify the calculation of the activation function. You can try to sacrifice a little quality due to a gain in speed, approximately replacing the calculation of the classical activation function with the calculation of a simpler function, which (on the available input data) gives approximately the same results. This is, generally speaking, a classical interpolation problem: we have a set of values ​​calculated by the original function A (s), and we select a simpler function that gives very similar values. Such a simple function a (s) can be an ordinary polynomial, or a polynomial with negative powers, or something else like that. I used four types of such functions:

$$$a (s) = b_0 + b_1 * s + b_2 * s ^ 2 + ... + b_n * s ^ n$;
$$$a (s) = b_0 + b_1 / s + b_2 / s ^ 2 + ... + b_n / s ^ n$;
$$$a (s) = b_0 + b_1 * s ^ {0,5} + b_2 * s ^ 1 + b_3 * s ^ {1,5} + ... + b_n * s ^ {0,5n}$;
$$$a (s) = b_0 + b_1 / s ^ {0.5} + b_2 / s ^ 1 + b_3 / s ^ {1,5} + ... + b_n / s ^ {0.5n}$;
')
Suppose that for each neuron we managed to replace the activation function with a slightly simpler one — this can be done, for example, by applying the least squares method. By itself, such a replacement of a very large gain, perhaps, will not give. But here you can try another trick:

1. Record analytically huge function NET (X), calculated by the network as a whole;
2. Replace the original functions A (s) in NET (X) with the replacement functions a (s) obtained for them;
3. Algebraically derived NET (X) to simplify (or rather, use some ready-made program code of symbolic simplification of expressions). This is already possible (at least, much simpler than we would have tried to simplify the network with the original functions, for example, with the exhibitors).

As a result, we get something simpler and, perhaps, a bit more mathematically visual - here you can already try to understand what kind of function the network has.

This is the option to explain the logic of the constructed network.

The described task, of course, only in words looks simple. To use it in my programs, I needed to write my own code for symbolic simplification of expressions. In addition, I solved a more complicated problem, assuming that each neuron with the function A (s) can have several alternative alternative activation functions. $$$a_k (s)$Therefore, the overall task has been reduced to the enumeration of variants of such functions and the symbolic simplification of the network for each such variant. Here only paralleling of calculations has already helped.

Result

The result pleased me. I accelerated a three-layer network (with three inputs) of eight neurons (with input weights and offsets) with activation functions “exponential sigmoid”. As time measurements showed, it was possible to get a gain of about 40% in time without significant loss in quality.

Illustrate. Here is the source network data:





And in the third, output layer:


If the inputs are designated as a, b and c, then, after replacements and simplifications, the network function NET is considered as:

double a2 = a*a; double b2 = b*b; double c2 = c*c; double a3 = a2*a; double b3 = b2*b; double c3 = c2*c; double z01 = sqrt(-1.6302e-02+7.9324e-01*a+9.65149e-01*b+5.64151e-01*c); double z06 = sqrt(1.583708e+00-8.907654e-01*a-2.844379e-01*a2+1.050942e+00*a3+1.178096e+01*b-1.865618e+00*b*a-3.145465e+00*b*a2-5.777153e+00*b2+3.138123e+00*b2*a-1.043599e+00*b3+1.32778e+00*c+5.849582e-01*c*a-3.440382e+00*c*a2+1.838371e+00*c*b+6.864703e+00*c*b*a-3.42434e+00*c*b2-3.013361e-01*c2+3.754167e+00*c2*a-3.745404e+00*c2*b-1.365524e+00*c3+1.014237e-01*z01); double NET = (-1.477593e+00)/(z06)+1.370237e+00-6.303167e-02*a-1.495051e-03*a2+2.33748e-02*a3+5.558024e-02*b+1.178189e-02*b*a-6.996071e-02*b*a2+1.837937e-02*b2+6.97974e-02*b2*a-2.321149e-02*b3+7.924241e-02*c+3.392287e-03*c*a-7.652018e-02*c*a2-1.214263e-02*c*b+1.526831e-01*c*b*a-7.616337e-02*c*b2-1.915279e-03*c2+8.349931e-02*c2*a-8.33044e-02*c2*b-3.037166e-02*c3+1.949161e-02*z01; 

Winning - again, 40% of the time, without much damage to quality. I think you can apply this approach in cases where the speed of the neural network is critical - for example, if it is calculated repeatedly, in a double or triple cycle. An example of such a problem : the numerical solution of the aerodynamic problem on a grid, and in each of its nodes the neural network calculates some useful forecast, for example, for a more accurate calculation of turbulent viscosity. Then we have an external cycle in time, a double or triple cycle in the coordinate is enclosed in it, and already there, inside, the calculation of the neural network sits. In this case, the simplification is more than appropriate and useful.