Hacker's guide to neural networks. Chapter 2: Machine Learning. We generalize SVM to a neural network.

Content:



An interesting fact is that SVM is just a separate type of very simple scheme (a scheme that calculates score = a * x + b * y + c, where a, b, c are weighting functions, and x, y are data entry points ). It can be easily expanded to more complex functions. For example, let's write a two-layer neural network that performs binary classification. The forward passage will look like this:

 //    x,y var n1 = Math.max(0, a1*x + b1*y + c1); //  1-   var n2 = Math.max(0, a2*x + b2*y + c2); // 2-  var n3 = Math.max(0, a3*x + b3*y + c3); // 3-  var score = a4*n1 + b4*n2 + c4*n3 + d4; //  

The above definition is a two-layer neural network with 3 hidden neurons (n1, n2, n3), which uses a non-linearity of a rectified linear element (Rectified Linear Unit or ReLU) for each hidden neuron. As you can see, we now have several parameters, which means that our classifier has become more complex and can complicate decision-making boundaries, and not just use a simple linear decision rule like, for example, SVM. Another way of presenting is that each of the three hidden neurons is a linear classifier, and at the moment we are creating an additional linear classifier on top of them. Now we will start to go deep :). So, let's teach this dual layer neural network. The code looks very similar to the SVM code example above, we only need to change the forward and backward pass:
')

 //    var a1 = Math.random() - 0.5; //    -0,5  0,5 // ...        for(var iter = 0; iter < 400; iter++) { //      var i = Math.floor(Math.random() * data.length); var x = data[i][0]; var y = data[i][1]; var label = labels[i]; //    var n1 = Math.max(0, a1*x + b1*y + c1); //  1-   var n2 = Math.max(0, a2*x + b2*y + c2); // 2-  var n3 = Math.max(0, a3*x + b3*y + c3); // 3-  var score = a4*n1 + b4*n2 + c4*n3 + d4; //  //    var pull = 0.0; if(label === 1 && score < 1) pull = 1; //    !  . if(label === -1 && score > -1) pull = -1; //    !  . //         //      «»  var dscore = pull; var da4 = n1 * dscore; var dn1 = a4 * dscore; var db4 = n2 * dscore; var dn2 = b4 * dscore; var dc4 = n3 * dscore; var dn3 = c4 * dscore; var dd4 = 1.0 * dscore; //  //     ReLU,   // ..      ,    «» var dn3 = n3 === 0 ? 0 : dn3; var dn2 = n2 === 0 ? 0 : dn2; var dn1 = n1 === 0 ? 0 : dn1; //      1 var da1 = x * dn1; var db1 = y * dn1; var dc1 = 1.0 * dn1; //      2 var da2 = x * dn2; var db2 = y * dn2; var dc2 = 1.0 * dn2; //      3 var da3 = x * dn3; var db3 = y * dn3; var dc3 = 1.0 * dn3; // !    ! //  ,         x,y //      .     //      ,   x,y //     ,    //  (..   ) ,    da1 += -a1; da2 += -a2; da3 += -a3; db1 += -b1; db2 += -b2; db3 += -b3; da4 += -a4; db4 += -b4; dc4 += -c4; // ,    var step_size = 0.01; a1 += step_size * da1; b1 += step_size * db1; c1 += step_size * dc1; a2 += step_size * da2; b2 += step_size * db2; c2 += step_size * dc2; a3 += step_size * da3; b3 += step_size * db3; c3 += step_size * dc3; a4 += step_size * da4; b4 += step_size * db4; c4 += step_size * dc4; d4 += step_size * dd4; //   ,     . // ! } 

This is how we train the neural network. Obviously, you want to gently break your code into blocks, but I specifically showed you this example in the hope that it explains everything clearly and clearly. Later we will consider the most optimal ways of applying these networks, and we will structure the code in a much more accurate, modular and more reasonable way.

In the meantime, I hope you have learned for yourself that a two-layer neural network is in fact not so difficult: we write the expression for the front passage, interpret the value at the end as a result, and then pull this value in positive or negative direction, depending on how this value should be for our specific example. Updating the parameter after back propagation of the error ensures that when we consider this particular example in the future, the network will rather give us the desired value, instead of the one it gave out before the update.