Neural network with SoftMax c # layer

Hi, in the last article I talked about the error back-propagation algorithm and gave an implementation that does not depend on the error function and on the neuron activation function. Several simple examples of the substitution of these very parameters were shown: minimizing the square of the Euclidean distance and log-likelihood for the sigmoid function and the hyperbolic tangent. This post will be a logical continuation of the past, in which I will consider a slightly non-standard example, namely the Softmax activation function to minimize cross-entropy . This model is relevant for the classification problem, when it is necessary to obtain at the output of a neural network the probabilities of belonging to the input image to one of the non-intersecting classes. Obviously, the total network output for all neurons of the output layer must be equal to one (as well as for the output images of the training set). However, it is not enough just to normalize the outputs, but to make the network model a probability distribution, and train it specifically for this. By the way, now on coursera.org there is a course on neural networks , it was he who helped to delve into understanding softmax, otherwise I would continue to use third-party implementations.

Using

I recommend to get acquainted with the previous post , since all the notation, interfaces and the learning algorithm itself are used without change here.
')

Softmax activation function

So, the first task that we face is to provide a way to model the network with a probability distribution. To do this, create a network of direct distribution, such that:

• the network contains a number of hidden layers, all neurons can have their own activation function;
• on the last layer is the number of neurons, which corresponds to the number of classes; All these neurons will be called softmax layer or group.

The softmax group neurons will have the following activation function (in this section I will omit the layer index, which means that it is the last and contains n neurons):

• , for i-th neuron

It can be seen from the formula that the output of each neuron depends on the adders of all other neurons of the softmax group, and the sum of the output values ​​of the whole group is equal to one. The beauty of this function is that the partial derivative of the i-th neuron is equal in its adder:

• 
  

We implement this function using the IFunction interface from the previous article :

Error function

In each training example, we will get a network output that models the probability distribution we need, and to compare two probability distributions, a correct measure is needed. As such a measure will be used cross entropy :

• 
• t - required outputs for the current training example
• y - real neural network outputs

And the total network error is calculated as:


To realize the elegance of the entire model, you need to see how the gradient is calculated from one of the output dimensions or a neuron. In the previous post, in the “output layer” section, it is described that the task is reduced to the calculation of dC / dz_i, we will continue from this point on:

• because the output of each neuron contains the adder of the current, we have to differentiate the entire amount. Due to the fact that the cost function depends only on the outputs of the neurons, and the outputs only on adders, it can be decomposed into two partial derivatives. Next, we consider individually each member of the sum (the main thing is to pay attention to the indices, in our case j runs through the neurons of the softmax group, and i is the current neuron)
• 

  ---

• 

  ---

• 

  ---

• 

The last transformation is obtained due to the fact that the sum of the values ​​of the output vector must be equal to one, according to the property of neurons of the softmax-group. This is an important requirement for the training set, otherwise the gradient will not be calculated correctly!

Let us turn to the implementation using the same representation as before:

Softmax layer

Generally speaking, a specific layer can not be done, just in the constructor of an ordinary network of direct distribution, create the last layer, with the activation function given above, and transmit to it in the designer a link to the softmax layer, but then, when calculating the output of each neuron, the denominator of the function will be calculated each time activation, but if you implement the double[] ComputeOutput(double[] inputVector) neural network properly:

 public double[] ComputeOutput(double[] inputVector) { double[] outputVector = inputVector; for (int i = 0; i < _layers.Length; i++) { outputVector = _layers[i].Compute(outputVector); } return outputVector; } 

then, because the network does not call the neuron's Compute method directly, but delegates this function to the layer, it can be done so that the denominator of the activation function is calculated once.

Total

So, the missing parts are ready, and you can assemble the designer . For example, I use the same implementation of a direct distribution network, just with a different constructor .