Error Propagation Algorithm with Regularization on c #
Hey. I want to continue the topic of implementation of machine learning methods on c #, and in this article I will talk about the back-propagation error algorithm for learning the direct-propagation neural network , and also give its implementation in C #. The peculiarity of this implementation is that the implementation of the algorithm is abstracted from the implementations of the objective function (the one that the neural network tries to minimize) and the activation functions of the neurons. The result will be a constructor with which you can play with various parameters of the network and the learning algorithm, see and compare the result. It is assumed that you are already familiar with the fact that such an artificial neural network (if not, I strongly recommend that you start by exploring Wikipedia or one of these articles ). Interesting? We climb under the cat.
To begin, consider the notation that I will use in the article, but for one thing we recall the basic concepts, I will not give pictures with neurons and layers, this is all completely in Wikipedia and here, in Habré. So, immediately into battle, the induced local field of a neuron (or just an adder) looks like this:
The activation function of the neuron, or transfer function of the value of the adder:
')
From the neuron we go to the network itself. A neural network is a model, it has parameters, and the task of the learning algorithm is to select such network parameters to minimize the value of the error function. The error function will be denoted by E. Model parameters are neuron weights:
- the weight of the j-th neuron of the layer n, which originates in the i-th neuron of the layer (n - 1).
Greek this
Let us denote the hyperparameter of the learning algorithm - the learning rate.
The change in weight is denoted by delta :
Thus, the new weight of the neuron is as follows:
It is worth mentioning that it is still possible (or rather, need) to add regularization to the change in weight. The regularization function R is a function of the parameters of the model, in our case, the weights of the neurons. Thus, the new error function looks like E + R , and the weight change formula is converted to the following:
Generally speaking, the implementation of regularization can also be abstracted from the learning algorithm, but so far I will not do this, because the current implementation of the learning algorithm is not the fastest, since otherwise, at each learning epoch (run of all the training examples) will have to be considered in one cycle accumulated error, and in the other - regularization. Another reason is that there are not many types of regularization (for example, I know only L1 and L2) that are used when teaching neural networks. In this implementation, I will use the L2 rate, and it will be an integral part of the learning algorithm.
To begin with we will stop on training modes. Weights can be changed in several ways:
Consider the situation with online learning, so it will be easier. So, the impulse came to network input
, the network has generated a response 
, although the correct response to x is 
.
Consider the partial derivative of the error function E :
Further reasoning is divided into two branches: for the last layer and for the remaining layers.
For the output layer, everything is simple; to correct the error, it suffices to calculate the derivative of the objective function by one of the weights and calculate the delta value. We take into account that the objective function completely depends only on the output value of the neuron, or the value of the activation function, and the activation function itself depends only on the adder.
Here you can see that to calculate the output layer error, it is necessary to calculate the value of the partial derivatives at the points, regardless of the target function or the neuron activation function.
But if the layer is not the output, then we need to accumulate the error values of all subsequent layers.
PS: I noticed that I forgot to put parentheses in superscripts to indicate that this is not a degree, but an index of a layer; keep this in mind while there is no degree anywhere.
What do we have:
The formulas are finished, let's move on to the implementation, and start with the concept of the error function. I have this in the form of a metric (in fact, it is). The CalculatePartialDerivaitveByV2Index method calculates the value of the partial derivative of a function for input vectors by an index variable from v2.
Thus, we can calculate the value of the partial derivative of the error function for the last layer from the real output of the network
.
For example, let's write a few implementations.
In a similar way we describe the function of neuron activation.
And examples.
In this section, we consider the presentation of the basic elements of the network; I will not give them the implementation, since she is obvious. The algorithm will be given for a fully connected “puffy” network, so the implementation of the network will need to be done accordingly.
So, the neuron looks like this.
Since we consider a fully connected “puffy” network, then Childs and Parents can not be implemented, but if you do a general algorithm, you will have to. Consider the fields that are particularly important for the learning algorithm:
The network layer looks simpler:
And the network view:
But we are considering a multilayered neural network, so a special presentation will be used:
The learning algorithm will be implemented through the pattern strategy :
For a more visual understanding, I will give a typical Train function of any neural network in the context of this implementation:
I use the following input format:
As can be seen from the code in the previous parts, the neural network works with
Finally, by showing the whole context, you can go to the actual implementation of the neural network learning algorithm.
Legend
To begin, consider the notation that I will use in the article, but for one thing we recall the basic concepts, I will not give pictures with neurons and layers, this is all completely in Wikipedia and here, in Habré. So, immediately into battle, the induced local field of a neuron (or just an adder) looks like this:
- the value of the linear combination of the weights vector and the vector of input values, the j-th neuron of the layer n- b - shift or displacement of the neuron; if it is agreed that the zero value of the input vector is always one
, the shift can be denoted as a weight with a zero index, and the formula can be simplified 
- the number of neurons in the layer n
The activation function of the neuron, or transfer function of the value of the adder:
- each neuron network can have its own activation function
- for all layers except the first one, the input vector will be the output vector of the previous layer, so that
')
From the neuron we go to the network itself. A neural network is a model, it has parameters, and the task of the learning algorithm is to select such network parameters to minimize the value of the error function. The error function will be denoted by E. Model parameters are neuron weights:
- the weight of the j-th neuron of the layer n, which originates in the i-th neuron of the layer (n - 1).Greek this
Let us denote the hyperparameter of the learning algorithm - the learning rate.The change in weight is denoted by delta :
- the direction of the gradient shows us in the direction of increasing the value of the function, but to minimize it we need to move in the opposite direction
Thus, the new weight of the neuron is as follows:
It is worth mentioning that it is still possible (or rather, need) to add regularization to the change in weight. The regularization function R is a function of the parameters of the model, in our case, the weights of the neurons. Thus, the new error function looks like E + R , and the weight change formula is converted to the following:
- lambda - learning hyperparameter, regularization coefficient (similar to learning speed)
- m - the size of the training sample
Generally speaking, the implementation of regularization can also be abstracted from the learning algorithm, but so far I will not do this, because the current implementation of the learning algorithm is not the fastest, since otherwise, at each learning epoch (run of all the training examples) will have to be considered in one cycle accumulated error, and in the other - regularization. Another reason is that there are not many types of regularization (for example, I know only L1 and L2) that are used when teaching neural networks. In this implementation, I will use the L2 rate, and it will be an integral part of the learning algorithm.
Error Propagation Algorithm
To begin with we will stop on training modes. Weights can be changed in several ways:
- either after each learning example (real-time learning, online learning, batchSize = 1)
- or accumulate changes for the entire training set, and then change all weights (full-batch, batchSize = trainingSet.Length)
- or after running a number of training examples (mini-batch, batchSize = any_number <trainingSet.Length)
Consider the situation with online learning, so it will be easier. So, the impulse came to network input
, the network has generated a response , although the correct response to x is .Consider the partial derivative of the error function E :
- but since the error function can be expressed only through an adder, and the adder depends on the weight, we can rewrite this record as follows
- this is how the partial derivative of the adder by weight is calculated, then you can rewrite the expression of the partial derivative of the objective function by weight; in the end we get
Further reasoning is divided into two branches: for the last layer and for the remaining layers.
Output layer
For the output layer, everything is simple; to correct the error, it suffices to calculate the derivative of the objective function by one of the weights and calculate the delta value. We take into account that the objective function completely depends only on the output value of the neuron, or the value of the activation function, and the activation function itself depends only on the adder.
Here you can see that to calculate the output layer error, it is necessary to calculate the value of the partial derivatives at the points, regardless of the target function or the neuron activation function.
Any hidden layer
But if the layer is not the output, then we need to accumulate the error values of all subsequent layers.
- further we will take into account that the adder of the next layer depends only on the outputs of the current layer, and the outputs of the current layer depend only on the adders of the current layer
- the derivative of the adder over the outputs of the previous layer will give us just the weight between the neurons, given this we rewrite the main conclusion
PS: I noticed that I forgot to put parentheses in superscripts to indicate that this is not a degree, but an index of a layer; keep this in mind while there is no degree anywhere.
What do we have:
- calculating the derivative of a certain activation function is just what you need
- calculation of the partial derivative of the objective function by the value of the adder of the next layer; everything is also simple here, because we are not on the last layer, and we calculate the changes in weights from the last to the first, so this value is already calculated at the previous step
- if the next layer is the last one, then we will calculate this value for the current layer, and thus we will extend the solution to the whole network
Implementation
Error function
The formulas are finished, let's move on to the implementation, and start with the concept of the error function. I have this in the form of a metric (in fact, it is). The CalculatePartialDerivaitveByV2Index method calculates the value of the partial derivative of a function for input vectors by an index variable from v2.
public interface IMetrics<T> { double Calculate(T[] v1, T[] v2); /// <summary> /// Calculate value of partial derivative by v2[v2Index] /// </summary> T CalculatePartialDerivaitveByV2Index(T[] v1, T[] v2, int v2Index); } Thus, we can calculate the value of the partial derivative of the error function for the last layer from the real output of the network
.For example, let's write a few implementations.
Minimizing half the Euclidean square distance
And the derivative will look like this:
And the derivative will look like this:
internal class HalfSquaredEuclidianDistance : IMetrics<T> { public override double Calculate(double[] v1, double[] v2) { double d = 0; for (int i = 0; i < v1.Length; i++) { d += (v1[i] - v2[i]) * (v1[i] - v2[i]); } return 0.5 * d; } public override double CalculatePartialDerivaitveByV2Index(double[] v1, double[] v2, int v2Index) { return v2[v2Index] - v1[v2Index]; } } Minimizing log likelihood
Here the main thing is not to forget that the logarithmic likelihood is calculated with a minus sign at the beginning, and the derivative will also be with a minus. I do not focus on checking or avoiding cases of division by zero, or the logarithm of zero.
internal class Loglikelihood : IMetrics<double> { public override double Calculate(double[] v1, double[] v2) { double d = 0; for (int i = 0; i < v1.Length; i++) { d += v1[i]*Math.Log(v2[i]) + (1 - v1[i])*Math.Log(1 - v2[i]); } return -d; } public override double CalculatePartialDerivaitveByV2Index(double[] v1, double[] v2, int v2Index) { return -(v1[v2Index]/v2[v2Index] - (1 - v1[v2Index])/(1 - v2[v2Index])); } } Here the main thing is not to forget that the logarithmic likelihood is calculated with a minus sign at the beginning, and the derivative will also be with a minus. I do not focus on checking or avoiding cases of division by zero, or the logarithm of zero.
Neuron activation function
In a similar way we describe the function of neuron activation.
public interface IFunction { double Compute(double x); double ComputeFirstDerivative(double x); } And examples.
Sigmoid
internal class SigmoidFunction : IFunction { private double _alpha = 1; internal SigmoidFunction(double alpha) { _alpha = alpha; } public double Compute(double x) { double r = (1 / (1 + Math.Exp(-1 * _alpha * x))); //return r == 1f ? 0.9999999f : r; return r; } public double ComputeFirstDerivative(double x) { return _alpha * this.Compute(x) * (1 - this.Compute(x)); } } Hyperbolic tangent
internal class HyperbolicTangensFunction : IFunction { private double _alpha = 1; internal HyperbolicTangensFunction(double alpha) { _alpha = alpha; } public double Compute(double x) { return (Math.Tanh(_alpha * x)); } public double ComputeFirstDerivative(double x) { double t = Math.Tanh(_alpha*x); return _alpha*(1 - t*t); } } Neuron, layer and network
In this section, we consider the presentation of the basic elements of the network; I will not give them the implementation, since she is obvious. The algorithm will be given for a fully connected “puffy” network, so the implementation of the network will need to be done accordingly.
So, the neuron looks like this.
public interface INeuron { /// <summary> /// Weights of the neuron /// </summary> double[] Weights { get; } /// <summary> /// Offset/bias of neuron (default is 0) /// </summary> double Bias { get; set; } /// <summary> /// Compute NET of the neuron by input vector /// </summary> /// <param name="inputVector">Input vector (must be the same dimension as was set in SetDimension)</param> /// <returns>NET of neuron</returns> double NET(double[] inputVector); /// <summary> /// Compute state of neuron /// </summary> /// <param name="inputVector">Input vector (must be the same dimension as was set in SetDimension)</param> /// <returns>State of neuron</returns> double Activate(double[] inputVector); /// <summary> /// Last calculated state in Activate /// </summary> double LastState { get; set; } /// <summary> /// Last calculated NET in NET /// </summary> double LastNET { get; set; } IList<INeuron> Childs { get; } IList<INeuron> Parents { get; } IFunction ActivationFunction { get; set; } double dEdz { get; set; } } Since we consider a fully connected “puffy” network, then Childs and Parents can not be implemented, but if you do a general algorithm, you will have to. Consider the fields that are particularly important for the learning algorithm:
- LastNET - neuron adder, the last calculated value is stored here
- LastState - neuron output, the last calculated value is stored here.
- dEdz is the neuron dE / dz mentioned above and is calculated depending on which layer the current neuron is on; partial derivative of the error function with respect to the neuron adder
The network layer looks simpler:
public interface ILayer { /// <summary> /// Compute output of the layer /// </summary> /// <param name="inputVector">Input vector</param> /// <returns>Output vector</returns> double[] Compute(double[] inputVector); /// <summary> /// Get last output of the layer /// </summary> double[] LastOutput { get; } /// <summary> /// Get neurons of the layer /// </summary> INeuron[] Neurons { get; } /// <summary> /// Get input dimension of neurons /// </summary> int InputDimension { get; } } And the network view:
public interface INeuralNetwork { /// <summary> /// Compute output vector by input vector /// </summary> /// <param name="inputVector">Input vector (double[])</param> /// <returns>Output vector (double[])</returns> double[] ComputeOutput(double[] inputVector); Stream Save(); /// <summary> /// Train network with given inputs and outputs /// </summary> /// <param name="inputs">Set of input vectors</param> /// <param name="outputs">Set if output vectors</param> void Train(IList<DataItem<double>> data); } But we are considering a multilayered neural network, so a special presentation will be used:
public interface IMultilayerNeuralNetwork : INeuralNetwork { /// <summary> /// Get array of layers of network /// </summary> ILayer[] Layers { get; } } Learning algorithm
The learning algorithm will be implemented through the pattern strategy :
public interface ILearningStrategy<T> { /// <summary> /// Train neural network /// </summary> /// <param name="network">Neural network for training</param> /// <param name="inputs">Set of input vectors</param> /// <param name="outputs">Set of output vectors</param> void Train(T network, IList<DataItem<double>> data); } For a more visual understanding, I will give a typical Train function of any neural network in the context of this implementation:
public void Train(IList<DataItem<double>> data) { _learningStrategy.Train(this, data); } Input Format
I use the following input format:
public class DataItem<T> { private T[] _input = null; private T[] _output = null; public DataItem() { } public DataItem(T[] input, T[] output) { _input = input; _output = output; } public T[] Input { get { return _input; } set { _input = value; } } public T[] Output { get { return _output; } set { _output = value; } } } As can be seen from the code in the previous parts, the neural network works with
DataItem.
, ( ), :
public class LearningAlgorithmConfig { public double LearningRate { get; set; } /// <summary> /// Size of the butch. -1 means fullbutch size. /// </summary> public int BatchSize { get; set; } public double RegularizationFactor { get; set; } public int MaxEpoches { get; set; } /// <summary> /// If cumulative error for all training examples is less then MinError, then algorithm stops /// </summary> public double MinError { get; set; } /// <summary> /// If cumulative error change for all training examples is less then MinErrorChange, then algorithm stops /// </summary> public double MinErrorChange { get; set; } /// <summary> /// Function to minimize /// </summary> public IMetrics<double> ErrorFunction { get; set; } } DataItem.
, ( ), :
public class LearningAlgorithmConfig { public double LearningRate { get; set; } /// <summary> /// Size of the butch. -1 means fullbutch size. /// </summary> public int BatchSize { get; set; } public double RegularizationFactor { get; set; } public int MaxEpoches { get; set; } /// <summary> /// If cumulative error for all training examples is less then MinError, then algorithm stops /// </summary> public double MinError { get; set; } /// <summary> /// If cumulative error change for all training examples is less then MinErrorChange, then algorithm stops /// </summary> public double MinErrorChange { get; set; } /// <summary> /// Function to minimize /// </summary> public IMetrics<double> ErrorFunction { get; set; } }
DataItem.
, ( ), :
public class LearningAlgorithmConfig { public double LearningRate { get; set; } /// <summary> /// Size of the butch. -1 means fullbutch size. /// </summary> public int BatchSize { get; set; } public double RegularizationFactor { get; set; } public int MaxEpoches { get; set; } /// <summary> /// If cumulative error for all training examples is less then MinError, then algorithm stops /// </summary> public double MinError { get; set; } /// <summary> /// If cumulative error change for all training examples is less then MinErrorChange, then algorithm stops /// </summary> public double MinErrorChange { get; set; } /// <summary> /// Function to minimize /// </summary> public IMetrics<double> ErrorFunction { get; set; } }Algorithm
Finally, by showing the whole context, you can go to the actual implementation of the neural network learning algorithm.
internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -) internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -) internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -) internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , ://process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), ://recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
: .
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , ://process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), ://recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
: .
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , ://process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), ://recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
: .
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , ://process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), ://recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
: .
, , , , . -)
internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -) internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , ://process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), ://recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
: .
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , ://process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), ://recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
: .
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , ://process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), ://recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
: .
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , ://process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), ://recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
: .
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , ://process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), ://recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
: .
, , , , . -)
internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -) internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -) internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -) internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)
internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).
( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");
, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }
, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);
:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }
:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }
, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());
, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);
. , , , . . , , , , , .
:
:
.
, , , , . -)Source: https://habr.com/ru/post/154369/
All Articles