Let's program the rosenblatt perceptron?

After a provocative article by Perceptron Rosenblatt - what is forgotten and invented by history? and one completely proving the absence of problems in the Rosenblatt perceptron, and even vice versa showing some interesting sides and possibilities. What is the role of the first “random” layer in the Rosenblatt perceptron? And indeed, reliable information about him, except in the original, is not possible to find. But even there it is quite difficult to describe how to program this perceptron. I will not post the full code. But let's try to go through a number of basics.

Let's start ... oh yes, I warn you, I will not be telling classically, but somewhat modernized ...


')
The code is in C #, but I think it’s better to treat it as a pseudocode. But if someone did not understand something, ask - I will explain.

And we begin with the sensors.

public class cSensor
{
public event EventHandler ChangeState ;
private sbyte state = 0 ;
public sbyte State
{
get { return state ; }
set
{
state = value ;
if ( state == 1 && ChangeState ! = null )
{
ChangeState ( this , new EventArgs ( ) ) ;
}
}
}
}

It's simple. The sensor has a state of 0 or 1. As soon as the sensor receives a unit, it sends an event. Next we have a synapse.

public delegate void BackCommunication ( sbyte Type ) ;
class cSinaps
{
private sbyte type ;
private BackCommunication hBackCommunication ;
public cSinaps ( sbyte tType, BackCommunication tBackCommunication )
{
type = tType ;
hBackCommunication = tBackCommunication ;
}
public void ChangeSensorState ( object sourse, EventArgs e )
{
hBackCommunication ( type ) ;
}
}

It has the type - it will excite or inhibit the activity of the sensor. And also the reaction to a change in the sensor (ChangeSensorState). Now the actual A-element in the middle layer.

public class cAssociation
{
// A-element activation level
public int ActivationLevel = 0 ;
// Synapses connected to this A-element
private cSinaps [ ] oSinaps ;

public cAssociation ( cSensor [ ] sensorsField, int SCount, Random RND )
{
int SinapsCount = 10 ;
oSinaps = new cSinaps [ SinapsCount ] ;

int tSinapsNumber = 0 ;
int tmpSensorNumber = 0 ;
sbyte tmpSensorType = 0 ;

for ( int j = 1 ; j < SinapsCount + 1 ; j ++ )
{
tmpSensorNumber = RND. Next ( SCount ) ;
if ( RND. Next ( 2 ) == 0 ) tmpSensorType = 1 ; else tmpSensorType = - 1 ;

oSinaps [ tSinapsNumber ] = new cSinaps ( tmpSensorType, AssumeSinapsSignal ) ;

sensorsField [ tmpSensorNumber ] . ChangeState + =
new EventHandler ( oSinaps [ tSinapsNumber ] . ChangeSensorState ) ;
tSinapsNumber ++ ;
}
}

void AssumeSinapsSignal ( sbyte Type )
{
ActivationLevel + = Type ;
}
}

When creating an A-element, it is necessary to form synapses associated with it, let them be 10. Randomly decide with which sensor to connect it and what type of synapse it will be (stimulating or inhibiting). And most importantly, we subscribe to the change in the value of the sensor in order to call at this moment AssumeSinapsSignal. And there we increase the level of activation or decrease depending on the type of synapse attached.

In general, everything, everything that was so difficultly told in What is the role of the first "random" layer in the Rosenblatt perceptron - we realized. We have in the set of A-elements a guaranteed linear representation of any arbitrary problem.

We now turn to learning the method of correcting errors in the second layer. Initially, the general algorithm, I think is clear without words.

public class cNeironNet
{
public cSensor [ ] SensorsField ; / * Touch field * /
public cAssociation [ ] AssociationsFiled ; / * Associative field * /
int ACount ;
int SCount ;
public ArrayList AHConnections ;
Random RND = new Random ( ) ;

public cNeironNet ( int argSCount, int argACount )
{
ACount = argACount ;
SCount = argSCount ;
SensorsField = new cSensor [ SCount ] ;
for ( int i = 0 ; i < SCount ; i ++ )
{
SensorsField [ i ] = new cSensor ( ) ;
}
AssociationsFiled = new cAssociation [ ACount ] ;
for ( int i = 0 ; i < ACount ; i ++ )
{
AssociationsFiled [ i ] = new cAssociation ( SensorsField, SCount, RND ) ;
}
}

/ * Add a new sample from the training set to processing * /
public ArrayList JoinStimul ( int [ ] tPerception, int [ ] tReaction )
{
for ( int i = 1 ; i < ACount + 1 ; i ++ )
{
AssociationsFiled [ i ] . ActivationLevel = 0 ;
}
for ( int i = 1 ; i < SCount + 1 ; i ++ )
{
SensorsField [ i ] . State = 0 ;
}
// Throw on the sensors obtained example
for ( int i = 0 ; i < SCount ; i ++ )
{
SensorsField [ i ] . State = tPerception [ i ] ;
}
// Remember which A-elements were active in this example
AHConnections = new ArrayList ( ) ;
for ( i = 0 ; i < ACount ; i ++ )
{
if ( AssociationsFiled [ i ] . ActivationLevel > 0 )
{
AHConnections. Add ( i ) ;
}
}
// Remember what the reaction should be for this example
SaveReaction ( tReaction ) ;
return AHConnections ;
}
/ * When all the examples are added, the perceptron is called to learn them * /
private void Storing ( )
{
// Do a lot of iterations
for ( int n = 1 ; n < 100000 + 1 ; n ++ )
{
// For each iteration, scroll through all the examples from the training set
for ( int i = 1 ; i < StimulCount + 1 ; i ++ )
{
// Activate the R-elements, i.e. counting outputs
RAktivization ( i ) ;
// Find out if the perceptron was wrong or not, if we made a mistake we send for training
bool e = GetError ( i ) ;
if ( e )
{
LearnedStimul ( i ) ;
Error ++ ; // The number of errors, if at the end of the iteration = 0, then jump out of training.
}
}
}
}
}

Activating R-elements is also simple. Sum the weights from the active A-elements, and forward through the threshold (= 0).

private void RAktivization ( int ReactionNumber )
{
int [ ] Summa = new int [ RCount + 1 ] ;
for ( int j = 1 ; j < RCount + 1 ; j ++ )
{
for ( i = 1 ; i < AHConnections [ ReactionNumber ] . Count + 1 ; i ++ )
{
Summa [ j ] + = Weight [ AHConnections [ ReactionNumber ] . Value [ i ] ] . Value [ j ] ;
}
}
for ( int i = 1 ; i < RCount + 1 ; i ++ )
{
if ( Summa [ i ] > 0 ) Reactions [ i ] = 1 ;
if ( Summa [ i ] <= 0 ) Reactions [ i ] = - 1 ;
}
}

Check there is an error or not below.

private int GetError ( int ReactionNumber )
{
int IsError = 0 ;
for ( int i = 1 ; i < RCount + 1 ; i ++ )
{
if ( Reactions [ i ] == NecessaryReactions [ ReactionNumber ] . Value [ i ] )
{
ReactionError [ i ] = 0 ;
}
else
{
IsError = 1 ;
ReactionError [ i ] = NecessaryReactions [ ReactionNumber ] . Value [ i ] ;
}
}
return IsError ;
}

Here we check the current reaction with the one that is, and prepare an array for learning about the reactionError reaction. Now we’ll focus on the last one - the actual learning.

private void LearnedStimul ( int ReactionNumber )
{
for ( int j = 1 ; j < RCount + 1 ; j ++ )
{
for ( int i = 1 ; i < AHConnections [ ReactionNumber ] . Count + 1 ; i ++ )
{
Weight [ AHConnections [ ReactionNumber ] . Value [ i ] ] . Value [ j ] + = ReactionError [ j ] ;
}
}
}

So that is all.

The only thing they ask me is: “apparently, this algorithm of teaching correction with an error is also stuck as the algorithm of back propagation of error, if the weights are zero?”. As you can see, here the training itself begins with zero weights. There are not any mathematical formulas - elementary increments or decrements. If the weight was 0, then if the error is corrected, it will become either +1 or -1, the weight can change the sign again after passing through zero, but it is physically impossible to get stuck at zero.

upd.

Below retran suggested using matrix and linq from functional programming to activate A-elements in place of my code in events according to a clearer OOP model.

public class Perceptron1
{
public int [ ] [ ] SAMatrix { get ; private set ; }
public Perceptron1 ( int sensorsCount, int associativeElementsCount )
{
var random = new Random ( ) ;
SAMatrix = new int [ associativeElementsCount ] [ ] ;
for ( var i = 0 ; i < associativeElementsCount ; i ++ )
{
SAMatrix [ i ] = new int [ sensorsCount ] ;
for ( var j = 0 ; j < 10 ; j ++ )
{
var sindex = random. Next ( sensorsCount ) ;
if ( random. Next ( 2 ) == 1 )
if ( random. Next ( 2 ) == 1 )
SAMatrix [ i ] [ sindex ] + = 1 ;
else
SAMatrix [ i ] [ sindex ] - = 1 ;
}
}
}
public int Activate ( int i, int [ ] inputs )
{
return ( SAMatrix [ i ] . Zip ( inputs, ( w, input ) => w * input ) . Sum ( ) > 0 ? 1 : 0 ) ;
}
}

Yes, of course the code is much shorter. But it’s less clear, it’s harder to make changes if the activation processes change. But most importantly, it works more than 100 times slower . So much for the charm of mathematical and functional programming :)

We test it this way (although in my code there are several bugs above, but they are easy to fix, if someone does this, + most of them fixed and updated the code above):

Random random = new Random ( ) ;
int [ ] Input = new int [ 1000 ] ;
int [ ] AActiv = new int [ 900 ] ;

TimeSpan BeginTime = DateTime. Now . TimeOfDay ;
Perceptron1 P1 = new Perceptron1 ( 1000 , 900 ) ;
for ( int i = 0 ; i < 100 ; i ++ )
{
for ( int j = 0 ; j < 1000 ; j ++ )
{
Input [ j ] = random. Next ( 2 ) ;
}
for ( int j = 0 ; j < 900 ; j ++ )
{
AActiv [ j ] = P1. Activate ( j, Input ) ;
}
}
TimeSpan locTime = DateTime. Now . TimeOfDay - BeginTime ;

// TimeSpan BeginTime = DateTime.Now.TimeOfDay;
// Perceptron2 P2 = new Perceptron2 (1000, 900);
// for (int i = 0; i <10000; i ++)
// {
// for (int j = 0; j <1000; j ++)
// {
// Input [j] = random.Next (2);
//}
// P2.JoinStimul (Input);
//}
// TimeSpan locTime = DateTime.Now.TimeOfDay - BeginTime;

Console. WriteLine ( locTime. ToString ( ) ) ;
Console. ReadLine ( ) ;