Deep learning and brain work: When will technological singularity

One of the most discussed topics today is the emergence of artificial intelligence (AI), comparable to human or even more advanced. The moment when this should happen is called the technological singularity.

In this article, we will compare in-depth training methods with the principles of our brain, find out (based on this source) how far our algorithms are from perfect AI, and try to understand when this technological singularity can occur.
')
According to the famous American futurologist Ray Kurzweil, the technological singularity should begin in 2030. His calculations are based on two estimates - the complexity of the structure of the human brain and the pace of technological progress. The problem is that his study was conducted in 2005 and has not been analyzed further since. Considering that neurobiology at that time contained less than 1% of the knowledge about the brain that we have today, such a forecast can hardly be considered reliable.

Over the past two years, scientists have proven that the "performance" of our brain is several orders of magnitude higher than we thought. For example, it has been shown that unexcited neurons can be trained using energy from proteins, and that neurons can independently synthesize the right proteins to help a person process the necessary information.

Mathematical models built in accordance with the principles of the brain, today can very accurately repeat the brain activity. But no more than that. Modern methods of modeling, unfortunately, do not take into account the latest discoveries of neuroscientists, so it does not make sense to draw any conclusions based on them.

It turns out that, on the one hand, new technologies allow us to test generalized hypotheses related to information processing. On the other hand, the data that we obtain in the course of such studies do not carry much meaning.

One of the most accurate models describing the behavior of neurons in the human brain is the cascade model of Poisson (linear-nonlinear Poisson cascade model, LNP). This model works on the principles similar to the deep learning, which is considered the most promising technology on the way to the development of AI.

Despite the fact that we know in which direction we need to move and which models we need to develop in the future, there are still a lot of restrictions that, according to scientists, delay the technological singularity for a few more decades.

The work of depth learning algorithms, unlike other technologies, is highly dependent on the speed of the Internet connection. It is known that the development of network technologies goes at a much slower pace compared to the development of computer components: according to statistics, the speed of the Internet connection doubles every three years. Last year, Mellanox developed switches that transmit data at 100 Gbps. Technological solutions of Mellanox cannot yet be applied on any supercomputer, but when this happens, we will not have to wait for the next breakthrough in the field of network technologies.

A similar situation develops with the development of RAM. Solutions such as 3D-memory, still provide an opportunity to increase the speed of the RAM. However, we are no longer able to improve the performance of traditional methods.

An equally serious problem concerns the consumption of electricity and its cost. The Tianhe-2 supercomputer, which, according to the TOP500 rating, is considered to be the most powerful computer in the world, consumes 24 MW of energy per day, which takes from 65 to 100 thousand dollars. This energy would be enough to provide electricity to 6,000 homes in Germany.

Moreover, there are a lot of physical limitations. For example, not so long ago was developed a transistor the size of several atoms. According to experts, in the next two years we will reach physical limits in the production of computer equipment. Moreover, at such a micro level, such quantum effects as, for example, quantum tunneling begin to appear.

It is easier to imagine this phenomenon, if it occurred at the macro level. Suppose you need to charge your phone, and you plug it into an outlet located near the TV. The tunneling effect assumes that in this case the electrons themselves would decide which device they should go to. Anyway, to avoid this phenomenon and start a new wave of technical progress, we need new materials, new technologies and new ideas.

What do depth learning and thought processes have in common?

Modern depth learning algorithms are far from AI systems, which can be seen in Hollywood blockbusters. To prove this, we compare the convolutional networks that are used in in-depth training with the information processing processes occurring in our brain.

If you are familiar with the basics of neuroscience, then you probably know that neurons transmit electrical signals through tubular structures called axons. When a neuron is excited, it transmits an electrical signal — called an action potential — along an axon, which, in turn, forks into several sections. These areas are called terminals. At the end of each terminal are certain proteins. They replace an electrical signal with a chemical reaction that causes neurotransmitters - substances that transmit important information for us - to jump from one neuron to another. The combination of the axon terminals and other receptors can be represented as an input layer of the convolutional network.

Neurons can have from five to several hundred branching processes called dendrites. Only in 2013, it was proved that the so-called dendritic spikes - action potentials arising in dendrites - play a large role in information processing.
Dendritic spikes occur in a situation where the depolarization of the dendrite reaches a fairly high value.

In this case, the electrical signal entering the body of the neuron can dramatically increase due to the potential-dependent channels that generate additional electrical potential. This process resembles the implementation of the max-pooling method in depth learning. Moreover, dendritic spikes play the same function both in the human brain and in the structure of the convolutional network. Appearing in the visual system, action potentials in dendrites help determine the position of an object in space and recognize images.



A is a neuron model that does not take into account dendritic commissures; B - modeling the usual dynamics of dendritic spikes; C - modeling of more complex dynamics of dendritic spikes with allowance for one-dimensional diffusion of particles, which resembles a convolution operation. Image © was published in an article by Anwar, Rooma, Nedelescu, Chen, Kuhn and De Schutter in Frontiers in Cellular Neuroscience

If the level of depolarization is high enough, this does not mean that the neuron will be excited. Immediately after excitation, the neuron has too high positive potential. In this regard, there may be a delay until the potential takes on the desired value.

In the LNP model, this delay is represented as an inhomogeneous Poisson process with a Poisson distribution. This means that the probability of excitation of a neuron from the first or second time is quite high, while the probability of a later excitation decreases exponentially.

These are just some of the examples of how deep learning methods reproduce the processes occurring in our brain. If you study this topic in more detail, then you will understand how the work of convolutional networks resembles the information processing processes occurring in our head. At the same time, deep learning and brain work have a number of significant differences, thanks to which we can assert that our computer systems are still far from artificial intelligence.

What is missing depth learning

The main disadvantage of existing models is that they do not take into account many details. Modern models do not use knowledge of biological information processing processes, including changes in proteins and genes - they are too difficult to understand.

Depth learning does not take into account, for example, axon-axon connections, since in them information is processed using special algorithms different from those mentioned above. In addition, in some cases, we simply do not have enough reliable information to create a mathematical model. For example, we do not know exactly how our brain is trained. We know that some learning algorithm with reinforcement is involved in this process, but we lack some details, which leads to the appearance of contradictory facts.

But do we really need these details? For comparison, the traditional physical model is used by physicists and engineers around the world. Thanks to this model, they manage to develop high-tech solutions that we can use in everyday life. At the same time, the physical theory is incomplete. Moreover, it had to be slightly changed after six months ago, physicists discovered a new particle at the Large Hadron Collider.

Another argument is given by the famous French scientist Yang Lekun. He claims that our aircraft are capable of moving through the air just as well as birds. But to describe the movement of the bird is extremely difficult, since it requires taking into account even the smallest details. The movement of the aircraft, in turn, is described only by the laws of aerodynamics. Why, then, deep learning cannot have a simpler structure compared to the human brain?

Obviously, we do not need to accurately reproduce all the thought processes in our algorithms in order to create an AI comparable to human. The only question is what knowledge will be enough for this.

So, we compared the mechanisms of in-depth learning with the processes occurring in the brain, and made sure that many of these processes (dendritic commissures, neuron delay) are also reproduced in convolutional networks. This gives us confidence that the development of depth learning methods is moving in the right direction.

Based on the current pace of technological progress, some scientists conclude that artificial intelligence will be created in the next 20 years. However, most likely, this will happen after 2080.

We will not be able to achieve success in the development of AI, if we can not overcome a number of limitations. We need to use energy more efficiently and organize a more efficient process for the production of machinery and components. Development and implementation of innovations that significantly increase the speed of RAM and Internet connections are becoming key factors for technological development. So we should not expect the emergence of advanced AI systems in the near future.

PS If you go back to the realities of today, this week we looked at the main trends in IaaS .

It is worth noting that in the past and at the beginning of this year we set a good pace in the work to expand the company's cloud space. We have not familiarized you with the equipment arriving at our cloud platforms for a long time and decided to correct this situation with the help of new material on unboxing Cisco Cisco UCS M4308 servers.