Is tribal enmity a natural failure?

What do computers teach us about how to get along with each other

At our office at Carnegie Mellon University, my colleague John Miller and I created a computer program that is prone to genocide.

We definitely did not strive for this. We have not studied rivalry or war. We were interested in the emergence of primitive forms of cooperation. So we created cars that lived in an imaginary society, and made them play a game with each other - it is known for generating complex social behavior just as well as a rotten banana producing fruit flies.
')
The game is called Prisoner's Dilemma . She has many varieties, but in fact - this is a story about two people choosing to cooperate with them or to deceive each other. If both cheat, both suffer. If both work together, both will be better. But if one cooperates, and the second cheats, the cheater will be even better.

The generalization of the game makes it attractive for a political philosopher, and strict specifics allows us to put it in the basis of a computer simulation. As a tool for investigating human behavior, it is equivalent to the inclined plane of Galileo or Mendel’s experiments with peas . Join the strike or wash across the picket line? Stop production so that prices do not fall, or will you dump and flood the market? Make an effort when working in a group, or relax, and leave the rest to work?

Our simulation was simple: in the virtual world, decision-makers who had a limited ability to reason played this game over and over again. We, like ruthless judges, awarded the thriving cars and punished the rest. Successful machines passed the strategy to next generations, which was disturbed by random variations that mimic natural mutations.

We have provided the machines with a simple language to think in and resources to provide memory and actions based on it. Each generation of machines collided in pairs with each other. That's how life seems to us: we constantly meet with trading partners, and with the consequences of how we deal with them. In our model of the world, two Robinson Crusoe met on the beach.

By launching the development of these small communities, we expected confirmation of what many consider to be the best strategy for playing the prisoner’s dilemma: tooth for tooth. The machine that uses this strategy starts with the fact that it keeps its promises, but takes revenge for the deception by a one-time deception on its part. Tooth-for-tooth is the rule of honor in our sandbox: treat others well, unless they give you reasons to treat them differently, and quickly forgive them.

But after examining the output of simulations in which strategies evolved freely in arbitrary directions, we found something completely different. After an early chaotic period, one of the cars quickly began to dominate, dominating the fictional world for hundreds of generations, and then at some point everything suddenly collapsed, and the world plunged into the chaos of conflicts, from which the next cycle grew. Archaeologists of such a world would have dug up thick layers of prosperity, alternating with the era of bones and ashes.

Instead of following the rules of the sandbox, ruled by cautious and proud co-operators, populations spawned strange configurations in which we did not see the point. Until one evening in the office, when we suddenly stumbled upon the truth. Dominated cars took players' actions for a code by which they could recognize the moment in which they encountered copies of themselves.

The simulations show the initial levels of probably random behavior, which at about the 300th generation was inferior to a high level of cooperation, coinciding with the complete dominance of one machine, leading the rest to extinction. This forced cooperation disappears at about the 450th generation. After that, the system switches between two extremes. Green and yellow stripes correspond to the era of high and low cooperation, respectively.

In the initial stages of the game, they worked out a clear scheme of work: cooperate, deceive, deceive, cooperate, deceive, cooperate (this is an example). If their opponent responded in exactly the same way, deceiving with them, cooperating with them, they eventually switched to the phase of continuous cooperation, to mutual advantage.

However, woe to those who do not know the code of honor. Any deviation from the expected sequence led to a general and ongoing war. Such a response could carry with them both cars in a kind of attack of a digital suicide bomber. Since it was very difficult to come across the desired sequence by chance, only the descendants of the ruling machines could flourish in the post-Codex era of disinterested cooperation. All the others were killed, including those who used the "tooth for tooth" strategy. This dominance continued until a lot of errors were accumulated in the code passed down from generation to generation, so that the dominant machines would cease to recognize each other. Then they fought on each other as fiercely as they did on outsiders, showing something similar to an autoimmune disease of the population.

In the period when the codes were respected, we called them shibololets , in memory of the linguistic discrimination mentioned in the book of Judges of Israel :

… And the Gileadites seized the crossing over Jordan from Ephraim, and when some of the surviving Ephraimites said, “Let me cross,” then the inhabitants of Gilead said to him: Are you not an Ephraimite? He said no. They said to him “say: shibbolet”, and he said: “sibboleth”, and could not speak out otherwise. Then they took him, and slaughtered him at the crossing of the Jordan. And there fell at that time of Ephraim forty-two thousand ... (Judges 12: 5-6).

Shibbolety - a common feature of the culture and conflicts of people. Finns, who could not utter the word yksi (“one”), were considered Russian during the Finnish civil war . Tourists in the business district of Manhattan are quickly recognized if they pronounce the name of Houston Street [Houston Street] in the same way that the name of the city of Houston [Houston] in Texas is pronounced.

And our machines used them so effectively to dominate the population that no one could survive. Even at the end of this era, their descendants inherited the ashes. The blind hand of evolution has found a simple, albeit terrible, solution.

It was a harsh and cruel social landscape. But we gave our machines very little to think about. How would two perfectly rational machines behave during conflict if they knew that each of them was perfectly rational? By the very nature of rationality, two fully rational creatures, faced with one problem, should behave the same way. Knowing this, everyone would choose cooperation - but not from altruism. Everyone would understand that in the event of a deception, his opponent would also choose a deception, which would both lose.

There is a spectrum between these two positions. At one end there are minimally calculating machines, limited zero positions of culture, which naturally degenerate into internecine enmity, tribalism . At the other end is the inevitable cooperation of a fully rational being.

Where on this line between rude machines and angelic rationality are human beings?

If we, people, are rational, or at least strive for it, then there is reason for optimism. Francis Fukuyama may have thought of something similar when he wrote his " The End of History " in 1992. Although his arguments are based on the works of nineteenth-century German philosophers, such as Nietzsche and Hegel, we can rewrite them in this way: a rather complicated situation in a person’s life ends up in a rational, liberal-democratic, capitalist order opposing a scattered set of enemies.

The arguments of Fukuyama were based not only on philosophical reasoning, but also on the comprehension of current events: the collapse of communism, the prosperity of electronic media, the seemingly seamless opening of borders, and the epic growth of stock markets.

Today, his work serves as a monument to the dreams of a previous era (one of the chapters was called “VCR victory”). Our cultures are evolving, but, apparently, not in the direction of harmony. The chaos of the XXI century is very similar to our simulations. Two decades after 9/11, even Western liberal democracies agree with gloomy models of human behavior and more sombre theorists than Fukuyama.



Karl Schmitt, for example, who considered the deliberative elements of democracy to be veiled authoritarian forms of government. Or Robert Michels, whose study of political inequality allowed him to see that democracy is a temporary stage in the evolution of a society heading for the rule of a small and closed group of elitist. While the intellectuals of both types of political extremism increasingly see the rational political order as a fantasy, shibbolets take on their role in defining racial, national, and religious boundaries and again become ineradicable properties of political life.

There is a big difference between these philosophers, as well as between the corresponding computer models - between the simple, cruel and irrational creatures that John Miller and I pretended to be, and the super-rational cooperatives that Fukuya saw as waiting for us at the end of history. But these models, at least, cause restrained optimism.

Researchers from the Institute for the Study of Machine Intelligence (MIRI) in Berkeley studied the behavior of rational, but resource-limited machines capable of studying each other's source code. It may seem that such transparency can solve the problem of cooperation: if you can predict what your opponent will do by simulating his code, you can solve it, then deception is not worth its price. But what if in the source code of the opponent there is a simulation of what I will do as a result of running this simulation, and he will try to take advantage of this knowledge? Without a symmetrical ideal rationality, this problem leads to serious distortions of thinking.

Some cars at the zoo MIRI may remind you of people you know. For example, CliqueBot collaborates with everyone with similar source code. Only codes that exactly match his code are important to him. FairBot, on the contrary, is trying to look for superficial differences and prove that his opponent will cooperate with him. Roughly speaking, FairBot says: “if I can prove that my opponent will cooperate with me, I will cooperate with him”.

How do these cars get along with each other? Although the general solution is the paradox of regression, studies of the predictive behavior of machines in the prisoner's dilemma provide a calming answer - mutual cooperation remains, at least, possible even for a player with limited resources. For example, FairBot can recognize fair machines in a similar way, even if their source code differs from it — which suggests that diversity and collaboration are not impossible, at least with a sufficiently high level of intelligence [Barasz, M., et al. Robust cooperation: Program equilibrium via provability logic. arXiv 1401.5577 (2014)].

Even from genocidal machines on the brutal edge of the spectrum, an inspirational lesson can be learned. They came from the depths of the PCB, being simulated on a Texas supercomputer. They have no biological excuses. It is possible, and we should not make excuses: if this behavior is so common that it appears even in the simplest simulations, perhaps we need neither fear nor idolize it, but treat it the same way as cancer or the flu.

What if we consider intertribal hostility, as a natural failure of any cognitive system based on silicon or carbon? Not as universal truth or inevitable sin, but as something that needs to be overcome?

Simon DiDio is an Assistant Professor at Carnegie Mellon University, Director of the Social Thinking Lab and Faculty of the Institute in Santa Fe.