The man taught the car to watch a movie. And then the real fantasy began.

Last week Warner Bros. Vimeo has sent a video hosting notice of copyright infringement under the Digital Millennium Copyright Act (DMCA). The notification included a regular list of illegally uploaded video materials owned by Warner. There were episodes of the show “Friends,” “Pretty Little Liars,” as well as two downloads from the video from Ridley Scott’s film Blade Runner.

A common example of copyright infringement? Not really. Warner made an amazing mistake. Part of the video (the statement from Warner has already been withdrawn) was not taken from the film. More precisely, it was taken, but in a form that the world has not yet seen.
')
It was part of a data-learning project using machine learning, in which Philip Dick’s classic tale about androids was restored from a handful of numbers.



In other words, the company Warner sent a DMCA request for an artificially restored video from the film, in which they talk about artificial creatures that are indistinguishable from people, because they could not distinguish between simulation and the real thing.

Parse "Blade Runner" using AI

Terence Broad [Terence Broad] - a researcher from London, working on a diploma in the specialty "creative computational methods." His thesis " Autocoding video frames " sounds boring until you begin to understand that it is the key to a strange interweaving of remix culture, Internet copyright and artificial intelligence, which led Warner to decide to send a request to delete an illegal video.

The goal of Broad was the application of “deep learning” - a fundamental AI technique that uses algorithmic machine learning - to video. He wanted to see what the AI ​​is capable of if he is taught to understand the data from the video sequence.

The video contains a huge amount of visual information. When you watch a video on a computer, all this information is pre-encoded and compressed, so that it can be decoded and unpacked. Without this, the files would be too large and would not fit on the hard disk.

Typically, video coding consists of an algorithm that uses a compression standard developed by people who chose all its parameters — how much data to compress, what format, how to package it, how to reduce various parameters like aspect ratio, sound, metadata, etc.

Broad wanted to teach the neural network to manage the video encoding process independently, without human intervention. A neural network is a computer simulation of the functions performed by the brain and the central nervous system. This is a mechanical form of AI, working on solving complex problems by the same methods as the central nervous system - using its various parts to collect information and transfer it to the entire system.

Broad hoped that, if successful, this new way could become "a new technique in the preparation of experimental images and videos." But before that, he had to teach the neural network to watch a movie — not the way people do it, but the way it suits the car.

Do coders dream about electricians (or how to teach AI to watch movies?)

Broad chose a neural network option called convolutional autoencoder. First, he set up a “ learned similarity metric ” to help the coder deal with the data of the Blade Runner. The metric gives the encoder selected movie frames, as well as “false” data, or data that is not part of the movie. By comparing the movie data with external garbage, the encoder learns to recognize the similar features of the data sets that came from the movie. In other words, he learned what the movie looks like.

Having learned to recognize the film data, the encoder reduced each frame to a representation as a number of 200 digits, and then reconstructed this number back into new frames, in order to achieve a match with the original. Broad chose a small file size, which made the result of the reconstruction very blurry. Finally, Broad instructed the encoder to restore the sequence of reconstructed frames so that they went in the same order as the frames in the original film.

In addition to Blade Runner, Broad taught his auto-encoder to watch Clouded, animated with rotoscoping. Both films are adaptations of the famous works of Philip Dick in the science fiction genre, and Broad decided that they are just suitable for his project.

For each of the films, the training was repeated six times, and each time Broad corrected the algorithm to help the machine more intelligently approach the task of reading the collected data. This is how the “Blade Runner” footage was chosen for the coder after the sixth workout. The picture shows two rows of before / after frames. On the left - the original, on the right - the interpretation of the encoder.



During the six workouts, Broad used only selected shots from two films. At the end of the sixth workout and fine tuning, Broad launched the neural network for the reconstruction of both films entirely on the basis of her knowledge. Here is an example of how her Cloudedness turned out:



Broad explained to Vox that the neural network version of the film was completely unique and based on what she saw in the original film. “Essentially, you see the movie from the point of view of the neural network. Therefore, reconstruction is an interpretation of the film by the system (and other films that I drove through the models), based on a limited representative “understanding”.

Why are the works of Philip Dick perfect for such a project?

Dick is a legendary science fiction writer whose work combined the study of social problems with metaphysical research and questions of the reality of our Universe. Many screen versions of his work include “Dissenting Opinion,” “Recall All,” “Changing Reality,” and a series from Amazon TV “The Man in High Castle ”.

And, of course, the famous story “Do Androids Dream About Electricians?”, Which inspired the creation of the film “Blade Runner” - an anti-utopian masterpiece of science fiction and one of the greatest films of all time. In the film, the work of Harrison Ford’s character Rick Deckard consists in tracking down and destroying “replicants” - a group of advanced androids who can almost ideally impersonate people. The hero's antagonist, Roy Batty, is one of the replicants - this famous role was played with a convincing "weariness of life" actor Rutger Hauer. Batty tries to cope with his humanity, at the same time fighting to prolong his life and trying to win a fight with Deckard before he “retires him”.



Dick was very concerned about the question of the gap between the "appearance of reality" and "real reality". In his dissertation, Broad indicated that, in his opinion, these two works by Dick for this simulation were very suitable:

For the study of these topics (the subjectivity of rationality) there can hardly be a more suitable film than Blade Runner (Blade Runner, 1982), which became one of the first works studying subjectivity and constantly concentrating attention on eyes, photographs and other perception symbols. .

Another film used as a model, “Blurred” (A Scanner Darkly, 2006), is another adaptation of the 1977 novel by Philip Dick. This story also explores the nature of reality, and its reconstruction by the neural network is particularly interesting, since each frame of the film has already been reconstructed by an animator, drawn over by hand.

That is, the use of “Blade Runner” in a project with artificially recreating material has a deep symbolic meaning. “I had a feeling that the first film recreated by the neural network should be“ Blade Runner ”.

Copyright Puzzle

All these difficulties and nuances of n / f culture and artificial learning did not reach the person who decided to send a request to remove material on behalf of Warner Bros. Perhaps because of this, after Vox contacted Warner, the latter conducted an internal investigation and restored two videos that were previously removed from the site.

However, Broad told Vox that the way he used the Blade Runner in his study of AI was unlikely to constitute an exemplary copyright infringement case: “No one has ever made a video in this way, and therefore the precedent for there is no such thing, and there is no legal definition as to whether such reconstructed video materials are a violation of copyright ”.

It doesn't matter if there are still copyright issues around his videos, Broad's experiments won't stop at Blade Runner. On the Medium website, in an article describing the project, he wrote that “I was surprised at how well the model behaved, as soon as I began to train it on Blade Runner,” and that it “would definitely carry out more experiments with training models at a larger number of films in the future to see what comes of it. "

The potential of machines to clearly and simply “read” and recreate video materials opens up amazing possibilities for both AI and video creation. Obviously, they have a long way to go until the moment when the Broad network will create some amazing video technology, but we can definitely say now - we have seen something that you would never believe.