Lumia Midsummer Night: How a photo installation was created at the MSND festival
On June 21, on the day of the summer solstice, Midsummer Night's Dream was held at the Serednikovo estate near Moscow. Every year, guests gather at the event in truly impressive masquerade costumes. However, today we want to tell a little about it.
Our cooperation with the organizers has become a good tradition. This year we decided to surprise visitors with an interactive installation - “bullet time” with a photo machine consisting of 60 Nokia Lumia 1520 smartphones. It allowed making impressive live pictures with the effect of frozen time in 360 ° format, which the guests of the festival could later enjoy. We talked about the technical aspects of the creation of this machine with our colleagues who were responsible for the direct implementation of the project.
We came to our colleagues from Hello Computer with a ready-made idea and ... deadlines. Fortunately, they readily went to work:
')
“ The project was allocated only three weeks, and after all the approvals and writing of the TOR, there are generally two. Therefore, we immediately tried to determine the most optimal solutions and platforms that could provide us with a fast development speed.
First of all, we started with the synchronization of phones. It is clear that forcing 60 devices connected via wireless to take a picture at the same time is a rather difficult task. Therefore, our choice fell on the architecture of "client-server ". "
As planned by the developers, the phones (clients) had to contact the server, synchronize time with it, calculating the corrections for the signal and get the time from the server when you need to take a picture.
The team that worked on the project included 7 people: two Windows / Windows Phone developers (one worked as a phone application, the other on a tablet); server developer who worked with the central server and utilities for gluing and sharing; project producer and assistant producer responsible for management; designer; Technical Director, on whose shoulders lay the preparation of TZ, coordination of work, prototyping, network and equipment design.
As it should be, the collective thought process began with a heated discussion, although the subject itself was Nokia Lumia 1520:
“ In our studio, only one person uses Windows Phone, so everyone else was interested in testing the devices and at the same time spending half a day on“ holivar ”on the topic:“ Do we need such big phones? ”,“ Is the Windows Phone platform good? ”And so on. d. ".
However, soon the project began to acquire its features from the technical side:
“ Since we decided from the very beginning with the architecture of the project, the main task for us was to remove as much logic as possible to the server. Setting up any kind of settings in 60 phones is a very time-consuming task.
We started with a simple NodeJS server prototype and a skeleton application for a Windows Phone 8.1 phone. Once a second, the server sent requests for smartphones to synchronize the current time and time of the moment of shooting. We also concluded that the difference between the “internal” time of the device and the timing it received from the server. This difference pleased us: it was only within 20-30 milliseconds . ”
At the first design stage, the developers had 5 smartphones, of which they assembled the first prototype. After a quick test, as usual, unforeseen difficulties emerged:
“ The first thing that caught my eye was the difference in color and illumination of images. And for the "glue" we needed to get the same picture on all devices.
Here, Windows Phone pleased us with good and complete documentation (for example, about what settings can be transferred to the phone) and ready-made examples. It was important for us to set the same white balance, shutter speed and ISO on all devices. And thanks to the advanced shooting settings in the Nokia Lumia 1520, which are usually found only in cameras, we were able to easily build a frame on each of the devices. We were also pleasantly surprised and the manual focus adjustment mode was useful.
All these settings were also received from the server . ”
Next, the team had to establish interaction between the devices and the server:
“ Pretty quickly we set up automatic sending of photos back to the server. At this point, we had a ready-made prototype of the application, which could be started to be tested on more phones.
We organized all client-server interaction according to the principles of REST API. In NodeJS (more precisely, in Express) this is done quite quickly and simply. We made the management interface in the form of a simple web page that opens from a local server. Data exchange implemented using weboket. Fortunately, we didn’t have to worry about cross-browser compatibility . ”
In parallel with writing an application for phones, the team worked on utilities for pasting photos into one video and then posting them on the network. As was clear from the number of devices involved in the project, the input of the utility should have received a set of 60 photos.
“ We decided not to reinvent the wheel and used ffmpeg for gluing and resizing photos. He pleasantly surprised us by the fact that he could do everything we needed: to glue the photo into a video with a change in the quality of compression and fps ;, change the size of the photo, impose a watermark . ”
An important task of the project was the ability to automatically post the resulting video to the network. For us, the social component of this project was a priority, so the team was required to teach the photo machine to automatically upload videos to the public stream, send links to the mail and enable MSND guests to share videos on social networks.
“ It was decided to take Coub as the video hosting for several reasons. Firstly, our short looped videos with music is the perfect format for Kouba. Secondly, this service has a convenient API for downloading, with which we have already worked before. The documentation, unfortunately, is hidden, but the guys send it on request. Thirdly, it was possible to easily and conveniently set up the feed and beautifully draw it. And, of course, each video could be shared online - this opportunity was provided by Coube itself, which saved development time.
From the previous project, we still have a suitable Python script. It worked quite simply: it scanned the correct folder and downloaded each new video file, after which it created a small text file with the video address and its status. We specifically abandoned the use of databases in favor of script flexibility and portability in any other project. Also, this script was able to send video to the mail . "
We also asked to add the ability to download the video to your device via the OneDrive service:
“ OneDrive has a clean and convenient API. Therefore, we simply added our script with a module for uploading video to the “cloud”. First, he passed the standard OAuth authorization, then uploaded files to our account and received a link to view and download. This link was saved to the same small file . ”
An important aspect of the work of all utilities was the support of offline mode. If the download of files for one reason or another did not succeed, the scripts silently tried to repeat it after a while.
The tablet on Windows RT - Lumia 2520 was chosen as a device on which guests could view their 360 ° images. The developers wrote an application that also accessed the server and received a list of all the videos from it. In the application, users could enter the mail, as well as get a QR code with a link to OneDrive or Coub.
When the software was ready, it was time to go to the field trials. To begin with, all 60 smartphones and the server were merged into one network. In order for the team not to get confused during the final assembly and further work with content, the server was assigned 254 addresses, and smartphones from 1 to 60. All devices were strictly numbered, and in the router each MAC address was assigned its own IP.
However, a one-time connection of 60 phones to one Wi-Fi network was not such a simple task:
“ We received all 60 smartphones 5 days before launch, and then the funniest“ tambourine dances ”began in this project.
We started tests on one of the most top routers (hereinafter, manufacturers and models of routers are not mentioned in order to avoid causing any reputational damage, author's note). He calmly withstood the connection of all 60 Lumia, but when he tried to get pictures from the devices, the files started to get lost. We were ready for this and had ready “plates” from another well-known manufacturer in order to distribute the customers evenly across three routers . ”
However, in this case, the team failed: the transfer of the first packet of photos took place without any problems, but the rest were lost in transmission or came with a delay of several minutes.
“ We started to panic - 4 days before launch, and with Wi-Fi everything is bad. We tried to dissolve everything on different non-intersecting channels. It's better, but not much. With that, the “weight” of one photo is only 500 kilobytes, that is, the total size of 60 photos was 30 megabytes + service traffic . ”
Began a long phase of testing. Engineers tried to understand how the situation changes when one or another number of phones is disconnected.
“ Probably, the conclusions reached by us will seem logical for specialists, but we are faced with such problems for the first time. Therefore, their decision cost us a lot of time.
The key mistake for us was that we tried to transfer all the photos at the same time. The signals interfered with each other, so the files were transferred very slowly. We were able to solve the problem by setting a delay of 2 seconds for each next device in the chain . ”
As a result, we came to a scheme in which every 20 devices were separated into a separate router and its own channel. This made it possible to transfer simultaneously 1 file per each router, taking into account a two-second delay, without any special problems. In such an architecture, sending one image took from 0.5 to 2 seconds (sometimes there were delays of up to 10 seconds), but the loss of files was already excluded from the list of problems.
“ Thus, it took us about 45 seconds to collect all the photos from the devices. Probably, this process could have been done faster, but under the conditions of a hard deadline there was no time for further experiments. Yes, and this speed, in principle, suited everyone: the change of guests inside the farm (the internal name of the installation among developers, author's note) and the preparation for the shooting took about the same amount of time ”.
The basis of the “farm” was the classic construction for shooting video in bullet time format: a curved spiral-shaped frame on which devices are mounted for shooting. In the course of the discussion of the project, some decorative improvements were proposed. One of them was, for example, a decorative fire in the center of the installation that was suitable for the subject of the event, through which guests had to jump at the time of the shooting.
Experimenting with the shape of the frame the team stopped at a four-meter radius and vertical position of the phones. This was explained both by the features of the shooting of the smartphones themselves and by the budget: an increase in the radius to 6 or 8 meters would lead to a significant increase in the cost of the project.
“ The main engineering task at this stage was the development of a mount that would allow very precisely adjusting the position of the phones in 2 planes of rotation. As a result, we stopped at a plastic mount with metal clips. The only thing left was to produce and assemble everything in time . ”
At the last stage, the team had to assemble the disparate components into one single system. They had only two days to install the equipment. The first day was completely given away for installation of the structure and Internet equipment, and the second day was for installation and configuration of smartphones, technical equipment and their testing.
“ The most difficult and time-consuming stage is the installation of smartphones. Filling software on all 60 devices takes about an hour. Now imagine what any bugs we found cost (well, that the client did not agree to 120 phones :)).
During the testing phase, we did encounter several errors with several times; as a result, we had to update the software on all devices again and again. Probably, it would be possible to make a system update by air, but there was no time for it anymore.
A separate story - installation 60 "Lumy". Smartphones had to be mounted very precisely, so that there were no jumps in the final videos. To do this, we made a special setting mode when smartphones displayed the image from the camera on the screen with a label in the center. In the center of the farm, we installed a rack with a bright target, which smartphones placed on. At first glance it seems that it is simple, but in reality it took us about 4 hours to set up the initial setting and another 3 hours to adjust . ”
All night and the following day, right up to the beginning of the event, the team was engaged in testing software, debugging modules and fixing software bugs.
Implementing a project of this level for a two-week period was a serious challenge for the development team.
“ This project was made for a specific event and had a clear“ deadline ”, so even a day delay would be a complete failure for all of us. We needed to clearly plan our resources.
We tried first to simplify the task as much as possible. Make a raw minimum, and then add to it "features" and additional functionality. And so in all aspects - from programming to design design.
In order to accurately meet the deadline all the processes had to run in parallel. We tried to make the components as autonomous as possible, so that developers do not depend on each other . ”
“ In fact, such projects are very charged. We worked together very closely for the whole two weeks. It was felt that everyone was sick for the result. Including Nokia: the client went to meet us and promptly made all the decisions.
In general, the project was a success, and we all implemented on time. Of course, there are a lot of aspects that can be improved.
For example, we could not achieve super accurate synchronization: some people were jerking on the video in a jump. In this case, you need to come up with some more reliable solution. We also have to work on the network and improve the speed of receiving photos, making it a separate process in the background and, possibly, moving the system to the 5 GHz standard.
As for the Windows Phone OS and Windows OS installed on the Nokia Lumia 1520, we were generally satisfied with them. Not being familiar with the development of these platforms, we treated them with skepticism. As it turned out, not only are they comfortable and pleasant to work with, Microsoft and Nokia are building a good infrastructure around them with tools and complete documentation . ”
Our cooperation with the organizers has become a good tradition. This year we decided to surprise visitors with an interactive installation - “bullet time” with a photo machine consisting of 60 Nokia Lumia 1520 smartphones. It allowed making impressive live pictures with the effect of frozen time in 360 ° format, which the guests of the festival could later enjoy. We talked about the technical aspects of the creation of this machine with our colleagues who were responsible for the direct implementation of the project.
Process
We came to our colleagues from Hello Computer with a ready-made idea and ... deadlines. Fortunately, they readily went to work:
')
“ The project was allocated only three weeks, and after all the approvals and writing of the TOR, there are generally two. Therefore, we immediately tried to determine the most optimal solutions and platforms that could provide us with a fast development speed.
First of all, we started with the synchronization of phones. It is clear that forcing 60 devices connected via wireless to take a picture at the same time is a rather difficult task. Therefore, our choice fell on the architecture of "client-server ". "
As planned by the developers, the phones (clients) had to contact the server, synchronize time with it, calculating the corrections for the signal and get the time from the server when you need to take a picture.
Team
The team that worked on the project included 7 people: two Windows / Windows Phone developers (one worked as a phone application, the other on a tablet); server developer who worked with the central server and utilities for gluing and sharing; project producer and assistant producer responsible for management; designer; Technical Director, on whose shoulders lay the preparation of TZ, coordination of work, prototyping, network and equipment design.
Architecture
As it should be, the collective thought process began with a heated discussion, although the subject itself was Nokia Lumia 1520:
“ In our studio, only one person uses Windows Phone, so everyone else was interested in testing the devices and at the same time spending half a day on“ holivar ”on the topic:“ Do we need such big phones? ”,“ Is the Windows Phone platform good? ”And so on. d. ".
However, soon the project began to acquire its features from the technical side:
“ Since we decided from the very beginning with the architecture of the project, the main task for us was to remove as much logic as possible to the server. Setting up any kind of settings in 60 phones is a very time-consuming task.
We started with a simple NodeJS server prototype and a skeleton application for a Windows Phone 8.1 phone. Once a second, the server sent requests for smartphones to synchronize the current time and time of the moment of shooting. We also concluded that the difference between the “internal” time of the device and the timing it received from the server. This difference pleased us: it was only within 20-30 milliseconds . ”
At the first design stage, the developers had 5 smartphones, of which they assembled the first prototype. After a quick test, as usual, unforeseen difficulties emerged:
“ The first thing that caught my eye was the difference in color and illumination of images. And for the "glue" we needed to get the same picture on all devices.
Here, Windows Phone pleased us with good and complete documentation (for example, about what settings can be transferred to the phone) and ready-made examples. It was important for us to set the same white balance, shutter speed and ISO on all devices. And thanks to the advanced shooting settings in the Nokia Lumia 1520, which are usually found only in cameras, we were able to easily build a frame on each of the devices. We were also pleasantly surprised and the manual focus adjustment mode was useful.
All these settings were also received from the server . ”
Next, the team had to establish interaction between the devices and the server:
“ Pretty quickly we set up automatic sending of photos back to the server. At this point, we had a ready-made prototype of the application, which could be started to be tested on more phones.
We organized all client-server interaction according to the principles of REST API. In NodeJS (more precisely, in Express) this is done quite quickly and simply. We made the management interface in the form of a simple web page that opens from a local server. Data exchange implemented using weboket. Fortunately, we didn’t have to worry about cross-browser compatibility . ”
Gluing and placing in the network
In parallel with writing an application for phones, the team worked on utilities for pasting photos into one video and then posting them on the network. As was clear from the number of devices involved in the project, the input of the utility should have received a set of 60 photos.
“ We decided not to reinvent the wheel and used ffmpeg for gluing and resizing photos. He pleasantly surprised us by the fact that he could do everything we needed: to glue the photo into a video with a change in the quality of compression and fps ;, change the size of the photo, impose a watermark . ”
An important task of the project was the ability to automatically post the resulting video to the network. For us, the social component of this project was a priority, so the team was required to teach the photo machine to automatically upload videos to the public stream, send links to the mail and enable MSND guests to share videos on social networks.
“ It was decided to take Coub as the video hosting for several reasons. Firstly, our short looped videos with music is the perfect format for Kouba. Secondly, this service has a convenient API for downloading, with which we have already worked before. The documentation, unfortunately, is hidden, but the guys send it on request. Thirdly, it was possible to easily and conveniently set up the feed and beautifully draw it. And, of course, each video could be shared online - this opportunity was provided by Coube itself, which saved development time.
From the previous project, we still have a suitable Python script. It worked quite simply: it scanned the correct folder and downloaded each new video file, after which it created a small text file with the video address and its status. We specifically abandoned the use of databases in favor of script flexibility and portability in any other project. Also, this script was able to send video to the mail . "
We also asked to add the ability to download the video to your device via the OneDrive service:
“ OneDrive has a clean and convenient API. Therefore, we simply added our script with a module for uploading video to the “cloud”. First, he passed the standard OAuth authorization, then uploaded files to our account and received a link to view and download. This link was saved to the same small file . ”
An important aspect of the work of all utilities was the support of offline mode. If the download of files for one reason or another did not succeed, the scripts silently tried to repeat it after a while.
The tablet
The tablet on Windows RT - Lumia 2520 was chosen as a device on which guests could view their 360 ° images. The developers wrote an application that also accessed the server and received a list of all the videos from it. In the application, users could enter the mail, as well as get a QR code with a link to OneDrive or Coub.
Network architecture
When the software was ready, it was time to go to the field trials. To begin with, all 60 smartphones and the server were merged into one network. In order for the team not to get confused during the final assembly and further work with content, the server was assigned 254 addresses, and smartphones from 1 to 60. All devices were strictly numbered, and in the router each MAC address was assigned its own IP.
However, a one-time connection of 60 phones to one Wi-Fi network was not such a simple task:
“ We received all 60 smartphones 5 days before launch, and then the funniest“ tambourine dances ”began in this project.
We started tests on one of the most top routers (hereinafter, manufacturers and models of routers are not mentioned in order to avoid causing any reputational damage, author's note). He calmly withstood the connection of all 60 Lumia, but when he tried to get pictures from the devices, the files started to get lost. We were ready for this and had ready “plates” from another well-known manufacturer in order to distribute the customers evenly across three routers . ”
However, in this case, the team failed: the transfer of the first packet of photos took place without any problems, but the rest were lost in transmission or came with a delay of several minutes.
“ We started to panic - 4 days before launch, and with Wi-Fi everything is bad. We tried to dissolve everything on different non-intersecting channels. It's better, but not much. With that, the “weight” of one photo is only 500 kilobytes, that is, the total size of 60 photos was 30 megabytes + service traffic . ”
Began a long phase of testing. Engineers tried to understand how the situation changes when one or another number of phones is disconnected.
“ Probably, the conclusions reached by us will seem logical for specialists, but we are faced with such problems for the first time. Therefore, their decision cost us a lot of time.
The key mistake for us was that we tried to transfer all the photos at the same time. The signals interfered with each other, so the files were transferred very slowly. We were able to solve the problem by setting a delay of 2 seconds for each next device in the chain . ”
As a result, we came to a scheme in which every 20 devices were separated into a separate router and its own channel. This made it possible to transfer simultaneously 1 file per each router, taking into account a two-second delay, without any special problems. In such an architecture, sending one image took from 0.5 to 2 seconds (sometimes there were delays of up to 10 seconds), but the loss of files was already excluded from the list of problems.
“ Thus, it took us about 45 seconds to collect all the photos from the devices. Probably, this process could have been done faster, but under the conditions of a hard deadline there was no time for further experiments. Yes, and this speed, in principle, suited everyone: the change of guests inside the farm (the internal name of the installation among developers, author's note) and the preparation for the shooting took about the same amount of time ”.
"Farm"
The basis of the “farm” was the classic construction for shooting video in bullet time format: a curved spiral-shaped frame on which devices are mounted for shooting. In the course of the discussion of the project, some decorative improvements were proposed. One of them was, for example, a decorative fire in the center of the installation that was suitable for the subject of the event, through which guests had to jump at the time of the shooting.
Experimenting with the shape of the frame the team stopped at a four-meter radius and vertical position of the phones. This was explained both by the features of the shooting of the smartphones themselves and by the budget: an increase in the radius to 6 or 8 meters would lead to a significant increase in the cost of the project.
“ The main engineering task at this stage was the development of a mount that would allow very precisely adjusting the position of the phones in 2 planes of rotation. As a result, we stopped at a plastic mount with metal clips. The only thing left was to produce and assemble everything in time . ”
X-day: montage
At the last stage, the team had to assemble the disparate components into one single system. They had only two days to install the equipment. The first day was completely given away for installation of the structure and Internet equipment, and the second day was for installation and configuration of smartphones, technical equipment and their testing.
“ The most difficult and time-consuming stage is the installation of smartphones. Filling software on all 60 devices takes about an hour. Now imagine what any bugs we found cost (well, that the client did not agree to 120 phones :)).
During the testing phase, we did encounter several errors with several times; as a result, we had to update the software on all devices again and again. Probably, it would be possible to make a system update by air, but there was no time for it anymore.
A separate story - installation 60 "Lumy". Smartphones had to be mounted very precisely, so that there were no jumps in the final videos. To do this, we made a special setting mode when smartphones displayed the image from the camera on the screen with a label in the center. In the center of the farm, we installed a rack with a bright target, which smartphones placed on. At first glance it seems that it is simple, but in reality it took us about 4 hours to set up the initial setting and another 3 hours to adjust . ”
All night and the following day, right up to the beginning of the event, the team was engaged in testing software, debugging modules and fixing software bugs.
Planning
Implementing a project of this level for a two-week period was a serious challenge for the development team.
“ This project was made for a specific event and had a clear“ deadline ”, so even a day delay would be a complete failure for all of us. We needed to clearly plan our resources.
We tried first to simplify the task as much as possible. Make a raw minimum, and then add to it "features" and additional functionality. And so in all aspects - from programming to design design.
In order to accurately meet the deadline all the processes had to run in parallel. We tried to make the components as autonomous as possible, so that developers do not depend on each other . ”
Impressions
“ In fact, such projects are very charged. We worked together very closely for the whole two weeks. It was felt that everyone was sick for the result. Including Nokia: the client went to meet us and promptly made all the decisions.
In general, the project was a success, and we all implemented on time. Of course, there are a lot of aspects that can be improved.
For example, we could not achieve super accurate synchronization: some people were jerking on the video in a jump. In this case, you need to come up with some more reliable solution. We also have to work on the network and improve the speed of receiving photos, making it a separate process in the background and, possibly, moving the system to the 5 GHz standard.
As for the Windows Phone OS and Windows OS installed on the Nokia Lumia 1520, we were generally satisfied with them. Not being familiar with the development of these platforms, we treated them with skepticism. As it turned out, not only are they comfortable and pleasant to work with, Microsoft and Nokia are building a good infrastructure around them with tools and complete documentation . ”
Source: https://habr.com/ru/post/230373/
All Articles