How to use frame-by-frame photography with a GoPro camera for the Mapillary service
Over the past six months, the popularity of the Mapillary web service has grown: over half a million photos have been uploaded, and more than 100 thousand pieces - in the last 10 days only!
At first glance it seems that only an ordinary smartphone is suitable for this business. But what if you could use an action camera, such as a GoPro, to collect photos for Mapillary? Mounted on a bike, helmet or even a car? As it turned out, this is possible.
And last week I took a small trip and collected about 1,700 photos in half an hour, using the TimeLapse function. On my 32GB SD card, it ate just 3.4 gigabytes. I could still ride for more than 4 hours and take about 150 thousand photos before my memory card is full!
However, this is not a trivial matter, so I think you need to explain in more detail how all this is done. The main problem is that GoPro does not contain a GPS receiver. How does the map service know where to display photos on the map? We ourselves must give him this information, so we will have to use a separate GPS receiver. Personally, I used my HTC One smartphone - the same one I used before to take regular photos for Mapillary.
')
In short, the procedure is as follows:
It seems to be not too difficult. Let us write out these actions in more detail on the example of my trip last week.
I secured the GoPro to the front left wing of my Mazda RX8 to get a good view of the middle of the road.
Then I fixed my HTC One smartphone in the car for a special fastener. Using the program " GoPro App ", I joined the camera and previewed a picture of the terrain, as GoPro sees it. This is convenient, but, in fact, not very necessary, because when the recording starts, the preview disappears. It is only necessary that the smartphone was in the car in the place where the signal of GPS-satellites is well caught. If you use the GoPro App, take a little time to adjust the time on the camera: the difference between the current time on devices should be as little as possible. As you will see later, you will have to take into account the time difference in order to synchronize the track and the photo. And the less correction you have to make, the easier it will be to do the work.
Let's check the camera settings again: the time-lapse shooting mode should be enabled. This is easily done using the GoPro App, but you can do it on GoPro itself. I chose 5MB, Medium, Narrow Field of View (Narrow) to get a look similar to the one I shoot using the phone. Equivalent focal length (in terms of 35-mm camera) will be approximately 20 mm. GoPro settings by default provide a wide field of view, with large optical distortions, so on this trip I gave up this mode. I plan to describe another scenario in another article when I took photos for Mapillary from a wide-angle 4K video stream. It was harder, so for now let's skip the story.
With the selected setting of 1 frame per second, the distance between photos will be 10–20 meters at a speed of 40–80 km / h. A Mapillary recommendation about one frame every two seconds is more suitable for a bike, and I plan, of course, to move faster than by bike!
Launch the GPS tracker application. In this case, I used the program geopaparazzi . In this application there is a button to start recording. I pressed this button and confirmed the suggested file name for saving the track. OK. Now everything is ready. It remains only to start shooting, and forward!
Rolling over, stop recording on the camera and on the GPS receiver. Now the real work begins! We need to geo-reference before we can load our material into Mapillary. In Geopaparazzi, I exported the track to a GPX file and sent it to myself by mail. From GoPro, I just pulled out a card and copied photos from it onto a laptop.
The first thing I wanted to do was to see what my trip looked like. I executed the command:
The result was a good high resolution image of the entire map. Note the use of the -D option to specify a larger gap between the markers on the map. This is necessary because geotag is configured by default for short tracks, like, for example, a fast bike ride. On the resulting image you can see the time and position of the key points of the trip. We need to enlarge the scale in several characteristic places, where we can manually check the time match on the camera and the GPS receiver and more precisely determine the accuracy of the clock between them. At the synchronization stage we will be able to correct this difference, and it is important to determine it correctly.
I executed the command:
I can find a photo where I drive under the bridge, and check the clocks.
The EXIF ​​data of this photo shows that it was taken at 12:39:11. Looking at the map, we see that we passed under the tunnel at 12:39:14. Thus, the error is 3 seconds. You can use this value in the process of geo-referencing, but it is better to first check with another photo.
I generated a map of the path through Morarp, because I can identify objects such as buildings and intersections. Do not use the crossroads where you stopped (like, for example, my turn to the right). Look for such landmarks, near which you were in motion. I searched for the first passage on the right, at the top of the map, and found a photo taken at 12:46:10.
I took note of several landmarks, about which I know exactly where they are: a warning sign in the middle, white marking at the intersection, a pillar of street lighting.
Another interesting way to check, which you can also use - if you were in a place where there are pictures of Google street view, you can compare two images.
Google’s street view pictures show a road sign, a markup at a crossroads, a tree and a lighting pole. But note that there is no fence, trees are growing instead. It is clear that the construction began after Google drove through this place. Google says the picture was taken in September 2011 - about 2.5 years ago - of course, something has changed.
From the data of the GPX track through Merarp one can see that the intersection passed at 12:46:13, which is 3 seconds later than it was recorded by the camera in the photo. And again we have an error of 3 seconds. This is good: it turns out that the error is the same throughout the entire track. You can continue and sync all 1500 photos using the command:
I set the time shift using the "-t -3" option and used the "-v" option to monitor the process. Since the script is a wrapper for the command line program “exif_file”, several processes start processing each file; this takes some time, but at the end all your photos will contain georeference information taken from GPX.
When the binding of geo-information to photos is completed, you can upload photos to mapillary.com. Enter the site, click on your name, select “Upload images”, click on the “Choose files” button. After selecting all the files, scroll down the page and click the “Start Uploading” button. The color scheme on the site is such that it is not always clear whether the download has started. Just scroll up the page and you will see red loading indicators under each image.
Finally, when the download is complete, click on your name, select “my uploads” ', and you will see new images for your track.
Click on your latest download to view photos in Mapillary!
Full processing of photos may take some time, so do not worry if they are not immediately available. Get back to them later.
The settings in GoPro for the described method of photographing are not for nothing called “Time Lapse” - “Slow Motion”. From the photos you can make a video. Because we shot one frame per second - if we make a video at 25 frames per second, we will get 25-fold acceleration. This is great! See what I did:
This video was made like this:
The geotag command was widely used in this article. This command is included in the ruby ​​gem "gpx2exif". The article used the features available in version 0.3.6. However, at the time of publication of the article version 0.3.1 was available. Therefore, I will explain how to install the latest version.
The first thing you need is Ruby. Installation depends on the operating system. I use Ubuntu 14.04 and RVM, here I will give instructions appropriate for my OS. And I recommend that you go to the sites ruby-lang.org and rvm.io for advice that is appropriate for your platform.
The part of the program that creates images in PNG format uses ImageMagick for its work. For Ubuntu, this means that you first need to install several dependencies:
When ruby ​​is installed, simply install the gem:
Then output the list with the command “gem list” to find out which version was installed as a result. If it is older than 0.3.5, then follow the instructions below.
Install git and then execute the commands:
If everything went well, then you have compiled and installed the latest version of Ruby Gem.
At first glance it seems that only an ordinary smartphone is suitable for this business. But what if you could use an action camera, such as a GoPro, to collect photos for Mapillary? Mounted on a bike, helmet or even a car? As it turned out, this is possible.
And last week I took a small trip and collected about 1,700 photos in half an hour, using the TimeLapse function. On my 32GB SD card, it ate just 3.4 gigabytes. I could still ride for more than 4 hours and take about 150 thousand photos before my memory card is full!
However, this is not a trivial matter, so I think you need to explain in more detail how all this is done. The main problem is that GoPro does not contain a GPS receiver. How does the map service know where to display photos on the map? We ourselves must give him this information, so we will have to use a separate GPS receiver. Personally, I used my HTC One smartphone - the same one I used before to take regular photos for Mapillary.
')
In short, the procedure is as follows:
- Mount the camera in a bicycle or car with a variety of fasteners.
- Run a program to record a GPS track. I used the program geopaparazzi, but there are many others that work just as well.
- Set up GoPro for single-frame shooting. I chose the settings: 5MP, medium, narrow, 1s.
- Launch GoPro and ride.
- Upload photos and GPX track to computer.
- Using the gpx2exif program, link photos to geographic coordinates by synchronizing the shooting time with the GPX track.
- Upload photos to Mapillary.
It seems to be not too difficult. Let us write out these actions in more detail on the example of my trip last week.
I secured the GoPro to the front left wing of my Mazda RX8 to get a good view of the middle of the road.
Then I fixed my HTC One smartphone in the car for a special fastener. Using the program " GoPro App ", I joined the camera and previewed a picture of the terrain, as GoPro sees it. This is convenient, but, in fact, not very necessary, because when the recording starts, the preview disappears. It is only necessary that the smartphone was in the car in the place where the signal of GPS-satellites is well caught. If you use the GoPro App, take a little time to adjust the time on the camera: the difference between the current time on devices should be as little as possible. As you will see later, you will have to take into account the time difference in order to synchronize the track and the photo. And the less correction you have to make, the easier it will be to do the work.
Let's check the camera settings again: the time-lapse shooting mode should be enabled. This is easily done using the GoPro App, but you can do it on GoPro itself. I chose 5MB, Medium, Narrow Field of View (Narrow) to get a look similar to the one I shoot using the phone. Equivalent focal length (in terms of 35-mm camera) will be approximately 20 mm. GoPro settings by default provide a wide field of view, with large optical distortions, so on this trip I gave up this mode. I plan to describe another scenario in another article when I took photos for Mapillary from a wide-angle 4K video stream. It was harder, so for now let's skip the story.
With the selected setting of 1 frame per second, the distance between photos will be 10–20 meters at a speed of 40–80 km / h. A Mapillary recommendation about one frame every two seconds is more suitable for a bike, and I plan, of course, to move faster than by bike!
Launch the GPS tracker application. In this case, I used the program geopaparazzi . In this application there is a button to start recording. I pressed this button and confirmed the suggested file name for saving the track. OK. Now everything is ready. It remains only to start shooting, and forward!
Rolling over, stop recording on the camera and on the GPS receiver. Now the real work begins! We need to geo-reference before we can load our material into Mapillary. In Geopaparazzi, I exported the track to a GPX file and sent it to myself by mail. From GoPro, I just pulled out a card and copied photos from it onto a laptop.
The first thing I wanted to do was to see what my trip looked like. I executed the command:
geotag -g väla_to_billesholm.gpx \ -o väla_to_billesholm.png \ -D 1 -s 2048x2048 The result was a good high resolution image of the entire map. Note the use of the -D option to specify a larger gap between the markers on the map. This is necessary because geotag is configured by default for short tracks, like, for example, a fast bike ride. On the resulting image you can see the time and position of the key points of the trip. We need to enlarge the scale in several characteristic places, where we can manually check the time match on the camera and the GPS receiver and more precisely determine the accuracy of the clock between them. At the synchronization stage we will be able to correct this difference, and it is important to determine it correctly.
I executed the command:
geotag -R 20140505T12:39:00+02-20140505T12:41:00+02 \ -g väla_to_billesholm.gpx \ -o krop.png -D 0.1 which generated this image:I can find a photo where I drive under the bridge, and check the clocks.
The EXIF ​​data of this photo shows that it was taken at 12:39:11. Looking at the map, we see that we passed under the tunnel at 12:39:14. Thus, the error is 3 seconds. You can use this value in the process of geo-referencing, but it is better to first check with another photo.
I generated a map of the path through Morarp, because I can identify objects such as buildings and intersections. Do not use the crossroads where you stopped (like, for example, my turn to the right). Look for such landmarks, near which you were in motion. I searched for the first passage on the right, at the top of the map, and found a photo taken at 12:46:10.
I took note of several landmarks, about which I know exactly where they are: a warning sign in the middle, white marking at the intersection, a pillar of street lighting.
Another interesting way to check, which you can also use - if you were in a place where there are pictures of Google street view, you can compare two images.
Google’s street view pictures show a road sign, a markup at a crossroads, a tree and a lighting pole. But note that there is no fence, trees are growing instead. It is clear that the construction began after Google drove through this place. Google says the picture was taken in September 2011 - about 2.5 years ago - of course, something has changed.
From the data of the GPX track through Merarp one can see that the intersection passed at 12:46:13, which is 3 seconds later than it was recorded by the camera in the photo. And again we have an error of 3 seconds. This is good: it turns out that the error is the same throughout the entire track. You can continue and sync all 1500 photos using the command:
geotag -g väla_to_billesholm.gpx \ 20140505_TimeLapse/*JPG -t 3 -v I set the time shift using the "-t -3" option and used the "-v" option to monitor the process. Since the script is a wrapper for the command line program “exif_file”, several processes start processing each file; this takes some time, but at the end all your photos will contain georeference information taken from GPX.
When the binding of geo-information to photos is completed, you can upload photos to mapillary.com. Enter the site, click on your name, select “Upload images”, click on the “Choose files” button. After selecting all the files, scroll down the page and click the “Start Uploading” button. The color scheme on the site is such that it is not always clear whether the download has started. Just scroll up the page and you will see red loading indicators under each image.
Finally, when the download is complete, click on your name, select “my uploads” ', and you will see new images for your track.
Click on your latest download to view photos in Mapillary!
Full processing of photos may take some time, so do not worry if they are not immediately available. Get back to them later.
And now - slow motion video
The settings in GoPro for the described method of photographing are not for nothing called “Time Lapse” - “Slow Motion”. From the photos you can make a video. Because we shot one frame per second - if we make a video at 25 frames per second, we will get 25-fold acceleration. This is great! See what I did:
This video was made like this:
- Rename all the photos so that the file names have numbers starting with 0000. I asked names like foo-0000.jpeg. To make my life easier, I wrote a ruby ​​script that creates hard links with the necessary names. You can then use the ffmpeg command to compile the video:
ffmpeg -f image2 -i foo-%04d.jpeg \ -r 25 -s 1280x960 ../myvid.mp4 - This command compresses 4-megapixel 4: 3 photos into a 960p HD movie.
- Then, using OpenShot video editor, I trimmed to 16: 9, added sound, added a map, and pinned to 720p resolution with medium quality for faster uploading to the web.
Install gpx2exif
The geotag command was widely used in this article. This command is included in the ruby ​​gem "gpx2exif". The article used the features available in version 0.3.6. However, at the time of publication of the article version 0.3.1 was available. Therefore, I will explain how to install the latest version.
Installing Ruby on Ubuntu
The first thing you need is Ruby. Installation depends on the operating system. I use Ubuntu 14.04 and RVM, here I will give instructions appropriate for my OS. And I recommend that you go to the sites ruby-lang.org and rvm.io for advice that is appropriate for your platform.
sudo apt-get install curl curl -sSL https://get.rvm.io | sudo bash -s stable --ruby sudo usermod -g rvm craig # logout and login to get RVM group source /etc/profile.d/rvm.sh The part of the program that creates images in PNG format uses ImageMagick for its work. For Ubuntu, this means that you first need to install several dependencies:
sudo apt-get install imagemagick imagemagick-doc libmagickwand-dev gem install rmagick # Requires libmagickwand-dev to compile Installing from rubygems.org
When ruby ​​is installed, simply install the gem:
gem install gpx2exif Then output the list with the command “gem list” to find out which version was installed as a result. If it is older than 0.3.5, then follow the instructions below.
Installing from github
Install git and then execute the commands:
git clone https://github.com/craigtaverner/gpx2exif cd gpx2exif bundle install rake build gem install pkg/gpx2exif-0.3.6.gem If everything went well, then you have compiled and installed the latest version of Ruby Gem.
Notes from the translator :
1) What is “Mapillary”? As Zverik put it: “Mapillary for images is like OpenStreetMap for maps.” In short, this is an analogue of Google street view and Yandex-panorama, which works on the principle of crowdsourcing. Users upload their geolocated photos, which are then displayed on the map along with a line of distance traveled. A content license permits the use of information obtained from photographs in OpenStreetMap.
2) For geo-referencing of photos there is also such a script on Python: https://github.com/mapillary/mapillary_tools/blob/master/python/geotag_from_gpx.py
3) About everything, mentioned in paragraphs 1 and 2, I learned from the article " Folk panoramas " in the blog of the user Zverik .
4) The most convenient way to fix the discrepancy in the clock is to take a picture of the clock on the GPS device with the camera.
Source: https://habr.com/ru/post/242463/
All Articles