Photo and video files in the home collection - processing and storage

Foreword

I think every amateur photographer like me has come up with the choice of the optimal way to process and store your photos. In this article I want to touch on this topic and offer my own version. I really want to hear other uses.
It so happened that it was comfortable for me to be and perform tasks in Linux-systems, therefore, all the Linux scripts. But the axis is not the main thing, the main thing is the correct approach.

I take pictures on an amateur SLR in RAW, then I translate everything into JPG, which are stored on the NAS along with the originals. Everything is sorted by date, individual events are tagged for easy future search. Also rented on videkameru in 1080p.
The SD card from the camera / camcorder is inserted into the nettop on Linux, working around the clock, which is the main actor. Further, through udev, the device name is assigned so that you can call the card in your own way and execute a script prepared in advance (for more details on creating udev rules, see here ). My version of the rules is as follows:
$ cat /etc/udev/rules.d/81-local.rules

KERNEL=="sdb1", SUBSYSTEM=="block", ATTR{size}=="63395840", NAME="SDcard_camcorder", RUN+="/usr/bin/zprocess_sd.sh" KERNEL=="sdb1", SUBSYSTEM=="block", ATTR{size}=="7919616", NAME="SDcard_camera", RUN+="/usr/bin/zprocess_sd.sh" 

')
I assign different names (NAME) to cards, depending on their size (ATTR {size}), after which the script (RUN) is launched. Device parameters can be found using udevinfo or udevadm (depending on the distribution):
udevadm info -a -p `udevadm info -q path -n /dev/sdb`
(instead of / dev / sdb put your device)
When drawing up rules, it is important to unambiguously identify the device, otherwise the command specified after RUN + = will be launched more than once, which will introduce confusion.
Udev runs a simple /usr/bin/zprocess.sh script that transfers information from the SD card and sorts:

 #!/bin/bash { sleep 10 su igor -c "/home/igor/scripts/move_from_sd.pl"; umount /dev/SDcard_cam*; su igor -c "/home/igor/scripts/sort.pl"; } & 

move_from_sd.pl moves the contents of the card to the to_sort folder on the NAS, writes to the log, sings five times at the end of the work - you can pull out the card:

 #!/usr/bin/perl -w use strict; use File::Copy; use File::Find; my $today = `date +%F_%T`; my $logfile = "/mnt/tank/media/move_from_sd_$today.log"; my @mount_string = split /\s+/, `mount|grep -i sdcard`; my $to_sort_folder = "/mnt/tank/media/to_sort/"; open LOGFILE, ">>$logfile" or die "Cannot create logfile: $!"; select LOGFILE; if (defined $mount_string[2]) { print "SD card is mounted on $mount_string[2]\n"; find(\&movefiles, $mount_string[2]); } else { print "SD card is not mounted!\n"; } sub movefiles { if (/(\.CR2$|\.MTS$)/) { print "Old file: $_\n"; my $newfile = $to_sort_folder . $_; print "New file: $newfile\nMoving file...\n"; if (-e $newfile) { $newfile = "$newfile"."_1"; move ($_, $newfile); print "Done! Had to rename to $newfile\n"; } elsif (move ($_, $newfile)) { print "Done!\n"; } else { print "moving $_ to $newfile failed: $!\n"; } } } `beep -D 300 -l 700 -r 5`; 

With this script, I also drop AVCHD files from the camcorder. Put your file types instead of CR2 and MTS. I'm not a programmer, so do not rashly for the code.

sort.pl sorts files into folders by date, month and year using exiftool:

 #!/usr/bin/perl -w use strict; my $today = `date +%F_%T`; my $logfile = "/mnt/tank/media/sort_$today.log"; my $raw_video_dir = "/mnt/tank/media/AVCHD"; my $video_dir = "/mnt/tank/media/Videos"; my $raw_photo_dir = "/mnt/tank/media/RAW"; my $photo_dir = "/mnt/tank/media/Pictures"; my $to_sort_dir = "/mnt/tank/media/to_sort"; open LOGFILE, ">>$logfile" or die "Cannot create logfile: $!"; select LOGFILE; foreach (<$to_sort_dir/*.MTS>) { if (! /\d+_\d+\.MTS/) { print "Renaming file $_..\n"; `exiftool '-FileName<DateTimeOriginal' -d %Y%m%d_%H%M%S%%-c.%%e $_`; } } foreach (<$to_sort_dir/*.MTS>) { chdir($raw_video_dir); print "Copying video file $_ into AVCHD ($raw_video_dir) folder..\n"; `exiftool -o . '-Directory<DateTimeOriginal' -d %Y/%m/%d $_`; chdir($video_dir); print "Moving video file $_ into Videos ($video_dir) folder..\n"; `exiftool '-Directory<DateTimeOriginal' -d %Y/%m/ $_`; } foreach (<$to_sort_dir/*.CR2>) { chdir($raw_photo_dir); print "Copying photo file $_ into RAW ($raw_photo_dir) folder..\n"; `exiftool -o . '-Directory<DateTimeOriginal' -d %Y/%m/%d $_`; chdir($photo_dir); print "Moving photo file $_ into Pictures ($photo_dir) folder..\n"; `exiftool '-Directory<DateTimeOriginal' -d %Y/%m/ $_`; } 

First, I rename the files from the camcorder to YYYYMMDD_HHMMSS. All originals from to_sort - RAW and AVCHD files - I copy in accordance with the date of the DateTimeOriginal snapshot per year / month / day, the RAW and AVCHD folders, respectively. Then I move everything to the Pictures and Video folders in the same way, but in a year / month. It turns out about the distribution of files in folders:
-> RAW -> 2011 -> 11 -> 12
IMG1234.CR2
IMG1236.CR2
-> AVCHD -> 2011 -> 11 -> 10
20111110_112323.MTS
-> Pictures -> 2011 -> 11
IMG1234.CR2
IMG1236.CR2
-> Videos -> 2011--11
20111110_112323.MTS
The conversion will be done inside the Pictures and Videos folders, but I will not touch the originals.

Conversion

The photo is converted in the Pictures folder. I use RAWTherapee for RAW processing.
In the program settings there is “Save Processing Parameters next to file”, therefore after changing the RAW file the same file with pp3 extension appears, which is a conversion profile. Conversion to JPG occurs 3 times a day across the krone by the following script, the RAW file itself is deleted upon completion:

 #/usr/bin/perl use strict; use warnings; use File::Find; my $today = `date +%F_%T`; my $logfile = "/mnt/tank/media/raw_conversion_$today.log"; my $photo_dir = "/mnt/tank/media/Pictures/"; my $rt_path = "/home/igor/RawTherapee/rawtherapee"; open LOGFILE, ">>$logfile" or die "Cannot create logfile: $!"; select LOGFILE; find(\&convert, $photo_dir); sub convert { if (/\.CR2$/) { print "----"x7 . "\n"; print "Converting $_ into jpeg..\n"; print (my $result = `$rt_path/rawtherapee -o . -S -j67 -c $_ 2>&1`); (my $picture = $_) =~ s/CR2/jpg/; my $pp3 = $_ . ".pp3"; if (-e "$picture") { unlink $_; unlink $pp3; } } } 

The conversion itself is the rawtherapee -o command . -S -j67 -c
converting to JPG with compression 67 in the same directory (-o.), in batch mode (-c). The -S option means that rawtherapee will skip the RAW file if there is no corresponding pp3 file (-S) (i.e., only viewed / modified RAW files). You can set the -s flag and specify through -p the name of the profile that will be used for conversion.

Video is converted in the Videos folder - all existing MTS files are converted to MKV using HandBrakeCLI . Runs twice a day on the krone.

 #!/usr/bin/perl -w use strict; use File::Find; my $today = `date +%F_%T`; my $logfile = "/mnt/tank/media/mts_conversion_$today.log"; my $video_dir = "/mnt/tank/media/Videos"; open LOGFILE, ">>$logfile" or die "Cannot create logfile: $!"; select LOGFILE; find(\&convert, $video_dir); sub convert { if (/MTS$/) { (my $outfile = $_) =~ s/MTS/mkv/; print "Converting $_ into $outfile ...\n"; `HandBrakeCLI -e x264 -q 20 -f mkv -i $_ -o $outfile`; unlink $_; } } 

As a result, in the Pictures and Videos folders are JPG and MKV.
Further viewing and rating, tagging and uploading to Picasa / Flickr is done in Shotwell .

Conclusion

In this way, I unloaded my working laptop, now all the conversion is done automatically on another host in the home network. All my photos and videos are stored on NAS on RAID-6, so I don’t worry if anything happens to my laptop.
As they say, Tim Towdy, so I will be glad to hear how you process and store your photo and video files, especially using the GNU / Linux OS.