I am far from a professional photographer, but I like to shoot, and from time to time my archive of several tens of thousands of photos of very different quality is replenished with photos from a DSLR, a "soap box" and three phones, as well as video recordings from phones and a video camera. What software I just did not try to maintain the archive of photos and videos in order! Adobe Lightroom, Apple iPhoto, Google Picasa ...

As a result, none of them solved the problem well, and I had to write my own. I thought that my experience could be useful to someone, from which this article was born.

If you are not only a photographer, but also a little programmer, then you can create a similar system in half an hour.
')

Formulation of the problem

So, what problem did I have to solve? How unusual are my expectations from a cataloger?

• I need a mechanism for parsing photos by year, month and date . If necessary, to quickly find an event in time.
• I need to identify duplicates during import and add only new to the archive.
• I need a free software, preferably not requiring installation, but ideally also a multi-platform . The idea is that the software should be on an external disk along with the photo archive, and run from the machine to which this disk is connected. The reason - the requirement for reliability and size of hard drives.
• I need software that would survive all reinstallations, changes to the operating system (even to another).
• I need a convenient mechanism for creating a backup locally , while it is important for me that the backup does not depend on software (that is, it was just a copy of the folder), and could easily be restored in whole or in part (subfolders).
• I need to separately keep RAW-versions of the photo (ideally generally on a separate disk) and share them with JPG-versions (even if they are of high resolution). At the same time, if it is necessary to raise the RAW version of the desired file quickly and easily Well, yes, you need to understand which file has this RAW version, and which one does not.
• Similarly, I need to create a video preview (.AVI) for AVCHD recordings from a Canon camera (.MTS) and keep the FullHD video (.MTS) separate from the preview video (.AVI) , since FullHD takes up a lot of space and is inconveniently fast. browse in search of the desired fragment, but when you need high quality for editing, where did you get it
• I need to upload photos to the network with whole folders (Yandex. Photos, Facebook)
• Have a current "mobile archive" of thumbnails with you on your mobile phone, tablet, laptop .

The most common requirements. Probably in the world there is ready-made software that allows you to do all this, but I could not find it. So I had to write myself.

Solution concept

So what is my approach?

• No databases, proprietary software . All photos are stored in an open file system in an orderly manner.
• Simple scripts are responsible for ordering , which can be created for any operating system and, if necessary, finalized.
• Using different hard drives for different tasks . External HDDs are for heavy files (.MTS, .RAW, maybe JPG originals), and for the laptop itself for lightweight renewable archives (previews). The originals of RAW and VIDEO can be stored on an external hard drive, as they tend to eat up all the disk space, are used extremely rarely, and also because they are important and it is a pity to lose them. JPG originals are stored either on an external hard disk or on a local one if there is confidence in it (Raid array, desktop PC hard disk). Previews (photos and videos separately) are stored on the laptop or on the local hard drive of the desktop PC and are convenient for uploading to the network.
  The file system structure for the above scheme:
  
  
  • Laptop hard disk, flash drive (medium reliability)
    • IMPORT /
      • Files_flashki /
        • IMG_2014.RAW
        • IMG_2014.JPG
        • IMG_2015.RAW
        • IMG_2016.JPG
        • MOV_0001.MOV
    • PREVIEW /
      • 2013 /
        • 05 /
          • sixteen/
            • IMG_2014.JPG
            • IMG_2015_RAW.JPG
        • 06 /
          • ten/
            • IMG_2016.JPG
    • PREVIEW_VIDEO /
      • 2013 /
        • 05 /
          • ten/
            • MOV_0001.MOV
  • Laptop hard drive, PC, external hard drive for lightweight originals (high reliability)
    • ORIG /
      • 2013 /
        • 05 /
          • sixteen/
            • IMG_2014.JPG
            • IMG_2015_RAW.JPG
        • 06 /
          • ten/
            • IMG_2016.JPG
  • External hard drive for heavy originals (high reliability)
    • RAW /
      • 2013 /
        • 05 /
          • sixteen/
            • IMG_2015.RAW
    • VIDEO /
      • 2013 /
        • 05 /
          • ten/
            • MOV_0001.MOV
  
  
  

  IMPORT folder

  Here everything new is copied from cameras and phones. In this heap we can find jpg-photos, RAW-photos, AVI-video, MOV-video in very different resolutions. When processing photos and videos are transferred from this folder to others (see below), and old, already imported photos are deleted from this folder. As a result, there may remain some kind of garbage that is processed manually.
  
  

  ORIG folder

  Folder for storing originals photos in JPG format. To structure photos and videos by year-month-day in this and subsequent folders, the hierarchy is year-month-day. That is, the photos for September 12, 2004 will be stored in the ORIG / 2004/09/12 / branch. My file names are the same as the original ones from IMPORT, but it is correct to make the file names unique, for example, in the form of the timestamp YYYYMMDDHHMMII.JPG, so that there are no duplicates when importing from different devices.
  
  

  RAW folder

  If a RAW photo is encountered, it is checked whether there is a native JPG option for it, and if it doesn’t exist, it is automatically created in ORIG / YYYY / MM / DD /, and the RAW photo itself is transferred to RAW / YYYY / MM / DD /. In this case, the name of the JPG version in ORIG indicates that there is a RAW version for this file: IMG_5905_RAW.JPG. Usually RAW-files are large, so it makes sense to keep this folder on an external disk.
  
  

  VIDEO folder

  The originals of the video files are transferred here. Usually they are quite large, so it makes sense to keep this folder on an external drive.
  
  

  PREVIEW and PREVIEW_VIDEO folders

  The thumbnails are created automatically when imported and put into their PREVIEW folder, keeping the hierarchy, something like PREVIEW / 2004/09/12 / comes out. They are dropped on mobile devices for a mobile photo catalog and uploaded to photo hosting sites.
  
  I drop the archive of thumbnails to mobile devices and periodically update them with regular copying of the latest folders. It is convenient to keep a folder with RAW and VIDEO on an external disk, as they tend to fill up any free disk space, and keep a folder with previews on the laptop - if they are suddenly lost due to some hard disk failure or loss of laptop, they are always can be done again, but on occasion you can quickly find the photo you want or show it to your friends.
  
  This is especially valuable in relation to FullHD-video from a video camera. For a home archive, it is enough to store video of screen quality, and in case of editing it would be good to take the original MOV in Full HD - for this I connect an external hard drive. Moreover, for me personally, absolutely all the laptops slow down to play Full HD MOV from a Canon camera (at least for the first few seconds while loading), which makes active use of original video for domestic purposes difficult. Therefore, it is very convenient when video previews are local, and FullHD versions are on an external disk.
  
  

  And what about the events?

  For event grouping, I use symbolic links to folders by day. That is, the trip to Madeira is stored as a folder EVENTS / Madeira / in which the folders are located 2012/05/01, 2012/05/02, ..., 2012/05/09. But there are several difficulties that this approach does not solve.
  
  The first is due to the fact that sometimes you want to associate with a trip to Madeira, not 3,500 photographs, but a couple of hundred, but selected ones. The second is that the topic is not always clearly beating by the day. Often there may be several topics during the day, and some topics may affect the second half of one day and the first half of the next, while the rest of the two days will relate to another event.
  
  Therefore, in addition to the method described above, there are two more: 1) make symbolic links to files, or 2) physically copy these files to the folder with the event. There is no mass creation of symbolic links in the operating system. Nevertheless, to implement such a method is still real and easy. Physical copying has its drawbacks - identical files begin to be stored in two places. Perhaps this is also a plus - you can kill similar photos, leave only the best, etc. You can make a script that traverses files in both folders and, with full identity, replaces the file with its symbolic link.
  
  My entire archive is stored on a hard disk, which can be connected to three computers - a MacBook, a computer with “Windows” and a computer with Ubuntu. Accordingly, the software should ideally be compatible with all three operating systems.
  
  It was a concept, everything below is my (by no means ideal) implementation.
  
  

  Implementation

  This block is already for those photographers who are not alien to programming and automation. The task is quite simple and with minimal knowledge about bash scripting you can figure it out.
  
  All my software is two scripts. One processes everything from the IMPORT folder and decomposes according to ORIG / YYYY / MM / DD /, VIDEO / YYYY / MM / DD /, RAW / YYYY / MM / DD / along the way creating folders if necessary. The second script scans ORIG, VIDEO, RAW and creates PREVIEW for photos and videos and puts it in the PREVIEW / YYYY / MM / DD folder.
  
  

  Importing and unfolding files in folders

  
  
  For the first script, you need a utility called exiftool. It allows you to pull out the date and time of shooting from the file's EXIF ​​information. It can be found for all three operating systems. Under ubuntu, it is installed from apt-get install packages libimage-exiftool-perl. Under Windows and MacOS, you can download it here: http://www.sno.phy.queensu.ca/~phil/exiftool/
  
  I chose bash scripting and perl as a tool for developing logic. First, because it is the same for MacOS and Ubuntu, and also with Cygwin portable, it will all work without significant edits under Windows. With your permission, I'll focus on MacOS and a little bit on Ubuntu, and on image processing, so as not to be sprayed at all.
  
  Below, I present strongly simplified scripts that perform the tasks.
  
  First, let's analyze the script that adds the specified .JPG file to the library:
  
  

  #./add_file_to_media_library.sh <filename> 

  
  
  As can be seen from the call syntax, it works with one file specified in the parameter. A small limitation for convenience - the path should be relative. That is, either "IMG_5949.JPG" (in the current directory), or MYPHOTOS / IMG_5949.JPG "(in the MYPHOTOS folder, which is set in the current directory).
  
  

  1. export PREFIX = "/ Volumes / Elements /" # is the path to the photo archive. This folder contains ORIG
  2. export FILENAME = " ` pwd` / $ 1 " # Add the full path to the file name.
  3. export OUT = ` exiftool " $ FILENAME " | grep "Create Date | head -n 1« ` ; # get the date the photos were created from EXIF
  4. echo $ OUT | grep "Create Date" ;
  5. # further we create from the date a path of the form ORIG / 2013/05/01 / - where ultimately we place the file
  6. export FOLDER = ` echo $ OUT | perl -i -npe "s /^(.*?): (\ d +) \: (\ d +) \: (\ d +) (. *?) $ / ORIG \ / \ 2 \ / \ 3 \ / \ 4 \ // g “ `
  7. # if there is no such way, create it
  8. echo mkdir -p " $ PREFIX $ FOLDER "
  9. mkdir -p " $ PREFIX $ FOLDER "
  10. echo -n $ FILENAME
  11. # since the path can be relative in the parameter, we pull out only the file name
  12. export NEWFILENAME = ` echo $ FILENAME | perl -i -npe "s / ^ (. *?) \ / ([^ \ /] +?) \. JPG $ / \ 2.JPG / g" ; `
  13. # check if there is already a file in the ORIG / 2013/05/01 / path
  14. if [ -f " $ PREFIX $ FOLDER $ NEWFILENAME " ]
  15. then
  16. # eat, then delete
  17. echo "exist ... $ PREFIX $ FOLDER $ NEWFILENAME "
  18. # here if necessary, we can enable removal from the IMPORT folder
  19. # but this business is dangerous, because the target file can be broken, zero size, etc. Then you can lose the photo. You can transfer the file somewhere, but I did not bother with it
  20. #rm $ FILENAME
  21. else
  22. # if there is no file, then move the file from the IMPORT folder to the ORIG / YYYY / MM / DD / folder
  23. echo mv \ " $ FILENAME \" \ " $ PREFIX $ FOLDER \"
  24. mv " $ FILENAME " " $ PREFIX $ FOLDER "
  25. fi

  
  
  
  
  As a result, all our software requires only exiftool, bash and perl and takes several lines on bash. It remains now to create a script that will call the above fragment, passing it the parameter "file name". There are lots of ways to do this, I use
  
  

   # find . -name «*.JPG» | perl -i -npe "s/^(.*?)[\n\r]+$/\.\/add_file_to_media_library.sh \"\1\"\n/g" ./add_file_to_media_library.sh «.//IMG_5790.JPG» ./add_file_to_media_library.sh «.//IMG_5802.JPG» ./add_file_to_media_library.sh «.//IMG_5794.JPG» 

  
  
  Accordingly, create add2media.sh:
  
  

   #!/bin/bash find . -name «*.JPG» | perl -i -npe "s/^(.*?)[\n\r]+$/\.\/add_file_to_media_library.sh \"\1\"\n/g" > /tmp/run.sh bash /tmp/run.sh rm /tmp/run.sh 

  
  
  (BTW is not a very elegant solution with creating run.sh. you could use both xargs and -exec in find, everything has its drawbacks and advantages)
  
  

  Creating a preview

  For the second script (creating thumbnails), you need a utility that can convert video files and images. For photos, this is the convert utility from the imagemagick package ( http://imagemagick.com ), for videos - ffmpeg or handbrake kits.
  
  Since the creation of thumbnails is related to the conversion of formats, this is not a quick deal, but does not require user input. You can leave for the night, at any time stop and at the next start the work will be restored from the previous point.
  
  

  1. #! / bin / bash
  2. # add to the file passed in the parameter path to the current directory
  3. export FILENAME = " ` pwd` / $ 1 ″
  4. # folder where PREVIEW directory is created. In my case, this is an external drive, you can use a local
  5. export PREFIX = "/ Volumes / Elements /"
  6. # since all the files in ORIG are already laid out on the shelves, we will believe the date from the hierarchy and once again not to contact exiftool
  7. export FOLDER = ` echo $ 1 | perl -i -npe "s / ^ (. *?) \ / (\ d \ d \ d \ d) \ / (\ d \ d) \ / (\ d \ d) \ / (. *?) $ / PREVIEW \ / \ 2 \ / \ 3 \ / \ 4 \ // g “ ` ;
  8. mkdir -p " $ PREFIX $ FOLDER "
  9. # pulling the file name out of the path
  10. export NEWFILENAME = ` echo $ FILENAME | perl -i -npe "s / ^ (. *?) \ / ([^ \ /] +?) \. JPG $ / \ 2.JPG / g" ; `
  11. echo $ {PREFIX} $ {FOLDER} $ {NEWFILENAME}
  12. # was a thumbnail created before?
  13. if [ -f " $ {PREFIX} $ {FOLDER} $ {NEWFILENAME} " ]
  14. then
  15. # yes, it was created. Nothing to do
  16. echo "... skipped (exist) $ {PREFIX} $ {FOLDER} $ {NEWFILENAME} " ;
  17. else
  18. # no, it was not created. Need to convert
  19. echo "... convert" ;
  20. convert " $ infile " -auto-orient -resize 1024 -quality 85 " $ {PREFIX} $ {FOLDER} $ {NEWFILENAME} "
  21. fi

  
  
  
  
  This script also assumes that the file name is passed as a parameter. In order to go through all the files in ORIG, it is used exactly the same as in the example for unfolding the file system scanner, only instead of calling ./add_file_to_media_library.sh there will be ./create_preview.sh
  
  

  Possible development

  Send photos to Facebook and Google Picasa. I also had developments in this regard automating the display from PREVIEW. For Facebook, the FBCMD utility is used , and the Google Command Line Tool is used to create albums and batch fill images on Picasa.
  
  In my opinion, various “features” of existing catalogolizers, like adding description of photos, tagging, faces, and so on. poorly suited for situations where one event generates a thousand shots and only one choice of them important-needed turns into serious work. But the use and modification of EXIF-information looks very interesting direction. For example, you can specify that from May 1 to 10, I was on Madeira, and all the images in ORIG in the corresponding EXIF ​​field acquire geographical coordinates. Upon subsequent import into online photo hosting sites, this information can be read and the photos are automatically attached to the site.
  
  I would also like to see the entire photo and video archive to be dropped at night somewhere in a protected cloud a la backup. It is also quite easy to automate in the approach described above, but has not yet reached out. And what catalogolizers do you use? are there anything in them that is not taken into account in my approach?
  
  Aliyev Rauf | http://RaufAliev.ru | Facebook | LinkedIn | r.aliev@gmail.com