Storage and automatic sorting of photos and other files. Work with file storage based on Synology NAS
I have long wanted to write about how I store my files, how I am doing backups, but I still could not reach. Recently an article appeared here, somewhat similar to mine, but with a different approach.
The article itself.
For many years I have been trying to find the perfect method for storing files. I think I found it, but there is always something to improve, if you have any thoughts on how to do it better, I read it with pleasure.
To begin with, I’ll tell you a few words about myself, I’m doing web development and taking pictures in my free time. Hence the conclusion that I need to store work and personal projects, photos, videos and other files.
')
I have about 680 GB of files, 90 percent of them are photos and videos.
Here is an approximate diagram of how and where all my files are stored.
Now more.
As you can see, the heart of everything is my NAS, namely Synology DS214, one of the simplest NAS's by Synology, however, all I need is coping.
The working machine is my macbook pro 13, 2015. I have 512GB there, but not all files are placed by itself, I keep only what I need at the moment. I synchronize all my personal files and folders with Dropbox, I know that this is not very reliable, but it performs only the synchronization function. And he does it best, at least from what I have tried. And I tried all the famous and not very clouds.
Synology also has its own cloud, you can deploy it on your NAS, I tried several times to transfer from Dropbox to Synology Cloud Station, but there were always problems with synchronization, always some errors, or not everything was synchronized.
All important files are stored in the Dropbox folder, sometimes I save something to the desktop so that I don’t lose anything to make a symlink to the Dropbox folder using the MacDropAny program.
The Download folder is not synchronized with me at all, but there is nothing important, only temporary files. If I download something important, then I copy it to the appropriate folder in Dropbox.
Files for work are stored on the working server, personal projects on GitLab are all simple here.
I also backup the entire system, excluding the Dropbox and Downloads folder by itself, so as not to take up space in vain. I make a backup of the system through Time Machine, an excellent tool, more than once rescued. I do on the same NAS-e, the benefit there is such a function. You can do on the external HDD of course, but it is not so convenient. You need to connect an external drive each time and start Time Machine yourself. Because of laziness, often such backups were done every few weeks. On the server, he automatically makes a backup, I do not even notice when he does it. I work from home, so I always have a fresh backup of my entire system. A copy is made several times a day, not counted how many times and how often.
This is where all the magic happens.
Synology has a great tool, it is called Cloud Sync, I think the name makes it clear what it does.
It can synchronize many cloud systems with each other, more precisely, then synchronize files from the NAS server with other clouds. Review of the program I think is on the network. I will not go into details. I'd better describe how I use it.
On the server I have a disk folder called Dropbox, it is a copy of my Dropbox account, Cloud Sync handles all this synchronization. If in Dropbox something happened with the files, it will happen on the server, it doesn’t matter, edit / delete / create. In general, the classic sync.
Further, I throw all these files onto my Yandex disk, I use it as a self-made backup disk, that is, I throw files there but do not delete anything from there, it turns out such a dump of files, but it helped me out a couple of times.
I send there only the “Photos” folder, also in the synchronization mode, I only do this for convenient viewing of the photo in Google photo and with the possibility of deleting the photo from there and they are deleted everywhere (except for the Yandex disk by itself). I will write about the photo below, there you can write a separate article.
But all this is not very reliable, if you accidentally deleted a file, it will be deleted everywhere and you can consider it lost. Of course, you can restore from a Yandex disk, but firstly, backup in one place is not very reliable by itself, and the Yandex disk itself is not the same service in which you can be 100% sure, even though it has never had any problems with it.
Therefore, I have always tried to store files somewhere else, already with a normal backup system.
Synology also has a tool for this, it is called HyperBackup, it backs up files either to other Synology servers, or to some kind of third-party solution cloud.
He also knows how to backup to external drives connected to the NAS-y, as I did until recently. But this is also not reliable, for example, if the fire is the end of the server and the HDD.
Here we are smoothly approaching another service, already from Synology itself. She has her own clouds for storing backups. It is sharpened specifically for HyperBackup, it makes a backup there every day, but this is a well thought-out backup, there are file versions, a timeline, and even clients for windows and mac os.
On this, with the storage of files all, I hope my files intact.
We now turn to sorting files.
Regular files, books, scans of documents and other unimportant files, I like everything sorted into folders by hand. Usually there are not many of them and I rarely open them.
The hardest thing with sorting photos and videos, I have a lot of them.
I take a photo from a few dozen to a few hundred a month. I take off on the SLR, on the drone and sometimes on the phone. Photos are personal and for stocks. Also sometimes I shoot home videos (not what you thought, just family videos, more often with a daughter). He, too, must somehow be stored and sorted so that there is no porridge.
I have a folder in the same Dropbox, called Sort Images, there are subfolders where all the photos and videos go, from there they are taken and sorted where necessary.
The sorting takes place on the NAS server, bash scripts work there that automatically run once a day and do their work. NAS is also responsible for their launch, there is a task scheduler that is responsible for running all scripts and other tasks. You can configure how often and when tasks will be launched, cron with an interface if easier.
Each folder has its own script. Now more about the folders:
Drone - here I get photos from the drone that I took for personal purposes. First, I process all the photos in the lightroom, then I export JPG to this folder. from there they go to another Dropbox folder, “Photo”.
There is a folder "Drone" and there they are already sorted by year and month. The scripts themselves create the necessary folders and rename the photos themselves according to my template, usually it is the date and time of taking the photos, I also add a random number at the end so that files with the same name are not obtained. I don’t remember why setting the seconds in the file name was not suitable for this purpose.
The tree looks like this: Photo / Drone / 2019/05 - May / 01 - May - 2019_19.25.53_37.jpg
Drone Video - I haven’t yet been filming a drone video, I need to learn a lot there, there is no time for that now, but I have already created a folder.
Images Operations - there are two folders inside, when files are caught there, they are simply either compressed on the maximum side up to 2000px for publishing on the Internet or flip images are made, I don’t need it, but I haven’t deleted the folder yet.
Panoramas - here it’s not hard to guess the panoramas, I keep them separate as this is a specific type of photo, I usually make them drone. I also make ordinary panoramas, but I also make 360 ​​panoramas and sometimes spheres, such kind of panoramas as small planets, I also make drone. From this folder, all photos also fall into Photo / Panoramas / 2019/01 - May - 2019_19.25.53_37.jpg. Here I do not do the sorting by months as there are not so many panoramas.
Personal Photo - Here I get photos that I make on a DSLR, usually these are family photos or travels, in general photos that are made as a memento for myself. I also process raw photos in lightroom and then export here.
From here they get here: Photo / 2019/05 - May / 01 - May - 2019_19.25.53_37.jpg
If I removed some kind of celebration or something else that it would be better to store separately, then in folder 2019 I create a folder with the name of the celebration and copy the photo there with my hands.
RAW - here are the source of the photo. I always shoot in RAW, I store all photos in JPG, but sometimes I also want to store RAW files, sometimes I want to process a frame differently. This is usually nature and only the best shots get there, not everything in a row.
Stock Photo - here I take off the photo for stocks, which I do either on the SLR or on the drone. Sorting is the same as in other photos, just in its own separate folder.
In the root directory of Dropbox, there is a folder Camera Uploads, this is the default folder in which the mobile app Dropbox takes there all the photos and videos. All photos of the wife from the phone are thrown off in this way. I also drop all my photos and videos from the phone here and from there I sort them into a separate folder. But I do it in a different way, more convenient for me. There is such a program for Android, FolderSync, it allows you to take all the photos from your mobile, upload them to Dropbox and then delete them from your phone. There are many settings, I recommend. The video from the phone is also included in this folder; they are also sorted like all photos, by year and month.
I collected all the scripts myself from any instructions on the Internet, I did not find ready-made solutions. I don’t know anything about bash scripts at all, maybe there are some mistakes or some things could be done better, but the most important thing is for me, they do their job and do what I need.
Scripts uploaded to github: https://github.com/pelinoleg/bash-spts
Previously, I used Hazel for mac os to sort photos, videos, it’s all by itself easier there, all tasks are created visually, you don’t need to write code, but there are two minuses. Firstly, you need to keep all the folders on your computer so that everything works well and secondly, if you suddenly switch to windows or linux, then there are no such programs. I tried to look for an alternative but they were all about nothing. A solution with scripts on the server is a more universal solution.
All scripts are configured to run once a day, usually at night. But if there is no time to wait and you need to somehow execute the necessary script now, there are two solutions, connect via SSH to the server and execute the necessary script or enter the admin area and also manually start the necessary script. All this seems to me not convenient, so I found a third solution. There is a program for Android, it can send ssh commands. I created several commands, each has its own button and now if I need to sort for example the photos that I took from the drone, I just press one button and the script works. The program is called SSHing, there are other similar ones, but for me this is the most convenient.
I also have several of my sites, they are more for show, almost no one goes there, but still it’s not a bad thing to do backup. I keep the sites on DigitalOcean, I installed aaPanel panel there. There is an opportunity to make backup copies of all files and all databases, but on the same disk.
It’s not the case to keep a backup on the same disk, so I also bang the script through the bash and copy everything to my server, archiving everything in one archive with the date in the title.
I hope at least someone will help the ways that I use and what shared.
As you can see from the article, I love automation and I try to automate everything I can, I didn’t describe many things from an automation point of view, as these are other topics and other articles.
The article itself.
For many years I have been trying to find the perfect method for storing files. I think I found it, but there is always something to improve, if you have any thoughts on how to do it better, I read it with pleasure.
To begin with, I’ll tell you a few words about myself, I’m doing web development and taking pictures in my free time. Hence the conclusion that I need to store work and personal projects, photos, videos and other files.
')
I have about 680 GB of files, 90 percent of them are photos and videos.
Circulation of files in my repositories:
Here is an approximate diagram of how and where all my files are stored.
Now more.
As you can see, the heart of everything is my NAS, namely Synology DS214, one of the simplest NAS's by Synology, however, all I need is coping.
Dropbox
The working machine is my macbook pro 13, 2015. I have 512GB there, but not all files are placed by itself, I keep only what I need at the moment. I synchronize all my personal files and folders with Dropbox, I know that this is not very reliable, but it performs only the synchronization function. And he does it best, at least from what I have tried. And I tried all the famous and not very clouds.
Synology also has its own cloud, you can deploy it on your NAS, I tried several times to transfer from Dropbox to Synology Cloud Station, but there were always problems with synchronization, always some errors, or not everything was synchronized.
All important files are stored in the Dropbox folder, sometimes I save something to the desktop so that I don’t lose anything to make a symlink to the Dropbox folder using the MacDropAny program.
The Download folder is not synchronized with me at all, but there is nothing important, only temporary files. If I download something important, then I copy it to the appropriate folder in Dropbox.
My Adventures with Dropbox
Once upon a time, somewhere in 2013-2014, I stored all the files in Dropbox-e and only in it, there were no backups. Then I did not have 1Tb, that is, I did not pay for it, I had about 25Gb, which I earned by inviting friends or other tasks.
One fine morning I turned on the computer and all my files disappeared, I also received a letter from Dropbox where they apologize and that my files were lost due to their fault. They gave me a link by which I could recover my files, but of course nothing recovered. So they gave me 1Tb for a year, after which I became their client, no matter how strange it sounds, but I never trusted them.
As I wrote above, I could not find a more suitable cloud for me, in the first place there were no synchronization problems, secondly, all sorts of services work only with Dropbox.
One fine morning I turned on the computer and all my files disappeared, I also received a letter from Dropbox where they apologize and that my files were lost due to their fault. They gave me a link by which I could recover my files, but of course nothing recovered. So they gave me 1Tb for a year, after which I became their client, no matter how strange it sounds, but I never trusted them.
As I wrote above, I could not find a more suitable cloud for me, in the first place there were no synchronization problems, secondly, all sorts of services work only with Dropbox.
Git
Files for work are stored on the working server, personal projects on GitLab are all simple here.
Time machine
I also backup the entire system, excluding the Dropbox and Downloads folder by itself, so as not to take up space in vain. I make a backup of the system through Time Machine, an excellent tool, more than once rescued. I do on the same NAS-e, the benefit there is such a function. You can do on the external HDD of course, but it is not so convenient. You need to connect an external drive each time and start Time Machine yourself. Because of laziness, often such backups were done every few weeks. On the server, he automatically makes a backup, I do not even notice when he does it. I work from home, so I always have a fresh backup of my entire system. A copy is made several times a day, not counted how many times and how often.
NAS
This is where all the magic happens.
Synology has a great tool, it is called Cloud Sync, I think the name makes it clear what it does.
It can synchronize many cloud systems with each other, more precisely, then synchronize files from the NAS server with other clouds. Review of the program I think is on the network. I will not go into details. I'd better describe how I use it.
On the server I have a disk folder called Dropbox, it is a copy of my Dropbox account, Cloud Sync handles all this synchronization. If in Dropbox something happened with the files, it will happen on the server, it doesn’t matter, edit / delete / create. In general, the classic sync.
Yandex disk
Further, I throw all these files onto my Yandex disk, I use it as a self-made backup disk, that is, I throw files there but do not delete anything from there, it turns out such a dump of files, but it helped me out a couple of times.
Google drive
I send there only the “Photos” folder, also in the synchronization mode, I only do this for convenient viewing of the photo in Google photo and with the possibility of deleting the photo from there and they are deleted everywhere (except for the Yandex disk by itself). I will write about the photo below, there you can write a separate article.
Hyperbackup
But all this is not very reliable, if you accidentally deleted a file, it will be deleted everywhere and you can consider it lost. Of course, you can restore from a Yandex disk, but firstly, backup in one place is not very reliable by itself, and the Yandex disk itself is not the same service in which you can be 100% sure, even though it has never had any problems with it.
Therefore, I have always tried to store files somewhere else, already with a normal backup system.
Synology also has a tool for this, it is called HyperBackup, it backs up files either to other Synology servers, or to some kind of third-party solution cloud.
He also knows how to backup to external drives connected to the NAS-y, as I did until recently. But this is also not reliable, for example, if the fire is the end of the server and the HDD.
Synology C2
Here we are smoothly approaching another service, already from Synology itself. She has her own clouds for storing backups. It is sharpened specifically for HyperBackup, it makes a backup there every day, but this is a well thought-out backup, there are file versions, a timeline, and even clients for windows and mac os.
On this, with the storage of files all, I hope my files intact.
We now turn to sorting files.
Regular files, books, scans of documents and other unimportant files, I like everything sorted into folders by hand. Usually there are not many of them and I rarely open them.
The hardest thing with sorting photos and videos, I have a lot of them.
I take a photo from a few dozen to a few hundred a month. I take off on the SLR, on the drone and sometimes on the phone. Photos are personal and for stocks. Also sometimes I shoot home videos (not what you thought, just family videos, more often with a daughter). He, too, must somehow be stored and sorted so that there is no porridge.
I have a folder in the same Dropbox, called Sort Images, there are subfolders where all the photos and videos go, from there they are taken and sorted where necessary.
The sorting takes place on the NAS server, bash scripts work there that automatically run once a day and do their work. NAS is also responsible for their launch, there is a task scheduler that is responsible for running all scripts and other tasks. You can configure how often and when tasks will be launched, cron with an interface if easier.
Each folder has its own script. Now more about the folders:
Drone - here I get photos from the drone that I took for personal purposes. First, I process all the photos in the lightroom, then I export JPG to this folder. from there they go to another Dropbox folder, “Photo”.
There is a folder "Drone" and there they are already sorted by year and month. The scripts themselves create the necessary folders and rename the photos themselves according to my template, usually it is the date and time of taking the photos, I also add a random number at the end so that files with the same name are not obtained. I don’t remember why setting the seconds in the file name was not suitable for this purpose.
The tree looks like this: Photo / Drone / 2019/05 - May / 01 - May - 2019_19.25.53_37.jpg
Drone Video - I haven’t yet been filming a drone video, I need to learn a lot there, there is no time for that now, but I have already created a folder.
Images Operations - there are two folders inside, when files are caught there, they are simply either compressed on the maximum side up to 2000px for publishing on the Internet or flip images are made, I don’t need it, but I haven’t deleted the folder yet.
Panoramas - here it’s not hard to guess the panoramas, I keep them separate as this is a specific type of photo, I usually make them drone. I also make ordinary panoramas, but I also make 360 ​​panoramas and sometimes spheres, such kind of panoramas as small planets, I also make drone. From this folder, all photos also fall into Photo / Panoramas / 2019/01 - May - 2019_19.25.53_37.jpg. Here I do not do the sorting by months as there are not so many panoramas.
Personal Photo - Here I get photos that I make on a DSLR, usually these are family photos or travels, in general photos that are made as a memento for myself. I also process raw photos in lightroom and then export here.
From here they get here: Photo / 2019/05 - May / 01 - May - 2019_19.25.53_37.jpg
If I removed some kind of celebration or something else that it would be better to store separately, then in folder 2019 I create a folder with the name of the celebration and copy the photo there with my hands.
RAW - here are the source of the photo. I always shoot in RAW, I store all photos in JPG, but sometimes I also want to store RAW files, sometimes I want to process a frame differently. This is usually nature and only the best shots get there, not everything in a row.
Stock Photo - here I take off the photo for stocks, which I do either on the SLR or on the drone. Sorting is the same as in other photos, just in its own separate folder.
In the root directory of Dropbox, there is a folder Camera Uploads, this is the default folder in which the mobile app Dropbox takes there all the photos and videos. All photos of the wife from the phone are thrown off in this way. I also drop all my photos and videos from the phone here and from there I sort them into a separate folder. But I do it in a different way, more convenient for me. There is such a program for Android, FolderSync, it allows you to take all the photos from your mobile, upload them to Dropbox and then delete them from your phone. There are many settings, I recommend. The video from the phone is also included in this folder; they are also sorted like all photos, by year and month.
I collected all the scripts myself from any instructions on the Internet, I did not find ready-made solutions. I don’t know anything about bash scripts at all, maybe there are some mistakes or some things could be done better, but the most important thing is for me, they do their job and do what I need.
Scripts uploaded to github: https://github.com/pelinoleg/bash-spts
Previously, I used Hazel for mac os to sort photos, videos, it’s all by itself easier there, all tasks are created visually, you don’t need to write code, but there are two minuses. Firstly, you need to keep all the folders on your computer so that everything works well and secondly, if you suddenly switch to windows or linux, then there are no such programs. I tried to look for an alternative but they were all about nothing. A solution with scripts on the server is a more universal solution.
All scripts are configured to run once a day, usually at night. But if there is no time to wait and you need to somehow execute the necessary script now, there are two solutions, connect via SSH to the server and execute the necessary script or enter the admin area and also manually start the necessary script. All this seems to me not convenient, so I found a third solution. There is a program for Android, it can send ssh commands. I created several commands, each has its own button and now if I need to sort for example the photos that I took from the drone, I just press one button and the script works. The program is called SSHing, there are other similar ones, but for me this is the most convenient.
I also have several of my sites, they are more for show, almost no one goes there, but still it’s not a bad thing to do backup. I keep the sites on DigitalOcean, I installed aaPanel panel there. There is an opportunity to make backup copies of all files and all databases, but on the same disk.
It’s not the case to keep a backup on the same disk, so I also bang the script through the bash and copy everything to my server, archiving everything in one archive with the date in the title.
I hope at least someone will help the ways that I use and what shared.
As you can see from the article, I love automation and I try to automate everything I can, I didn’t describe many things from an automation point of view, as these are other topics and other articles.
Source: https://habr.com/ru/post/452798/
All Articles