Once again about the electronic library for PocketBook

Greetings to you, Habr!

This article is just a detailed commentary on the text “Electronic library for PocketBook: automatic processing” by dsd_corp , since I, being ethereal (readonly) in spirit, cannot (for now) leave the usual comments. However, I will not discuss the policy of Habr, but let me get down to business.

First, I would like to thank the author for his article. I learned a lot of useful information about the PocketBook 360 ° I have used for a long time and put a trash bin on it in relative order. The script, as promised by the author, turned out to be portable (I have Linux), except for a slight inconvenience with encodings.
')
Below I just give my comments.

1. The directory where the actual books are stored is best named so that it starts with a dot. The device itself follows the ancient yuniksovoy tradition of "files from the point - for sharpening" and does not show them. That is, the script should change $storagename to, for example, '.zipstorage' .
2. I found that the names of the files created are written in cp1251 encoding (have they really not been converted to Unicode in Windows yet?). It seemed to me very inconvenient, so I tried to edit the script to make a kosher utf8. Initially naively made
   
   sed -i 's/windows-1251/utf8/g' *php *.inc
   
   etc., but it turned out bad, because the script does some pretty non-trivial things, in particular, it understands what the original encoding of the input fb2-files and correctly recodes. In addition, I see text in PHP almost for the first time in my life. Therefore, I returned everything as it was and just did
   
   convmv -f cp1251 -t utf8 -r --notest out_dir/dest/
   
   every time after the execution of the script.
3. The fact that references by authors are laid out too deeply (the letters A-Z / first letter / first three letters / author) seemed uncomfortable to me. At least, when there are not very many books, it is not justified. I figured out how to do better, namely: to balance the tree, limiting the number of elements in the upper levels, so that they fit on one page (for example, it is 10 at my settings). Then the path will look like 'Ave-Arts / Aksakov', that is, it will turn out approximately as in the naming of the volumes of the encyclopedia. Depending on the number of books, additional levels may be needed, of which there should be about a logarithm of the number of authors. It seems that such a thing is scientifically called a prefix tree. I have not yet implemented this feature, but simply shortened the initial layout to 'first letter / author'.
4. Sometimes (probably, on broken fb2-files) the script can behave inadequately - it clogs up all the memory and starts swapping. I did not debug; it is probably easier for me to rewrite the functionality in a more familiar language than to deal with PHP. True, I'm not sure that I will get at least to this, especially since I have overtaken almost all of my library, and so far everything is satisfactory.
5. Experimenting with how link files work, I didn’t immediately understand how file systems are named from the device point of view. Namely, an external SD card is mounted in / mnt / ext2, and an internal memory is mounted in / mnt / ext1. (When I experimented, I wrote directly to the internal memory, and the links with '/ mnt / ext2' did not work for me.) This naming is rather counterintuitive - ext2 causes a definite association with the name of the file system (however, vfat is still on the card). Therefore, when applying the script, it is necessary to remember that it prepares files for the first section of the external card. Otherwise, in the end you need to do something like this:
   
   find out_dir/dest/ -name '*.flk' -exec sed -i 's/ext2/ext1/' '{}' \;
   
   I also tried to use relative, rather than absolute flk-links, but they seem to not work at all. If anyone knows anything about relative links, let them share.
   
6. About copying efficiency. I did not experience any special problems (only one and a half gig), but there are some considerations for optimization. First, the idea of ​​ziping files seems dubious - we have a device with a very slow processor and relatively fast storage media. Therefore, the time of opening files may deteriorate due to compression. However, tests are needed here. Secondly, to speed up pouring onto a card, you can prepare an image of a partition and then copy it onto the card entirely with the dd utility. How to make an image of a partition on a file I even almost remember - for this you need a losetup utility that turns a file into a device that you can use to mkfs and mount.

UPD

How to copy an image

As a result of the work of the cataloger, a cloud of small files appears, which are then copied for a long time to the file system of the flash drive. You can try to speed up this process by creating a disk (or even on tmpfs, if you like) copies of the image of a flash drive partition, and then copy the entire image. The benefit from this is not obvious, because as a result you will have to copy bytes than if you copy by file. In addition, unnecessary depreciation of the flash drive may increase. Nevertheless, I present a recipe suitable for any (not too ancient) Linux. (Commands beginning with a sharp require root's rights, the rest can be made an ordinary user.)

1. First, you need to know the exact size of the desired section of the flash drive. Suppose the device is defined as / dev / sdc:
   
   # fdisk -l /dev/sdc
...
Disk /dev/sdc: 4 GB, 4127195136 bytes
64 heads, 32 sectors/track, 3936 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 1 3934 4028400 83 Linux
   
   Here we have a partition size of 4028400 blocks of 512 bytes.
   
2. Next, you need to create a file on disk exactly the same size. Since the dd utility has the bs (block size) option, we do not even need to multiply 4028400 by 512. However, in this case, the block size does not matter for dd, the main thing is that bs * count is exactly the required size in bytes.
   
   $ dd if=/dev/zero of=image bs=512 count=4028400
   
3. Create a file system on the resulting file:
   $ /sbin/mkdosfs image
   
4. Mount the image file as a device (this requires the nuclear loop module, which is in the cores of all modern distributions by default), copy the files, unmount:
   $ mkdir mnt
# mount -o loop image mnt
# cp -r ...... mnt
# umount mnt
$ rmdir mnt
   
5. Now copy the image file to the partition location (it’s important not to make a mistake with the device name):
   # dd if=image of=/dev/sdc1
   

Done! You can mount the USB flash drive and make sure that its contents are as expected.

It happens that the device has broken blocks, then everything becomes more complicated. In this case, you can copy the image of the section to a file, instead of creating it again (steps 1-3):

# dd if=/dev/sdc1 of=image conv=noerror

The idea is that the file system stores meta information about its bad blocks, so we can copy the data to the image file, without fear that they will end up on the BB. However, I did not check it and I can be mistaken.

Aerobatics is to keep the current image of the flash drive on a home computer and have a utility that economically copies the difference between the new image and the old one to the USB flash drive.