Blocks. The internal structure of the Caché database file. Part 3

As in the two previous parts ( part 1 , part 2 ), I continue to acquaint you with the internal structure of the Caché databases. This time I will talk about what else you can find out and what my project Caché Blocks Explorer can help with.



I think many people thought that which is shown in the picture (clickable). When I wanted to portray the fragmentation of globals, for me the prototype was various numerous disk defragmentation utilities. And, I hope, I managed to make an equally useful product.

This utility displays a block map. Each square represents a block, and its color corresponds to a specific global, which is specified in the legend. The block itself also shows how much it is filled with data, which allows you to create a visual representation of the filling of the database file. The display of the global level blocks and the block map have not yet been implemented - they, like all empty blocks, will be displayed in white.
')
You can select a database and the block map download will start right away. Information is not loaded sequentially, but bypassing the tree of blocks, which may make it look something like the image below.


We continue to work with the database that we used in the previous article . I deleted all globals, we do not need them. And it generated new data based on the Sample classes package from the SAMPLES area. For this, I configured a mapping package to my HABR area.


Executed data generation command.

d ##class(Sample.Utils).Generate(20000) 

The result on our map is:



You can note that the blocks do not start filling from the very beginning of the file. From the 16th block, the blocks of upper level pointers begin, and from the 50th data blocks. 16 and 50 are default values ​​and can be changed if desired. The NewGlobalPointerBlock property in the SYS.Database class is responsible for starting the block of pointers; this property sets the default value for new globals. For already created globals, you can change it in to the % Library.GlobalEdit class via the PointerBlock property. The block from which data blocks will begin is specified in the NewGlobalGrowthBlock property of the SYS.Database class, for a single global, this GrowthBlock property in the % Library.GlobalEdit class. It makes sense to change these values ​​only for those globals that do not yet have data, because changing this property does not affect the current location of the block of upper pointers or data blocks.

Here we see that we have the most data in the global ^ Sample.PersonD with 989 blocks of 83% filling. In second place ^ Sample.PersonI 573 blocks, and this global is already filled by 70%. We can select any global we are interested in to see exactly which blocks are involved under this global. And for the global ^ Sample.PersonI, we can see that some of the blocks are almost empty. It is also clear that the blocks of these two globals are mixed. This is understandable, because when creating new objects both globals are filled: one with data, the other with indices to the Sample.Person table.



Now that we have test data, we can look at the possibilities of manipulating the database that Caché provides us and see the result in a vivid way. To begin with, we will do a small purge in the data in order to create some visibility of activity, when the data can not only be added but also deleted. Therefore we will execute the code which will delete some random data.

  set id="" set first=$order(^Sample.PersonD(""),1) set last=$order(^Sample.PersonD(""),-1) for id=first:$random(5)+1:last { do ##class(Sample.Person).%DeleteId(id) } 

After running this code, we got the result, which can be seen below. There were several empty blocks, the blocks are now filled to 64-67%.



To work with the database, we have the opportunity to use the ^ DATABASE utility in the% SYS area. We use some of its features.



To begin with, since we have blocks with a small filling, we will compress all globals in the database and look at the result.





As you can see, as a result of the compression of globals, we received the requested 90% fill, or values ​​close to this. As a result of compression, it also turned out that previously empty blocks are now filled with data transferred from other blocks. Compression of all globals in the database can be done using the ^ DATABASE utility (item 7), or by executing the following command, passing the first parameter to the database path:

 d ##class(SYS.Database).CompactDatabase("c:\intersystems\ensemble\mgr\habr\") 

It is also possible to move empty blocks to the end of the database. Such a need may arise, for example, if we delete a large portion of data, and we want to subsequently reduce the size of the database file. To demonstrate this, repeat the deletion of data from our database. That's what I did after the removal.




Here we see the appearance of empty blocks. Caché provides the ability to move these empty blocks to the end of the database file, and further it is possible to reduce the size of the database. In order to move empty blocks, we will run the FileCompact method from the SYS.Database class in the system area, or with the help of the ^ DATABASE utility item 13. This method takes three parameters: the path to the database, the desired amount of free space at the end of the file (default is 0 ), and the return parameter is how much free space is obtained.

 do ##class(SYS.Database).FileCompact("c:\intersystems\ensemble\mgr\habr\",999) 

And that's what happened with us, we now have no voids, the voids in the beginning are not counted, because they correspond to the settings (where to start blocks of top-level pointers and data blocks).

Defragmentation

Now we can go on to defrag our globals. This process will rearrange the blocks of each global in order. In the process of the utility may require free space at the end of the database file, and if necessary it will be added. You can defragment using the ^ DATABASE utility item 14, or with the following command:

 d ##class(SYS.Database).Defragment("c:\intersystems\ensemble\mgr\habr\") 

Free up space

Now we see all our globals lined up in turns. But for defragmentation, all the same it took to increase free space in the database file. And we can free this place using paragraph 12 in the ^ DATABASE utility or with the command:

 d ##class(SYS.Database).ReturnUnusedSpace("c:\intersystems\ensemble\mgr\habr\") 

Now our database takes up much less space, and only 1MB of free space remains in the database file. The ability to defragment globals in the database and manage free space by moving and freeing free blocks appeared relatively recently. Previously, if you need to defragment and reduce the size of the database file, you had to use the ^ GBLOCKCOPY utility. It performed block-by-copy from one database to another newly created one, with the ability to select specific globals. This utility is still available.