Dive into BerkeleyDB JE. Introduction to DPL API

Introduction

A little bit about sabzh. BerkeleyDB is a high-performance embedded DBMS that comes in the form of a library for various programming languages. This solution assumes the storage of key-value pairs, the ability to assign several values to one key is also supported. BerkeleyDB supports multi-threading, replication, and more. Attention of this article will be paid primarily to the use of the library provided by Sleepycat Software in the bearded 90s. This article will cover the main aspects of working with the DPL (Direct Persistence Layer) API.

Note : All examples in this article will be given in the Kotlin language.

Entity description

For a start, let's look at how to describe entities. Fortunately, it is very similar to JPA. All entities are reflected as classes with annotations @Persistent and @Entity , each of which allows you to specify in an explicit form the version of the described entity. Within the framework of this article, we will use only the @Entity annotation, in the following ones - light will be shed on @Persitent

Simple entity example

 @Entity(version = SampleDBO.schema) class SampleDBO private constructor() { companion object { const val schema = 1 } @PrimaryKey lateinit var id: String private set @SecondaryKey(relate = Relationship.MANY_TO_ONE) lateinit var name: String private set constructor(id: String, name: String): this() { this.id = id this.name = name } }

Note : for the key with the @PrimaryKey annotation of the java.lang.Long type, you can also specify the sequence parameter, which will create a separate sequence for generating the identifiers of your entities. Alas, in Kotlin does not work.

')

It is worth noting separately that: firstly, all entities need to leave the default private constructor for the library to work correctly, secondly, the @SecondaryKey annotation @SecondaryKey be present in each entity field that we want to index later on. In this case, this is the name field.

Using constraints

To use constraints in entities, the creators offer a completely straightforward way - to perform verification inside accessors. Modify the example above for clarity.

Constraint example

 @Entity(version = SampleDBO.schema) class SampleDBO private constructor() { companion object { const val schema = 1 } @PrimaryKey lateinit var id: String private set @SecondaryKey(relate = Relationship.MANY_TO_ONE) var name: String? = null private set(value) { if(value == null) { throw IllegalArgumentException("Illegal name passed: ${value}. Non-null constraint failed") } if(value.length < 4 || value.length > 16) { throw IllegalArgumentException("Illegal name passed: ${value}. Expected length in 4..16, but was: ${value.length}") } } constructor(id: String, name: String): this() { this.id = id this.name = name } }

Relationship between entities

BerkeleyDB JE supports all types of relationships:

1: 1 Relationship.ONE_TO_ONE
1: N Relationship.ONE_TO_MANY
N: 1 Relationship.MANY_TO_ONE
N: M Relationship.MANY_TO_MANY

To describe the relationship between entities, the same @SecondaryKey with three additional parameters:

relatedEntity - class of entity, the relation to which is described
onRelatedEntityDelete - behavior when deleting an entity (interrupting a transaction, zeroing references, cascading deletion)
name - the entity field that acts as foreign key

Relationship between entities on the example of a primitive shop

 @Entity(version = CustomerDBO.schema) class CustomerDBO private constructor() { companion object { const val schema = 1 } @PrimaryKey() var id: String? = null private set @SecondaryKey(relate = Relationship.ONE_TO_ONE) lateinit var email: String private set var balance: Long = 0L constructor(email: String, balance: Long): this() { this.email = email this.balance = balance } constructor(id: String, email: String, balance: Long): this(email, balance) { this.id = id } override fun toString(): String { return "CustomerDBO(id=$id, email=$email, balance=$balance)" } }

 @Entity(version = ProductDBO.schema) class ProductDBO { companion object { const val schema = 1 } @PrimaryKey() var id: String? = null private set @SecondaryKey(relate = Relationship.MANY_TO_ONE) lateinit var name: String private set var price: Long = 0L var amount: Long = 0L private constructor(): super() constructor(name: String, price: Long, amount: Long): this() { this.name = name this.price = price this.amount = amount } constructor(id: String, name: String, price: Long, amount: Long): this(name, price, amount) { this.id = id } override fun toString(): String { return "ProductDBO(id=$id, name=$name, price=$price, amount=$amount)" } }

 @Entity(version = ProductChunkDBO.schema) class ProductChunkDBO { companion object { const val schema = 1 } @PrimaryKey() var id: String? = null private set @SecondaryKey(relate = Relationship.MANY_TO_ONE, relatedEntity = OrderDBO::class, onRelatedEntityDelete = DeleteAction.CASCADE) var orderId: String? = null private set @SecondaryKey(relate = Relationship.MANY_TO_ONE, relatedEntity = ProductDBO::class, onRelatedEntityDelete = DeleteAction.CASCADE) var itemId: String? = null private set var amount: Long = 0L private constructor() constructor(orderId: String, itemId: String, amount: Long): this() { this.orderId = orderId this.itemId = itemId this.amount = amount } constructor(id: String, orderId: String, itemId: String, amount: Long): this(orderId, itemId, amount) { this.id = id } override fun toString(): String { return "ProductChunkDBO(id=$id, orderId=$orderId, itemId=$itemId, amount=$amount)" } }

 @Entity(version = OrderDBO.schema) class OrderDBO { companion object { const val schema = 1 } @PrimaryKey() var id: String? = null private set @SecondaryKey(relate = Relationship.MANY_TO_ONE, relatedEntity = CustomerDBO::class, onRelatedEntityDelete = DeleteAction.CASCADE) var customerId: String? = null private set @SecondaryKey(relate = Relationship.ONE_TO_MANY, relatedEntity = ProductChunkDBO::class, onRelatedEntityDelete = DeleteAction.NULLIFY) var itemChunkIds: MutableSet<String> = HashSet() private set var isExecuted: Boolean = false private set private constructor() constructor(customerId: String, itemChunkIds: List<String> = emptyList()): this() { this.customerId = customerId this.itemChunkIds.addAll(itemChunkIds) } constructor(id: String, customerId: String, itemChunkIds: List<String> = emptyList()): this(customerId, itemChunkIds) { this.id = id } fun setExecuted() { this.isExecuted = true } override fun toString(): String { return "OrderDBO(id=$id, customerId=$customerId, itemChunkIds=$itemChunkIds, isExecuted=$isExecuted)" } }

Configuration

BerkeleyDB JE provides extensive configuration options. This article will cover the minimum settings required for writing a client application, and in the future, as far as possible, the light will be shed on more advanced features.

To begin with, consider the entry points to the component that will work with the database. In our case, these will be the Environment and EntityStore . Each of them provides an impressive list of different options.

Environment

Setting work with the environment involves the definition of standard parameters. In the simplest form, something like this will be released:

  val environment by lazy { Environment(dir, EnvironmentConfig().apply { transactional = true allowCreate = true nodeName = "SampleNode_1" cacheSize = Runtime.getRuntime().maxMemory() / 8 offHeapCacheSize = dir.freeSpace / 8 }) }

transactional - set to true if you want to use transactions
allowCreate - set to true if the environment is to be created, if it is not found in the specified directory
nodeName - set the name for a configurable Environment; A very nice option, if the application will use several Environment, and you ~~do not~~ want ~~to shoot yourself a leg to~~ have readable logs
cacheSize - the amount of memory that will be allocated under the in-memory cache
offHeapCacheSize - the amount of memory that will be allocated for disk cache

EntityStore

If the application uses the DPL API, the main class for working with the database will be EntityStore. The standard configuration is as follows:

  val store by lazy { EntityStore(environment, name, StoreConfig().apply { transactional = true allowCreate = true }) }

Indexes, data access

In order to understand how indexes work, the easiest way to consider this SQL query is:

 SELECT * FROM customers ORDER BY email;

In BerkeleyDB JE, this query can be implemented as follows: the first thing that is required is, in fact, create two indices. The first one is basic, it should correspond to the @PrimaryKey our entity. The second is secondary, corresponding to the field, the ordering of which is performed ( note — the field should, as was said above, be annotated as @SecondaryKey ).

  val primaryIndex: PrimaryIndex<String, CustomerDBO> by lazy { entityStore.getPrimaryIndex(String::class.java, CustomerDBO::class.java) } val emailIndex: SecondaryIndex<String, String, CustomerDBO> by lazy { entityStore.getSecondaryIndex(primaryIndex, String::class.java, "email") }

Data retrieval is performed in the usual way - using the cursor interface (in our case - EntityCursor )

  fun read(): List<CustomerDBO> = emailIndex.entities().use { cursor -> mutableListOf<CustomerDBO>().apply { var currentPosition = 0 val count = cursor.count() add(cursor.first() ?: return@apply) currentPosition++ while(currentPosition < count) { add(cursor.next() ?: return@apply) currentPosition++ } } }

Relations & Conditions

A common task is to get entities using the relationship between their tables. Consider this question on the example of the following SQL query:

 SELECT * FROM orders WHERE customer_id = ?;

And his presentation in the framework of Berkeley:

  fun readByCustomerId(customerId: String): List<OrderDBO> = customerIdIndex.subIndex(customerId).entities().use { cursor -> mutableListOf<OrderDBO>().apply { var currentPosition = 0 val count = cursor.count() add(cursor.first() ?: return@apply) currentPosition++ while(currentPosition < count) { add(cursor.next() ?: return@apply) currentPosition++ } } }

Unfortunately, this option is possible only under one condition. To create a query with several conditions you will need to use a more complex structure.

Modified Buyer

 @Entity(version = CustomerDBO.schema) class CustomerDBO private constructor() { companion object { const val schema = 1 } @PrimaryKey() var id: String? = null private set @SecondaryKey(relate = Relationship.ONE_TO_ONE) lateinit var email: String private set @SecondaryKey(relate = Relationship.MANY_TO_ONE) lateinit var country: String private set @SecondaryKey(relate = Relationship.MANY_TO_ONE) lateinit var city: String private set var balance: Long = 0L constructor(email: String, country: String, city: String, balance: Long): this() { this.email = email this.country = country this.city = city this.balance = balance } constructor(id: String, email: String, country: String, city: String, balance: Long): this(email, country, city, balance) { this.id = id } }

New indexes

  val countryIndex: SecondaryIndex<String, String, CustomerDBO> by lazy { entityStore.getSecondaryIndex(primaryIndex, String::class.java, "country") } val cityIndex: SecondaryIndex<String, String, CustomerDBO> by lazy { entityStore.getSecondaryIndex(primaryIndex, String::class.java, "city") }

Sample query with two conditions (SQL)

 SELECT * FROM customers WHERE country = ? AND city = ?;

Sample request with two conditions

  fun readByCountryAndCity(country: String, city: String): List<CustomerDBO> { val join = EntityJoin<String, CustomerDBO>(primaryIndex) join.addCondition(countryIndex, country) join.addCondition(cityIndex, city) return join.entities().use { cursor -> mutableListOf<CustomerDBO>().apply { var currentPosition = 0 val count = cursor.count() add(cursor.first() ?: return@apply) currentPosition++ while(currentPosition < count) { add(cursor.next() ?: return@apply) currentPosition++ } } } }

As you can see from the examples - quite a dreary syntax, but it is quite possible to live.

Range queries

With this type of query, everything is transparent, the indexes have an overload of the function

fun <E> entities(fromKey: K, fromInclusive: Boolean, toKey: K, toInclusive: Boolean):
    

      
      
      
    

  EntityCursor<E>

fun <E> entities(fromKey: K, fromInclusive: Boolean, toKey: K, toInclusive: Boolean):
    

      
      
      
    

  EntityCursor<E>

fun <E> entities(fromKey: K, fromInclusive: Boolean, toKey: K, toInclusive: Boolean):
    

      
      
      
    

  EntityCursor<E>

which provides the ability to use the cursor, iterating over the desired data sample. This method works quite quickly, since indexes are used, is relatively convenient, and, in my opinion, does not require separate comments.

Instead of conclusion

This is the first article from the planned cycle by BerkeleyDB. Based on its goal - to acquaint the reader with the basics of working with the Java Edition library, to consider the main features that are necessary for routine actions. Subsequent articles will cover more interesting details of working with this library if the article turns out to be interesting to someone.

Since I have very little experience with Berkeley, I would be grateful for the criticism and corrections in the comments if I made some flaws somewhere.

(10.12.2017) UPD1: in the latest version of Berkeley JE compatible with Android, transactions do not work correctly, and you need to make sure in advance that you can create a checkpoint after changing the cache. To do this, you can use the Environment :: sync function, which is blocking (sic!) And very expensive (sic!) ^ 2 writes the current cache changes to disk.

(10.12.2017) UPD2: It is worth noting that when competing with the database (judging from personal experience), it is recommended to disable the log cleaner. This is done through the Environment configuration. This setting may change without any additional changes in working with the database, and no data will be lost.

Source: https://habr.com/ru/post/336098/

All Articles