slowpoke is not the fastest database

Hello.

slowpoke is a key / value data store, written in the golang standard library. Slowpoke has a minimalist, user-friendly api and is suitable for solving a fairly wide range of tasks.

You can write the value in slowpoke using the Set command:
')

slowpoke.Set("db/some.db", []byte("foo"), []byte("bar")) 

The storage unit in slowpoke is a file. In this example, the directory “db” will be created, with the file “some.db”, in which three bytes will be placed (“bar”).

Strictly speaking, each file in slowpoke is an independent database. Each database is served in a separate goroutine. Opening / reading the file is done automatically. Those. if this database exists, it will be opened and read. If not - created. This feature allows you not to think about the state of the database when working with it and use it, for example, in http request handlers or in other goroutine.

The slowpoke takes an array of bytes as the key and value. In the example above, strings are essentially used as keys and values. We write in slowpoke numbers:

 for i := 0; i < 40; i++ { id := make([]byte, 4) binary.BigEndian.PutUint32(id, uint32(i)) slowpoke.Set("numeric.db", id, nil) } 

We converted numbers into the BigEndian format, which will allow us to correctly take into account the sorting of keys during further work. The value in this example is not specified, and only an array of keys will be created (the size of “numeric.db” will be 0 bytes). Speaking of keys, keys in slowpoke are stored in memory, but stored on disk.

This allows you to quickly operate with keys on the one hand, and close the table if there is a shortage of RAM on the other hand. For this reason, large keys are not recommended. I assume that the optimal total size of all keys for one table will be 10 megabytes. If this size is exceeded, it is better to split the database into several tables. Values ​​are not stored in memory and can be of any size (pictures, movies). The total size of the table of values ​​can not exceed uint32. Some advantages of this approach (separate storage of keys and values) are described in the article www.usenix.org/system/files/conference/fast16/fast16-papers-lu.pdf (this article focuses on ssd disks Lsm tree and other architectural delights not used in slowpoke)

Let's go back to slowpoke. In addition to numbers, strings, files, etc., you can save objects by serializing them with one of the methods. An example of serialization in json:

 binary, _ := json.Marshal(post) slowpoke.Set("json",key,binary) 

Or use the gob package built into golang: golang.org/pkg/encoding/gob

All write commands are slowpoke atomic, and are completed with the sync command (data synchronization). The fact is that modern file systems, when writing to a file, actually write to the buffer. And if the operating system crashes, the buffer will be lost. Most databases have a nosync mode (it can be called differently, but the point is that the synchronization operation is very slow, this is especially noticeable on older hard drives and this mode is used to win benchmarks and to speed up writing). A good overview on the topic of Crash Vulnerability.

In slowpoke, there is no nosync mode, so:

To insert multiple entries in slowpoke, use the Sets command.

This command is recommended for writing a packet of keys / values. In this case, the data will be written to the file buffer and synchronized only once, at the end of the recording. What dramatically increases the speed of recording on the one hand, makes it possible not to sacrifice data integrity in general, on the other hand. An example of using the Coman Sets:

  var pairs [][]byte for i := 0; i < 40; i++ { id := make([]byte, 4) binary.BigEndian.PutUint32(id, uint32(i)) post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i), Category: "Category:" + strconv.Itoa(i/10)} b, _ := json.Marshal(post) pairs = append(pairs, id) pairs = append(pairs, b) } //store posts fast slowpoke.Sets(posts, pairs) 

In the example above, first an array of key / value pairs is formed, then it is written.

We learned to save data in slowpoke using the set and sets commands. But before moving on to the reading commands, consider the command to select keys from slowpoke.

Keys is built in to work with keys in slowpoke.

Keys can be extracted from slowpoke:

• In direct order
• In reverse order
• With offset
• From a specific value
• Prefixed by

Command example:

 //get last 2 post key with offset 2 limit := uint32(2) offset := uint32(2) order := false //descending keys, _ := slowpoke.Keys(posts, nil, limit, offset, order) 

There are a number of rules for the Keys team:

If limit is not specified, all keys will be returned.

If the from field is specified (not equal to nil), the keys will be returned after this value (the value itself will not be included).

If there are no keys matching the conditions, an empty array will be returned.

The keys inside are represented as an array of bytes. Accordingly, for getting correctly sorted numbers, for example, you need to convert them to BigEndian.

The Keys command manipulates data in memory and does not access the disk. Inside, in slowpoke, the keys are duplicated in arrays (slice), each slice is created under the table and sorted at the time of the call of Keys (if necessary). When sampling by prefix / value, a binary search is used (if possible). The Keys team is preferred over Get / Gets for the reasons stated above.

In addition to the possibility of padzhinatsiya, sampling the next value, the command Keys allows you to select keys for a given prefix. For example, tags: tag: id or email: username keys can be stored in the database. In this case, it is necessary to pass in the from field a value with the symbol * at the end, for example [] byte ("sex: *"). I also remind you that the keys can be created without values, which can be convenient for storing indexes in memory (even if the keys are without values, they will be stored on disk, so that they can be restored in case of a failure or table closure).

To select values, paired with the command Keys - it is convenient to use the command Gets.

The Gets command receives an array of keys as input (i.e. the result of the Keys command). Example:

  keys, _ := slowpoke.Keys(posts, nil, limit, offset, order) //get key/ values res := slowpoke.Gets(posts, keys) 

The result of the Gets command is an array of key / value pairs. Notice the gets:

- the only team that does not return errors
- if one of the keys is not present, it will be skipped.
- if there are none of the keys - an empty array will be returned

Also, the Sets command accepts key / value pairs as input. What can be used to save data in another table, for example.

Keys - returns an array of keys
Gets - takes an array of keys, returns an array of pairs
Sets - takes an array of pairs

Probably needless to say that the functions for working with groups of values ​​are faster than their counterparts for working with a single value.

To select a single value, use the Get command.

  res, err := slowpoke.Get(file, key) 

If there is no value, Get returns nil and the error key is not found.

The CloseAll () command closes all active databases and should be called upon completion of work, for example:

  sigchan := make(chan os.Signal, 10) signal.Notify(sigchan, os.Interrupt) <-sigchan //        slowpoke.CloseAll() 

Either way: defer slowpoke.CloseAll ()

We looked at the basic commands of slowpoke, but there are still advanced commands.

The Open command opens the database and reads the keys into memory.

Team Close - closes the database and unloads keys from memory.

There is no need to call these commands manually if the size of the RAM allows you to store all the keys of the public databases. These commands can be useful in the following cases.

Example 1: you keep logs

The log for the time interval is assembled into an array of pairs and inserted using the Sets command
And it is recorded, for example, in a file corresponding to the date of the logs, for example “logs / 20170101.db”
At the end of the work with this interval, the database “logs / 20170101.db” can be closed and unloaded from memory using the lose command. With any request to this database - it will be automatically opened and read.

Example 2: blockchain transaction chain

Suppose each transaction has a number. It may be appropriate to store each pool of transactions in a separate file, for example 1000.db - corresponds to transactions from 1 to 1000, and unload from memory, at the end of the recording, if the call to this block is relatively rare.

Example 3: hosting sites

You can store the data of each site in a separate folder, for example sites / mysite.com.db
and unload from memory after updates, if the reading of them is relatively rare. Since the keys in slowpoke are stored separately from the values, reading unloaded keys from the file into memory is a quick operation.

there is also a command to delete the database DeleteFile ()

Architecture

slowpoke is written in the golang standard library. For storage of keys are used map and slice. Scaling is achieved by running each database in a separate goroutine.

Values ​​are stored as is, without overhead and only on disk, references to addresses of values ​​are stored in memory. Neither BTree nor LSM Tree are used in the project. Also, mmap technology is not used (and is not planned). As practice shows, the more complex the database architecture, the more problems will have to face the growth of the database. Since nothing is free.

The downside of the simplicity of the architecture is slowpoke slower performance compared with market leaders, at least on typical measurements:

 //macbook 2017 slowpoke vs boltdb //The 100 Set took 19.440075ms to run./19.272079ms //The 100 Sets took 1.139579ms to run./? //The 100 Get took 671.343µs to run./211.878µs //The 100 Gets took 206.775µs to run./? //The 100 Keys took 36.214µs to run./? //The second 100 Keys took 5.632µs to run./? 

For example, in databases using LSM Tree (rocksdb, leveldb, cassandra, badger) the record will be faster.

In databases using mmap - (LMDB, Boltdb, SophiaDb) reading values ​​will be faster (slowpoke does not cache values)

However, in general, to my considerable surprise (as can be seen from the name of the database), for all the simplicity of the architecture, the loss was minimal, and on a number of tasks slowpoke even overtakes its more clever competitors. A number of benchmarks are in the project and in separate repositories.

To use slowpoke in programming languages ​​other than golang, an example of a small database server is written on http. Also slowly saw a larger project on this engine .

GRPC server with backdoor as REST API

At the moment, slowpoke is young and not widely spread, mostly used by friends, on their personal pet projects (such as telegram bot, tggram.com, in some places), as api for snatchnews.com (in a mobile application, via rest api). Theoretically, it would fit well into projects involving the storage of large amounts of data - ipfs.io or Ethereum as an alternative to more complex leveldb / badger databases.

I would be glad to pool rekvestam.