Redis in production

I would like to talk about some of the features of Redis when used on a combat server. Alternatives will be considered when saving data to a disk, allowing to achieve different degrees of reliability in case of failures. There will also be shown configuration examples for backup and monitoring. It uses Redis 2.2.11 on Amazon EC2 with Ubuntu 10.10 installed.

Backup

')
We use Redis to build custom feeds. Recovery of all feeds from scratch in case of data loss takes a considerable amount of time, so we make backups. Even if you use Redis as a caching server, in some cases it can be a long time to warm up the cache. Therefore, it is always recommended to make backup copies.

Redis makes RDB snapshots based on the following parameters in redis.conf :

save 900 100 save 300 1000 save 60 100000 dir /var/redis/ dbfilename dump.rdb rdbcompression yes 

There are two persistence modes: RDB and AOF . It is worth noting that in both modes a very reliable way of recording information on a disk is used, which practically excludes situations of data loss in case of a hardware failure. How much data you lose depends only on the choice of persistence mode. RDB allows you to adjust this parameter flexibly, but on average, if it fails, it can be lost for about an hour. In this mode, Redis first writes a complete snapshot of the database into a temporary file and only after the end of the recording on the disk renames it to working. This eliminates data loss due to the atomicity of the rename () system call.

In the case of AOF , Redis keeps a log of operations that clients perform and writes them to a file (by default, every second). AOF is an abbreviation for the Append Only File, which means that Redis does not change the data already recorded, but only adds new data to the end. Due to the fact that when using AOF , Redis defaults to writing data to disk every second, the maximum that you lose in case of failure when using this mode is 1 second.

In our project, RDB is used because the hour of data is uncritical for us to lose. In addition, in the worst case, data can be recovered from the main DBMS.

More about persistence in Redis :

http://redis.io/topics/persistence
http://antirez.com/post/redis-persistence-demystified.html

For backups, we use a great backup gem , which has Redis support.

 # encoding: utf-8 Backup::Model.new(:my_backup, 'My Backup') do split_into_chunks_of 500 database Redis do |db| db.name = "dump" db.path = "/var/redis" db.host = "localhost" db.additional_options = [] db.invoke_save = false end compress_with Gzip do |compression| compression.best = true compression.fast = false end notify_by Mail do |mail| # ... end store_with S3 do |s3| # ... end end 

We specifically disable invoke_save in the config because we don’t want the main stream Redis to be blocked by writing to the disk during each backup.

Monitoring

Monit is used for monitoring, which is configured to notify on all occurring events. The settings for it are quite simple:

 set mailserver localhost set mail-format { from: monit-app1@example.com } set alert support@example.ru but not on { action pid ppid } check process redis with pidfile /var/run/redis.pid start program = "/usr/bin/redis-server /etc/redis/redis.conf" stop program = "/usr/bin/redis-cli -p 6379 shutdown" group redis 

Lack of memory with BGSAVE

With large amounts of stored data, problems are possible with RDB snapshots. For recording, use the BGSAVE command, which fork the current process and in this fork data is written to disk. Thus, the main thread is not blocked and the recording occurs asynchronously. The problem is that on UNIX systems, when fork () is called, the contents of the memory that the parent process uses are also copied into the child process. Suppose if Redis currently occupies 2Gb of memory, and only 1Gb of free memory remains in the system, then when executing the BGSAVE command, the following error is possible:

 [18696] 28 Mar 12:26:54 # Can't save in background: fork: Cannot allocate memory

In modern systems, when copying memory for forks, the Copy on Write method is used. Memory is copied only when writing to the corresponding section. Redis forks the process only to save the data asynchronously, this fork does not change them at all, which means we can safely set the system parameter vm.overcommit_memory to 1. This parameter is responsible for the possibility of allocating more memory than is available. Add to /etc/sysctl.conf line:

 vm.overcommit_memory = 1 

And re-read the config:

 # sysctl -p 

More about this problem:

http://groups.google.com/group/redis-db/browse_thread/thread/dc4876861b174358
Background under Linux even if it’s a lot of free RAM!