Redis Persistence

Persistence Options

-------------------

- RDB persistence perform point-in-time snapshots of dataset at a given inerval

- AOF persistence logs replays command agian at server startup

- Redis can write logs in the background when it gets too big

- Persistence can be disabled (data will exist only while server us running)

- AOF and RDB can be combined

NOTE:

Redis is planning to unify both AOF and RDB in the future

Interaction between AOF and RDB

-------------------------------

- redis >= 2.4 avoids running AOF rewrite and RDB snapshot at the same time

  to prevent heavy disk I/O

- when snapshotting is in progress and user issues a BGREWRITEAOF, server will

  reply OK and will start it once snapshotting is finished

- if both enabled, AOF will be used to reconstruct dataset since it is

  guaranteed to be more complete

Backing up redis

----------------

- It is safe to backup RDB file while server is running because RDB file is

  never modified once produced; it only gets modified during redis restart or

  after a snapshot operation

- One way of backing up is sending hourly and daily snapshots to separate

  directories

- RDB file backups can be encrypted using "gpg"

RDB

Advantages

----------

a. Compact single file point-in-time representation of your redis data

b. Perfect for backups

c. Good for disaster recovery

d. Can be transported encrypted

e. Faster restarts compared to AOF

f. Different versions can be saved via snapshotting

g. Performance is maximize by forking - parent forks a child that does the job

Disadvantages

-------------

a. Prone to data loss during unplanned shutdown (solution to minimize loss is to

   create snapshot every 5 minutes or so)

b. Forking is time consuming on large datasets

Snapshots:

- By default, snapshot of dataset is saved to /var/lib/redis/dump.rdb

- To manually save snapshot, use "SAVE" or "BGSAVE" command

Configurations

--------------

save 60 100

dump dataset to disk every 60 seconds if atleast 1000 keys changed

How snapshots work

------------------

1. Forks a child

2. Child writes dataset to temporary RDB file

3. When done, child replaces the old RDB file with the temporary one

AOF

Advantages

----------

a. More durable (resolves issues of data loss)

b. Has different fsync policies: no fsync, fsync/second, fsync/query

c. Log is append-only log, no corruption since redis-check-aof tool can fix it

d. Can automatically rewrite AOF in the background when it gets too big

e. Has a log of all operations that can easily parsed

f. AOF file can be easily exported

Disadvantages

-------------

a. AOF files are bigger compared to RDB for same dataset

b. Can be slower compared to RDB depending on the fsync policy used

c. Contains some rare bugs

How it works?

-------------

- Every time redis receives a command, it appends (SET) it to the AOF

  (/var/lib/redis/appendonly.aof)

- AOF file contains series of commands executed to arrive at a given key

- When redis restarts, it will replay AOF to rebuild the state

- Important features

    a. BGREWRITEAOF - writes the shortest sequence of command needed to rebuild

                      current dataset in memory

                    - solve issues on big AOFs (e.g if a counter is incremented

                      100 times, you'll end up w/ a single key in the dataset

                      containing the final value; 99 of those are not needed to

                      rebuild the dataset)

Configuration

-------------

- Can be turned on by using "appendonly yes"

How durable is AOF

------------------

1. fsync for every new command (very slow but very safe)

2. fsync every second (very fast, can lose 1 second of data) -- RECOMMENDED!

3. no fsync (faster but not safe)

What can I do if AOF got corrupted?

-----------------------------------

1. Make backup of AOF

2. Fix by: redis-check-aof --fix

3. Restart server w/ fixed file

How log rewriting works in AOF?

-------------------------------

a. redis forks (parent and child)

b. child starts writing AOF to a temporary file

c. parent writes new changes to memory and old file

d. when child is done, parent appends new changes to temporary file

e. redis renames old to new file

f. redis starts appending to new file

Switching to from RDB to AOF

----------------------------

Redis >= 2.2:

- backup latest .rdb file

- transfer backup to safe place

- execute: redis-cli config set appendonly yes

- execute: redis-cli config save ""  # turns off snapshotting persistence

- make sure number of keys is still the same

- make sure .aof file is created

- make settings persistent by updating redis.conf

Redis 2.0:

- backup latest .rdb file

- transfer backup to safe place

- stop all writes to db

- execute: redis-cli bgrewriteaof  # creates .aof file

- wait for redis to finishe generating .aof

- stop server

- enable aof persistence in redis.conf

- start server

- make sure .aof file is created

- make settings persistent by updating redis.conf