Chapter 14. Backup and Recovery

Standard backup best practices apply when creating your OpenStack backup policy. For example, how often to back up your data is closely related to how quickly you need to recover from data loss.

[Note]Note

If you cannot have any data loss at all, you should also focus on a highly available deployment. The OpenStack High Availability Guide offers suggestions for elimination of a single point of failure that could cause system downtime. While it is not a completely prescriptive document, it offers methods and techniques for avoiding downtime and data loss.

Other backup considerations include:

Just as important as a backup policy is a recovery policy (or at least recovery testing).

 Database Backups

The example OpenStack architecture designates the cloud controller as the MySQL server. This MySQL server hosts the databases for nova, glance, cinder, and keystone. With all of these databases in one place, it's very easy to create a database backup:

Select Text
1
<a id="d6e6208" class="indexterm"># mysqldump --opt --all-databases > openstack.sql</a>

If you only want to backup a single database, you can instead run:

Select Text
1
<a id="d6e6208" class="indexterm"># mysqldump --opt nova > nova.sql</a>

where nova is the database you want to back up.

You can easily automate this process by creating a cron job that runs the following script once per day:

Select Text
1
2
3
4
5
6
7
<a id="d6e6208" class="indexterm">#!/bin/bash
backup_dir="/var/lib/backups/mysql"
filename="${backup_dir}/mysql-`hostname`-`eval date +%Y%m%d`.sql.gz"
# Dump the entire MySQL database
/usr/bin/mysqldump --opt --all-databases | gzip > $filename
# Delete backups older than 7 days
find $backup_dir -ctime +7 -type f -delete</a>

This script dumps the entire MySQL database and deletes any backups older than seven days.

 File System Backups

This section discusses which files and directories should be backed up regularly, organized by service.

 Identity

/etc/keystone and /var/log/keystone follow the same rules as other components.

/var/lib/keystone, although it should not contain any data being used, can also be backed up just in case.

 Block Storage

/etc/cinder and /var/log/cinder follow the same rules as other components.

/var/lib/cinder should also be backed up.

 Object Storage

/etc/swift is very important to have backed up. This directory contains the swift configuration files as well as the ring files and ring builder files, which if lost, render the data on your cluster inaccessible. A best practice is to copy the builder files to all storage nodes along with the ring files. Multiple backup copies are spread throughout your storage cluster.

 Recovering Backups

Recovering backups is a fairly simple process. To begin, first ensure that the service you are recovering is not running. For example, to do a full recovery of nova on the cloud controller, first stop all nova services:

Select Text
1
2
3
4
5
6
<a id="d6e6300" class="indexterm"># stop nova-api
# stop nova-cert
# stop nova-consoleauth
# stop nova-novncproxy
# stop nova-objectstore
# stop nova-scheduler</a>

Now you can import a previously backed-up database:

Select Text
1
<a id="d6e6300" class="indexterm"># mysql nova < nova.sql</a>

You can also restore backed-up nova directories:

Select Text
1
2
<a id="d6e6300" class="indexterm"># mv /etc/nova{,.orig}
# cp -a /path/to/backup/nova /etc/</a>

Once the files are restored, start everything back up:

Select Text
1
2
3
4
5
6
<a id="d6e6300" class="indexterm"># start mysql
# for i in nova-api nova-cert nova-consoleauth nova-novncproxy
nova-objectstore nova-scheduler
> do
> start $i
> done</a>

Other services follow the same process, with their respective directories and databases.

 Summary

Backup and subsequent recovery is one of the first tasks system administrators learn. However, each system has different items that need attention. By taking care of your database, image service, and appropriate file system locations, you can be assured that you can handle any event requiring recovery.

Questions? Discuss on ask.openstack.org
Found an error? Report a bug against this page


loading table of contents...