Backing up MySQL and MongoDB data on a multi-node stack

In playbooks/openstack you'll find a backup.yml playbook that will take consistent Cinder snapshots of the MySQL and MongoDB volumes of backend instances, and also rotate them to a configurable limit. It's designed to run very quickly, in a non-disruptive manner so as not to incur any downtime for users.

It is meant to be run from the deploy node of a multi-node Heat stack, with OpenStack authentication variables properly set. See playbooks/openstack/group_vars/backend_servers.example for a list of variables that must be changed, including the number of snapshots that will be maintained in rotation for each instance and database.

It is recommended that this playbook be run with:

--limit <secondary_backend_node>

With the sample multi-node Heat template, it would either be --limit 192.168.122.112 or --limit 192.168.122.113. It should not be targetted on the primary backend node (192.168.122.111), because in order to get a consistent snapshot, prior to snapshotting it will stop the MariaDB service and sync the filesystem. (MongoDB doesn't require special handling due to the fact that its journal is stored in the same volume as the database being snapshotted.)

Note: the snapshot script expects the backend Cinder volume names to follow a certain pattern: the MySQL volume must contain "mysql_data" in its name, and the MongoDB one must contain "mongodb_data". The provided multi-node Heat template follows this precise pattern.

Automating the backup

To run the backup playbook on a schedule, create a cron file such as /etc/cron.d/backup. If you wish to run the backup daily at 01:05 AM, for example, copy the following lines into the file:

# /etc/cron.d/backup: cron entry for backups

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
HOME=/home/ubuntu

05 1 * * * ubuntu cd /var/tmp/edx-configuration/playbooks; ansible-playbook -i openstack/inventory.py openstack/backup.yml --limit 192.168.122.112 2>&1 >> /var/log/backup.log

You'll find a sample backup.cron in playbooks/openstack, alongside the backup playbook.

You will also need to create an SSH key without a passphrase for the ubuntu user on the deploy node, and distribute it to the other nodes. This is so that Ansible can connect to the backend node without human intervention. (You can skip this step if you've already done it as part of initial stack deployment.)

# Create a new key pair
export KEYFILE=~/.ssh/id_rsa
ssh-keygen -t rsa -N "" -f $KEYFILE

# Disable strict host key checking
echo "StrictHostKeyChecking no" > ~/.ssh/config

# Deploy it
ips=$(~/edx-configuration/playbooks/openstack/inventory.py --list | grep 192 | tr -d ' ",')
for ip in $ips; do
    ssh-keyscan $ip >> ~/.ssh/known_hosts
    ssh-copy-id -i $KEYFILE $ip
done