Backing up MySQL and MongoDB data on a multi-node stack
In playbooks/openstack
you'll find a backup.yml
playbook that will take
consistent Cinder snapshots of the MySQL and MongoDB volumes of backend
instances, and also rotate them to a configurable limit. It's designed to run
very quickly, in a non-disruptive manner so as not to incur any downtime for
users.
It is meant to be run from the deploy
node of a multi-node Heat stack, with
OpenStack authentication variables properly set. See
playbooks/openstack/group_vars/backend_servers.example
for a list of
variables that must be changed, including the number of snapshots that will be
maintained in rotation for each instance and database.
It is recommended that this playbook be run with:
--limit <secondary_backend_node>
With the sample multi-node Heat template, it would either be --limit
192.168.122.112
or --limit 192.168.122.113
. It should not be targetted on
the primary backend node (192.168.122.111
), because in order to get a
consistent snapshot, prior to snapshotting it will stop the MariaDB service and
sync the filesystem. (MongoDB doesn't require special handling due to the fact
that its journal is stored in the same volume as the database being
snapshotted.)
Note: the snapshot script expects the backend Cinder volume names to follow a certain pattern: the MySQL volume must contain "mysql_data" in its name, and the MongoDB one must contain "mongodb_data". The provided multi-node Heat template follows this precise pattern.
Automating the backup
To run the backup playbook on a schedule, create a cron file such as
/etc/cron.d/backup
. If you wish to run the backup daily at 01:05 AM, for
example, copy the following lines into the file:
# /etc/cron.d/backup: cron entry for backups SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin HOME=/home/ubuntu 05 1 * * * ubuntu cd /var/tmp/edx-configuration/playbooks; ansible-playbook -i openstack/inventory.py openstack/backup.yml --limit 192.168.122.112 2>&1 >> /var/log/backup.log
You'll find a sample backup.cron
in playbooks/openstack
, alongside the
backup playbook.
You will also need to create an SSH key without a passphrase for the ubuntu user on the deploy node, and distribute it to the other nodes. This is so that Ansible can connect to the backend node without human intervention. (You can skip this step if you've already done it as part of initial stack deployment.)
# Create a new key pair export KEYFILE=~/.ssh/id_rsa ssh-keygen -t rsa -N "" -f $KEYFILE # Disable strict host key checking echo "StrictHostKeyChecking no" > ~/.ssh/config # Deploy it ips=$(~/edx-configuration/playbooks/openstack/inventory.py --list | grep 192 | tr -d ' ",') for ip in $ips; do ssh-keyscan $ip >> ~/.ssh/known_hosts ssh-copy-id -i $KEYFILE $ip done