openstack-backup

OpenStack Backup Operations

A cloud without tested backups is a cloud waiting to lose data. Backup operations protect the entire stack -- from the databases that store state, to the configurations that define behavior, to the volumes that hold user data. The backup hierarchy ensures that any component can be recovered independently, from a single Keystone Fernet key to a complete cloud rebuild.

Backup planning starts with two numbers: RPO (Recovery Point Objective) -- how much data loss is acceptable (e.g., "no more than 1 hour of changes"), and RTO (Recovery Time Objective) -- how long recovery can take (e.g., "service restored within 4 hours"). These numbers drive every decision: backup frequency, storage location, retention depth, and automation investment.

In NASA SE terms, backup maps to Phase E (Operations & Sustainment) as a core sustainment activity, and Phase F (Closeout) for data archive procedures. The backup hierarchy follows the same defense-in-depth philosophy NASA applies to mission-critical data: multiple copies, multiple formats, verified recoverability.

Deploy

Backup Infrastructure Setup

Backup scripts location:

# Create backup directory structure
mkdir -p /opt/openstack-backups/{database,config,volumes,images,snapshots}
mkdir -p /opt/openstack-backups/scripts
mkdir -p /opt/openstack-backups/logs

# Set ownership and permissions
chown -R root:root /opt/openstack-backups
chmod 700 /opt/openstack-backups

Kolla-Ansible built-in database backup:

# MariaDB backup (built into Kolla-Ansible)
kolla-ansible -i inventory mariadb_backup

# Backup location: /var/lib/docker/volumes/mariadb/_data/backups/
# File: mysqlbackup-<timestamp>.qp.xb.gz

Cron scheduling for automated backups:

# /etc/cron.d/openstack-backup
# Daily database backup at 02:00
0 2 * * * root /opt/openstack-backups/scripts/backup-databases.sh >> /opt/openstack-backups/logs/db-backup.log 2>&1

# Daily config backup at 02:30
30 2 * * * root /opt/openstack-backups/scripts/backup-configs.sh >> /opt/openstack-backups/logs/config-backup.log 2>&1

# Weekly volume snapshots (Sunday 03:00)
0 3 * * 0 root /opt/openstack-backups/scripts/backup-volumes.sh >> /opt/openstack-backups/logs/volume-backup.log 2>&1

Backup storage targets:

| Target | Use Case | Configuration | |--------|----------|---------------| | Local disk | Fast restore, primary copy | /opt/openstack-backups/ | | NFS mount | Off-host copy, disaster recovery | Mount at /mnt/backup-nfs/ | | Swift (S3-compatible) | Object storage archive, long-term retention | Use openstack object create or s3cmd |

Configure

MariaDB Backup Strategy

Full backup (weekly):

#!/bin/bash
# backup-databases.sh -- full MariaDB backup with mariabackup
BACKUP_DIR="/opt/openstack-backups/database"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="${BACKUP_DIR}/full_${TIMESTAMP}"

# Use Kolla-Ansible's built-in backup
kolla-ansible -i /etc/kolla/inventory mariadb_backup

# Or manual mariabackup via container
docker exec mariadb mariabackup --backup \
  --target-dir=/backup/full_${TIMESTAMP} \
  --user=root --password=${DB_ROOT_PASSWORD}

echo "Full backup completed: ${BACKUP_FILE}" >> /opt/openstack-backups/logs/db-backup.log

Incremental backup (daily, between full backups):

# Incremental based on last full backup
LAST_FULL=$(ls -d ${BACKUP_DIR}/full_* | sort | tail -1)
docker exec mariadb mariabackup --backup \
  --target-dir=/backup/inc_${TIMESTAMP} \
  --incremental-basedir=${LAST_FULL} \
  --user=root --password=${DB_ROOT_PASSWORD}

Service-Specific Backups

RabbitMQ definitions export:

# Export queue definitions, exchanges, bindings
docker exec rabbitmq rabbitmqctl export_definitions /tmp/rabbitmq_definitions.json
docker cp rabbitmq:/tmp/rabbitmq_definitions.json \
  /opt/openstack-backups/config/rabbitmq_definitions_$(date +%Y%m%d).json

Keystone Fernet key backup:

# Critical: Fernet keys are required to validate existing tokens
docker cp keystone_api:/etc/kolla/keystone/fernet-keys/ \
  /opt/openstack-backups/config/fernet-keys_$(date +%Y%m%d)/

Cinder volume snapshot policies:

# Create snapshot of all in-use volumes
for vol_id in $(openstack volume list --status in-use -f value -c ID); do
  openstack volume snapshot create --volume ${vol_id} \
    --description "Scheduled backup $(date +%Y%m%d)" \
    backup_${vol_id}_$(date +%Y%m%d)
done

Glance image export:

# Export critical images (base OS, golden images)
for image_id in $(openstack image list --status active -f value -c ID); do
  openstack image save --file /opt/openstack-backups/images/${image_id}.raw ${image_id}
done

Nova instance snapshot:

# Snapshot critical instances
openstack server image create --name "backup_$(date +%Y%m%d)" <instance-id>

Configuration File Backup

#!/bin/bash
# backup-configs.sh -- backup all Kolla-Ansible configurations
BACKUP_DIR="/opt/openstack-backups/config/kolla_$(date +%Y%m%d)"
mkdir -p ${BACKUP_DIR}

# Core configuration files
cp /etc/kolla/globals.yml ${BACKUP_DIR}/
cp /etc/kolla/passwords.yml ${BACKUP_DIR}/
cp -r /etc/kolla/inventory ${BACKUP_DIR}/
cp /etc/kolla/admin-openrc.sh ${BACKUP_DIR}/

# Per-service configuration overrides
for svc in keystone nova neutron cinder glance swift heat horizon; do
  [ -d /etc/kolla/${svc} ] && cp -r /etc/kolla/${svc} ${BACKUP_DIR}/
done

# Encrypt sensitive files
gpg --symmetric --cipher-algo AES256 \
  --output ${BACKUP_DIR}/passwords.yml.gpg \
  ${BACKUP_DIR}/passwords.yml
rm ${BACKUP_DIR}/passwords.yml

echo "Config backup completed: ${BACKUP_DIR}"

Backup Encryption

GPG encryption for backup files:

# Encrypt a backup file
gpg --symmetric --cipher-algo AES256 \
  --batch --passphrase-file /root/.backup-passphrase \
  --output backup.tar.gz.gpg backup.tar.gz

# Decrypt
gpg --decrypt --batch --passphrase-file /root/.backup-passphrase \
  --output backup.tar.gz backup.tar.gz.gpg

OpenSSL alternative:

# Encrypt
openssl enc -aes-256-cbc -salt -pbkdf2 \
  -in backup.tar.gz -out backup.tar.gz.enc \
  -pass file:/root/.backup-passphrase

# Decrypt
openssl enc -aes-256-cbc -d -pbkdf2 \
  -in backup.tar.gz.enc -out backup.tar.gz \
  -pass file:/root/.backup-passphrase

Retention Policies

| Tier | Frequency | Retention | Storage Target | |------|-----------|-----------|---------------| | Daily | Every day at 02:00 | 7 days | Local disk | | Weekly | Sunday 03:00 (full DB) | 4 weeks | NFS mount | | Monthly | First Sunday of month | 12 months | Swift/S3 archive |

Rotation script:

# Purge backups older than retention period
find /opt/openstack-backups/database -name "*.gz" -mtime +7 -delete
find /mnt/backup-nfs/database -name "*.gz" -mtime +28 -delete
# Monthly backups on Swift are managed by object expiry headers

Backup Storage Sizing

| Component | Typical Size | Growth Rate | 30-day Estimate | |-----------|-------------|-------------|-----------------| | MariaDB full | 500MB-2GB | ~50MB/day | ~3.5GB | | MariaDB incremental | 50-200MB | Per day | ~4.2GB | | Kolla configs | 10-50MB | Stable | ~50MB | | Fernet keys | <1MB | Stable | <1MB | | Volume snapshots | Varies by usage | Per volume | Plan per tenant | | Glance images | 1-10GB per image | Per upload | Plan per image |

Operate

Scheduled Backup Verification

Restore test procedure (monthly):

# 1. Create isolated test database container
docker run -d --name mariadb_restore_test \
  -e MYSQL_ROOT_PASSWORD=testpass \
  mariadb:latest

# 2. Restore latest backup into test container
docker exec mariadb_restore_test mariabackup --prepare \
  --target-dir=/backup/latest_full

docker exec mariadb_restore_test mariabackup --copy-back \
  --target-dir=/backup/latest_full

# 3. Verify data integrity
docker exec mariadb_restore_test mysql -u root -ptestpass \
  -e