openstack-kolla-ansible-ops

Kolla-Ansible Day-2 Operations -- Infrastructure Lifecycle Management

Kolla-Ansible is not just a deployment tool -- it is the infrastructure lifecycle manager. The initial kolla-ansible deploy is a one-time event. Everything after that -- reconfiguration, upgrades, container management, maintenance -- is what this skill covers. Operators will use these procedures repeatedly throughout the cloud's operational life.

The operational command set:

| Command | Purpose | When to Use | |---------|---------|-------------| | kolla-ansible reconfigure | Regenerate configs and restart affected services | After globals.yml or config override changes | | kolla-ansible upgrade | Pull new images and upgrade services | Minor or major OpenStack release upgrades | | kolla-ansible prechecks | Validate configuration before applying | Always run before reconfigure or upgrade | | kolla-ansible stop | Stop all or specific service containers | Maintenance, troubleshooting | | kolla-ansible deploy | Deploy services (also used for redeployment) | After stop, or for new services | | kolla-ansible pull | Pull container images without deploying | Pre-stage images before upgrade window |

Relationship to the kolla-ansible deployment skill: The deployment skill (Phase 313) covers the initial bootstrap, deploy, and post-deploy verification that brings the cloud online for the first time. This skill picks up where deployment left off. They share the same inventory and globals.yml, but the operational context is different: deployment is about getting services running; operations is about keeping them running, changing their configuration, and upgrading them.

Container lifecycle in Kolla: Every OpenStack service runs in a Docker container managed by Kolla-Ansible. The container lifecycle is: image pull (from registry or local build) -> deploy (create container with config volumes) -> reconfigure (regenerate config, restart container) -> upgrade (pull new image, recreate container, run migrations). Understanding this lifecycle is essential for day-2 operations.

Deploy

Operational Prerequisites

Kolla-Ansible virtual environment maintenance:

# The Kolla-Ansible venv should be maintained separately from system Python
source /path/to/kolla-venv/bin/activate

# Verify kolla-ansible version matches deployed release
kolla-ansible --version
pip show kolla-ansible | grep Version

Inventory file management:

# Inventory defines which hosts run which services
# For single-node: all-in-one inventory
# For multi-node: multinode inventory with role assignments

# Verify inventory connectivity
ansible -i /etc/kolla/inventory all -m ping

globals.yml change tracking:

# globals.yml is the primary configuration file -- track all changes in git
cd /etc/kolla
git init  # If not already tracked
git add globals.yml
git commit -m "baseline: initial globals.yml configuration"

# Before every change, commit the current state
git add globals.yml
git commit -m "pre-change: document current state before <change-description>"

Password management:

# Initial password generation (done once during deployment)
kolla-ansible -i /etc/kolla/inventory genpasswd

# Passwords are stored in /etc/kolla/passwords.yml
# After initial generation, rotate passwords manually
# Never regenerate all passwords on a running system

Image management:

# Check current image versions
docker images | grep kolla

# Pull latest images for current release
kolla-ansible -i /etc/kolla/inventory pull

# Pull images for a specific service
kolla-ansible -i /etc/kolla/inventory pull --tags nova

Configure

Service Reconfiguration Workflow

Reconfiguration is the most common operational action. It regenerates configuration files from templates and globals.yml, then restarts affected containers.

# Step 1: Make the change in globals.yml or config overrides
vim /etc/kolla/globals.yml

# Step 2: Validate the change
kolla-ansible -i /etc/kolla/inventory prechecks

# Step 3: Apply the change
kolla-ansible -i /etc/kolla/inventory reconfigure

# Step 4: Verify service health
openstack service list
openstack endpoint list

Partial Reconfiguration with --tags

Target specific services to minimize disruption:

# Reconfigure only Nova
kolla-ansible -i /etc/kolla/inventory reconfigure --tags nova

# Reconfigure only Neutron
kolla-ansible -i /etc/kolla/inventory reconfigure --tags neutron

# Reconfigure only Keystone
kolla-ansible -i /etc/kolla/inventory reconfigure --tags keystone

# Multiple services
kolla-ansible -i /etc/kolla/inventory reconfigure --tags nova,neutron

Config Validation Before Apply

Always run prechecks before reconfigure or upgrade:

# Full prechecks
kolla-ansible -i /etc/kolla/inventory prechecks

# Service-specific prechecks
kolla-ansible -i /etc/kolla/inventory prechecks --tags nova

# Prechecks verify:
# - Container runtime is available
# - Required images exist
# - Configuration templates render without errors
# - Port conflicts are detected
# - Service dependencies are met

Custom Config Overrides

Kolla-Ansible supports per-service configuration overrides that persist across reconfigures:

# Override directory structure
/etc/kolla/config/
  nova/
    nova.conf         # Merged into nova.conf
    nova-compute.conf # Merged into nova-compute.conf
  neutron/
    neutron.conf      # Merged into neutron.conf
    ml2_conf.ini      # Merged into ml2_conf.ini
  keystone/
    keystone.conf     # Merged into keystone.conf

# Example: /etc/kolla/config/nova/nova.conf
[DEFAULT]
debug = True

[scheduler]
max_attempts = 5

globals.yml Change Matrix

What changes require which operational action:

| Change Category | Example | Required Action | |----------------|---------|-----------------| | Service enable/disable | enable_heat: "yes" | deploy (new service) or stop + remove | | Backend change | neutron_plugin_agent: "ovn" | Full redeploy of affected services | | Network config | neutron_external_interface | reconfigure --tags neutron | | Logging | enable_central_logging: "yes" | reconfigure | | TLS | kolla_enable_tls_internal: "yes" | reconfigure (generates certs, updates all services) | | Allocation ratios | nova_cpu_allocation_ratio | reconfigure --tags nova | | Image tag | openstack_release | upgrade (not reconfigure) |

Operate

Service Reconfigure

Complete procedure for a configuration change:

1. Document the change: git -C /etc/kolla diff globals.yml
2. Commit pre-change state: git -C /etc/kolla add globals.yml && git -C /etc/kolla commit -m "pre-change: <description>"
3. Make the change: edit globals.yml or config overrides
4. Run prechecks: kolla-ansible -i /etc/kolla/inventory prechecks
5. Run reconfigure: kolla-ansible -i /etc/kolla/inventory reconfigure (or with --tags)
6. Verify service health: openstack service list && openstack endpoint list
7. Commit post-change state: git -C /etc/kolla add globals.yml && git -C /etc/kolla commit -m "post-change: <description>"

Minor Upgrade (Patch)

Within the same major OpenStack release (e.g., 2024.1 patch):

1. Backup databases: kolla-ansible -i /etc/kolla/inventory mariadb_backup
2. Pull new images: kolla-ansible -i /etc/kolla/inventory pull
3. Run prechecks: kolla-ansible -i /etc/kolla/inventory prechecks
4. Run upgrade: kolla-ansible -i /etc/kolla/inventory upgrade
5. Verify all services: openstack service list -- all services should be enabled and endpoints reachable
6. Run smoke test: create a test instance, verify networking, delete test instance

Major Upgrade

Across OpenStack releases (e.g., 2024.1 to 2024.2):

1. Pre-upgrade backup:

```bash # Database backup kolla-ansible -i /etc/kolla/inventory mariadb_backup # Verify backup: ls -la /var/lib/docker/volumes/mariadb_backup/_data/