openstack-monitoring

OpenStack Monitoring Operations

Monitoring is the nervous system of a cloud deployment. Without it, operators fly blind -- problems are discovered by users instead of by automation. The monitoring stack provides continuous visibility into service health, resource utilization, and performance trends. When an operator asks "is the cloud healthy?" the answer comes from monitoring, not from guesswork.

The stack has three pillars: Prometheus collects and stores metrics (time-series data from every service, node, and container), Grafana visualizes those metrics (dashboards per service, alerting overview, capacity trends), and Alertmanager routes notifications when metrics cross thresholds (email, webhook, PagerDuty). Together they form a closed loop: collect, visualize, alert, act.

In NASA SE terms, monitoring maps to Phase E (Operations & Sustainment) -- continuous technical assessment through operational health monitoring, SLA tracking, and performance baseline comparison. The SURGEON agent is the primary consumer of this skill, using monitoring data for cloud health assessment. The GUARD agent consumes security-related metrics for posture evaluation.

Deploy

Kolla-Ansible Monitoring Deployment

globals.yml settings:

# Enable monitoring stack
enable_prometheus: "yes"
enable_grafana: "yes"

# Optional: enable centralized logging
enable_central_logging: "yes"
enable_fluentd: "yes"

# Grafana admin credentials
grafana_admin_password: "{{ vault_grafana_admin_password }}"

Deployment:

# Deploy monitoring alongside other services
kolla-ansible -i inventory deploy --tags prometheus,grafana

# Or as part of full deployment
kolla-ansible -i inventory deploy

Container verification:

# Verify all monitoring containers are running
docker ps --filter "name=prometheus" --format "table {{.Names}}\t{{.Status}}"
docker ps --filter "name=grafana" --format "table {{.Names}}\t{{.Status}}"
docker ps --filter "name=alertmanager" --format "table {{.Names}}\t{{.Status}}"
docker ps --filter "name=node_exporter" --format "table {{.Names}}\t{{.Status}}"
# Expected: prometheus_server, grafana, alertmanager, node_exporter (all Up)

# Verify Prometheus is scraping targets
curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | head -20
# All targets should show "health": "up"

Prometheus target configuration:

Kolla-Ansible auto-configures scrape targets for deployed services. Verify the target list:

# Check active targets
curl -s http://localhost:9090/api/v1/targets | python3 -c "
import sys, json
data = json.load(sys.stdin)
for group in data['data']['activeTargets']:
    print(f\"{group['labels'].get('job','unknown'):30s} {group['health']:6s} {group['lastScrape']}\")
"

Grafana data source setup:

Kolla-Ansible registers Prometheus as the default Grafana data source. Verify:

# Check Grafana health
curl -s http://localhost:3000/api/health
# Expected: {"commit":"...","database":"ok","version":"..."}

# Check data source
curl -s -u admin:${GRAFANA_PASSWORD} http://localhost:3000/api/datasources
# Should list Prometheus data source

Initial dashboard import:

Kolla-Ansible ships default dashboards. For additional dashboards, use Grafana provisioning:

# Dashboards are stored in /etc/kolla/grafana/provisioning/dashboards/
# Add custom dashboards as JSON files to this directory
# Grafana auto-loads on container restart
docker restart grafana

Configure

Prometheus Scrape Targets

Configure scrape jobs for all OpenStack service endpoints and infrastructure components:

OpenStack API endpoints (per-service):

| Service | Exporter/Endpoint | Default Port | Metrics | |---------|-------------------|--------------|---------| | Keystone | keystone_exporter or /healthcheck | 5000 | Auth latency, token issuance rate | | Nova | nova_exporter or API /os-services | 8774 | Instance count, hypervisor stats | | Neutron | neutron_exporter or API /v2.0/agents | 9696 | Agent status, network/subnet counts | | Cinder | cinder_exporter or API /os-services | 8776 | Volume count, snapshot stats | | Glance | API /healthcheck | 9292 | Image count, upload throughput | | Swift | swift_exporter | 8080 | Object count, container stats | | Heat | API /healthcheck | 8004 | Stack count, operation success rate | | Horizon | HTTP probe | 443 | Dashboard availability, response time |

Infrastructure exporters:

| Component | Exporter | Default Port | Key Metrics | |-----------|----------|--------------|-------------| | HAProxy | haproxy_exporter | 9101 | Backend status, request rate, error rate | | MariaDB | mysqld_exporter | 9104 | Connections, query rate, replication lag | | RabbitMQ | rabbitmq_exporter | 9419 | Queue depth, message rate, consumer count | | Node | node_exporter | 9100 | CPU, memory, disk, network per host | | Container | cAdvisor | 8080 | Container CPU/memory/network per service |

Grafana Dashboard Provisioning

Organize dashboards by service domain:

Per-service dashboards:

• Nova Compute: Hypervisor utilization, instance lifecycle states, API latency, scheduler placement
• Neutron Networking: Agent health, network/port counts, floating IP utilization, bandwidth
• Cinder Storage: Volume provisioned/used, snapshot counts, IOPS, backend capacity
• Keystone Auth: Token issuance rate, auth latency, failed auth attempts, active sessions

Infrastructure dashboards:

• Node Overview: CPU, memory, disk I/O, network throughput per host
• HAProxy: Backend health, active connections, error rates, response codes
• MariaDB: Query rate, slow queries, connections, buffer pool usage
• RabbitMQ: Queue depth, message throughput, consumer status, memory usage

Alertmanager Configuration

Notification channels:

# /etc/kolla/prometheus/alertmanager.yml
route:
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
    - match:
        severity: warning
      receiver: 'email-ops'

receivers:
  - name: 'default'
    email_configs:
      - to: 'ops-team@example.com'
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: '<pagerduty-integration-key>'
  - name: 'email-ops'
    email_configs:
      - to: 'ops-team@example.com'
        send_resolved: true

Alert Rule Definitions

Service health alerts:

# /etc/kolla/prometheus/rules/service-health.yml
groups:
  - name: openstack_services
    rules:
      - alert: ServiceDown
        expr: up{job=~"openstack-.*"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} is down"
          description: "{{ $labels.instance }} has been unreachable for 2 minutes"

      - alert: HighAPILatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High API latency on {{ $labels.job }}"

      - alert: ResourceExhaustion
        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.10
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Disk space below 10% on {{ $labels.instance }}"

      - alert: CertificateExpiringSoon
        expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "TLS certificate expires within 30 days on {{ $labels.instance }}"

Retention and Storage Sizing

Prometheus retention:

# prometheus.yml or command-line flags
--storage.tsdb.retention.time=30d    # Keep 30 days of data
--storage.tsdb.retention.size=50GB   # Or cap at 50GB, whichever hits first

**Storage esti