openstack-keystone

OpenStack Keystone Identity Service

Keystone is the identity service for OpenStack. Every API call to every OpenStack service passes through Keystone for authentication and authorization. It is the first service deployed and the last decommissioned. If Keystone is down, the entire cloud is down.

Keystone provides five core functions: identity (users, groups), resources (projects, domains), assignment (roles mapped to users on projects), token (authentication proof with configurable lifetime), and catalog (service endpoint registry). Understanding Keystone means understanding how every OpenStack service discovers and trusts every other service.

Deploy

Kolla-Ansible Configuration

globals.yml settings:

# Required -- set strong passwords
keystone_admin_password: "{{ vault_keystone_admin_password }}"
keystone_database_password: "{{ vault_keystone_database_password }}"

# TLS (recommended for production)
kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "yes"
kolla_copy_ca_into_containers: "yes"

# Token provider (Fernet is default and recommended)
keystone_token_provider: "fernet"

# Optional tuning
keystone_token_expiration: 3600  # seconds, default 1 hour

Deployment sequence:

# 1. Bootstrap -- creates databases, service users, initial endpoints
kolla-ansible -i inventory bootstrap-servers

# 2. Deploy Keystone (runs as part of full deploy or targeted)
kolla-ansible -i inventory deploy --tags keystone

# 3. Post-deploy -- creates admin credentials file
kolla-ansible -i inventory post-deploy

Container verification:

# Verify Keystone containers are running
docker ps --filter "name=keystone" --format "table {{.Names}}\t{{.Status}}"
# Expected: keystone_api, keystone_fernet (both Up)

# Verify service responds
source /etc/kolla/admin-openrc.sh
openstack token issue  # Must return a valid token table

Service catalog registration:

Kolla-Ansible auto-registers Keystone in the service catalog. Verify:

openstack service list  # Should show "identity" service
openstack endpoint list --service keystone
# Expected: 3 endpoints (public, internal, admin) per region

Initial admin setup:

# admin-openrc.sh is generated by post-deploy
source /etc/kolla/admin-openrc.sh

# Verify admin project and user exist
openstack project show admin
openstack user show admin
openstack role assignment list --user admin --project admin
# admin should have "admin" role on "admin" project

Configure

Fernet Token Provider

Fernet tokens are cryptographic tokens validated without database lookup. They require synchronized key repositories across Keystone nodes.

Key repository: /etc/kolla/keystone/fernet-keys/ (inside container)

Key types:

• Staging key (index 0): Used for decryption only; becomes primary on rotation
• Primary key (highest index): Used for encryption; tokens it created remain valid
• Secondary keys (middle indices): Decrypt only; pruned after max_active_keys exceeded

Rotation procedure:

# Rotate Fernet keys (Kolla-Ansible managed)
kolla-ansible -i inventory keystone_fernet_rotate

# Manual rotation (if needed)
docker exec keystone_api keystone-manage fernet_rotate \
  --keystone-user keystone --keystone-group keystone

Configuration: keystone.conf [fernet_tokens]

• max_active_keys: Default 3. Set to (token_expiration / rotation_interval) + 2

Domain, Project, and User Hierarchy

Domain (organizational boundary)
  +-- Project (resource container, formerly "tenant")
       +-- User (identity with credentials)
       +-- Group (collection of users)
       +-- Role Assignment (user/group + role on this project)

Best practices:

• Create a domain per organization or department
• Use the "default" domain for service accounts
• Never modify the "admin" project; create separate operator projects
• Assign roles to groups, not individual users, for scalability

RBAC Policies

Policy files: /etc/kolla/keystone/policy.yaml (override defaults)

# Example: restrict user creation to domain-scoped admins
"identity:create_user": "rule:admin_required and domain_id:%(target.user.domain_id)s"

# Example: allow project members to list users in their project
"identity:list_users": "role:member and project_id:%(scope.project.id)s"

Policy evaluation order:

1. Check explicit rule match in policy.yaml
2. Fall back to code default (oslo.policy)
3. Deny if no rule matches

Verify policy changes:

# Test a specific policy rule
openstack --os-auth-url http://keystone:5000/v3 \
  --os-username testuser --os-password testpass \
  --os-project-name testproject --os-user-domain-name default \
  --os-project-domain-name default user list

Service Catalog Management

# List all registered services and endpoints
openstack service list
openstack endpoint list

# Register a new service
openstack service create --name nova --description "Compute" compute
openstack endpoint create --region RegionOne \
  compute public http://controller:8774/v2.1
openstack endpoint create --region RegionOne \
  compute internal http://controller:8774/v2.1
openstack endpoint create --region RegionOne \
  compute admin http://controller:8774/v2.1

Federation (SAML/OIDC)

Federation allows external identity providers to authenticate OpenStack users.

Key concepts:

• Identity Provider (IdP): External auth source (AD, Okta, Keycloak)
• Mapping: Rules that translate IdP assertions to Keystone users/groups/projects
• Protocol: SAML 2.0 or OpenID Connect
• Service Provider (SP): Keystone acting as relying party

Configuration reference: keystone.conf [federation], [saml], [openid]

Credential Encryption

# Initialize credential encryption (first time)
docker exec keystone_api keystone-manage credential_setup \
  --keystone-user keystone --keystone-group keystone

# Rotate credential encryption keys
docker exec keystone_api keystone-manage credential_rotate \
  --keystone-user keystone --keystone-group keystone

# Migrate credentials to new encryption
docker exec keystone_api keystone-manage credential_migrate \
  --keystone-user keystone --keystone-group keystone

Operate

User and Project Management

# Create domain
openstack domain create --description "Engineering" engineering

# Create project in domain
openstack project create --domain engineering --description "Dev team" dev-team

# Create user in domain
openstack user create --domain engineering --password-prompt \
  --email dev@example.com devuser

# Assign role
openstack role add --project dev-team --user devuser --user-domain engineering member

# List role assignments
openstack role assignment list --project dev-team --names

Token Debugging

# Issue a token and inspect it
openstack token issue -f json

# Check token expiration
openstack token issue -c expires -f value
# Compare with system clock -- clock skew causes premature expiration

# Revoke a specific token
openstack token revoke <token-id>

Fernet Key Rotation Procedure

Schedule: Rotate at half the token expiration interval. Default token lifetime is 1 hour, so rotate every 30 minutes in production.

# 1. Check current key count
docker exec keystone_api ls /etc/kolla/keystone/fernet-keys/ | wc -l

# 2. Rotate keys
kolla-ansible -i inventory keystone_fernet_rotate

# 3. Verify rotation
docker exec keystone_api ls -la /etc/kolla/keystone/fernet-keys/
# New highest-numbered key should be primary

# 4. Verify tokens still work
openstack token issue

Backup and Restore

# Backup Keystone database
docker exec mariadb mysqldump -u root -p keystone > keystone_backup.sql

# Backup Fernet keys
docker cp keystone_api:/etc/kolla/keystone/fernet-keys/ ./fernet-keys-backup/

# Restore database
docker exec -i mariadb mysql -u root -p keystone < keystone_backu