doc-verifier

Documentation Verifier

This skill guides agents performing documentation verification for cloud infrastructure. Documentation drift -- the gap between what docs say and what the system does -- is the primary failure mode for operations manuals. The doc-verifier detects drift using NASA's TAID (Test, Analysis, Inspection, Demonstration) verification methods applied to documentation artifacts.

Documentation drift is insidious because it compounds silently. A single undetected drift item erodes operator trust. Once operators stop trusting documentation, they stop reading it and start improvising. The doc-verifier breaks this cycle by making verification systematic and repeatable.

Drift Detection Methods

Four categories of documentation drift exist, each with distinct detection strategies.

Configuration Drift

Definition: Document says setting X=A but the running system has X=B.

Detection strategy:

1. Parse documented configuration values from procedures (grep for =, :, or YAML key-value patterns)
2. Read actual configuration files (/etc/kolla/*/ for Kolla-Ansible deployments, globals.yml)
3. Query API for runtime configuration where available (openstack configuration show)
4. Compare documented values against actual values
5. Flag mismatches with severity based on operational impact

Common sources: Kolla-Ansible globals.yml changes, service reconfiguration, security hardening, upgrades.

Detection patterns:

• OPS-KEYSTONE-*: Compare documented Fernet key rotation interval against keystone.conf [fernet_tokens]
• OPS-NOVA-*: Compare documented compute driver against nova.conf [DEFAULT] compute_driver
• OPS-NEUTRON-*: Compare documented ML2 plugin against ml2_conf.ini [ml2] mechanism_drivers

Endpoint Drift

Definition: Document references a URL, port, or API endpoint that has changed or is unreachable.

Detection strategy:

1. Extract documented endpoints (URLs, host:port patterns, API paths)
2. Verify reachability using non-destructive probes (curl -s -o /dev/null -w "%{http_code}")
3. Verify service catalog entries match documented endpoints (openstack endpoint list)
4. Check port availability where direct HTTP probes are inappropriate

Common sources: TLS migration (HTTP to HTTPS), port changes, VIP address changes, service catalog updates.

Procedure Drift

Definition: Documented steps no longer produce the documented results.

Detection strategy:

1. Parse procedure steps to extract commands and expected output
2. Execute read-only commands against the running system
3. Compare actual output to documented expected output
4. For destructive commands, verify preconditions and rollback sections exist without executing

Common sources: CLI argument changes across OpenStack releases, output format changes, deprecated subcommands, renamed resources.

Detection patterns:

• RB-KEYSTONE-*: Run openstack token issue and compare output fields to documented fields
• RB-NOVA-*: Run openstack server list and verify column headers match documentation
• RB-NEUTRON-*: Run openstack network list and verify output structure

Version Drift

Definition: Document references a software version that has been upgraded or changed.

Detection strategy:

1. Extract version references from documentation (OpenStack release names, package versions, container image tags)
2. Compare against installed versions (openstack versions show, rpm -qa | grep openstack, docker image ls)
3. Check Kolla-Ansible container image tags against documented tags
4. Flag version mismatches with assessment of whether procedures are still valid for the new version

Common sources: OpenStack minor releases, Kolla container rebuilds, base OS updates, Python package updates.

Verification Methods (TAID Applied to Documentation)

NASA SP-6105 SS 5.3 defines four verification methods. Applied to documentation verification:

Test

Execute documented commands and compare output to documented expected results.

Approach:

• For each procedure, extract commands (lines starting with $, #, or indented code blocks)
• Extract expected output (lines following "Expected:" or "Output:" or indented below commands)
• Execute the command on the running system
• Compare actual output to expected output using the appropriate comparison strategy

Comparison strategies:

• Exact match: Deterministic output (version strings, configuration values, boolean results)
• Pattern match: Dynamic values using regex (timestamps, UUIDs, request IDs, IP addresses)
• Presence check: Structural elements that must exist but whose values vary (column headers, section markers)
• Subset check: Output must contain documented elements but may have additional items

Analysis

Compare documented configuration values against running configuration without executing commands.

Approach:

• Parse documented config values from procedures and operations manual
• Read actual configuration files from disk or API
• Compute the diff between documented and actual values
• Assess whether differences affect procedure validity

Targets: globals.yml, service .conf files, policy files, network configurations, certificate parameters.

Inspection

Review documentation structure for completeness against the required format.

**Checklist for operations procedures (OPS-* format):**

• [ ] Procedure ID present and matches OPS-{SERVICE}-{NNN} pattern
• [ ] Title clearly states what the procedure accomplishes
• [ ] SE Phase reference present
• [ ] NPR reference present
• [ ] Last verified date and system version present
• [ ] Purpose section exists
• [ ] Preconditions section exists and lists system state requirements
• [ ] Safety considerations section exists
• [ ] Procedure section has numbered steps with expected results
• [ ] Verification section exists with confirmation steps
• [ ] Rollback section exists with undo steps
• [ ] References section exists

**Checklist for runbooks (RB-* format):**

• [ ] Runbook ID present and matches RB-{SERVICE}-{NNN} pattern
• [ ] SE Phase Reference present
• [ ] Last Verified Against date and version present
• [ ] Preconditions section exists
• [ ] Procedure section has numbered steps with expected output
• [ ] Each step has "If not" guidance for unexpected results
• [ ] Verification section exists
• [ ] Rollback section exists
• [ ] Related Runbooks section exists

Demonstration

Walk through an end-to-end documented scenario and confirm each step works as described.

Approach:

• Select a complete operational scenario (e.g., "Create project, configure network, launch instance")
• Follow the documented procedure exactly as written
• Record actual results at each step
• Compare end state to documented end state
• Document any deviations with severity assessment

Demonstration scenarios (non-destructive):

• Authentication flow: openstack token issue through documented steps
• Resource listing: openstack server list, openstack network list, openstack volume list
• Service health: openstack compute service list, openstack network agent list

Automated Verification Framework

Script Structure

For each documented procedure, build a verification script:

INPUT:  Procedure document (Markdown)
PARSE:  Extract (command, expected_output) pairs
FILTER: Remove destructive commands (create, delete, set, update, migrate)
RUN:    Execute read-only commands against running system
COMPARE: Actual vs documented using appropriate strategy
REPORT: Per-procedure pass/fail with specific drift items

Drift Severity Classification

| Severity | Definition | Example | |----------|-----------|---------| | Critical | Procedure will fail if followed as written | Documented command uses deprecated flag; documented port is wrong | | Warning | Procedure works but output differs from documentation | Column headers changed; additi