doc-claim-validator

Documentation Claim Validator

Verify that what documentation says is actually true by extracting testable claims and checking them against the codebase. Complements doc-maintenance (which handles structural health) by handling semantic accuracy.

When to Use

• After significant code changes (refactors, renames, API changes)
• Before releases — catch docs that describe removed or changed behavior
• When onboarding devs report "the docs are wrong"
• As a periodic trust audit on project documentation
• After running doc-maintenance to go deeper than structural checks

Quick Reference

| Resource | Purpose | Load when | |----------|---------|-----------| | scripts/extract_claims.py | Deterministic claim extraction from markdown | Always (Phase 1) | | scripts/verify_claims.py | Automated verification against codebase | Always (Phase 2) | | references/claim-taxonomy.md | Full taxonomy of claim types with examples | Triaging unclear claims |

Workflow Overview

Phase 1: Extract    → Pull verifiable claims from docs (deterministic script)
Phase 2: Verify     → Check claims against codebase (automated + AI)
Phase 3: Report     → Classify failures by severity and type
Phase 4: Remediate  → Fix or flag broken claims

Phase 1: Extract Claims

Run the extraction script to parse all markdown files and pull out verifiable assertions:

python3 skills/doc-claim-validator/scripts/extract_claims.py [--json] [--root PATH] [--scope docs|manual|all]

The script extracts these claim types from markdown:

| Type | What it captures | Example in docs | |------|-----------------|-----------------| | file_path | Inline code matching file path patterns | ` src/auth/login.ts | | command | Code blocks or inline code with shell commands | npm run build | | code_ref | Function, class, method references in inline code | authenticate() | | import | Import/require statements in code blocks | import { Router } from 'express' | | config | Configuration keys, env vars, settings | MAX_RETRIES=3 | | url | External links (http/https) | docs | | architectural | Verb-anchored prose claims about technology, integrations, or architectural patterns | "Uses Redis for caching", "follows the actor model", "delegated to Auth0" | | dependency | Package/library name claims | "Uses Redis for caching" | | behavioral` | Assertions about what code does | "The system retries 3 times" |

The first 7 types are extracted deterministically. The script uses verb-anchored regex for architectural (rules like uses X, built with X, follows the X pattern, delegated to X, via X, depends on X) — this catches anchorless prose claims that previously slipped through.

The last 2 (dependency, behavioral) require AI analysis and are handled in Phase 2. behavioral in particular is not regex-extracted because behavioral claims are free-form prose ("the cache invalidates when the user logs out") that doesn't pattern-match cleanly — the behavioral verifier discovers and verifies them in one pass.

Output: A structured list of claims with source file, line number, claim type, and the literal text of the claim.

Phase 2: Verify Claims

Step 2a — Automated verification

Run the verification script on the extracted claims:

python3 skills/doc-claim-validator/scripts/verify_claims.py [--json] [--root PATH] [--claims-file PATH] [--check-staleness]

Pass --check-staleness to enable git-based drift analysis (see below).

The script checks each claim type differently:

| Claim type | Verification method | Pass condition | |------------|-------------------|---------------| | file_path | os.path.exists() | File exists at referenced path | | command | shutil.which() + script check | Binary exists or script file exists | | code_ref | grep -r for function/class name | Symbol found in codebase | | import | Check module exists in project or deps | Module resolvable | | config | Grep for config key in source | Key found in config files or code | | url | HTTP HEAD request (optional, off by default) | Returns 2xx/3xx |

Pass --check-urls to enable URL verification (slow, requires network).

Step 2b — AI-assisted verification

After the automated pass, dispatch agents to verify claims the script cannot. Three of four verifiers run on general-purpose + sonnet — behavioral, architectural, and code-example verification all require multi-file reasoning that haiku's excerpt-read pattern strains under. The dependency verifier stays on Explore + haiku because it's pure pattern matching against manifest files.

Dispatch strategy: per-docfile batching

For behavioral and architectural verifiers, dispatch one sonnet call per markdown file containing claims of that type, with all claims from that file batched into a single prompt. This keeps each call's context budget on a small number of related claims (cross-referencing within the doc improves verification) while keeping total call count tied to doc-set size rather than claim count. For a project with ~50 docs and ~150 architectural claims, expect ~10–20 sonnet calls (only docs with claims trigger calls), not 150.

For release audits where precision matters more than cost, run with per-claim dispatch — one sonnet call per claim, each with the full doc as context. Higher cost, higher precision.

Verifiers

Verifier 1 — Dependency claim verifier (subagent_type: "Explore", model: "haiku"): Read package.json, requirements.txt, go.mod, Cargo.toml, or equivalent dependency manifests. Cross-reference any doc claims about libraries, frameworks, or services used. Report claims that reference dependencies not in the project. Stays on haiku because pattern-matching against manifests doesn't benefit from sonnet's reasoning.

Verifier 2 — Behavioral claim verifier (subagent_type: "general-purpose", model: "sonnet", per-docfile): For each markdown file in scope, dispatch a sonnet agent with the file content. The agent (a) discovers behavioral claims in the file ("retries 3 times", "caches for 5 minutes", "validates input before processing", "the cache invalidates when the user logs out"), (b) finds the relevant code via grep / codanna / Read, (c) verifies whether the claim matches the implementation. Report each claim with confirmed / contradicted / unverifiable / conditional status. Sonnet is needed because behavioral verification often requires tracing across multiple files (handler → middleware → config) and distinguishing happy-path from error-path behavior.

Verifier 3 — Architectural claim verifier (subagent_type: "general-purpose", model: "sonnet", per-docfile): For each markdown file with extracted architectural claims, dispatch a sonnet agent with the file content and the list of pre-extracted claims. The agent verifies each claim by:

• For uses/built/depends/via frames: check the named technology in

dependency manifests, config files, and source imports.

• For delegated frames: check for SDK imports or HTTP integrations matching

the named service.

• For follows/uses_pattern frames: check directory structure, class names,

and code organization for the named architectural pattern (e.g., CQRS: separate command/query handlers + event store; hexagonal: adapters/ports dirs; saga: orchestrator class with named transitions).

Report each claim with confirmed / contradicted / unverifiable / conditional status. Sonnet is needed because architectural patterns aren't 1:1 with any single file — verification requires reading enough of the codebase to recognize the pattern.

Verifier 4 — Code example verifier (subagent_type: "general-purpose", model: "sonnet", per-docfile): For code blocks in docs that show usage examples, verify the function signatures, parameter names, return types, and import paths match t