recipe-quality-audit

recipe-quality-audit

Audit a single NVCA recipe's quality and produce a structured scorecard.

Security model

• This skill operates on local repository files only — no network access required.
• Source document downloads (Tier 2 checks) use ensureSourceDocx() which fetches from known template source URLs only.
• No credentials or API keys are needed.

Usage

Run the audit for a specific recipe:

Audit the recipe: nvca-certificate-of-incorporation

Or audit all recipes:

Audit all NVCA recipes and update the quality tracker

Checks

Tier 1: Structural (no source download needed)

| # | Check | How | |---|-------|-----| | S1 | File inventory | Does recipe have metadata.yaml, replacements.json, clean.json? Optional: computed.json, normalize.json, selections.json | | S2 | Metadata schema valid | Run existing validateRecipeMetadata() from src/core/metadata.ts | | S3 | Field-to-replacement coverage | For each field in metadata, is there a replacement key referencing {field_name}? | | S4 | Ambiguous keys | Flag replacement keys < 8 chars without context qualifier (e.g., [name], [its]) | | S5 | Smart quote coverage | Keys with apostrophes should have smart-quote variants (or patcher normalizes — check patcher has normalizeQuotes) | | S6 | Source SHA present | source_sha256 in metadata.yaml | | S7 | Test fixture exists | integration-tests/fixtures/{recipe-id}-*.json exists |

Tier 2: Behavioral (requires source download)

| # | Check | How | |---|-------|-----| | B1 | Source download + scan | Count all \[[_A-Z] prefixed bracket patterns in source (underscore-fill or capitalized placeholders) | | B2 | Replacement coverage ratio | (keys with match in source) / (total bracket patterns). Target: >80% | | B3 | Unmatched underscore patterns | [___+] patterns in source not in replacements.json | | B4 | Clean effectiveness | After clean, no footnotes, no "Note to Drafter", no preamble |

Tier 3: Fill quality (requires fill run)

| # | Check | How | |---|-------|-----| | F1 | Default-only fill | Fill with defaults, run verifyOutput, count blank placeholders | | F2 | Full-values fill | Fill with all fields from test fixture, assert all verify checks pass | | F3 | Formatting anomaly count | Check 8 from verifier (single-char underlined runs) | | F4 | Zero-match replacement keys | Keys that existed in replacements.json but matched nothing in the source |

Output: Quality Scorecard

{
  "recipe_id": "nvca-voting-agreement",
  "maturity": "beta",
  "scores": { "structural": "6/7", "behavioral": "3/4", "fill": "0/4", "total": "9/15" },
  "checks": [
    { "id": "S1", "name": "File inventory", "passed": true, "details": "metadata.yaml, replacements.json, clean.json present" },
    { "id": "S7", "name": "Test fixture exists", "passed": false, "details": "No fixture matching integration-tests/fixtures/nvca-voting-agreement-*.json" }
  ],
  "field_coverage": { "metadata_fields": 14, "replacement_refs": 10, "uncovered": 4 },
  "recommendations": [
    "Add test fixture for fill testing (S7)",
    "Add replacement keys for 4 uncovered fields (S3)"
  ]
}

Maturity Tiers

• scaffold: metadata-only, can't fill
• beta: has replacements + clean, score < 11/15 OR no test fixture
• production: score >= 11/15 AND has test fixture AND has computed.json (if conditional sections exist in source)

Workflow

When running the audit:

1. Read content/recipes/QUALITY_TRACKER.md for current state
2. Run Tier 1 checks (always possible)
3. Run Tier 2 checks if source document is available (use ensureSourceDocx())
4. Run Tier 3 checks if a test fixture exists
5. Compute scorecard and maturity tier
6. Output the scorecard as JSON
7. Update the quality tracker if requested

Implementation Notes

• Use validateRecipeMetadata() from src/core/metadata.ts for S2
• Use ensureSourceDocx() from src/core/recipe/downloader.ts for B1-B4
• Use runRecipe() from src/core/recipe/index.ts for F1-F4
• Bracket pattern detection uses \[[_A-Z] prefix to avoid counting citations and legal references
• Zero-match keys come from PatchResult.zeroMatchKeys returned by the patcher
• Cross-reference zero-match keys with cleanConfig.removeRanges and cleanConfig.removeParagraphPatterns to suppress expected zero-matches