testing-guide

Testing Guide

What to test, how to test it, and what NOT to test — for a plugin made of prompt files, Python glue, and configuration.

Philosophy: GenAI-First Testing

Traditional unit tests work for deterministic logic. But most bugs in this project are drift — docs diverge from code, agents contradict commands, component counts go stale. GenAI congruence tests catch these. Unit tests don't.

Decision rule: Can you write assert x == y and it won't break next week? → Unit test. Otherwise → GenAI test or structural test.

Three Test Patterns

1. Judge Pattern (single artifact evaluation)

An LLM evaluates one artifact against criteria. Use for: doc completeness, security posture, architectural intent.

pytestmark = [pytest.mark.genai]

def test_agents_documented_in_claude_md(self, genai):
    agents_on_disk = list_agents()
    claude_md = Path("CLAUDE.md").read_text()
    result = genai.judge(
        question="Does CLAUDE.md document all active agents?",
        context=f"Agents on disk: {agents_on_disk}\nCLAUDE.md:\n{claude_md[:3000]}",
        criteria="All active agents should be referenced. Score by coverage %."
    )
    assert result["score"] >= 5, f"Gap: {result['reasoning']}"

2. Congruence Pattern (two-source cross-reference)

The most valuable pattern. An LLM checks two files that should agree. Use for: command↔agent alignment, FORBIDDEN lists, config↔reality.

def test_implement_and_implementer_share_forbidden_list(self, genai):
    implement = Path("commands/implement.md").read_text()
    implementer = Path("agents/implementer.md").read_text()
    result = genai.judge(
        question="Do these files have matching FORBIDDEN behavior lists?",
        context=f"implement.md:\n{implement[:5000]}\nimplementer.md:\n{implementer[:5000]}",
        criteria="Both should define same enforcement gates. Score 10=identical, 0=contradictory."
    )
    assert result["score"] >= 5

Analytic Rubric Pattern (decomposed per-criterion evaluation)

More reliable than holistic scoring. Each criterion is evaluated independently with a binary MET/UNMET judgment. Use for: security posture, enforcement quality, multi-faceted assessments.

def test_security_posture_analytic(self, genai):
    result = genai.judge_analytic(
        question="Evaluate the security posture of this codebase",
        context=f"Hook samples:\n{hook_content[:5000]}",
        criteria=[
            {"name": "No hardcoded secrets", "description": "No real API keys or tokens in source", "max_points": 1},
            {"name": "Named exit codes", "description": "Hooks use named constants, not bare numbers", "max_points": 1},
            {"name": "Path validation", "description": "File operations validate paths", "max_points": 1},
        ],
    )
    assert result["total_score"] >= 2, f"{result['total_score']}/{result['max_score']}: {result['reasoning']}"

Return value: {"criteria_results": [...], "total_score": N, "max_score": N, "pass": bool, "band": str, "reasoning": str}

When to use: Multi-faceted evaluations where you need to know which specific criteria passed or failed. Each criterion gets its own LLM call for independent judgment.

Consistency Check Pattern (multi-round agreement)

For high-stakes judgments where a single LLM evaluation might be unreliable. Runs multiple rounds and checks for agreement. Uses median score as the final result.

def test_pipeline_completeness_consistent(self, genai):
    result = genai.judge_consistent(
        question="Does implement.md define a complete SDLC pipeline?",
        context=f"implement.md:\n{content[:6000]}",
        criteria="Pipeline should have research, plan, test, implement, review, security, docs steps.",
        rounds=3,
    )
    assert result["final_score"] >= 7, f"median={result['final_score']}, agreement={result['agreement']}"

Return value: {"rounds": [...], "agreement": bool, "scores": [...], "final_score": median, "pass": bool, "band": str, "reasoning": str}

When to use: Critical assessments where false positives/negatives are costly. Agreement=False signals the evaluation needs human review.

Temperature Guidance

All ask() calls default to temperature=0 for deterministic, reproducible judging. Override only when you need creative/diverse outputs:

# Default: temperature=0 (deterministic judging)
response = genai.ask("Evaluate this code", temperature=0)

# Override for creative tasks like edge case generation
response = genai.ask("Generate unusual test inputs", temperature=0.7)

3. Cross-Validation Pattern (two sources that must match)

No LLM needed. When two configs/files must stay in sync, read both and compare directly. Catches the #1 recurring bug class: adding something to one place but not the other.

def test_policy_and_hook_in_sync(self):
    """Policy always_allowed and hook NATIVE_TOOLS must be identical."""
    policy_tools = set(json.load(open(POLICY_FILE))["tools"]["always_allowed"])
    hook_tools = hook.NATIVE_TOOLS
    # Check BOTH directions
    assert policy_tools - hook_tools == set(), f"In policy not hook: {policy_tools - hook_tools}"
    assert hook_tools - policy_tools == set(), f"In hook not policy: {hook_tools - policy_tools}"

When to use: Any time two files define overlapping data — permissions↔hook, manifest↔disk, config↔worktree copy, command frontmatter↔policy. Key principle: Read both sources dynamically. Never hardcode expected values in the test itself.

4. Structural Pattern (dynamic filesystem discovery)

No LLM needed. Discover components dynamically and assert structural properties. Use for: component existence, manifest sync, skill loading.

def test_all_active_skills_have_content(self):
    skills_dir = Path("plugins/autonomous-dev/skills")
    for skill in skills_dir.iterdir():
        if skill.name == "archived" or not skill.is_dir():
            continue
        skill_md = skill / "SKILL.md"
        assert skill_md.exists(), f"Skill {skill.name} missing SKILL.md"
        assert len(skill_md.read_text()) > 100, f"Skill {skill.name} is a hollow shell"

5. Property-Based Pattern (hypothesis invariants)

Define properties that must always hold, instead of testing specific examples. Catches 23-37% more bugs than example-based tests. Use for: pure functions, serialization, data transformations, parsers.

from hypothesis import given, strategies as st

@given(st.lists(st.integers()))
def test_sort_preserves_elements(arr):
    """Invariant: sorting never loses or adds elements."""
    result = sorted(arr)
    assert set(result) == set(arr)
    assert len(result) == len(arr)

@given(st.dictionaries(st.text(min_size=1), st.text()))
def test_config_roundtrip(config):
    """Invariant: serialize → deserialize = identity."""
    assert json.loads(json.dumps(config)) == config

When to use: Pure functions, roundtrips, idempotent operations, parsers. When NOT to use: Agent prompts (use GenAI judge), filesystem checks (use structural).

PBT Candidate Selection — When to Use Property-Based Testing

Good candidates (pure functions with testable invariants):

• Input validation functions (validate_agent_name, validate_message, sanitize_*)
• Scoring/normalization functions (normalize_severity, compute_priority)
• Serialize/deserialize roundtrips (settings fix-then-validate, JSON encode/decode)
• Set operations and state machines (circuit breaker threshold, denial counts)
• Mathematical functions with known identities (Fibonacci recurrence)
• Parsers with structural guarantees (acceptance criteria extraction)

Bad candidates (avoid PBT for these):

• Agent prompt behavior — use GenAI judge tests instead; Hypothesis cannot meaningfully generate LLM prompts
• File system operations — use structural tests with tmp_path; filesystem side effects break Hypothesis shrinking
• Config values