nw-tdd-methodology

Outside-In TDD Methodology

TDD cycle — 3-phase canonical (ADR-025, 2026-05-07)

Current canonical: DELIVER cycle is 3-phase: RED → GREEN → COMMIT.

• RED absorbs PREPARE + RED_ACCEPTANCE + RED_UNIT (legacy 5-phase). Writes PBT unit tests targeting production code; unskips the corresponding AT scenario authored upstream by DISTILL. Exits via the fail-for-right-reason gate — both PBT unit + AT must fail with a semantically-correct error (AssertionError / expected-exception-not-thrown), not a collection error / import error / skip marker. The gate preserves RED→GREEN discipline atomically without separate phase boundaries.
• GREEN: implement minimum production code making PBT unit + AT pass. Exit gate all-tests-pass.
• COMMIT: stage + commit with Step-Id: trailer. Pre-commit hooks enforce style; F-DES-COMMIT-PHASE-CRAFTER-DEAD-PATH guidance in commit.yaml addresses adapter-probe annotation requirement.

DISTILL retains canonical AT authorship (per nw-distill Mandate 7). RED phase in DELIVER does NOT write acceptance scenarios from scratch — it only unskips the scaffolds DISTILL produced.

Legacy (5-phase v4 contract, ADR-024 era) — PREPARE / RED_ACCEPTANCE / RED_UNIT / GREEN / COMMIT — preserved for audit-log replay of pre-2026-05-07 commits. Future features use 3-phase canon. References to RED_ACCEPTANCE / RED_UNIT below describe the legacy contract; new work treats them as merged inside RED.

Paradigm Mandate — Property-Based + State-Delta (STANDING, 2026-05-05)

Default test-writing paradigm for UNIT + ACCEPTANCE tests — not optional, not "when applicable".

Test-level applicability matrix

| Level | Default paradigm | Rationale | |---|---|---| | Unit | Property-based + state-delta — single-example is FALLBACK only | Property tests cover equivalence classes; the state-delta universe forbids hidden mutations on adjacent slots | | Acceptance (Gherkin) | Property: framing with quantified preconditions; classic Scenario: is FALLBACK | Acceptance tests document system invariants; properties express the spec better than picked examples | | Integration | UNCHANGED — single-example test verifies WIRING | The contract is "wires connect correctly", not "all input shapes succeed". One representative call suffices | | E2E | UNCHANGED — single-example end-to-end happy path | The contract is "complete flow connects", not "all flows are equivalent". One golden walkthrough suffices |

Mandate (unit + acceptance levels)

Every unit and acceptance test you write MUST be:

1. Property-based by default — use Hypothesis @given strategies to explore equivalence classes, NOT single-fixture examples. A property test asserting an invariant over N generated inputs replaces N example tests with stronger semantic coverage.

1. State-delta over single-property assertion — capture the FULL observable state surface (universe), declare the expected delta with predicates (prepended_with, set_to, unchanged, containing, idempotent_after, legacy_healed, normalized_to, appended_with), and call assert_state_delta(before, after, universe, expected, strict=True). strict=True forbids hidden mutations on adjacent slots — this is what catches bugs that pinned-fixture asserts miss.

from hypothesis import given, settings, strategies as st
from nwave_ai.state_delta import assert_state_delta, set_to, unchanged

@given(domain_input=domain_specific_strategy())
@settings(max_examples=100, deadline=None)
def test_pbt_invariant(domain_input):
    before = capture_full_state()
    perform_action(domain_input)
    after = capture_full_state()
    assert_state_delta(
        before, after,
        universe={"slot.a", "slot.b", "slot.c", "slot.d"},
        expected={"slot.a": set_to(expected_from(domain_input)), "slot.b": unchanged()},
        strict=True,
    )

1. Acceptance tests express PROPERTIES of the system — Gherkin scenarios should be framed as Property: <invariant statement> with quantified preconditions ("a set of N tasks with arbitrary timestamps") and invariant outcomes ("monotonically descending by timestamp"), instead of single-example Scenario: blocks. Step definitions internally use @given strategies + state-delta assertions.

OLD pattern (banned by default):

Scenario: Operator sees three tasks ordered by recency
  Given tasks A, B, C with timestamps T1 < T2 < T3
  When the operator runs `prism board`
  Then the board shows: B (T2), C (T3), A (T1) — wait, ordered descending: C, B, A

NEW pattern (required default):

Property: Board column order reflects recency
  Given a set of N tasks with arbitrary timestamps
  When the operator runs `prism board`
  Then the column order is monotonically descending by timestamp
  And no task appears twice in the column
  And every input task appears exactly once in the output

Fallback — when property-framing genuinely cannot express the contract (e.g., flow-specific UI tests with puntual outcomes, golden-file diffs, error messages with exact strings):

• Write classic example-based test
• Document WHY property-framing failed in a one-line # bypass: comment
• Compensate by adding stronger PBT at unit/integration level

Forbidden bypass paths (insufficient justification):

• "Single-property is enough" — NO. Always declare the universe, even if only one slot is checked. unchanged() predicate covers the rest.
• "Mock-based interaction test" — NO. Mocks are still observable state; their call-recording surface is part of the universe (mock.call_count, mock.last_call_args).

Exempt categories (still apply paradigm where it adds value, but not mandatory):

• Pure-function tests with single output and no side effects
• Schema/AST validators with single output
• Smoke imports

Goals

| Goal | Old paradigm | New paradigm | |---|---|---| | Number of tests | N example-tests per contract | 1 PBT covers N+ examples | | Token consumption | High (N test bodies, N test names) | Low (one body, one strategy) | | Coverage | Pinned by chosen examples | Discovered via Hypothesis shrinking | | Bug-finding | Limited to imagined cases | Includes edge cases author didn't think of | | Documentation value | Examples may diverge from spec | Property = invariant = living spec | | Speed | Slower (more tests to run) | Faster (fewer tests, same coverage) |

Empirical efficacy framework — debt-payoff curve (NOT instantaneous hit rate)

Reframing (Ale 2026-05-05): paradigm efficacy is measured via the debt-payoff curve over a surface's lifetime, not via instantaneous hit rate snapshots.

Three stages

| Stage | Surface state | Expected hit rate | Meaning | |---|---|---|---| | Stage 0 | New code, paradigm-from-day-zero | N/A — debt never accumulates | Healthy by construction. Tests catch hidden mutations as they emerge | | Stage 1 | Bug-prone, debt-accumulated, never-cured | 33–75% (state-delta migration) | Debt-payoff phase. High yield = years of single-property-asserts being unmasked | | Stage 2a | Stage-1-completed surface, PBT amplification | ~0% by design | Maintenance mode. Debt is paid; PBT now catches in-flight regressions, not retroactive ones | | Stage 2b | Bug-prone, never-migrated, PBT amplification | ~75% (per hardening empirical) | Same as Stage 1 — surface still has accumulated assumptions |

Empirical evidence (2026-05-05 pilot, 2 instances)

| Pilot | Stage | Surface | Hit rate | Source | |---|---|---|---|---| | Stage 1 state-delta migration | Stage 1 | installer plugin tests, never cured | 4/9 = 44% | | Stage 2a PBT amplification | Stage 2a | plugin code post Stage 1 | 0/3 = 0% (post commit 29daeb102) |

Reading the data correctly: the master 0% is NOT failure — it's confirmation that Stage 1 already extracted the debt. Stage 2a on a cured surface validates the surface stays healthy. Stage 2b on an uncured surface re-confir