test-fix-loop

Test-Fix Loop

Autonomous test-fix iteration loop. Run the test suite, diagnose failures, apply fixes to implementation code, and re-run until all tests pass or a termination condition is met. This is a recovery mechanism for unexpected failures -- not a replacement for RED/GREEN/REFACTOR (use atdd-developer for TDD discipline).

When to Use

• After implementation produces unexpected test failures in unrelated modules
• When soleur:work GREEN phase fails and manual diagnosis is tedious
• To batch-fix multiple test failures across a codebase
• NOT for writing new tests (use atdd-developer)
• NOT for linting, type-checking, or non-test failures

Phase 0: Detect and Confirm

Detect Test Runner

Auto-detect the test command from project files in priority order:

1. CLAUDE.md -- explicit test command (highest priority)
2. package.json -- scripts.test field
3. Cargo.toml -- cargo test
4. Makefile / Justfile -- test target
5. Gemfile / Rakefile -- bundle exec rake test or bin/rails test
6. pyproject.toml -- pytest
7. go.mod -- go test ./...

If $ARGUMENTS contains a custom test command, use it instead of auto-detection. If $ARGUMENTS contains a number, use it as max iterations (default: 5). If no runner is detected, ask the user for the test command.

Require Clean Working Tree

Run git status --porcelain. If output is non-empty, STOP and tell the user to commit their changes first.

Pre-flight Confirmation

<decision_gate> API budget. Each iteration of this loop consumes one main-model turn (parse failures → cluster → fix → re-run) against the Anthropic API key in your Claude Code session. The max iterations cap (default 5, configurable via $ARGUMENTS) is the only cost ceiling — a runaway against a perpetually-flaky test command or an infinite-regression chain runs up to the cap before terminating. Soleur does not bill or proxy these calls — Anthropic does, against the key in your session. The Soleur LICENSE (BSL 1.1) disclaims warranty for runtime cost; you operate this loop against your own budget.

Show the user: detected test command, max iterations, current branch. Get one confirmation before starting the loop. This is the only approval gate -- no per-iteration approval. </decision_gate>

Phase 1: Test-Fix Loop

Record the current commit SHA as <initial-sha> before entering the loop. This is the rollback target if the loop terminates on failure after multiple iterations.

Run the initial test suite and capture its exit code (rc) — the verdict is rc, not the parsed output. Exit with "All tests already pass. Nothing to fix." only when rc is 0. A run that exits non-zero while parsing to zero failures is not a pass: this repo's test-all.sh exits 3 when zero suites failed and at least one was terminated with a signal-shaped status, rendered [KILLED] rather than [FAIL] (see its EXIT CONTRACT block). Handle that through the Suite terminated row in §2 — never by entering the loop and never by reporting success.

For each iteration (up to max iterations):

1. Parse Failures

Extract failure summaries from test output: test name and error message only (one line each). Discard full stack traces and passing test output to minimize context consumption.

Distinguish build/compilation errors from test failures. If the suite fails to compile, treat the entire build error as a single cluster and fix the compilation issue first.

Count ^[KILLED] lines separately from failures, and record both alongside rc. A terminated suite emits no assertion output, so it parses to zero failures and produces zero clusters: "non-zero rc, nothing parseable" is a real shape, and it is unresolved coverage, not an empty failure set. Never let it fall through to "all tests pass".

2. Check Termination Conditions

Before attempting fixes, check whether to stop. Rows are checked top to bottom; the first match wins.

The count driving every delta row below is failures + killed, and a suite that is KILLED in either of the two iterations being compared is excluded from the delta rather than counted as fixed. A failures-only count is arithmetically unsafe here: a suite that FAILED in iteration N and is KILLED in N+1 lowers it, which reads as a fabricated improvement; when the same suite completes again in N+2 the count jumps back, which reads as Regression and would git reset --hard HEAD over real fixes on the strength of a signal artifact.

| Condition | Detection | Action | |-----------|-----------|--------| | Suite terminated (unresolved) | Any ^[KILLED] line in the output, or runner rc 3 | Do NOT stage, do NOT report success, do NOT reset. Re-run that suite alone; if it is KILLED again, STOP and report UNRESOLVED naming the suite | | All tests pass | rc 0 (zero failures and zero killed) | Stage fixes with git add -A, report success | | Max iterations | iteration == limit | git reset --hard <initial-sha> (revert all iterations), report | | Regression | Failure count increased vs previous iteration | git reset --hard HEAD (discard uncommitted fixes), report | | Circular fix | Failure name set matches any prior iteration | git reset --hard <initial-sha> (revert all iterations), report | | Non-convergence | Failure count unchanged for 2 consecutive iterations | git reset --hard <initial-sha> (revert all iterations), report | | Build error persists | Same compilation error after fix attempt | git reset --hard <initial-sha> (revert all iterations), report |

If a termination condition triggers, skip to the Diagnostic Report.

3. Cluster and Diagnose

Cluster failures by file or module (max 5 groups, sorted by failure count descending). If more than 5 modules fail, take the top 5 and note the skipped modules.

For each cluster, apply the diagnostic-first rule:

• Read the failing test to understand expected behavior
• Read the implementation code referenced by the error
• Identify the root cause before proposing a fix

4. Checkpoint and Fix

<critical_sequence> Commit the current working tree as a rollback checkpoint before applying fixes. Skip on iteration 1 if the tree is clean -- <initial-sha> already serves as the rollback point.

git add -A && git commit -m "test-fix-loop: checkpoint iteration N"

Apply fixes to implementation code only. NEVER modify test files, add skip annotations, delete tests, or weaken assertions.

Re-run the full test suite after applying fixes.

Evaluate the result (same failures + killed count and the same row order as §2):

• Any ^[KILLED] line or rc 3: do NOT stage, do NOT report success, do NOT reset — take the Suite terminated row
• All pass (rc 0): stage all fixes with git add -A, report success, STOP
• Failures decreased: continue to next iteration (fixes stay in working tree; the next iteration's checkpoint commits them)
• Regression: git reset --hard HEAD (discard uncommitted fixes, return to checkpoint), STOP
• Circular or non-convergence: git reset --hard <initial-sha> (revert ALL iterations), STOP
• Max iterations reached: git reset --hard <initial-sha> (revert ALL iterations), STOP

</critical_sequence>

Diagnostic Report

On termination (success or failure), write a report to stdout:

• Result: SUCCESS, REGRESSION, CIRCULAR, MAX_ITERATIONS, NON_CONVERGENCE, or UNRESOLVED
• Iterations completed: N out of max
• Termination reason: one-line explanation
• Iteration history: failure count per iteration with delta
• Remaining failures: test name and error message for each (if not success)
• Fixes applied: files modified and what changed (last iteration)
• Recommendation: what the user should investigate next (if not success)

On success, fixes are staged but NOT committed. The user reviews and commits via /ship or manually.

Key Principles

• Diagnose before fixing -- never guess at the root cause
• Fix implementation code only -- tests define the co