playwright-test-diagnosis

Goal

Turn heal-traces.ndjson into a concise root-cause report with statement-level evidence.

Inputs To Locate

• heal-traces/<executionId>/<playwrightTestId>/<attempt>/heal-traces.ndjson
• Sibling artefacts in the same <attempt>/ dir: trace.zip,

screenshots/stmt-*.png, videos/<file>.webm (when present)

• heal-traces/executions.ndjson — append-only run index; the latest

line is the most recent run

• heal-traces/<executionId>/execution.json — per-run manifest

(totals, git, duration, per-test summaries)

• The test file mentioned in the trace (statement.loc and source text)

If multiple trace files exist, prioritize the latest failing attempt. <executionId> is an auto-generated uuidv4 per run; pick the newest entry in executions.ndjson to find the most recent execution dir.

Workflow

1. Find the test-result event and note final status, duration, stderr, and stdout.
2. Scan statement events in order and identify:
3. first failing statement (status != ok)
4. last successful statement before failure
5. slow statements (durationMs outliers)
6. Extract high-signal fields from failing/nearby statements:
7. source
8. loc.line and file context
9. captured variable values
10. serialized error payload
11. screenshot references (if present)
12. Build a timeline:
13. setup/context
14. trigger statement
15. error manifestation
16. propagation/teardown
17. Correlate with external signals:
18. network/API response mismatches
19. assertion target mismatch vs highlighted locator screenshot
20. helper-function internal state that does not surface in standard Playwright trace
21. Propose a minimal next action:
22. test fix
23. app fix
24. synchronization/waiting fix
25. selector/locator fix

Heuristics

• Prefer the earliest causal mismatch, not the final thrown assertion.
• Treat statement-level values as source of truth when they contradict log text.
• Call out hidden helper failures explicitly (they are often invisible in action-level traces).
• For flaky behavior, compare passing and failing runs at the first diverging statement.
• Keep confidence explicit: high, medium, or low.

Output Format

Use this structure:

## Test Failure Analysis

- Test: <name>
- Status: <failed/passed>
- Confidence: <high|medium|low>

### What failed

- <1-2 bullets with exact failing statement and error>

### Root cause hypothesis

- <most likely cause tied to statement-level evidence>

### Evidence

- <statement source + line + key variable values>
- <timing or API correlation>
- <screenshot/locator evidence if available>

### Recommended fix

- <smallest practical change>

### Verification

- Re-run: <command>
- Expectation: <what should change in trace>

Quick Commands

Use fast filters when needed:

• failing lines: rg '"status":"(fail|error)"' heal-traces -g "*.ndjson"
• slow statements: rg '"durationMs":[1-9][0-9]{3,}' heal-traces -g "*.ndjson"
• error payloads: rg '"error"|\"stderr\"' heal-traces -g "*.ndjson"
• latest run dir: tail -n 1 heal-traces/executions.ndjson → read executionId