regression-root-cause-analyzer

Regression Root Cause Analyzer

Overview

Systematically investigate failing regression tests to identify root causes by analyzing code changes, error messages, test dependencies, and common failure patterns.

Workflow

1. Gather Initial Information

Collect essential details about the test failure:

Test failure details:

• Which test(s) are failing?
• Error message and stack trace
• When did tests start failing? (commit, PR, date)
• Do tests fail consistently or intermittently?
• Do tests fail locally, in CI, or both?

Quick commands to gather info:

# Run the failing test
pytest path/to/test_file.py::test_name -v

# Get recent commits
git log --oneline -10

# Check current branch and status
git status
git branch

# See what changed recently
git log --since="1 week ago" --oneline

# Check for uncommitted changes
git diff

2. Analyze Error Messages

Parse the error message and stack trace for clues. See [failure-patterns.md](references/failure-patterns.md) for common patterns.

Error type indicators:

ImportError / ModuleNotFoundError:

ImportError: cannot import name 'UserService' from 'app.services'

• Likely cause: Module or class renamed/moved
• Investigation: Check recent commits to app/services/

AttributeError:

AttributeError: 'User' object has no attribute 'email_address'

• Likely cause: Property renamed or removed
• Investigation: Check User class definition changes

TypeError (arguments):

TypeError: process_data() got an unexpected keyword argument 'format'

• Likely cause: Function signature changed
• Investigation: Find process_data definition and recent changes

AssertionError:

AssertionError: assert 3 == 2

• Likely cause: Logic change or test data change
• Investigation: Understand what the assertion checks

KeyError / IndexError:

KeyError: 'status'

• Likely cause: Data structure changed
• Investigation: Check response/data format changes

3. Identify Code Changes

Use git to find what changed since tests last passed.

Find When Tests Broke

# If you know the last good commit
git diff <last-good-commit> <current-commit>

# Check specific file changes
git log -p path/to/file.py

# See what changed in last N commits
git log -p -n 5

# Find commits that touched specific function
git log -S "function_name" -p

Analyze Relevant Changes

Priority order for investigation:

1. Direct changes to tested code - Changes to the file/class being tested
2. Changes to imported dependencies - Files imported by tested code
3. Changes to test fixtures/mocks - Test setup code
4. Changes to test infrastructure - pytest config, test runners
5. Dependency version changes - requirements.txt, package.json

Commands to find changes:

# What files changed recently?
git diff --name-only HEAD~5..HEAD

# Changes to specific file
git diff HEAD~5..HEAD path/to/file.py

# Changes to test file
git diff HEAD~5..HEAD path/to/test_file.py

# Changes to requirements
git diff HEAD~5..HEAD requirements.txt package.json

4. Investigate Specific Failure Patterns

Match the error to common patterns:

Pattern: API Signature Change

Symptoms:

• TypeError: missing required argument
• TypeError: got unexpected keyword argument

Investigation steps:

1. Find function definition: grep -r "def function_name" .
2. Check git history: git log -p -S "def function_name"
3. Compare signatures before/after
4. Update test calls to match new signature

Pattern: Return Type Change

Symptoms:

• AttributeError when accessing return value
• TypeError: 'NoneType' object is not iterable

Investigation steps:

1. Check function return statements
2. Look for changes from return [] to return None
3. Check if error handling changed
4. Update test to handle new return type

Pattern: Dependency Version Change

Symptoms:

• Tests fail after pip install or npm install
• Different behavior in CI vs local

Investigation steps:

1. Check dependency files: git diff HEAD~5..HEAD requirements.txt
2. Review changelogs for updated packages
3. Look for deprecation warnings in test output
4. Pin to previous version to confirm

Pattern: Test Isolation Issue

Symptoms:

• Tests pass individually but fail when run together
• Order-dependent failures

Investigation steps:

1. Run tests individually: pytest test_file.py::test_one
2. Run in different orders
3. Check for shared state (global variables, database, files)
4. Look for missing cleanup in teardown

5. Generate Root Cause Report

Produce a structured markdown report:

Root Cause Analysis: [Test Name]

Summary

• Test: test_user_registration
• Status: Failing since commit abc123
• Error: TypeError: process_user() got an unexpected keyword argument 'email_format'

Root Cause

The process_user() function signature changed in commit abc123. The parameter email_format was renamed to email_type.

Evidence:

• Commit abc123 modified app/users.py
• Function signature before: def process_user(data, email_format="html")
• Function signature after: def process_user(data, email_type="html")
• Test still calls: process_user(user_data, email_format="html")

Likelihood: High (95%)

This is the direct cause of the TypeError.

Fix Required

Update test to use new parameter name:

# Before
result = process_user(user_data, email_format="html")

# After
result = process_user(user_data, email_type="html")

Related Changes

• Commit abc123: Renamed email_format to email_type throughout codebase
• 5 other test files also need updates

Additional Notes

The function behavior is otherwise unchanged. Only the parameter name differs.

Alternative Hypotheses

Hypothesis 2: Test Data Changed

Likelihood: Low (10%)

The test fixture might have changed, but review shows fixtures are unchanged.

Hypothesis 3: Environment Difference

Likelihood: Very Low (5%)

Could be environment-related, but error is consistent locally and in CI.

Reproduction Steps

1. Checkout commit abc123
2. Run: pytest tests/test_users.py::test_user_registration
3. Observe TypeError

Verification Steps

1. Apply suggested fix
2. Run test: pytest tests/test_users.py::test_user_registration
3. Verify test passes

6. Verify the Root Cause

Before finalizing the analysis:

Test the hypothesis:

1. Apply the proposed fix
2. Run the test to confirm it passes
3. Run related tests to ensure no new breaks

If fix doesn't work:

1. Review alternative hypotheses
2. Gather more information
3. Expand investigation to related areas

Commands to verify:

# Run the specific failing test
pytest path/to/test.py::test_name -v

# Run all related tests
pytest path/to/test.py -v

# Run with verbose output
pytest path/to/test.py::test_name -vv

# Run with print statements visible
pytest path/to/test.py::test_name -s

Investigation Strategies

Strategy 1: Binary Search Through Commits

Find exact commit that broke tests:

# Start bisect
git bisect start

# Mark current (broken) commit
git bisect bad

# Mark last known good commit
git bisect good <commit-hash>

# Git will checkout middle commit
# Run tests, then mark good or bad
pytest tests/

# If tests pass
git bisect good

# If tests fail
git bisect bad

# Repeat until git finds the breaking commit

Strategy 2: Compare Working vs Broken

Diff approach:

# Compare file between commits
git diff <good-commit>:<path> <bad-commit>:<path>

# Show file at specific commit
git show <commit>:path/to/file.py

Checkout approach:

# Temporarily checkout old version
git checkout <good-commit> path/to/file.py

# Run tests
pytest tests/

# Restore current version
git checkout HEAD path/to/file.py

Strategy 3: Isolate the Problem

**Minimal