test-suite-prioritizer

Test Suite Prioritizer

Intelligently prioritize test execution based on code changes and project impact.

Core Capabilities

This skill helps optimize test execution by:

1. Analyzing code changes - Identifying which files were modified
2. Mapping test coverage - Determining which tests cover changed code
3. Calculating priority scores - Ranking tests by impact and relevance
4. Generating execution order - Creating an optimized test run sequence
5. Providing reasoning - Explaining why each test is prioritized

Test Prioritization Workflow

Step 1: Identify Code Changes

Determine what code has been modified to focus testing efforts.

Sources of Change Information:

• Git diff (most recent commit or uncommitted changes)
• Pull request file list
• Specific files mentioned by user
• Modified files in working directory

Collect Changed Files:

# Recent commit changes
git diff --name-only HEAD~1 HEAD

# Uncommitted changes
git diff --name-only

# Changes in current branch vs main
git diff --name-only main...HEAD

# Files in last N commits
git diff --name-only HEAD~N HEAD

Example Output:

src/auth/login.py
src/payment/processor.py
src/models/user.py
tests/test_auth.py

Step 2: Analyze Test Coverage

Map which tests cover the changed code.

Strategies:

1. Direct Test-to-Code Mapping

• Tests in same directory as changed code
• Tests with similar names (e.g., user.py → test_user.py)
• Integration tests that import changed modules

2. Static Analysis

• Parse test files for imports
• Identify which modules each test uses
• Map dependencies transitively

3. Coverage Data (if available)

• Use pytest-cov, coverage.py, JaCoCo output
• Identify tests that execute changed lines
• Most accurate but requires prior test run

Example Mapping:

Changed: src/auth/login.py
Covered by:
  - tests/test_auth.py::test_login_success
  - tests/test_auth.py::test_login_failure
  - tests/integration/test_user_flow.py::test_complete_registration
  - tests/e2e/test_authentication_flow.py::test_full_auth_cycle

Changed: src/payment/processor.py
Covered by:
  - tests/test_payment.py::test_process_payment
  - tests/test_payment.py::test_refund
  - tests/integration/test_checkout.py::test_complete_purchase

Step 3: Calculate Priority Scores

Assign priority scores based on change impact.

Scoring Formula:

Priority Score = Base Score × Change Impact Factor

Base Score factors:
- Direct coverage of changed file: +10 points
- Indirect coverage (imports changed file): +5 points
- Integration test covering changed area: +3 points
- Test in same module: +2 points

Change Impact Factor (multiply):
- Critical path code (auth, payments): ×1.5
- Recently failed test: ×1.3
- High historical failure rate: ×1.2
- Core business logic: ×1.2
- UI/cosmetic changes: ×0.8

Example Scoring:

# Example: test_login_success
base_score = 10  # Directly tests login.py (changed)
impact_factor = 1.5  # Auth is critical path
priority_score = 10 × 1.5 = 15

# Example: test_user_flow
base_score = 5 + 3  # Imports login.py + integration test
impact_factor = 1.2  # Core business logic
priority_score = 8 × 1.2 = 9.6

# Example: test_ui_styling
base_score = 2  # Same module as changed file
impact_factor = 0.8  # UI/cosmetic
priority_score = 2 × 0.8 = 1.6

Step 4: Generate Prioritized Test List

Create ordered execution sequence with scores and reasoning.

Output Format:

# Test Execution Priority List

## Summary
- Total tests analyzed: 45
- High priority (score ≥ 10): 8 tests
- Medium priority (score 5-9): 15 tests
- Low priority (score < 5): 22 tests
- Estimated time for high priority tests: ~2 minutes

## Recommended Execution Order

### Priority 1: Critical Tests (Run First)

1. **tests/test_auth.py::test_login_success** (Score: 15.0)
   - Directly tests modified file: src/auth/login.py
   - Critical path: Authentication
   - Reason: Core security functionality changed

2. **tests/test_auth.py::test_login_failure** (Score: 15.0)
   - Directly tests modified file: src/auth/login.py
   - Critical path: Authentication
   - Reason: Error handling for authentication changed

3. **tests/test_payment.py::test_process_payment** (Score: 15.0)
   - Directly tests modified file: src/payment/processor.py
   - Critical path: Payment processing
   - Reason: Payment logic modified, high business impact

### Priority 2: High Impact Tests

4. **tests/integration/test_user_flow.py::test_complete_registration** (Score: 9.6)
   - Indirect coverage: Imports src/auth/login.py
   - Integration test: End-to-end user flow
   - Reason: Validates auth changes in realistic scenario

5. **tests/integration/test_checkout.py::test_complete_purchase** (Score: 9.6)
   - Indirect coverage: Imports src/payment/processor.py
   - Integration test: Full checkout flow
   - Reason: Validates payment changes with real workflow

6. **tests/test_models.py::test_user_model** (Score: 7.0)
   - Related file: src/models/user.py modified
   - Dependency: Auth and payment depend on user model
   - Reason: Foundation for changed functionality

### Priority 3: Supporting Tests

7. **tests/test_validators.py::test_email_validation** (Score: 4.0)
   - Indirect dependency: Used by auth module
   - Reason: Input validation for authentication

8. **tests/e2e/test_authentication_flow.py::test_full_auth_cycle** (Score: 3.9)
   - E2E coverage: Complete authentication flow
   - Reason: Comprehensive validation but slow to execute

[... remaining tests with lower scores ...]

## Quick Commands

Run high priority tests only (8 tests, ~2 min):

pytest tests/test_auth.py::test_login_success \ tests/test_auth.py::test_login_failure \ tests/test_payment.py::test_process_payment \ tests/integration/test_user_flow.py::test_complete_registration \ tests/integration/test_checkout.py::test_complete_purchase \ tests/test_models.py::test_user_model \ tests/test_validators.py::test_email_validation \ tests/e2e/test_authentication_flow.py::test_full_auth_cycle

Run critical tests only (3 tests, ~30 sec):

pytest tests/test_auth.py tests/test_payment.py::test_process_payment

## Time Estimates

- Critical tests (3): ~30 seconds
- High priority tests (8): ~2 minutes
- All prioritized tests (23): ~5 minutes
- Full test suite (45): ~15 minutes

## Recommendations

1. **Pre-commit**: Run critical tests (score ≥ 15) before committing
2. **CI fast lane**: Run high priority tests (score ≥ 9) in first stage
3. **Full validation**: Run all tests in parallel after merge

Step 5: Handle Special Cases

Adjust prioritization for specific scenarios.

Time-Constrained Testing:

## 1-Minute Quick Check (Top 3 Critical Tests)

pytest tests/test_auth.py::test_login_success \
       tests/test_auth.py::test_login_failure \
       tests/test_payment.py::test_process_payment

These cover the most critical changed functionality.

Post-Refactoring:

## Post-Refactor Validation

Changed file: src/utils/helpers.py (used by 25 modules)

Priority:
1. All tests directly importing helpers.py (18 tests)
2. Integration tests using helper functions (8 tests)
3. Tests in dependent modules (25 tests)

Reason: Wide-reaching refactor requires comprehensive validation.

Flaky Test Handling:

## Note: Flaky Tests Detected

The following tests have high failure rates but low priority scores:
- tests/test_cache.py::test_concurrent_access (30% failure rate)
- tests/test_api.py::test_rate_limiting (25% failure rate)

Recommendation:
- Fix flaky tests before relying on prioritization
- Consider quarantining or mocking timing-dependent code

Step 6: Provide Execution Guidance

Offer practical commands and strategies.

For CI/CD Pipelines:

# .github/workflows/ci.yml example
jobs:
  quick-tests:
    name: Critical Tests (Fast Fee