skill-forge-benchmark

Skill Benchmarking & Performance Tracking

Measure and compare skill performance across iterations with statistical rigor using multiple trials, variance analysis, and trend tracking.

Process

Step 1: Define Benchmark Configuration

Accept configuration as:

• Existing eval set: Path to evals/evals.json (from /skill-forge eval)
• Benchmark config: Custom config with trial count and thresholds

Benchmark config schema:

{
  "skill_name": "my-skill",
  "skill_path": "./my-skill",
  "eval_set_path": "./evals/evals.json",
  "trials_per_eval": 3,
  "baseline_type": "no_skill",
  "previous_benchmark": null,
  "thresholds": {
    "min_pass_rate": 0.8,
    "max_avg_tokens": 100000,
    "max_avg_duration_seconds": 120,
    "min_improvement_ratio": 1.0
  }
}

Step 2: Execute Benchmark Runs

For each eval, run trials_per_eval times (default: 3) to get reliable metrics:

1. Execute with-skill runs (3x per eval)
2. Execute baseline runs (3x per eval)
3. Capture per-run: pass/fail, token count, duration
4. Save each run's timing.json and grading.json

Use agents/skill-forge-executor.md for parallel execution where possible.

Step 3: Aggregate Results

Run python scripts/aggregate_benchmark.py <workspace>/iteration-<N> --skill-name <name>:

Output benchmark.json schema:

{
  "skill_name": "my-skill",
  "iteration": 1,
  "timestamp": "2026-03-06T12:00:00Z",
  "summary": {
    "total_evals": 10,
    "with_skill": {
      "pass_rate": 0.87,
      "pass_rate_std": 0.05,
      "avg_tokens": 45000,
      "avg_duration_seconds": 34.2
    },
    "baseline": {
      "pass_rate": 0.60,
      "pass_rate_std": 0.08,
      "avg_tokens": 62000,
      "avg_duration_seconds": 52.1
    },
    "improvement_ratio": 1.45,
    "token_savings_ratio": 0.73,
    "time_savings_ratio": 0.66
  },
  "per_eval": [
    {
      "eval_id": 0,
      "eval_name": "basic-trigger",
      "with_skill": {"pass_rate": 1.0, "avg_tokens": 30000, "avg_duration_seconds": 20.1},
      "baseline": {"pass_rate": 0.67, "avg_tokens": 50000, "avg_duration_seconds": 45.0},
      "trials": 3
    }
  ],
  "thresholds_met": {
    "min_pass_rate": true,
    "max_avg_tokens": true,
    "max_avg_duration_seconds": true,
    "min_improvement_ratio": true
  }
}

Step 4: Compare with Previous Iterations

If previous_benchmark is provided or prior iteration-<N-1> exists:

1. Load previous benchmark.json
2. Calculate delta per metric:
3. Pass rate change
4. Token usage change
5. Duration change
6. New regressions (evals that passed before but fail now)
7. New improvements (evals that failed before but pass now)

Step 5: Generate Benchmark Report

# Benchmark Report: [skill-name]

## Iteration [N] vs [N-1]

### Summary
| Metric | Current | Previous | Delta | Threshold | Status |
|--------|---------|----------|-------|-----------|--------|
| Pass Rate | 87% | 78% | +9% | >= 80% | PASS |
| Avg Tokens | 45K | 52K | -13% | <= 100K | PASS |
| Avg Time | 34s | 41s | -17% | <= 120s | PASS |
| Improvement | 1.45x | 1.30x | +0.15x | >= 1.0x | PASS |

### Regressions (Action Required)
| Eval | Previous | Current | Notes |
|------|----------|---------|-------|
| eval-5 | PASS | FAIL | Output missing required section |

### Improvements
| Eval | Previous | Current | Notes |
|------|----------|---------|-------|
| eval-3 | FAIL | PASS | Error handling now works |

### Per-Eval Detail
[Full breakdown table]

### Variance Analysis
| Eval | Pass Rate | Std Dev | Trials | Reliability |
|------|-----------|---------|--------|-------------|
| eval-0 | 100% | 0.00 | 3 | High |
| eval-1 | 67% | 0.47 | 3 | Low (investigate) |

### Recommendations
[Based on regressions, low-reliability evals, and threshold failures]

Step 6: Threshold Gating

If any threshold fails:

1. Flag as FAIL with specific threshold details
2. List which evals caused the failure
3. Recommend running /skill-forge evolve to address issues
4. Do NOT approve for publish until thresholds pass

Error Handling

• Flaky trials: If a trial times out or crashes, exclude it from variance calculation and note "trials_completed" vs "trials_requested" in per-eval results
• Insufficient trials: If fewer than 2 trials complete for an eval, flag variance as "unreliable" in the report
• Missing baseline: If baseline runs fail entirely, report with-skill results only and skip improvement_ratio
• Threshold edge cases: If pass_rate equals the threshold exactly, treat as PASS

Integration with Other Sub-Skills

• skill-forge-eval: Provides the eval set and grading infrastructure
• skill-forge-evolve: Receives benchmark failures as improvement targets
• skill-forge-publish: Requires benchmark pass (score >= thresholds) before publish
• skill-forge-review: Can include benchmark summary in review report