/harness:evolve
Run the propose-evaluate-iterate loop. LangSmith is the evaluation backend, git worktrees provide isolation.
Setup
.evolver.json must exist. If not, tell user to run harness:setup.
TOOLS="${EVOLVER_TOOLS:-$([ -d ".evolver/tools" ] && echo ".evolver/tools" || echo "$HOME/.evolver/tools")}"
EVOLVER_PY="${EVOLVER_PY:-$([ -f "$HOME/.evolver/venv/bin/python" ] && echo "$HOME/.evolver/venv/bin/python" || echo "python3")}"
Never pass LANGSMITH_API_KEY inline. Tools resolve it automatically via _common.ensure_langsmith_api_key().
Arguments
--iterations N(default: ask or 5)--mode light|balanced|heavy— override mode from config--no-interactive— skip prompts, use defaults (for cron/background runs)
If interactive, ask iterations (3/5/10), target score (0.8/0.9/0.95/none), and execution mode (interactive/background).
Mode Parameters
MODES = {
"light": {"proposers": 2, "waves": 1, "concurrency": 5, "timeout": 60, "sample": 10, "analysis": "summary", "pairwise": False, "archive": "winner"},
"balanced": {"proposers": 3, "waves": 2, "concurrency": 3, "timeout": 120, "sample": None, "analysis": "summary", "pairwise": "if_close", "archive": "all"},
"heavy": {"proposers": 5, "waves": 2, "concurrency": 3, "timeout": 300, "sample": None, "analysis": "full", "pairwise": True, "archive": "all"},
}
Read mode from config, allow --mode override:
MODE=$(python3 -c "import json; print(json.load(open('.evolver.json')).get('mode', 'balanced'))")
If not --no-interactive, confirm or switch:
{
"question": "Mode: {MODE}. Continue?",
"header": "Mode",
"options": [
{"label": "Yes, continue with {MODE}"},
{"label": "Switch to light (~2 min/iter)"},
{"label": "Switch to balanced (~8 min/iter)"},
{"label": "Switch to heavy (~25 min/iter)"}
]
}
If changed, update config and re-read MODE.
Pre-Loop
Preflight
$EVOLVER_PY $TOOLS/preflight.py --config .evolver.json
Validates API key, config schema, LangSmith state, dataset health, and canary in one pass. If it fails, ask user: fix and retry, continue anyway, or abort. If health issues are auto-correctable, run /harness:health first.
Baseline LLM-Judge
If LLM evaluators (correctness, conciseness) are configured but baseline only has code-based scores, spawn the evaluator agent on the baseline experiment. Re-read and update best_score in .evolver.json after scoring.
Resolve Project Directory
Read project_dir from config. If non-empty, all worktree paths include it: {worktree}/{project_dir}/.
The Loop (per iteration)
0. Read State + Start Iteration Trace
BEST=$(python3 -c "import json; b=json.load(open('.evolver.json')).get('best_experiment'); print(b if b else '')")
PROJECT_DIR=$(python3 -c "import json; print(json.load(open('.evolver.json')).get('project_dir', ''))")
ITER_START=$(date +%s)
Start iteration trace (logs to LangSmith for observability):
ITER_TRACE=$($EVOLVER_PY $TOOLS/log_iteration.py --config .evolver.json --action start --version v{NNN} 2>/dev/null)
ITER_RUN_ID=$(echo "$ITER_TRACE" | python3 -c "import sys,json; print(json.load(sys.stdin).get('run_id',''))" 2>/dev/null)
ITER_DOTTED_ORDER=$(echo "$ITER_TRACE" | python3 -c "import sys,json; print(json.load(sys.stdin).get('dotted_order',''))" 2>/dev/null)
If log_iteration.py fails (no LangSmith, no key), the loop continues — tracing is optional.
If $BEST is empty (no baseline ran), skip data gathering — proposers work from code analysis only.
1. Gather Data (parallel)
Analysis format depends on mode (MODES[MODE]["analysis"]):
if [ -n "$BEST" ]; then
ANALYSIS_FMT=$(python3 -c "m={'light':'summary','balanced':'summary','heavy':'full'}; print(m.get('$MODE','summary'))")
$EVOLVER_PY $TOOLS/trace_insights.py --from-experiment "$BEST" --format $ANALYSIS_FMT --output trace_insights.json &
$EVOLVER_PY $TOOLS/read_results.py --experiment "$BEST" --config .evolver.json --split train --format $ANALYSIS_FMT --output best_results.json &
wait
fi
2. Generate Strategy + Lenses
From trace_insights.json, best_results.json, evolution_memory.md, production_seed.json:
strategy.md — Current iteration data ONLY. No stale info. Contents: target files, failure clusters (latest experiment), top 3 promoted memory insights (rec >= 2), approaches to avoid, top 3 failing examples with judge feedback. Cap at 1500 tokens.
lenses.json — Investigation questions for proposers:
- One per failure cluster (max 3), one architecture, one production, one evolution_memory, one open
- If
evolution_archive/has 3+ iterations, onearchive_branchlens that suggests revisiting a losing candidate's approach - Sort by severity, cap at 5 lenses
3. Spawn Proposers (mode-dependent)
Proposer count: MODES[MODE]["proposers"] (light=2, balanced=3, heavy=5). Cap lenses at this number.
Waves: MODES[MODE]["waves"] (light=1 single wave, balanced/heavy=2 two-wave).
Build IDENTICAL shared prefix (objective + files_to_read + context) for KV-cache sharing. Only the <lens> block differs — place it LAST. Include evolution_archive/ in <files_to_read> so proposers can grep prior candidates.
IMPORTANT: After each proposer worktree is created, copy untracked files and set trace nesting. Always use absolute paths:
SRC="$(dirname "$(git rev-parse --git-common-dir)")"
[ -n "$PROJECT_DIR" ] && SRC="$SRC/$PROJECT_DIR"
# If langsmith-tracing companion is installed, proposer traces nest under iteration:
[ -n "$ITER_DOTTED_ORDER" ] && export CC_LANGSMITH_PARENT_DOTTED_ORDER="$ITER_DOTTED_ORDER"
# For each worktree (after Agent creates it, before agent reads files):
cp "$SRC/.evolver.json" "$WT_PROJECT/.evolver.json"
[ -f "$SRC/.env" ] && cp "$SRC/.env" "$WT_PROJECT/.env"
[ -d "$SRC/evolution_archive" ] && cp -r "$SRC/evolution_archive" "$WT_PROJECT/evolution_archive"
Do NOT suppress stderr with 2>/dev/null — if the copy fails, you need to see the error.
Wave 1 — critical + high severity lenses, run independently in parallel:
Agent(
subagent_type: "harness-proposer",
isolation: "worktree",
run_in_background: true,
prompt: "{SHARED_PREFIX}\n\n<lens>\n{lens.question}\nSource: {lens.source}\n</lens>"
)
Wait for wave 1 to complete. Report each completion as it happens.
Wave 2 — medium + open lenses, see wave 1 results before starting:
Add to the shared context for wave 2 proposers:
<prior_proposals>
Wave 1 proposers completed:
- Proposer {id} ({lens}): {approach from proposal.md} — {committed/abstained}
...
</prior_proposals>
Wave 2 proposers see what wave 1 tried and can build on it, avoid duplication, or take complementary approaches. Research shows +14% quality when agents observe prior outputs.
If only 1-2 lenses total, run as single wave.
4. Evaluate Candidates
Run evaluations with mode parameters. run_eval.py auto-copies config files to worktrees:
CONCURRENCY=$(python3 -c "m={'light':5,'balanced':3,'heavy':3}; print(m.get('$MODE',3))")
TIMEOUT=$(python3 -c "m={'light':60,'balanced':120,'heavy':300}; print(m.get('$MODE',120))")
SAMPLE=$(python3 -c "m={'light':'10','balanced':'','heavy':''}; s=m.get('$MODE',''); print(f'--sample {s} --sample-split train' if s else '')")
for WT in {worktree_paths_with_commits}; do
WT_PROJECT="$WT"
[ -n "$PROJECT_DIR" ] && WT_PROJECT="$WT/$PROJECT_DIR"
$EVOLVER_PY $TOOLS/run_eval.py --config "$SRC/.evolver.json" --worktree-path "$WT_PROJECT" --experiment-prefix v{NNN}-{id} --concurrency $CONCURRENCY --timeout $TIMEOUT $SAMPLE &
done
wait # CRITICAL: wait for ALL evals before judge
Note: $SRC is set via git rev-parse --git-common-dir — resolves to the main repo root even when CWD is inside a worktree (--show-toplevel returns the worktree root, which is wrong).
Auto-spawn LLM-as-judge — check if LLM evaluators