/harness:setup

Set up the Harness Evolver v3 in a project. Explores the codebase, configures LangSmith, runs baseline evaluation.

Prerequisites

Check for LangSmith API key — it can be in the environment, the credentials file, or .env:

python3 -c "
import os, platform
key = os.environ.get('LANGSMITH_API_KEY', '')
if not key:
    creds = os.path.expanduser('~/Library/Application Support/langsmith-cli/credentials') if platform.system() == 'Darwin' else os.path.expanduser('~/.config/langsmith-cli/credentials')
    if os.path.exists(creds):
        for line in open(creds):
            if line.strip().startswith('LANGSMITH_API_KEY='):
                key = line.strip().split('=',1)[1].strip()
    if not key and os.path.exists('.env'):
        for line in open('.env'):
            if line.strip().startswith('LANGSMITH_API_KEY=') and not line.strip().startswith('#'):
                key = line.strip().split('=',1)[1].strip().strip('\"').strip(\"'\")
print('OK' if key else 'MISSING')
"

If MISSING: "Set your LangSmith API key: export LANGSMITH_API_KEY=lsv2_pt_... or run npx harness-evolver@latest to configure."

The tools auto-load the key from the credentials file, but the env var takes precedence.

Resolve Tool Path and Python

# Prefer env vars set by plugin hook; fallback to legacy npx paths
TOOLS="${EVOLVER_TOOLS:-$([ -d ".evolver/tools" ] && echo ".evolver/tools" || echo "$HOME/.evolver/tools")}"
EVOLVER_PY="${EVOLVER_PY:-$([ -f "$HOME/.evolver/venv/bin/python" ] && echo "$HOME/.evolver/venv/bin/python" || echo "python3")}"

Use $EVOLVER_PY instead of python3 for ALL tool invocations. This ensures the venv with langsmith is used.

IMPORTANT: Never pass LANGSMITH_API_KEY inline in Bash commands. The key is loaded automatically by the SessionStart hook (from credentials file or environment) and by each Python tool's ensure_langsmith_api_key(). Passing it inline exposes it in the output. If the key is missing, tell the user to run export LANGSMITH_API_KEY=lsv2_pt_... instead.

Phase 1: Explore Project (automatic)

find . -maxdepth 3 -type f -name "*.py" -not -path "*/.venv/*" -not -path "*/node_modules/*" -not -path "*/__pycache__/*" | head -30

Monorepo detection: if the project root has multiple subdirectories with their own main.py or pyproject.toml, it's a monorepo. Use AskUserQuestion to ask WHICH app to optimize before proceeding — do NOT scan everything.

Look for:

• Entry points: files with if __name__, or named main.py, app.py, agent.py, graph.py, pipeline.py
• Existing LangSmith config: LANGCHAIN_PROJECT / LANGSMITH_PROJECT in env or .env
• Existing test data: JSON files with inputs, CSV files, etc.
• Dependencies: requirements.txt, pyproject.toml

To identify the framework, read the entry point file and its immediate imports. The proposer agents will use Context7 MCP for detailed documentation lookup — you don't need to detect every library, just identify the main framework (LangGraph, CrewAI, OpenAI Agents SDK, etc.) from the imports you see.

Detect virtual environments — check for venvs in the project or parent directories:

# Check common venv locations
for venv_dir in .venv venv ../.venv ../venv; do
    if [ -f "$venv_dir/bin/python" ]; then
        echo "VENV_FOUND: $venv_dir/bin/python"
        break
    fi
done

If a venv is found, use it for the entry point instead of bare python. The agent's dependencies are likely installed there, not in the system Python. For example: ../.venv/bin/python agent.py {input} instead of python agent.py {input}.

Identify the run command — how to execute the agent. Use {input} as a placeholder for the JSON file path:

• .venv/bin/python main.py {input} — if venv detected (preferred)
• python main.py {input} — agent reads JSON file from positional arg
• python main.py --input {input} — agent reads JSON file from --input flag
• python main.py --query {input_json} — agent receives inline JSON string

The runner writes {"input": "user question..."} to a temp .json file and replaces {input} with the file path. If the entry point already contains --input (without placeholder), the runner appends the file path as the next argument.

If no placeholder and no --input flag detected, the runner appends --input <path> --output <path>.

Phase 2: Confirm Configuration (interactive)

Present all detected configuration in one view with smart defaults and ask for confirmation.

Use AskUserQuestion:

{
  "questions": [{
    "question": "Here's the configuration for your project:\n\n**Entry point**: {command}\n**Framework**: {framework}\n**Python**: {venv_path or 'system python3'}\n**Optimization goals**: accuracy (correctness evaluator)\n**Test data**: generate 30 examples with AI\n\nDoes this look good?",
    "header": "Setup Configuration",
    "multiSelect": false,
    "options": [
      {"label": "Looks good, proceed", "description": "Use these settings and start setup"},
      {"label": "Customize goals", "description": "Choose different optimization goals"},
      {"label": "I have test data", "description": "Use existing JSON file or LangSmith project"},
      {"label": "Let me adjust everything", "description": "Change entry point, framework, goals, and data source"}
    ]
  }]
}

If "Looks good, proceed": Use defaults — goals=accuracy, data=generate 30 with testgen. Skip straight to Phase 3.

If "Customize goals": Ask the goals question, then proceed to Phase 3 with testgen as default data source.

Use AskUserQuestion:

{
  "questions": [{
    "question": "What do you want to optimize?",
    "header": "Goals",
    "multiSelect": true,
    "options": [
      {"label": "Accuracy", "description": "Correctness of outputs — LLM-as-judge evaluator"},
      {"label": "Latency", "description": "Response time — track and minimize"},
      {"label": "Token efficiency", "description": "Fewer tokens for same quality"},
      {"label": "Error handling", "description": "Reduce failures, timeouts, crashes"}
    ]
  }]
}

Map selections to evaluator configuration for setup.py.

Phase 2.5: Mode Selection

{
  "questions": [{
    "question": "Evolution mode?",
    "header": "Mode",
    "multiSelect": false,
    "options": [
      {"label": "light", "description": "20 examples, 2 proposers, ~2 min/iter. Good for testing."},
      {"label": "balanced (Recommended)", "description": "30 examples, 3 proposers, ~8 min/iter. Best trade-off."},
      {"label": "heavy", "description": "50 examples, 5 proposers, ~25 min/iter. Maximum quality."}
    ]
  }]
}

Pass selection to setup.py as --mode light|balanced|heavy.

The mode determines testgen count:

• light: generate 20 examples
• balanced: generate 30 examples (default, current behavior)
• heavy: generate 50 examples

If "I have test data": Ask the data source question, then proceed to Phase 3 with accuracy as default goal.

Use AskUserQuestion with preview:

{
  "questions": [{
    "question": "Where should test inputs come from?",
    "header": "Test data",
    "multiSelect": false,
    "options": [
      {
        "label": "Import from LangSmith",
        "description": "Use real production traces as test inputs",
        "preview": "## Import from LangSmith\n\nFetches up to 100 recent traces from your production project.\nPrioritizes traces with negative feedback.\nCreates a LangSmith Dataset with real user inputs.\n\nRequires: an existing LangSmith project with traces."
      },
      {
        "label": "I have a file",
        "description": "Point to an existing file with test inputs",
        "preview": "## Provide Test Data\n\nSupported formats:\n- JSON array of inputs\n- JSON with {\"inputs\": {...}} objects\n- CSV with input columns\n\nExample:\n```json\n[\n  {\"input\": \"What is P