Harness Engineering

Design the system around AI agents for reliable, safe production use.

Agent Runtime Model

Treat the agent harness as the kernel around an LLM OS:

Usage Template

Prompt

Use harness-engineering for this agent workflow. Design permissions, tools, feedback loops, observability, and failure handling.

Use Case

• Moving an agent workflow from ad hoc prompting toward a reliable runtime architecture.

Expected Result

• The agent produces a harness design with permission tiers, tool boundaries, logs, evals, and recovery paths.

Output Example

• A runtime spec with permission matrix, tool allowlist, approval gates, logs, evals, and incident response.

Verification Case

• The design names what the agent can do automatically, what needs approval, and what is denied.

Verified Effect

• An ad hoc agent workflow becomes a controlled runtime with explicit permissions, observability, and failure handling.

When to Use

• Deploying agents to production
• Setting up permissions/guardrails/approval workflows
• Designing multi-agent systems
• Agent behaved unpredictably → needs better constraints
• Configuring auto-mode or permission tiers

Three Domains

Six Components

1. Context & Knowledge Layer

• Curated access to code, docs, schemas, logs
• Use CLAUDE.md for project-level context
• Use context-manager for token budgeting
• Never inject raw 10K+ token files
• Persist reusable outputs to wiki, docs, logs, or state files; chat history is not durable memory

2. Tooling & API Surface

Three-Tier Permission Model:

Tier 3 heuristic:

1. Can destroy data irreversibly? → BLOCK
2. Accesses credentials? → BLOCK
3. Affects shared infrastructure? → BLOCK
4. Target inferred (not explicit)? → BLOCK

3. Architectural Constraints

• Linters, structural tests
• File/directory access boundaries
• Token budget: never exceed 80% context window

4. Feedback & Validation Loops

• Write-Test-Fix cycle
• Generator + Evaluator pattern (GAN-inspired)
• Deny-and-continue: try safer alternative on block
• Escalate to human after 3 consecutive blocks

5. Observability & Governance

• Log every tool call + result
• Metrics: success rate, revert rate, token usage
• Cost tracking per session
• Evidence ledger: verification command, source link, screenshot, test result, or diff for each completion claim

6. Maintenance

• Periodic agents: dead code, outdated docs, architectural drift
• Wiki lint for broken links/missing frontmatter
• Context garbage collection: summarize completed work, close idle processes, and record residual risk

GAN-Inspired Multi-Agent Pattern

"Tuning a standalone evaluator to be skeptical is far more tractable than making a generator critical of its own work."

Generator ←──sprint contract──→ Evaluator
    │                              │
    ▼                              ▼
 Produces output              Playwright/Test

Implementation:

1. Planner → 1-sentence prompt → full spec
2. Generator → builds in sprints
3. Evaluator → tests, files bugs
4. Feedback loop → 5-15 iterations

Physical → Cognitive Analogies

Multi-Agent Patterns

Permissioned Tool Design

Define every tool like a system call:

Prefer narrow tools with explicit inputs over broad shell/API access. If broad access is unavoidable, wrap it with approval gates and post-action verification.

Closed-Loop System (Aladdin)

Data Lake → Risk Engine → Optimizer → Stress Test → OMS → Compliance → Feedback Loop
     ↑                                                              ↓
     └──────────────────── 反馈闭环 ────────────────────────────────┘

Key: Closed loop is the moat — not any single component. MECE principle applies.

Quality Gates

• Each agent has defined permission scope
• Three-tier permission model documented
• Write-Test-Fix feedback loop in place
• Tool calls logged and observable
• Token costs tracked per session
• High-risk actions require approval
• Multi-agent isolation prevents interference
• Generator and Evaluator separate for complex tasks
• Reusable outputs have durable write-back outside chat history
• Tool contracts include bounds, failure path, evidence, and audit trail