calibrate

/calibrate -- Score predictions vs outcomes

The user wants to check what actually happened after a sprint's recommendations were implemented.

Arguments

$ARGUMENTS

Expected format: /calibrate --outcome "what happened" or /calibrate <claim_id> "actual result"

Instructions

1. Parse the outcome: The user provides outcome data as free text or claim-specific results.

1. Match outcomes to predictions: Use wheat_search to find the original estimate, recommendation, or risk claims that predicted something. Compare prediction to actual outcome.

1. Create calibration claims as cal### claims with evidence tier production (these are real outcomes):
2. If prediction was accurate: factual claim noting the match
3. If prediction was wrong: factual claim noting the delta (predicted X, actual Y)
4. If prediction was partially right: estimate claim with the refined numbers

1. Compute accuracy scorecard:
2. Group by evidence tier: what % of stated vs web vs documented vs tested claims were accurate?
3. Group by claim type: are estimates less accurate than factual claims?
4. This validates whether the evidence tier system is predictive

1. Run wheat_compile.

1. Print scorecard:

``` Calibration results: Predictions scored: <N> Accurate: <N> (<percent>) Partially accurate: <N> Wrong: <N>

Accuracy by evidence tier: stated: <percent> web: <percent> documented: <percent> tested: <percent>

Next steps: /brief -- recompile with calibrated data /research <topic> -- investigate where predictions went wrong ```