result-to-claim

Result-to-Claim Gate

🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It is

verdict-bearing — it judges whether results support a claim. Re-running that

verdict on a wall-clock timer adds no new signal (the verdict changes only

when the results change, not when the clock ticks). What you actually want

to schedule is the external wait that precedes it — experiments done → then

run this gate once. See

[shared-references/external-cadence.md](../shared-references/external-cadence.md).

Experiments produce numbers; this gate decides what those numbers mean. Collect results from available sources, get a Codex judgment, then auto-route based on the verdict.

Context: $ARGUMENTS

When to Use

• After a set of experiments completes (main results, not just sanity checks)
• Before committing to claims in a paper or review response
• When results are ambiguous and you need an objective second opinion

Workflow

Step 1: Collect Results

Gather experiment data from whatever sources are available in the project:

1. W&B (preferred): wandb.Api().run("<entity>/<project>/<run_id>").history() — metrics, training curves, comparisons
2. EXPERIMENT_LOG.md: full results table with baselines and verdicts
3. EXPERIMENT_TRACKER.md: check which experiments are DONE vs still running
4. Log files: ssh server "tail -100 /path/to/training.log" if no other source
5. docs/research_contract.md: intended claims and experiment design

Assemble the key information:

• What experiments were run (method, dataset, config)
• Main metrics and baseline comparisons (deltas)
• The intended claim these experiments were designed to test
• Any known confounds or caveats

Step 1.5: Deterministic evidence pre-check (before spending a Codex call)

For every claim that cites a specific number + a source file, verify the evidence exists mechanically — no model call — to catch hallucinated evidence before the jury runs (see [shared-references/evidence-precheck.md](../shared-references/evidence-precheck.md)).

1. Build the claims list. From the cited numbers and their result files, write [{"id", "value", "source"}, ...] to .aris/claims.json (source is the result file/glob relative to the project root; value is the cited number or string).

2. Run the pre-check — this is a real step, not a suggestion. Execute the block below (resolver per integration-contract §2, Policy B: warn-and-skip if the helper is unresolved — never block the audit):

# Policy B = warn-and-skip: nothing here may abort the audit. cd is non-fatal, the
# helper run is explicitly non-blocking, no pipefail-fragile pipe.
cd "$(git rev-parse --show-toplevel 2>/dev/null || pwd)" 2>/dev/null || true
if [ -z "${ARIS_REPO:-}" ] && [ -f .aris/installed-skills.txt ]; then
    ARIS_REPO=$(awk -F'\t' '$1=="repo_root"{print $2; exit}' .aris/installed-skills.txt 2>/dev/null) || true
fi
EVIDENCE_CHECK=".aris/tools/evidence_check.py"
[ -f "$EVIDENCE_CHECK" ] || EVIDENCE_CHECK="tools/evidence_check.py"
[ -f "$EVIDENCE_CHECK" ] || { [ -n "${ARIS_REPO:-}" ] && EVIDENCE_CHECK="$ARIS_REPO/tools/evidence_check.py"; }
[ -f "$EVIDENCE_CHECK" ] || EVIDENCE_CHECK=""

mkdir -p .aris
if [ -n "$EVIDENCE_CHECK" ]; then
    # NB: evidence_check exits 1 when it FINDS hallucinated evidence (value_not_found /
    # path_missing) — that is the useful signal, NOT a failure. So judge success by
    # whether valid JSON was produced, never by exit code. `|| true` keeps set -e calm.
    python3 "$EVIDENCE_CHECK" . --batch .aris/claims.json > .aris/evidence_precheck.json 2>.aris/evidence_precheck.err || true
    if [ -s .aris/evidence_precheck.json ] && python3 -c "import json,sys;json.load(open('.aris/evidence_precheck.json'))" 2>/dev/null; then
        cat .aris/evidence_precheck.json
    else
        echo "WARN: evidence_check produced no valid output (see .aris/evidence_precheck.err);" >&2
        echo "      pre-check skipped (Policy B); the Codex jury still runs." >&2
    fi
else
    echo "WARN: evidence_check.py not resolved at .aris/tools/, tools/, or \$ARIS_REPO/tools/." >&2
    echo "      Pre-check skipped (Policy B); the Codex jury still runs. Fix: rerun" >&2
    echo "      bash tools/install_aris.sh, export ARIS_REPO, or copy the helper to tools/." >&2
fi

The output is {"results": [{id, value, source, status, ...}], "summary": {status: n}} with status ∈ {verified, value_not_found, path_missing, unparseable}.

3. Act on the statuses. Any claim returned value_not_found or path_missing is hallucinated evidence — mark it claim_supported: no with integrity_status: evidence_not_found immediately; do NOT spend a Codex call defending a number that isn't in the data. unparseable claims (no usable value/source) just go to the jury normally.

4. Carry the per-claim status into Step 2. Feed a small evidence pre-check: <id> → verified | value_not_found | path_missing | unparseable table (from .aris/evidence_precheck.json) into the Step-2 Codex prompt so the jury knows which claims have real evidence to read. If the pre-check was skipped (helper unresolved), say so in that slot rather than omitting it.

verified here means only that the cited evidence exists — whether it supports the claim is still the Codex jury's call in Step 2 (a deterministic gate DRIVES, it does not ACQUIT).

Step 2: Codex Judgment

Send the collected results to Codex for objective evaluation:

mcp__codex__codex:
  config: {"model_reasoning_effort": "xhigh"}
  prompt: |
    RESULT-TO-CLAIM EVALUATION

    I need you to judge whether experimental results support the intended claim.

    Intended claim: [the claim these experiments test]

    Experiments run:
    [list experiments with method, dataset, metrics]

    Results:
    [paste key numbers, comparison deltas, significance]

    Evidence pre-check (deterministic, from Step 1.5):
    [per-claim: <id> → verified | value_not_found | path_missing.
     A value_not_found/path_missing means the cited number is NOT in its result
     file — treat that claim as having no evidence; do not defend it. `verified`
     means the number exists in the file — YOU still judge whether it supports
     the claim.]

    Baselines:
    [baseline numbers and sources — reproduced or from paper]

    Known caveats:
    [any confounding factors, limited datasets, missing comparisons]

    Please evaluate:
    1. claim_supported: yes | partial | no
    2. what_results_support: what the data actually shows
    3. what_results_dont_support: where the data falls short of the claim
    4. missing_evidence: specific evidence gaps
    5. suggested_claim_revision: if the claim should be strengthened, weakened, or reframed
    6. next_experiments_needed: specific experiments to fill gaps (if any)
    7. confidence: high | medium | low

    Be honest. Do not inflate claims beyond what the data supports.
    A single positive result on one dataset does not support a general claim.

Step 3: Parse and Normalize

Extract structured fields from Codex response:

- claim_supported: yes | partial | no
- what_results_support: "..."
- what_results_dont_support: "..."
- missing_evidence: "..."
- suggested_claim_revision: "..."
- next_experiments_needed: "..."
- confidence: high | medium | low

Step 3.5: Check Experiment Integrity (if audit exists)

Skip this step if EXPERIMENT_AUDIT.json does not exist.

if EXPERIMENT_AUDIT.json exists:
    read integrity_status from file
    attach to verdict output:
        integrity_status: pass | warn | fail

    if integrity_status == "fail":
        append to verdict: "[INTEGRITY CONCERN] — audit found issues, see EXPERIMENT_AUDIT.md"
        downgrade confidence to "low" regardless of Codex judgment

    if integrity_status == "warn":
        append to verdict: "[INTEGRITY: WARN] — audit flagged potential issues"
else:
    integrity_sta