paper-claim-audit skill

Paper Claim Audit: Zero-Context Evidence Verification

Codex assurance: write review_independence: same-family and

acceptance_status: provisional into base audit JSON. A fresh Codex PASS may

advance the pipeline but cannot produce submission-ready yes. Missing/failed

review emits BLOCKED; overlay/deterministic acceptance uses accepted.

Verify that every claim in the paper matches raw evidence for: $ARGUMENTS

Why This Exists

The executor writes experiments AND writes the paper. It "knows" what the results should be. This creates confirmation bias:

• Rounding 84.7% up to 85.3%
• Reporting best seed instead of average
• Citing metrics from a different experiment config
• Claiming "improves by 15%" when the delta is actually 12.8%

A fresh reviewer with zero prior context catches these because it has no expectations — it just compares paper text vs raw files.

How This Differs From Other Audit Skills

Core Principle

Zero-context, fresh reviewer. The auditor receives ONLY:

• Paper .tex files (the claims)
• Raw result files (the evidence)

It does NOT receive:

• ❌ EXPERIMENT_LOG.md
• ❌ EXPERIMENT_TRACKER.md
• ❌ AUTO_REVIEW.md
• ❌ NARRATIVE_REPORT.md
• ❌ Any executor summary or interpretation
• ❌ Any prior audit results
• ❌ Any conversation history

This is stricter than reviewer-independence — it's zero-context evidence audit.

Workflow

Step 1: Collect Files (Executor — Codex)

Locate paper and result files WITHOUT reading or interpreting them.

Paper files (claims) — paths shown relative to the shell's working directory so you can find them with ls; when writing them into auditedinputhashes, use paths relative to the paper dir (no paper/ prefix) per the "Submission Artifact Emission" section below:

paper/main.tex                # → hash key: main.tex
paper/sections/*.tex          # → hash key: sections/*.tex
paper/tables/*.tex (if separate)   # → hash key: tables/*.tex

Result files (evidence):

results/*.json, results/*.jsonl, results/*.csv, results/*.tsv
outputs/*.json, outputs/*.csv
wandb-summary.json (if exists)
**/metrics.json, **/eval_results.json
**/config.yaml, **/args.json (experiment configs)

Exclude (no summaries, no interpretations):

EXPERIMENT_LOG.md, EXPERIMENT_TRACKER.md, AUTO_REVIEW*.md
NARRATIVE_REPORT.md, PAPER_PLAN.md, findings.md
Any .md file that is an executor-written summary

Step 2: Fresh Reviewer Audit (GPT-6-Astra — NEW thread, no reply)

CRITICAL: Use a fresh reviewer agent every run. Never reuse an old reviewer context for this audit.

spawn_agent:
  model: gpt-6-astra
  reasoning_effort: ultra
  message: |
    You are a paper-to-evidence auditor. You have ZERO prior context about
    this research. You will receive only paper source files and raw result
    files. Your job is to verify that every number in the paper exactly
    matches the raw evidence.

    Paper files to read:
    [list .tex file paths]

    Result files to read:
    [list .json/.csv/.yaml file paths]

    ## Audit Protocol

    ### A. Extract Every Quantitative Claim
    For each number, percentage, comparison, or scope statement in the paper:
    - Location (section, table, caption, or inline text)
    - Exact claim text
    - The number or comparison being made

    ### B. Trace Each Claim to Evidence
    For each extracted claim, find the supporting raw data:
    - Which result file contains this number?
    - What is the EXACT value in that file?
    - Match status: exact_match / rounding_ok / mismatch

    ### C. Check These Specific Failure Modes

    1. **Number inflation**: Paper says 85.3%, raw file says 84.7%
       Rule: only standard rounding to displayed precision is allowed

    2. **Best-seed cherry-pick**: Paper says "achieves

Step 3: Write Report (Executor — Codex)

Parse the reviewer's response and write PAPERCLAIMAUDIT.md:

# Paper Claim Audit Report

**Date**: [today]
**Auditor**: GPT-6-Astra ultra (fresh zero-context thread)
**Paper**: [paper title from tex]

## Overall Verdict: [PASS | WARN | FAIL]

## Claims Verified: [N total]
- exact_match: [count]
- rounding_ok: [count]
- ambiguous_mapping: [count]
- missing_evidence: [count]
- mismatch: [count]

## Issues Found

### [FAIL/WARN] Claim #N: [description]
- **Location**: Section X / Table Y / Figure Z
- **Paper says**: "..."
- **Evidence shows**: ...
- **Status**: [status]
- **Fix**: [specific correction needed]

## All Claims (detailed)

| # | Location | Paper Value | Evidence Value | Status |
|---|----------|-------------|---------------|--------|
| 1 | Table 2 | 85.3% | 85.28% | rounding_ok |
| 2 | Abstract | "15% improvement" | 12.8% | number_mismatch |
| ... |

Also write PAPERCLAIMAUDIT.json for machine consumption.

Step 4: Print Summary

📋 Paper Claim Audit Complete

  Claims verified: 24
  exact_match:     18
  rounding_ok:      3
  ambiguous:         1
  ⚠️ mismatch:      2

  Overall: ⚠️ WARN

  See PAPER_CLAIM_AUDIT.md for details.

When to Run

1. After /paper-write — first check before improvement loop
2. After /auto-paper-improvement-loop — recheck if improvement loop changed numbers
3. Before submission — final verification

Integration with Other Skills

Read by /auto-paper-improvement-loop (if exists)

if PAPER_CLAIM_AUDIT.json exists:
    read mismatched claims
    fix them as priority items in the improvement round

Advisory, Never Blocking

Same pattern as /experiment-audit:

• PASS → continue normally
• WARN → print warning, continue, flag draft as "check numbers before submission"
• FAIL → print alert, continue, but do NOT mark as submission-ready

Render HTML view (auto, when RENDER_HTML = true, default)

After writing paper/PAPERCLAIMAUDIT.md and paper/PAPERCLAIMAUDIT.json, invoke /render-html on the audit report:

/render-html "paper/PAPER_CLAIM_AUDIT.md" --json "paper/PAPER_CLAIM_AUDIT.json"

Uses full review gate (audit-class artifact; base Codex review is fresh same-family provisional). Output: paper/PAPERCLAIMAUDIT.html with embedded source SHA256 + .review.json sidecar.

Non-blocking: if /render-html fails (helper missing, secondary Codex agent unavailable, file write error), log the failure and treat the audit as complete — the JSON + MD verdict files are canonical; the HTML view is a human-reader convenience.

Skip if RENDER_HTML = false is set in AGENTS.md / CLAUDE.md or passed as — render html: false.

Key Rules

• Fresh thread EVERY run. Never use a continuation reply. Never carry context.
• Zero executor interpretation. Only file paths. No summaries.
• Only raw results. No EXPERIMENTLOG, no AUTOREVIEW, no human summaries.
• Rounding rule. Only standard rounding to displayed precision. 84.7% → 84.7% or 85% is OK. 84.7% → 85.3% is NOT OK.
• Review class. Base Codex reviewer is same-family provisional; only an overlay may record cross-family accepted.

Review Tracing

After each reviewer agent call, save the trace following shared-references/review-tracing.md (Policy C — forensic; never silently skip). Use savetrace.sh (resolved per the chain in shared-references/integration-contract.md §2) or write files directly to .aris/traces//run/. Respect the --- trace: parameter (default: full).

Submission Artifact Emission

This skill always writes paper/PAPERCLAIMAUDIT.json, regardless of caller or detector outcome. A detector-negative run (paper has no numeric claims) emits verdict NOTAPPLICABLE; a paper-with-numeric-claims-but-no- raw-results run emits BLOCKED. Silent skip is forbidden — paper-writing Phase 6 and verifypaper_audits.sh both rely on this artifact existing at a predictable path.

The artifact conforms to the schema in shared-references/assurance-contract.md:

{
  "audit_skill":      "paper-claim-audit",
  "verdict":          "PASS | WARN | FAIL | NOT_APPLICABLE | BLOCKED | ERROR",
  "reason_code":      "all_numbers_match | rounding_drift | missing_raw_results | ...",
  "summary":          "One-line human-readable verdict summary.",
  "audited_input_hashes": {
    "main.tex":                              "sha256:...",
    "sections/5.evidence.tex":               "sha256:...",
    "/abs/path/to/results/run_2026_04_19.json": "sha256:..."
  },
  "trace_path":       ".aris/traces/paper-claim-audit/<date>_run<NN>/",
  "thread_id":        "<codex mcp thread id>",
  "executor_model":   "codex-gpt-6-astra",
  "executor_family":  "openai",
  "reviewer_model":   "gpt-6-astra",
  "reviewer_family":  "openai",
  "review_independence": "same-family",
  "acceptance_status": "provisional",
  "reviewer_reasoning": "ultra",
  "generated_at":     "<UTC ISO-8601>",
  "details": {
    "total_claims":   <int>,
    "mismatches":     [ ... per-claim issue records ... ],
    "result_files":   [ ... raw files consulted ... ]
  }
}

auditedinputhashes scope

Hash the declared input set passed into this audit invocation — i.e. the exact .tex files and raw result / config files this run read — not a repo-wide union and not the reviewer's self-reported subset. If a caller passed only main.tex + a single result file, hash those two files and no others. The external verifier rehashes these entries; any mismatch flags STALE.

Path convention (must match what verifypaperaudits.sh expects): keys are paths relative to the paper directory (the arg passed to the verifier) for in-paper files — so main.tex, not paper/main.tex — and absolute paths for out-of-paper files such as external results/ dirs. The verifier resolves relative entries via os.path.join(paper_dir, key); prefixing with paper/ produces paper/paper/main.tex and false-fails as STALE.