eval-integrity skill

Evaluation Integrity

Provide a read-only audit of whether an evaluation can support its stated claim. Do not alter code, data, prompts, labels, or study operations. Begin with exactly one verdict line in the form Verdict: PASS, Verdict: FAIL, or Verdict: INCONCLUSIVE. Emit no other verdict line and do not restate the verdict later in the report. A favorable score is not itself evidence of integrity.

Route correctly

review-work.

• NOT for code review: route implementation, correctness, security, or maintainability review to

data-scientist.

• NOT for data processing: route cleaning, transforming, joining, modeling, or analyzing data to

result from it.

• Use this skill for the independence and validity of the measurement setup, not for producing a

Audit method

1. State the claim and map the evaluation

Write the claim in testable form: what comparison, metric, threshold, or causal statement is being made; for which population; and under which decision rule. Map every role before judging it:

Trace shared ownership, shared data, shared prompts, shared labels, and shared selection decisions across this map. Several models do not establish independence merely because there are several of them; identify distinct training, tuning, prompt design, labels, data, and decision authority.

2. Check the ground truth

Identify the independent ground truth that could confirm the claim without reusing the system's own preferences, outputs, labels, or optimization target. Test whether it is:

• external to the system under test and its designer;
• collected or adjudicated without seeing the treatment identity when blinding is feasible;
• appropriate to the claim rather than a proxy selected because it favors one arm; and
• fixed before comparison, or accompanied by a disclosed multiplicity and selection procedure.

If no independent ground truth exists, do not substitute agreement among related systems. Mark the claim INCONCLUSIVE unless the stated claim is explicitly limited to that related system's score.

3. Probe independence failures

Check only risks relevant to the mapped design:

retrieval, tuning, or post-hoc selection.

• Contamination: benchmark items, answers, or near duplicates reached training, prompt design,

unequal, or the control cannot answer the causal comparison.

• Invalid controls: arms differ beyond the intended treatment, allocation is biased, exposure is

success without external confirmation.

• Circular judging: the evaluated system, its designer, or a derived rubric defines its own

scorer in a way that makes agreement self-fulfilling.

• Scorer-label coupling: labels, thresholds, reward signals, or rubric choices are reused by the

tuned, or influenced by the comparison designer without an independent safeguard.

• Evaluator-designer coupling: the evaluator, judge prompt, or adjudication process is selected,

annotators, or correlated annotation policies invalidate assumed independence.

• Split or annotator dependence: overlapping sources, duplicates, temporal leakage, shared

evaluators or analysts toward the expected outcome.

• Hypothesis priming: task framing, examples, pre-registration gaps, or interim results steer

behavior or ratings change because of that knowledge.

• Observer and demand effects: subjects or observers infer the desired treatment or outcome, and

For each concern, distinguish documented evidence from an assumption. Missing provenance for data, splits, labels, prompts, evaluator selection, or scoring is INCONCLUSIVE, not evidence that the design was independent.

4. Reach the verdict

material unresolved dependency.

• PASS: evidence supports an independent, valid measurement path for the stated claim, with no

the stated claim.

• FAIL: evidence shows a dependency or invalid design that materially invalidates or overstates

established, or remaining uncertainty prevents either result.

• INCONCLUSIVE: provenance or evidence is missing, the ground truth is not independently

Do not convert uncertainty into PASS because a metric looks plausible, a result is statistically significant, or multiple related evaluators agree.

Report

Report the verdict first. Then report only findings that fired; do not pad the report with risks that were checked and did not fire. For each fired finding include:

adjudication, or control change that would address it.

1. Evidence: the observed artifact, provenance record, or explicit absence of one.
2. Impact: which claim or inference the issue weakens or invalidates.
3. Minimal independence fix: the smallest separation, blind, held-out source, external

1. Missing evidence: what must be supplied to resolve the finding, if applicable.
2. Residual risk: what remains uncertain after the minimal fix.

Keep conclusions bounded to the evidence. This audit advises; it does not certify a system, recompute results, or approve deployment.