Mmcp.market

differential-audit skill

by pedrohcgs·pedrohcgs/claude-code-my-workflow·1.6k stars·MIT

Compare two implementations of the same thing — a port (R↔Python↔Stata), a reimplementation, a replication package, a refactor, or a new version against the old — so that agreement means something. Freeze inputs first, inventory every expected output, test the comparator itself, compare every channel (not just the headline number), and give each divergence a stable ID and a smallest witness. Use for cross-language parity, replication, upgrade/regression gates, or whenever "the numbers match" is about to license a claim.

A100/100content scan

Is the differential-audit skill safe?

Clean: nothing in its files matched our rules. We read 1 file in the folder on 2026-09-28.

No findings.

Install the differential-audit skill

A skill is a folder. Copy it into your agent's skills folder and the agent loads it when the task matches its description. 

git clone --depth 1 https://github.com/pedrohcgs/claude-code-my-workflow.git /tmp/claude-code-my-workflow
mkdir -p ~/.claude/skills
cp -r /tmp/claude-code-my-workflow/.claude/skills/differential-audit ~/.claude/skills/differential-audit
available in every project

In the Claude apps, zip the folder and upload it from the Skills settings. The folder on GitHub

The instructions your agent would load

SKILL.md as published, without the frontmatter. Read it on GitHub

Make agreement mean something

Two implementations agreeing proves they satisfy a prespecified contract. It does not prove either is correct, and it never validates the method or its assumptions. Both can be wrong in the same way — especially when one was written by reading the other. Design the comparison so that agreement is informative and disagreement is legible.

Rule: freeze before you compare; test the comparator before you trust it.

1. State the claim and the reference

Write down: what is being compared, which side is the reference, and what agreement would and would not establish. "Matches the R package" is a conformance claim, not a correctness claim. Say so explicitly, so nobody later reads parity as validation.

2. Freeze the inputs before inspecting anything

Record and fix: data versions or hashes, code and package versions, random seeds or realized sample splits, options and defaults, the outputs to be compared, and the acceptance thresholds. Freezing after a first look invites tolerance drift toward whatever the run produced.

Do not compare defaults across systems as if only the language changed. Map the choices explicitly — a "default" is a substantive modeling decision that usually differs between implementations.

3. Declare tolerance classes, and make them binding

Do not carry a single fuzzy epsilon. Classify each output:

  • EXACT — names, ordering, sample masks, counts, statuses, return/error codes, warning classes, option defaults. Byte-equal after documented normalization.
  • Scalar numeric — deterministic estimates, standard errors, p-values, critical values. State absolute and relative tolerances and the justification.
  • Matrix/vector — covariance matrices, influence summaries, weight vectors, plot data.
  • Stochastic — must meet a prespecified error-rate criterion with uncertainty reported.

A looser tolerance may be used only through a recorded approved divergence with a reason. Silent widening is the most common way a parity gate stops testing anything.

4. Crosswalk and output inventory

Write the mapping between the two implementations, plus an inventory of every expected output with expected row/cell counts. Every declared object must have a live comparison or an explicit out-of-scope reason. Without an inventory, both sides can silently omit the same result and the comparison reports success.

5. Build cases that isolate mechanisms

Not just the happy path:

  • analytically solvable or known-truth cases;
  • relevant data problems (missing, unbalanced, near-collinear, extreme-but-valid weights, shuffled row order, ties);
  • compound cases combining several problems;
  • published examples;
  • randomized valid designs across the supported surface, not a handful of fixtures.

Fixed fixtures are necessary but not sufficient — they test what the author already thought of.

6. Test the comparator itself

Before trusting a green result, feed the comparison a wrong value, a missing result, a misaligned row, and an empty result. It must fail, not skip. A comparator that silently passes over what it cannot reconcile turns every subsequent green into noise. (See vaccinate.)

7. Compare every declared channel

Not only the headline coefficient: estimates, uncertainty measures, sample counts, labels and ordering, diagnostics, warnings, and failure statuses. Divergent warnings and differing error behavior are real defects — they change what a user does next.

8. Give every divergence an ID and a smallest witness

For each difference: a stable identifier, the smallest reproducible case, and a classification — defect / intentional difference / limitation of the reference / unresolved. Unresolved stays red; it is not averaged away or waived. A fix must turn its witness green and survive a rerun of the full audit, so a local patch does not break something else.

9. Adversarial expansion by someone else

Fixtures written by the implementer test the implementer's mental model. Have an independent reviewer add designs not shared in advance, and preserve any that reveal bugs or materially increase coverage as permanent fixtures. This is the cheapest defense against a suite that passes because it was written to pass.

10. Close with separate reviews and an honest scope statement

End with distinct scientific and implementation sign-off, recording: checks run, open findings, accepted differences, explicit non-claims, and who approved release. State plainly that the audit establishes conformance to the frozen contract — not the truth of the method.

Minimum checklist

  1. Name the reference; state what agreement would and would not establish.
  2. Freeze versions, hashes, seeds, options, outputs, tolerances.
  3. Declare binding tolerance classes; record any approved divergence.
  4. Crosswalk + expected-output inventory with counts.
  5. Known-truth, dirty, compound, and randomized cases.
  6. Seed comparator faults — it must fail, not skip.
  7. Compare all channels, including warnings and failures.
  8. Stable ID + smallest witness + classification per divergence; unresolved stays red.
  9. Independent reviewer adds unseen designs.
  10. Separate sign-offs; state the non-claims.

Cross-references

  • provenance-and-ground-truth.md — ranked oracles, declared precedence, the divergence taxonomy
  • .claude/rules/replication-protocol.md — the tolerance contract

More skills from pedrohcgs/claude-code-my-workflow

  • Aadjudicate-reviewTurn an incoming set of findings — from an AI reviewer, a referee report, a code review, a linter, or a second model — into verified fixes, without letting a confident misread damage correct work. Every finding is a CANDIDATE until checked against the actual source. Use whenever you receive review comments, audit findings, or a critique you did not write yourself, especially when the reviewer is a model or when the volume is too large to check by feel.
  • Aaudit-reproducibilityEnforce the replication-protocol.md rule by cross-checking numeric claims in a manuscript against the actual R / Stata / Python outputs. Report PASS/FAIL per claim against tolerance thresholds. Use before submission and before releasing a replication package.
  • Ablast-radiusBefore and after changing anything shared — a function's return value, a signature, a schema, a label set, a config default, a constant, a file format — find every consumer and actually run them. Catches the change that looks purely additive but silently breaks a contract in a file you never opened. Use when editing shared code, adding a field/column/return element, renaming, changing units or defaults, or touching a pipeline that produces reported numbers.
  • Acapture-environmentSnapshot the computational environment for a replication package — detects the analysis stack (R / Stata / Python) and emits the right lockfiles (renv.lock + sessionInfo.txt, requirements.txt / environment.yml / uv.lock, Stata version + ado package list), records seeds and RNG kind, optionally writes a pinning Dockerfile, and produces a paste-ready "Computational requirements" block. Use when user says "capture the environment", "snapshot my dependencies", "pin the versions", "make a renv.lock / requirements.txt", "make this byte-reproducible", or before releasing a replication package to openICPSR / the AEA Data Editor.
  • AchallengeStress-test a finding against the choices you did not make. Enumerates the discrete forks a competent analyst could have taken (measure definition, sample filter, control set, clustering level, weighting, functional form), runs the specification grid, and reports the distribution rather than a point estimate — then attacks the identifying assumption with named, computable sensitivity statistics. Use when the user says "is this robust", "challenge this result", "specification curve", "multiverse", "how sensitive is this", "what if I'd used a different measure", "stress-test my estimate", or before a result becomes a headline claim. NOT a reviewer of prose or code — it challenges the CLAIM.
  • AcheckpointSave a structured state snapshot before stopping or handing off. Captures the active plan, recent decisions, file pointers (with line numbers), open questions, and the next 1–3 actions into a checkpoint file under `quality_reports/checkpoints/`. Optionally proposes `[LEARN]` entries to add to MEMORY.md. Use when user says "checkpoint", "save state", "snapshot before I stop", "where am I", "wrap up the session for handoff", or before a long break / model switch / collaborator handoff. Companion to (NOT replacement for) the narrative session-log workflow.
  • Acoauthor-briefGenerate a co-author / collaborator handoff brief for a multi-author, multi-machine project — summarizing what changed since the last brief (git delta), the current state of each artifact (manuscript, analysis, slides), open questions, how to reproduce locally, and any restricted-data access steps. Use when user says "coauthor brief", "handoff brief", "bring my coauthor up to speed", "what changed since last week", "onboard a collaborator", "write a handoff for [name]", or before sending a co-author the repo. NOT a commit or a checkpoint — it is the cross-machine, cross-person summary `meta-governance.md` only partially covers.
  • AcommitCommit the current work — runs the quality, consistency and passport gates, branches off main if needed, stages specific files, and writes a commit whose subject states what is now true. Pushes and opens a pull request only with --pr or when the user asks; never merges — a merge happens only when the user explicitly says to merge. Use ONLY on explicit commit intent — user says "commit", "let's commit this", "open a PR", or prefixes with `/commit`. Do NOT auto-invoke on vague end-of-task phrases ("we're done", "wrap up") — those require explicit confirmation first. Never force-pushes or skips hooks.
  • Acompile-latexCompile a Beamer LaTeX slide deck with XeLaTeX (3 passes + bibtex). Use when user says "compile", "build the slides", "rebuild the PDF", "run latex", "render the tex", or asks why a `.tex` file isn't producing a PDF. Operates on `Slides/*.tex`.
  • Acompress-sessionDistill the current conversation into a structured note (decisions made, open questions, file pointers with line numbers, next 1–3 actions) and save to `quality_reports/session_logs/` before auto-compression. Differs from `/checkpoint` (explicit stop-point snapshot) and from auto-compaction (which truncates rather than distills). Use when context is approaching auto-compact threshold, when a long pipeline has accumulated many decisions, or when the user says "compress", "distil this session", "before we hit auto-compact", "structured handoff before context resets".
  • Acontext-statusShow current context status and session health. Use to check how much context has been used, whether auto-compact is approaching, and what state will be preserved.
  • Acreate-lectureCreate a new Beamer lecture `.tex` from source papers and materials, with notation consistency checks and the project's preamble wired in. Use when user says "create a lecture on X", "new lecture from these papers", "start a deck on topic Y", "scaffold a new Beamer file", "build me a lecture from these PDFs". Scaffolds the full deck — NOT for compiling existing `.tex` (use `/compile-latex`).

All agent skills → · MCP servers