falsepass MCP server

falsepass

<!-- mcp-name: io.github.Super-Eric/falsepass -->

English | 中文

Probe RL environment verifiers for false passes (a known-bad completion is rewarded) and false fails (a known-good completion is not).

Website and docs: https://super-eric.github.io/falsepass/ · write-up: Your reward function is an assertion

In one public environment, echoing the question back earned full reward on every task: each question listed all of its candidate answers, and the grader only checked that the right one appeared somewhere in the reply, so a policy could collect full reward by listing everything. falsepass finds this kind of hole before training does.

An RL environment is a task dataset plus a verifier that turns a model's output into reward. falsepass treats that verifier as the system under test: it calls the environment's own reward code on completions whose correctness is known in advance and reports every disagreement. No model, no rollouts, no GPU.

It works with three frameworks, published or still in a local folder:

Install

Needs uv on the PATH (every audit runs in its own throw-away virtualenv) and Python 3.11 or newer.

uvx falsepass demo              # try it, nothing to install
uv tool install falsepass       # or keep `falsepass` on the PATH
pip install falsepass           # or into an existing environment
uvx --from git+https://github.com/Super-Eric/falsepass falsepass --version   # the unreleased main branch

Quick start

falsepass demo                            # audits a bundled toy environment with one planted defect (about a minute)
falsepass init owner/environment          # a Hub environment: package, version and verifiers pin read from the Hub
falsepass init ./my-environment           # or your own, from its folder (pyproject.toml); edits are picked up on every run
falsepass audit my-environment            # probe it; results in audits/my-environment/summary.md
falsepass results my-environment --json   # headline numbers and failing probes, for scripts and agents

demo writes the toy environment to ./falsepass-demo and audits it. Its grader pays when the reference appears anywhere in the answer, and the summary shows that one line from several sides: the static scan points at it, and the probes that list several answers (P5), retract the answer (P9) or deny it inside the answer slot (P15) are paid. A condensed excerpt of the real output (full file: examples/toy-env/summary.md):

Shortened. The full README is on GitHub.