LEEVAR reliability battery MCP server

LEEVAR Battery

An 18-test, 6-dimension reliability battery for AI agents. We run it, we grade it, and any dimension your sample cannot evidence comes back NOT TESTED instead of a guess.

Point it at a transcript or a live endpoint and get an honest A–F back — per dimension, with the coverage it was computed on.

curl -s https://rzmsvvalhaqvxbuhpdnb.supabase.co/functions/v1/api-scan \
  -H 'content-type: application/json' \
  -d '{"agent":{"name":"my-agent"},"mode":"transcript","email":"you@example.com",
       "transcript":["<a real work sample, not marketing copy>"]}'

No account. No API key. 5 free scans a month. Results are emailed and queryable.

Why NOT TESTED is the point

Most agent evals average over whatever they managed to measure. If a probe found no evidence, that silence quietly becomes a number, and the number becomes a grade.

This battery refuses. A dimension your sample cannot evidence is reported NOT TESTED and excluded from the composite — and an agent with unverified dimensions does not clear the gate, however high the rest scored.

We know the failure mode first-hand: our own grader once scored a fabricated $150 refund 100/100 on truthfulness, because every claim was "directly based on tool call results". The claims were. The tool results were fiction. That is why transcript-mode evidence is treated as the agent's own testimony.

When we refuse to grade

This is the part worth reading before you spend a call on us.

We issue a letter only when at least two thirds of the probes find evidence in what you send — 13 of 18, as the battery stands today. Below that you get PARTIAL: every dimension we could measure, scored honestly, the rest marked NOT TESTED, and the reference average labelled so nobody mistakes it for a verdict.

PARTIAL — 12/18 tests evidenced, no grade issued

We are not grading this agent. Only 12 of 18 probes found evidence in your
samples, and an average over 12 tests is not a reliability grade — it is a
coin toss with a letter on it.

For reference only, over the 12 evidenced tests: 96.2/100.
Do not deploy on this number.

That is a real scan, and it is one of ours — the one in examples/, rendered by the code in this repository. 96.2 is higher than a scan we published an A for. The gate refused it anyway, on coverage. (The report on the day said 96.6; why the two differ.)

What clears the line: real transcripts rather than marketing copy, and enough of them to exercise the behaviour — a long thread for context handling, repeated runs for consistency, an induced failure for recovery. A thin sample does not produce a generous score; it produces NOT TESTED.

Run it yourself — the scanner is in this repository

Everything above describes the hosted run. The engine behind it is here, under MIT, and it grades locally against a model you choose. No account, no key of ours, nothing leaves your machine unless you point it at a provider.

# Deno 2.x — one binary, no package install, no node_modules
curl -fsSL https://deno.land/install.sh | sh

git clone https://github.com/forevercrab321-svg/leevar-battery
cd leevar-battery

deno task demo      # the full 18-test battery, no model, no network, no cost

The demo grades a sample conversation that contains the agent's own TOOLRESULT lines. Six probes are refused before any judge is called, twelve find evidence, and the run ends with no grade** — which is the whole point, reproduced offline, in under a second, with nothing to configure:

## PARTIAL — 12/18 tests evidenced, no grade issued

Then point it at a real model and a real transcript:

# see where your transcript would go, and what it could cost, before sending it
deno task scan --transcript ./samples.json --provider anthropic --dry-run

# provider  anthropic
# endpoint  https://api.anthropic.com/v1/messages

Shortened. The full README is on GitHub.