Agent Eval MCP server

Agent Evaluation

<!-- mcp-name: io.github.RudrenduPaul/agent-eval --> <!-- Ownership-proof string for registry.modelcontextprotocol.io publishing. Do not remove. -->

Statistical regression testing for LLM agents: run version A and version B 50x each and get a p-value, an effect size, and a 95% confidence interval on whether behavior actually shifted.

pip install agent-regress-cli

(uv add and the npm/npx wrapper are covered in Install below.)

The test that every eval framework skips

You changed a prompt. Your evals still pass. But your agent's tool accuracy dropped from 84% to 70%.

Is that a real regression? Or is it LLM run-to-run noise?

Threshold testing cannot answer that question. agent-eval can.

Run your agent 50x on version A, 50x on version B. Get a p-value, an effect size, and a 95% confidence interval on whether behavior actually shifted.

p=0.003, Cohen's d=-0.61 -> REGRESSED (deploy blocked)
p=0.410, Cohen's d=0.021 -> STABLE (safe to ship)

[!NOTE]

Promptfoo, one of the most widely used open-source LLM eval frameworks, was acquired by OpenAI in March 2026, staying open source but folding its team into OpenAI's Frontier platform. agent-eval is Apache 2.0-licensed, self-hostable, and has no commercial dependency. The statistical core (Mann-Whitney U, bootstrap CI, Cohen's d) will never be paywalled.

Install

pip install agent-regress-cli
# or
uv add agent-regress-cli
# or, from Node/npx (thin wrapper around the same Python CLI)
npx agent-regress-cli

Quickstart

In 30 seconds (CLI)

Already have per-run scores from your own harness? Point the CLI at two JSON arrays of scores, one per version:

pip install agent-regress-cli

agent-regress compare \
  --version-a-results v1_scores.json \
  --version-b-results v2_scores.json \
  --metric tool_accuracy

# ============================================================
# agent-regress Report -- tool_accuracy
# ============================================================
# Verdict:    REGRESSED
# p-value:    0.0000
# Cohen's d:  -2.193
# 95% CI:     [-0.213, -0.148]
#
# Version A:  0.8470 +/- 0.0525  (n=50)
# Version B:  0.6685 +/- 0.1025  (n=50)
# Delta:      -0.1786
# ============================================================

Add --json --fail-on-regression to get clean, parseable output and a non-zero exit code on REGRESSED, for wiring straight into CI.

Shortened. The full README is on GitHub.