retriEVAL MCP server

retriEVAL

LLM evaluation as an MCP server. Score your AI's outputs for faithfulness, relevancy, and hallucination from inside any MCP client — no pipeline, no test harness. Every result comes back with a link to a dashboard that keeps the history.

Try it live (no signup) · Watch the 2-minute demo · Dashboard

Why

Five customer-support answers, scored on two metrics:

Every answer was on-topic and well-written. Two of them contradicted the policy they were supposedly grounded in — one promised free return shipping the policy doesn't offer, another invented a free overnight replacement. Reviewing by eye, you'd sign off on all five.

That gap is the point. Relevancy asks did it answer the question. Faithfulness asks is it actually in the source. You need both, and the second one catches the expensive failures.

Connect

Self-hosted. Clone it, point it at a judge, run it — your data never leaves your machine and there's no service to sign up for.

git clone https://github.com/hcarrillo001/retrieval-mcp
cd retrieval-mcp
pip install -r requirements.txt

export ANTHROPIC_API_KEY=sk-ant-...        # or a local judge, below
python server.py                            # stdio, for Claude Desktop / Cursor

Then just ask:

Score these cases with faithfulness: [{"input": "...", "actualoutput": "...", "retrievalcontext": ["..."]}]

Pass your cases inline and nothing is stored — one call, no setup step. See Run locally (stdio) for client config, and Deploy as HTTP if you want your own always-on instance with a dashboard.

Want to try it before installing anything? There's a live sandbox at retrieval-mcp.com — no signup, runs on a free judge, nothing saved.

Keeping everything local: set RETRIEVALJUDGEBACKEND=ollama and the judge runs on your machine too, so no data leaves your network at any point. Useful if you're evaluating anything you can't send to a third party.

What you get

model, so nothing has to leave your network

• 9 built-in metrics plus custom metrics you author in plain English
• Swappable judges — Anthropic, Groq, Gemini, OpenRouter, or a local Ollama

• Golden sets from files, URLs, inline JSON, JSONL, CSV, or TSV
• Run history in Supabase with shareable permalinks and run comparison
• A spend cap, because a judge-based tool can otherwise run up a bill

Honest limitations

scores as a signal rather than ground truth.

• Judge agreement hasn't been validated against human labels yet, so treat

model outputs means loading a second set. Splitting them is the next change.

• Golden sets currently hold their own outputs, so re-running one against new

restart. Runs persist; those don't.

• Golden sets and authored metrics live in the server process and are lost on

Metrics (DeepEval-aligned)

faithfulness · answerrelevancy · contextualprecision · contextualrecall · contextualrelevancy · hallucination · bias · toxicity · summarization — plus authored G-Eval metrics you define in plain language. All are normalized so higher = better (bias/toxicity report the clean fraction), and each reasons before scoring.

Versatile golden sets

loadgoldenset accepts a file path (including uploaded files), an http(s) URL, an inline JSON array, or JSONL text, in JSON / JSONL / CSV / TSV. Field names are auto-normalized (question→input, answer→actualoutput, groundtruth→expectedoutput, contexts→context, passages→retrievalcontext, …), so most public benchmarks load as-is.

Short replies by default

Shortened. The full README is on GitHub.