MCP.Market

Llmtest MCP server

by tjacquesson·io.github.tjacquesson/llmtest-mcp·v0.6.1
No reviews yet

Benchmark AI models on real prompts. Find cheaper, faster alternatives across 340+ models.

B72/100grade B
Write a reviewInstall
What users say
No reviews yet
Be the first
Safety scan
B72/100

full report

Adoption
New

0 stars40 downloads/wk

Reviews

Write one

Nobody has reviewed Llmtest yet.

If you have run it, two minutes of your experience saves the next person an afternoon.

Llmtest tools (15, 2 write)

write = sends, deletes, buys or posts

Read from the package source without running it. The installed server may list more.

  • disable_autopilot

    Turn off autopilot. Already-accepted optimizations stay active; only future runs are paused. Use enable_autopilot to turn it back on.

  • enable_autopilot

    Turn on autopilot. Once enabled, LLMTest will automatically optimize eligible flows every 7 days, auto-accept changes that pass all safety gates, and auto-revert if drift detection later sees regressions. Cost: each optimization run costs ~$1-2, billed to LLMTest credits. Runs are skipped silently if the account has no credits. Prerequisites: verified email, account age >= 14 days, and >=20 real c

  • get_account

    Check your credit balance and usage stats.

  • get_autopilot_status

    Show whether autopilot is enabled and whether the account meets the prerequisites. Autopilot automatically optimizes prompts and models for eligible flows every 7 days. Checks: optimization scheduler (weekly), drift detection (daily scan, per-optimization weekly). Prerequisites for a flow to be auto-optimized: - Autopilot enabled on the account - Email verified - Account age >= 14 days (trust ramp

  • get_suggestions

    Get pending recommendations to switch to a cheaper or faster model without losing quality.

  • list_active_optimizations

    List auto-accepted optimizations still within their 24h revert window. Each includes the flow, strategy, savings, and hours remaining to revert.

  • list_flows

    Show all AI features in this project with their current model, call volume, latency, and cost.

  • list_new_models

    Show new and trending models that are worth benchmarking soon.

  • list_samples

    Show how many test samples are stored for each flow.

  • optimize_prompt

    Analyze a flow's prompt and automatically generate an optimized version. Tests optimized prompt against baseline using real samples and a multi-judge AI (Claude Sonnet + GPT-4o, position-swapped, self-consistency). Also tests cheaper models. REQUIRES: at least 3 samples on the flow. If the flow has none (zero traffic), BEFORE calling this tool you should: (1) ask the user to describe the feature,

  • revert_optimization

    Revert an auto-accepted optimization. Only works within the 24h revert window. Restores the previous active prompt. Use list_active_optimizations to find IDs.

  • run_benchmarkwrite action

    Benchmark a flow against alternative models. Use optimize_for to focus the search: "cost" tests cheaper models, "quality" tests potentially better models, "speed" tests faster models, "balanced" (default) tests a diverse spread. Use challengers to test specific models (e.g. ["anthropic/claude-sonnet-4", "google/gemini-2.5-flash-preview"]). If omitted, challengers are selected automatically based o

  • seed_samples

    Register test samples for a flow so it can be benchmarked without real traffic. Use this for pre-launch projects or when you want to benchmark a specific scenario. Each sample must be an array of messages in OpenAI chat format: [{ "role": "system", "content": "..." }, { "role": "user", "content": "..." }] To use: read the codebase to find the prompt for this flow, then generate 5-10 realistic and

  • status

    Show LLMTest proxy status and a summary of monitored AI activity.

  • update_suggestionwrite action

    Accept or dismiss a model-switch suggestion by ID.

Public scan report

scanner v0.1.9 · 2026-09-27 · same rubric, same numbers if you re-run it

no findings
  • Code scan3 source files scanned25/25
  • –Live reliabilityno gateway calls yet and no remote to proben/a
  • –Tool poisoningtools not inspected (local package is not executed); not countedn/a
  • Auth qualitystatic API keys via environment variables6/15
  • Maintenancelast push 119 days ago8/15
  • Maintainer identityregistry namespace matches repository owner; GitHub account older than a year8/10
Overall 72/100. Components that don't apply are left out of the denominator. Any critical finding is an F.RubricAppeal a findingJSON

How Llmtest compares

Against every graded MCP server, from our own scans and uptime checks. Updated after each scan run.

Safety score
72/100
higher than 38% of 48,255 servers
Tools
15
the typical server has 7
Tools that change things
13%
typical: 0%. Fewer write tools means less an agent can break
Downloads a week
40
more than 31% of 12,191 servers with a package

What its tools let an agent do: 11 read · 3 change · 1 run or deploy

Install Llmtest in Claude Code, Cursor or VS Code

Runs npx -y llmtest-mcp on your machine. Read the scan report first; the gateway never runs local packages.

claude mcp add llmtest-mcp -- npx -y llmtest-mcp
Add to Cursor

What the publisher says

From the Llmtest repository's README, as published. We do not edit it. Read it on GitHub

LLMTest MCP Server

<!-- mcp-name: io.github.tjacquesson/llmtest-mcp -->

MCP server that benchmarks AI models on your actual prompts and finds cheaper, faster alternatives. Works with Claude Code, Cursor, Windsurf, and any MCP-compatible tool.

Quick Start

1. Get your API key

Sign up at llmtest.io and grab your API key from the dashboard.

2. Add to your tool

Claude Code:

claude mcp add llmtest -- npx llmtest-mcp

Then set your key:

export LLMTEST_API_KEY=llmt_your_key_here

Cursor / Windsurf / Other MCP clients:

Add to your MCP config file:

{
  "mcpServers": {
    "llmtest": {
      "command": "npx",
      "args": ["llmtest-mcp"],
      "env": {
        "LLMTEST_API_KEY": "llmt_your_key_here"
      }
    }
  }
}

3. Talk to your AI

Just ask in natural language:

  • "Check my LLMTest status"
  • "Find cheaper models for my AI calls"
  • "Run a benchmark on my blog-writer flow"
  • "What models are trending?"

How It Works

LLMTest is a proxy that sits between your app and AI providers. Point your app at https://llmtest.io/v1 instead of calling OpenAI/Anthropic directly, and LLMTest tracks your usage, benchmarks alternatives, and suggests cost savings.

This MCP server gives your AI assistant access to LLMTest's tools so it can manage everything for you.

Available Tools

Autopilot

Autopilot automatically optimizes your flows on a weekly cadence. Changes that pass every safety gate go live with a 24-hour revert window. Drift detection keeps checking after that and rolls back if quality slips.

To enable from your IDE: ask your AI assistant something like "enable LLMTest autopilot". It will call enableautopilot. Use getautopilot_status to confirm prerequisites.

Prerequisites (checked per flow each cycle):

  • Autopilot enabled on the account
  • Email verified
  • Account age ≥ 14 days (trust ramp)
  • Flow has ≥ 20 real calls in the last 7 days
  • Flow not optimized by autopilot in the last 14 days (cooldown)
  • Positive credit balance (~$1–2 per run)

Safety gates (all must pass for auto-accept): 95% CI lower bound > 50% win rate, multi-judge agreement ≥ 80%, ≥ 20% total savings, no length-bias warning, golden-set regression check.

Revert: 24h window after auto-accept. After that, only drift detection can roll back.

Typical Workflow

Pre-launch (no traffic yet):

  1. Tell your AI: "I'm building a support chatbot using gpt-4o"
  2. It seeds realistic test samples with seed_samples
  3. It runs run_benchmark to compare models
  4. It shows you get_suggestions with cheaper alternatives

Shortened. The full README is on GitHub.

Nothing above is checked by us. What we check is on the safety report.

Llmtest: common questions

Is Llmtest MCP server safe?
Mostly: it is graded B (72/100). Read the Llmtest safety report
How do I install Llmtest?
It runs on your machine. Copy the Claude Code, Cursor, VS Code or Claude Desktop config from the install section.
Does Llmtest need an API key?
Yes. The registry entry asks for LLMTEST_API_KEY.
Is Llmtest maintained?
The last commit was 157 days ago (2026-05-31). The latest release is v0.6.1.
How does Llmtest compare with other MCP servers?
safety score 72/100 (higher than 38% of 48,255 servers); tools 15 (the typical server has 7); tools that change things 13% (typical: 0%. Fewer write tools means less an agent can break). Full comparison.