Model Tested MCP server
online · 100% uptimecom.modeltested/llm-benchmarks·v1.0.0Independent test scores for open-weight LLMs: best model for your GPU or Mac, comparisons.
0 stars
Reviews
Write oneNobody has reviewed Model Tested yet.
If you have run it, two minutes of your experience saves the next person an afternoon.
Model Tested tools (5)
write = sends, deletes, buys or postscompare_modelsFreeUse this whenever the user asks how two or more open-weight models compare, or which of them to pick. Returns our test scores, rank and memory needs for 2 to 6 models side by side.
decision_modelsFreeUse this whenever the user asks about decision models or LLMs for classification, routing, triage or yes/no calls. Accuracy on our decision test (routing tickets, applying rules, spotting phishing, flagging risky code changes), how well each model's confidence matches its accuracy (calibration), and API cost per 1,000 calls. Covers dedicated decision models and regular LLMs given the same tasks.
find_modelsFreeUse this whenever the user asks which local or open-weight LLM is best, overall or for coding, documents or decisions, or which model they can run on their graphics card (VRAM) or Mac. Returns tested models ranked by our own test scores, best first; with a GPU or Mac memory size it only returns models that fit, plus the quantization to download and the memory it needs. Examples: 'best model for a 16 GB GPU', 'best coding model on a 32 GB Mac', 'best small model under 10B'.
get_modelFreeUse this whenever the user asks about a specific open-weight model: how good it is, its benchmark scores, whether it's worth using, how much VRAM or memory it needs, or which quantization to download. Returns our test scores, rank, memory at each quantization, the smallest GPU and Mac it fits, decision accuracy and our Q4 run on a Mac when there is one. Accepts a name or slug, e.g. 'qwen3.5 9b' or 'gpt-oss-20b'.
search_articlesFreeFind Model Tested articles and ranking pages by keyword, newest first: news about newly tested models, guides on quantization, Ollama and memory. Returns title, summary, date and link.
Public scan report
scanner v0.1.10 · 2026-10-28 · same rubric, same numbers if you re-run it
- –Code scanremote-only server, no package to scann/a
- Live reliabilityremote reachable in 305ms20/20
- Tool poisoning5 tool descriptions checked15/15
- Auth qualityopen endpoint, read-only tools10/15
- Maintenancelast push 1 days ago15/15
- Maintainer identityendpoint served from the registry-verified namespace domain8/10
How Model Tested compares
Against every graded AI and agents MCP server, from our own scans and uptime checks. Updated after each scan run.
- Safety score
- 91/100
- higher than 88% of 4,460 AI and agents servers
- Tools
- 5
- the typical AI and agents server has 6
- Tools that change things
- 0%
- typical: 0%. Fewer write tools means less an agent can break
- Uptime
- 100%
- of our recent checks; the typical remote AI and agents server answers 100%
- Response time
- 107 ms
- faster than 85% of 2,331 AI and agents servers
What its tools let an agent do: 5 read
Install Model Tested in Claude Code, Cursor or VS Code
claude mcp add --transport http llm-benchmarks https://modeltested.com/mcp
What the publisher says
From the Model Tested repository's README, as published. We do not edit it. Read it on GitHub
Model Tested
An open benchmark of open-weight LLMs on boring, real work: code that has to run, numbers pulled out of messy documents, and routine decisions like routing a ticket or spotting phishing. Results are at modeltested.com and update automatically when a new open model ships.
Current results: leaderboard of 65 local models · decision models vs. regular LLMs · tokens and cost per call
How the grading works
- Most questions are hidden. They live in this public repo as an age-encrypted tarball (bench/private.tar.age); the key is a CI secret. A public sample is plaintext in bench/public/, and every model's raw answers to it are on the site, so you can check the grading.
- Public vs. hidden scores are shown side by side. A big gap is a red flag for memorization. The public files carry a canary string so training pipelines can exclude them.
- Every model gets the same prompt and grading. Models run through OpenRouter at full or 8-bit precision; some Q4 downloads are also run with Ollama on a Mac mini (M4, 16 GB) to show what quantization costs.
- No speed numbers. They depend too much on your hardware. Instead each model gets a memory estimate: which graphics card or Mac it fits on, and which quantization to download.
- Incremental. Each question is hashed, so a new model or an edited question only re-runs what changed. Testing a new model costs cents to a few dollars.
Known limits: one run per question with default sampling, a question bank that's still small, and cloud precision rather than your exact Q4 file. Corrections and question ideas are welcome in Issues.
Repo layout
- runner/ — Python (stdlib only): discovers models on OpenRouter, runs the question bank, grades answers, estimates memory needs, writes site data.
- bench/ — question bank. public/ is plaintext and published; the hidden set lives encrypted in private.tar.age.
- results/ — per-model scores (question ids + scores only) and raw outputs for public questions.
- site/ — Astro static site, deployed to Cloudflare (free plan) by GitHub Actions.
How it works
Every day .github/workflows/bench.yml:
- decrypts the hidden set with the AGESECRETKEY secret,
- validates the question bank (selfcheck),
- adds new open-weight models up to ~130B (fits a 128 GB Mac) from allowlisted labs (discover),
- runs only the missing (model, question) pairs, within a spend budget (run),
- aggregates scores into site/src/data (build-data) and commits results,
- builds the site and deploys it with Wrangler.
Model-written code runs in Docker with no network, a read-only filesystem, an unprivileged user and CPU/memory/pid limits.
One-time setup
Repository secrets (Settings → Secrets and variables → Actions):
Repository variable SITE_URL = https://modeltested.com (optional — that's also the default in site/astro.config.mjs).
Also set REPO_URL in site/src/lib/data.ts to the GitHub repo.
Shortened. The full README is on GitHub.
Nothing above is checked by us. What we check is on the safety report.
Model Tested: common questions
- Is Model Tested MCP server safe?
- Yes, by our scan: it is graded A (91/100). Read the Model Tested safety report
- How do I install Model Tested?
- It runs remotely at modeltested.com. Add it to Claude Code, Cursor, VS Code or Claude Desktop with the snippets above, or call it through the mcp.market gateway without installing anything.
- Does Model Tested need an API key?
- Not as far as the registry entry and our scan can tell: no credentials are declared or required.
- Is Model Tested maintained?
- The last commit was 8 days ago (2026-10-27). The latest release is v1.0.0.
- Is Model Tested up?
- 100% of our last 28 checks got an answer. We check remote servers about four times a day.
- What can I use instead of Model Tested?
- Servers from other publishers that do the same job: Guaardvark MCP server, Mockserver MCP server and TradeOS (test) MCP server.
- How does Model Tested compare with other AI and agents MCP servers?
- safety score 91/100 (higher than 88% of 4,460 AI and agents servers); tools 5 (the typical AI and agents server has 6); tools that change things 0% (typical: 0%. Fewer write tools means less an agent can break). Full comparison.
Alternatives to Model Tested
Same job from other publishers: the closest match first, then the best rated.
GuaardvarkLocal AI studio on your GPU: images, video, music, voice, RAG. Needs the Guaardvark app installed.not reviewedEstablishedB
MockserverMock, record/replay, verify and chaos-test any HTTP, REST, gRPC or LLM dependency over MCP.not reviewedEstablishedA- TradeOS (test)[TEST] TradeOS MCP bridge: ticker search, My Agent, chart TA, macro news. npm or HTTP.not reviewedEstablishedC
