AI Benchmark Index MCP server

benchmark-core

AI Benchmark Index · npm · API · MCP

benchmark-core is an open-source TypeScript engine for reproducible AI model benchmarks. One definition produces an SDK, command-line interface (CLI), Model Context Protocol (MCP) server, and HTTP API.

It measures whatever your tasks and deterministic grader define. The included example measures 20 arithmetic answers and one tool-use episode with a transient order-service fault.

npx benchmark-core run --model offline --provider mock --json

{"model":"mock:1","mean":0.5238,"confidenceInterval95":{"low":0.3333,"high":0.7143},"totals":{"tasks":21,"passed":11,"failed":0}}

The example data comes from a committed seeded generator. Seed 20261015 reproduces every public task.

Install

benchmark-core requires Node.js 20 or newer.

npm install benchmark-core

The package has one runtime dependency: the official MCP TypeScript SDK.

Define a benchmark

import { defineBenchmark, exactMatch, run, mockAdapter } from "benchmark-core";

const benchmark = defineBenchmark({
  id: "benchmark-colors",
  name: "Color recall",
  tagline: "Checks exact color recall.",
  tags: ["science"],
  category: "memory",
  whatItMeasures: "Exact recall of a named color.",
  version: "0.1.0",
  status: "alpha",
  tasks: [{ id: "sky", kind: "single", prompt: "Name the sky color.", expected: "blue" }],
  grade: (task, submission) => exactMatch(task.expected, submission.answer ?? ""),
  provenance: [{
    name: "Maintainer-written task",
    kind: "human-written",
    url: "https://github.com/example/benchmark-colors",
    licence: "MIT",
    howToGet: "Read data/tasks.jsonl.",
    retrievedAt: "2026-10-15T00:00:00.000Z",
    contaminationNotes: "Public task; use a hidden split for final comparisons.",
    refresh: "Review and add tasks through pull requests."
  }],
  gradingMethod: "Exact string match."
});

const report = await run(benchmark, mockAdapter(), { concurrency: 2, seed: 7 });
console.log(report.mean);

tasks can be replaced by an asynchronous loadTasks() function. Task IDs must be unique, and each task declares kind: "single" | "episode".

How it works

defineBenchmark({ tasks, grade, environment, provenance })
                  │
        ┌─────────┼──────────┐
        │         │          │
       SDK       CLI      MCP server
        │         │          │
        └─────────┴──── HTTP handler
                  │
             RunReport JSON

The runner preserves task order while it uses bounded concurrency. Each row records attempts, latency, score, reason, usage, and errors. Episode rows also contain the transcript, tool log, final state, and abort reason.

The 95% confidence interval uses 2,000 seeded bootstrap resamples. Identical scores and seeds produce identical intervals.

Runtime surfaces

CLI

A benchmark package calls createCli(benchmark, { bin }). The returned main(argv) supports:

• list
• show
• run --model --provider with optional episode limits
• score
• card [output.svg]
• provenance
• index-entry
• serve [port]
• mcp

Every command accepts --json. Exit code 0 means success, 1 means invalid usage, and 2 means a run or runtime failure.

Model Context Protocol

Shortened. The full README is on GitHub.