Ai Model Experiments MCP server

@pipeworx/ai-model-experiments

Model Lab — run the same prompt(s) across many AI models at once (Anthropic, OpenAI, Google, Meta, Mistral, DeepSeek, Qwen and more via OpenRouter) and compare their outputs, cost, and latency side by side, with an optional AI-written comparison summary when the run completes.

Part of Pipeworx — an MCP gateway connecting AI agents to 1558+ live data sources.

Tools

• experimentmodels(search?, mincontext?, maxpricepermtok?, limit?) — browse ~300 available model ids with context window and our billed per-token price (provider cost × 1.5). Use the returned ids in experimentcreate.
• experiment_estimate(prompts[], models[], reps?, params?, summary?) — free dry-run: cell count + estimated billed cost range for a spec, before creating it.
• experimentcreate(name?, prompts[], models[], reps?, params?, summary?, maxspendusd) — creates and starts an experiment (prompts × models × reps). Async: returns experimentid immediately; execution happens on a separate cron worker within ~1 minute. Requires prepaid balance ≥ maxspendusd.
• experimentstatus(experimentid) — cell counts by state, spend vs cap, whether complete. Poll this after create.
• experimentresults(experimentid, include_outputs?) — per-model aggregates (latency, tokens, cost, error rate), per-cell outputs, and the AI-written comparison summary.
• experiment_list(limit?) — caller's experiments, newest first.
• experimentcancel(experimentid) — skips pending cells (unbilled); in-flight cells finish and bill.
• experimenttopup(amountusd?) — current balance + the x402 top-up flow.

Auth

Prepaid only — no free tier, no BYO-key mode. Every call to experimentcreate requires a Pipeworx platform credit balance ≥ maxspendusd (1 credit = $0.0001). Top up via x402: POST https://gateway.pipeworx.io/credits/topup?amountusd=N returns an HTTP 402 payment challenge (USDC on Base); retry with a PAYMENT-SIGNATURE header to settle, and credits land instantly. See experiment_topup for the exact flow and current balance.

Billing is provider cost × 1.5, floored at $0.10/experiment. Model prices returned by experiment_models are already the billed (marked-up) price, never the raw provider cost.

Execution is handled by a separate cron worker (workers/experiment-runner, fires every minute) — a created experiment is not synchronous. Poll experimentstatus rather than expecting experimentcreate to block until done.

Data sources

• — the model catalog (id, context window, provider pricing) that experiment_models wraps and re-prices.
• Inference for each cell is executed against OpenRouter's chat completions API by workers/experiment-runner, which also reads OpenRouter's /generation endpoint for the actual per-call cost so the 1.5× markup is billed on real cost, not an estimate.

Full build plan, architecture, and current live-vs-planned status: docs/model-lab-plan.md.

Quick Start

Add to your MCP client (Claude Desktop, Cursor, Windsurf, etc.):

{
  "mcpServers": {
    "ai-model-experiments": {
      "url": "https://gateway.pipeworx.io/ai-model-experiments/mcp"
    }
  }
}

What this endpoint actually serves

tools/list at https://gateway.pipeworx.io/ai-model-experiments/mcp returns the tools in the table above plus the shared Pipeworx meta-tools — askpipeworx, discovertools, search_within, remember/recall and the rest of the gateway-wide set. So the tool count you see is larger than this table: a single-pack endpoint currently lists roughly 30 shared tools alongside the pack's own. The connection's initialize response states its exact scope, and is the authoritative answer for a given day.

Shortened. The full README is on GitHub.