Nonobench MCP server

Nonobench

A benchmark suite for evaluating LLM reasoning capabilities on Nonogram (Picross) puzzle solving across different grid sizes. Results are published at nonobench.com.

Built by Maurice Kleine.

What is a Nonogram?

Nonograms (also known as Picross, Griddlers, or Paint by Numbers) are logic puzzles where you fill in cells on a grid based on numeric clues for each row and column. The clues indicate consecutive groups of filled cells, separated by at least one empty cell. Solving these puzzles requires logical deduction and constraint satisfaction - making them an excellent test of LLM reasoning abilities.

Project Structure

nonobench/
├── bench/          # Benchmark runner, results database and exporter
└── visualizer/     # TanStack Start dashboard on Cloudflare Workers (nonobench.com), also home of the puzzle set

Prerequisites

• Bun runtime (v1.4+)
• Node.js v24+ (for the visualizer)
• OpenRouter API key

Quick Start

1. Clone the Repository

git clone https://github.com/mauricekleine/nonobench.git
cd nonobench

2. Set Up Environment Variables

Create a .env file in the bench/ directory (see bench/.env.example):

OPENROUTER_API_KEY=your_openrouter_api_key_here

This is the only variable needed. The visualizer builds without any.

3. Running Benchmarks

cd bench
bun install
bun run bench                       # prints the plan and exits, no API calls
bun run bench --model <name>        # run one model (repeat --model for more)
bun run bench --all-missing         # run every configured model with missing work
bun run bench --model <name> --sizes 20x20  # opt in to the extended tier

Runs are incremental and append-only: a model/puzzle pair that already has a successful result is never run again, and the database refuses to overwrite it. Failed runs are retried on the next invocation. Results are stored in bench/results.db (SQLite).

Useful flags:

• --max-cost stops launching new puzzles once this session's spend reaches the amount. Requests already in flight still finish, so a session can overshoot by up to --parallel requests per selected model. Use the OpenRouter key's own limit as the hard ceiling.
• --parallel sets concurrent requests per model (default 10); lower it for rate-limited providers.
• --limit runs only the first n puzzles of each size, for pilots against a scratch database (NONOBENCH_DB=/tmp/copy.db).

Output modes

New runs ask for the answer as strict structured output (a JSON schema, only routed to endpoints that enforce it), so models cannot wrap the grid in prose. For a few models the schema-enforcing endpoints measurably hurt answers; those run in text mode instead (outputMode: "text" in bench/constants.ts, chosen by a 5x5 A/B with the benchmark prompt). Every run records its mode, and grading is identical for both: the answer must satisfy every clue. The runner also stops a model that solves none of the 5x5 puzzles with structured output, and one whose early runs mostly report zero reasoning tokens, since both point at the harness rather than the model.

After benchmarking, export results for the visualizer:

bun run export

This writes visualizer/app/results.json (aggregates) and visualizer/public/results-raw.json (every run, including prompts and outputs).

Other scripts:

• bun test - parser, grader, database-policy and puzzle checks
• bun run typecheck - TypeScript check
• bun run regrade - read-only comparison of stored grades against the current grader

Running Against a Local Model

The runner can benchmark a model that you serve yourself. Any server with an OpenAI-compatible /v1 endpoint works: vLLM, SGLang, llama.cpp, LM Studio, and Ollama.

Shortened. The full README is on GitHub.