GitHub (read-only) MCP server

GitHub MCP Server: Teaching an AI to Pick the Right Tool

<!-- mcp-name: io.github.AbdulMuhaiminKhan/github-mcp -->

A read-only Model Context Protocol server that lets Claude (or any MCP client) answer questions about my GitHub repos, issues, pull requests and commits. It comes with a benchmark that measures how often the model picks the right tool with the right arguments.

Rewriting the tool descriptions took a local 7B model from 63% to 95% correct, and from 60% to 90% on held-out questions it was never tuned on. An ablation shows where those 32 points came from, and most of them were not the prose:

Every miss left in v2 is the model asking for something the API doesn't offer: merged PRs, sorting by stars, or closed issues from the open-issues tool. Better wording can't fix those. The next step is to support them in the schema.

Explore every question in the results dashboard →

What I built

getrepository, listopenissues, searchissues, listpullrequests, getrecentcommits.

• MCP server (Python, mcp SDK 2.x) with 6 read-only tools over stdio: list_repositories,

The token stays in the server process and never reaches the model.

• Least-privilege access: a fine-grained, read-only GitHub token and readOnlyHint annotations.

secondary rate-limit handling, ETag caching (304s don't count against the rate limit), exact totals for list tools, and error messages that tell the model how to recover.

• Production behaviour: schema-validated inputs, retries with backoff and jitter, primary and

and commits from two authors. A fake GitHub API serves the same data offline, so the benchmark also runs in CI with no token.

• A benchmark with known answers: a seed script creates two repos with 15 issues, 5 pull requests

multi-step agent runs whose final answers are checked against the seeded facts (13).

• Three evaluations: tool selection with argument grading (60 questions), a held-out set (20), and

only if held-out accuracy doesn't drop.

• An ablation (v1 → v1b → v2) that separates input handling from description wording.
• An automatic description optimizer that rewrites descriptions from failures and keeps a change

scores 100% when the right call is made).

• A harness that benchmarks any MCP server, not just this one (examples/).
• A results dashboard (docs/) with every question, call and error from the runs.
• Tests and CI on Python 3.10–3.13 (ruff, mypy, pytest, and an oracle check that the harness

Architecture

flowchart LR
    U[User] -->|question| C["Claude Desktop<br/>(MCP client)"]
    C <-->|"JSON-RPC over stdio<br/>tools/list · tools/call"| S["github-mcp<br/>Python MCP server"]
    S -->|"HTTPS · read-only token<br/>ETags · retries"| G[(GitHub REST API)]
    E["eval/run_eval.py"] -.->|"same MCP protocol"| S
    E -.->|"question + tool schemas"| L["LLM under test<br/>(Ollama or Claude)"]
    S -.->|"GITHUB_API_URL (CI)"| F[("fake GitHub<br/>seeded repos")]

Shortened. The full README is on GitHub.