Forgejudge MCP server

ForgeJudge

An open, always-on leaderboard and CI gate for autonomous coding agents — every patch runs in a sandbox, every run has a public trace, every regression fails the build.

▶ Live leaderboard: forgejudge.ahmedhobeishy.tech · playground · methodology · model swap · MCP registry

<!-- mcp-name: io.github.ahmedEid1/forgejudge --> <!-- ^ ownership proof for the MCP registry (validated against this PyPI long-description). -->

Current numbers (hidden-test = the agent never sees the failing test; $0 free tier; same harness, swap the model; 18 tasks × 3 seeds = 54 runs/model, 162 total):

| Model | pass@1 | pass@3 |

|---|---|---|

| gpt-oss-120b | 90.7% | 100% |

| llama-3.3-70b | 88.9% | 94.4% |

| llama-3.1-8b | 48.1% | 66.7% |

The score rises with the better model while the harness stays fixed (model-swap proof), and pass@3 > pass@1 shows real run-to-run variance — which is exactly why the CI gate is multi-seed. Every run deep-links its Langfuse trace.

ForgeJudge is the only open-source autonomous software-engineering agent that proves its quality in public on every commit: a hand-rolled single-agent solver, a deterministic execution-as-judge harness, an always-on leaderboard with per-run traces, and a CI gate that blocks regressions — all on a $0 / self-hostable stack against a contamination-resistant, intrinsically-verifiable golden set.

The engineered harness, observability, and gate are the deliverable — not a high resolution rate. A $0 free-model agent will score modestly by design. We prove value with a model-swap comparison: the score rises with a better model while the harness stays fixed.

How it works

flowchart TD
    G["Golden set · Git-canonical<br/>18 intrinsically-verifiable, mutation-hardened<br/>make-CI-green tasks"]

    subgraph SOLVER["Single-agent solver"]
        direction LR
        L["localize<br/>(BM25)"] --> R["repair<br/>(LLM router · critic · syntax edit-gate)"] --> V["validate<br/>(run tests)"]
    end

    G --> SOLVER
    SOLVER --> PATCH["unified diff"]
    SOLVER -. "every step traced" .-> TRACE["OTel → Langfuse<br/>per-run public trace"]

    PATCH --> H["Deterministic harness, in a sandbox<br/>apply test_patch + candidate patch · run F2P / P2P<br/><b>RESOLVED iff</b> every FAIL_TO_PASS passes AND every PASS_TO_PASS stays green<br/>swebench-equivalent · stricter on skips · cheat-resistant"]

    H --> STORE["Run store<br/>Neon + pgvector"]
    STORE --> LB["Leaderboard<br/>pass@1 / pass@3 · cost · tokens · trace link"]
    H --> GATE["Multi-seed CI gate<br/>a PR that lowers the resolution rate fails the build"]

    style G stroke:#3fb950,stroke-width:2px
    style H stroke:#4cc2ff,stroke-width:2px
    style GATE stroke:#f0883e,stroke-width:2px

Shortened. The full README is on GitHub.