QueryPilot MCP server

QueryPilot

<!-- mcp-name: io.github.nickklos10/querypilot -->

Eval-driven SQL reliability for AI agents.

QueryPilot helps agents safely generate, validate, repair, execute, and regression-test SQL against real fixture databases.

Why QueryPilot Exists

Read-only SQL access for agents is becoming a commodity. Tools that let an agent list tables, read schemas, and run validated SELECTs already exist. What is much harder — and what QueryPilot focuses on — is making that access measurably reliable: proving the SQL the agent generates is correct, safe, fast, and not regressing.

Every change to QueryPilot, your prompts, or your model can be measured against an execution-truth eval suite. Suites can be authored by hand or auto-generated by replaying your audit log as a regression set, so the same queries that worked in production yesterday have to keep working tomorrow.

Quick Demo

python3 -m venv .venv
.venv/bin/pip install -e ".[dev,eval]"
.venv/bin/querypilot eval init           # scaffold suites/ and .eval/
.venv/bin/querypilot eval run \
    --suite suites/smoke.yaml \
    --generator demo \
    --report eval-out.json
.venv/bin/querypilot eval check \
    --report eval-out.json \
    --baseline .eval/baseline.json \
    --threshold 0.9 \
    --require-safety 1.0

Sample output (abridged — see the full report at the top of this README):

QueryPilot Eval Report
Suite:     smoke
Generator: demo

Overall
  ✅  Pass rate                       3 / 3 (100%)
  ✅  Safety pass rate                0 / 0 (100%)
  ✅  Correctness                     3 / 3 (100%)
  ✅  P95 latency                     18 ms

✅ No threshold violations.

The bundled suites/smoke.yaml runs against a tiny SQLite fixture (tests/fixtures/demo.db) so the harness works end-to-end without an LLM key. To benchmark a real generator, use --generator openai or --generator anthropic.

Audit-Log Replay

querypilot eval replay turns a JSONL audit log written by JSONLAuditSink into a BenchmarkSuite whose gold SQL is the SQL that previously executed. Re-running that suite gates accuracy regressions against your own production traffic — the unique-to-QueryPilot capability the eval positioning rests on.

querypilot eval replay \
    --audit-jsonl audit.jsonl \
    --fixture-db sqlite:///tests/fixtures/demo.db \
    --output suites/replay.yaml
querypilot eval run --suite suites/replay.yaml --generator demo --report replay-out.json

Conservative defaults: only successful ask records, non-empty results, no active access policy. --include-failures, --include-masked, --include-empty relax each filter.

CI Gate

querypilot eval check compares a SuiteReport JSON against thresholds and a committed baseline, exiting non-zero on regression. A sample GitHub Actions workflow ships at .github/workflows/eval.yml:

- run: querypilot eval run --suite suites/smoke.yaml --generator demo --report eval-out.json
- run: querypilot eval check --report eval-out.json --baseline .eval/baseline.json --threshold 0.9 --require-safety 1.0

When a regression is detected the output explains which cases regressed and how:

Regression detected.

Pass rate:
  baseline: 96%
  current:  89%

Shortened. The full README is on GitHub.