digline MCP server

digline

Regression testing for LLM applications — with the baseline in your repository, not on someone's server.

Your prompt worked on Tuesday. On Thursday it works a little less — not enough to break, enough for a user to notice in two weeks. No ordinary test catches it, because there is no correct output to compare against, only a better or a worse one.

digline gives you an approved reference — the baseline — and on every change tells you whether you are below it: which case, which check, by how much. The baseline is a JSON file in your repository, so it goes through code review and it rolls back with git. No server, no account, no network call you have not configured yourself.

$ digline compare --suite suite.py --run latest
2 checks got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference.

how-do-i-return · llm_rubric · Score fell from 1.000000 to 0.700000.
how-do-i-return · contains · Went from passing to failing (1.000000 → 0.000000).

Why digline

Most evaluation tools tell you whether an output is below a threshold. digline also tells you whether it is worse than it was — the drift from 0.91 to 0.78 that trips no threshold and is the first thing a user feels.

The suite is Python, not YAML: a judge is an object, a target is a function, and what may leave a perimeter is declared in code — none of which a configuration file expresses without reinventing a language. Where a suite is plain data it can be TOML instead, and the two forms build the same objects — what TOML cannot express, it refuses by name rather than half-supporting. Built for teams shipping LLM features for someone else, who have to show a customer what was tested, when, under which commit, and who approved it.

Wondering how digline differs from promptfoo, DeepEval, or observability platforms? See How digline compares.

Quickstart

With uv (recommended):

uv init && uv add digline

or with pip in an existing environment: pip install digline.

Requires Python 3.12+, which uv fetches for you if you do not have it. On the pip path an older interpreter says ERROR: No matching distribution found with from versions: none, which does not say why — that is what it means.

If you use Claude Code, digline also installs as a plugin. What it ships first is the judgement layer. The operating-digline skill carries the same rules as AGENTS.md, and loads when the agent is about to run a suite or read a comparison. Beside it is the MCP server, which measures, reads and explains, and cannot promote a baseline. Two lines plus one dependency: the server imports your suite, so it runs from your project's .venv.

Shortened. The full README is on GitHub.