NoiseFloor — market attention and evidence checks MCP server

noisefloor

Is this number real, or is it noise?

Zero dependencies. Standard library only. Works as a Python package or as an MCP server.

The problem

You ship a change and check the A/B test. Then you check again an hour later. Then tomorrow. You stop when it looks like a win.

That procedure lies to you, and not by a little:

Both arms had the same 10% conversion rate, so every "winner" was false. A test tuned to be wrong 5% of the time was wrong 40% of the time, purely because someone looked more than once.

This isn't a discipline problem. Peeking is the rational thing to do when a bad variant is costing money. It's a maths problem, and it has a solution.

Reproduce the table with pytest tests/test_calibration.py.

Install

pip install noisefloor

Use

from noisefloor import experiment

experiment.compare(a_successes=500, a_total=5000,
                   b_successes=750, b_total=5000)
# {'decided': True, 'winner': 'B',
#  'reading': "B wins: its interval sits entirely above A's",
#  'A': {'rate': 0.1, 'interval': [0.08366, 0.11809]},
#  'B': {'rate': 0.15, 'interval': [0.13049, 0.17101]}, ...}

Run it after every single observation if you like. The guarantee holds at every sample size simultaneously, so stopping early, stopping late, or stopping because your manager walked past all cost you nothing.

The other four checks

from noisefloor import change, coverage, forecast, multiple

# Did this metric actually change? Catches collapses as well as spikes.
change.scan(daily_signups)
# {'state': 'changed', 'direction': 'down',
#  'reading': 'CHANGED (down), 8.5x its usual spread', ...}

# Did the metric move, or did my sample size move?
coverage.check(conversion_rate, sessions_per_day)
# {'verdict': 'SAMPLING_ARTIFACT',
#  'reading': 'your sample size changed by -41% at the same time and it tracks
#              this metric historically (correlation 0.70)...'}

# What should the next reading be?
forecast.next_value(latency_p95)

# How good have these forecasts actually been?
forecast.score(latency_p95)
# {'empirical_coverage': 0.803, 'calibrated': True, 'n_misses': 197,
#  'worst_misses': [...]}   # misses are always published

# I watch 40 metrics. Which genuinely stand out?
multiple.select({'signups': 3.2, 'latency': 812.0, 'errors': 1.1, ...})
# {'selected': ['latency'],
#  'reading': '1 of 40 metrics worth looking at: latency'}

As an MCP server

Assistants read metrics constantly and assert significance the way people do — by eyeballing a change and calling it. This gives them a way to check.

{
  "mcpServers": {
    "noisefloor": { "command": "uvx", "args": ["--from", "noisefloor", "noisefloor-mcp"] }
  }
}

mcp-name: io.github.beepboop2025/noisefloor

Six tools: abtest, diditchange, realorsampling, forecastnext, scoreforecasts, whichmetrics_matter.

Why these methods

Every check is anytime-valid or distribution-free — the two properties that survive contact with how dashboards are really used: looked at whenever someone feels like it, and stopped when they see what they want.

Shortened. The full README is on GitHub.