Screaming Frog SEO Spider MCP Server (headless)

Screaming Frog SEO Spider MCP Server (headless)

A headless MCP (Model Context Protocol) server for Screaming Frog SEO Spider. It drives the SF command line and saved crawl database directly, so Claude (or any MCP-compatible client) can run crawls, export crawl data, and analyze the results with the Screaming Frog GUI closed: on your laptop, on a server, or inside scheduled audits and CI pipelines.

This is a community project, not affiliated with Screaming Frog. Since SEO Spider v24 there is also an official MCP built into the app. The two work differently and solve different problems.

How this differs from the official Screaming Frog MCP

Screaming Frog shipped an official MCP server in SEO Spider v24. It's substantial: around 29 tools covering crawl control (start, pause, resume, progress), reports and bulk exports with field selection, URL-level inspection, screenshots, embeddings exports, and optionally a Node.js script runner with npm and filesystem read/write tools. It runs in two modes, either a Streamable HTTP server inside the open app, or a STDIO mode where the MCP client launches the Spider itself, headless. Setup is documented for Claude Desktop and LM Studio.

If you want maximum capability in an interactive session (visualizations, crawl comparison, screenshots, scripted post-processing of exports), use the official MCP. It does far more, and it's maintained by the vendor.

This server makes a different trade: it's a small, deliberately limited wrapper around SF's CLI and the saved crawl database, built for runs where nobody is watching.

Locked-down by design. Nine read-and-export tools, nothing else. No script runner, no npm install, no filesystem write access. The official MCP offers all three, and its own docs note that enabling the Node runtime "allows the execution of arbitrary code on your system" and should only be granted to a fully trusted client. There's also an SFALLOWEDDOMAINS allowlist to restrict what an agent is able to crawl. When an agent runs unattended on a schedule, a tool surface this small is a feature.

Installs anywhere, plainly. A pip/uv-installable Python package with a one-line stdio config on any MCP client (Claude Code, Cursor, whatever) on macOS, Linux, or Windows. The official STDIO mode ships as a Claude Desktop extension (.mcpb); the HTTP mode means opening the app and starting the server from its settings.

Light process model. Screaming Frog only runs while a tool actually needs it. The official server is the Spider application running for the whole session, whichever mode you pick.

A few tools the official set doesn't have: aggregatecrawldata for counts and distributions computed server-side (the official path to "how many 404s" is a full export, or a Node script), deletecrawl and storagesummary for cleaning up SF's crawl database (their sfclearcrawl clears a paused crawl, it doesn't manage stored ones), regex filtering across any column of any export via readcrawldata, and sf_check pre-flight diagnostics that catch license problems and GUI database locks before you waste a crawl.

Shortened. The full README is on GitHub.