Commoncrawl MCP server

@pipeworx/commoncrawl

Common Crawl's public web archive — find every time a URL was crawled since 2008 and read back the exact page bytes that were captured.

Part of Pipeworx — an MCP gateway connecting AI agents to 1679+ live data sources.

Tools

(CC-MAIN-2026-34 and friends) with the capture window each one covers. contains_date answers "which snapshot would have seen my page on that day".

• commoncrawlcrawls(limit?, containsdate?) — the monthly crawl collections

CDX index lookup for a URL, host or whole domain. Returns capture timestamp, HTTP status, MIME type, detected language, content digest, and the WARC filename + offset + length that the next tool needs.

• commoncrawlindexsearch(url, crawl?, match_type?, limit?, page?, filter?, from?, to?) —

one archived record back by byte range: WARC headers, the captured HTTP response headers, and the page body as crawled.

• commoncrawlfetchrecord(filename, offset, length, maxbodybytes?) — reads

Auth

Keyless.

Data sources

• — the crawl collection list.
• — the CDX index, JSON-lines.
• — WARC records, HTTP range reads.

Things the next person would otherwise rediscover

url=example.com query alternated between 200 and nginx 502 inside a minute, so it is load, not the query. ccFetch retries a 5xx twice with a short backoff. Without that a caller reads transient nginx noise as "Common Crawl has no record of this URL", which is a different and wrong answer.

• The CDX host 502s sporadically under load. Measured 2026-09-17: the same

empty result set with an explicit note, so an empty captures array is never silently indistinguishable from an upstream failure.

• "No captures" is a 404 with an English sentence, not JSON. Handled as an

offset..offset+length-1 decompresses standalone — no need to stream the whole 1 GB WARC. DecompressionStream('gzip') handles it in the Workers runtime.

• Each indexed record is its own gzip member, so a byte-range read of

@cloudflare/workers-types: its TextDecoderConstructorOptions requires ignoreBOM too. Pass no options; non-fatal is the default anyway.

• new TextDecoder('utf-8', { fatal: false }) does not typecheck against

and is the first thing to 502. Narrow with match_type: "host" plus a filter such as =status:200.

• A broad match_type: "domain" query over a large domain is expensive upstream

Related packs

crawlgraph is our own link graph over sites we crawl. This pack reads the Common Crawl Foundation's corpus — different data, different questions.

Quick Start

Add to your MCP client (Claude Desktop, Cursor, Windsurf, etc.):

{
  "mcpServers": {
    "commoncrawl": {
      "url": "https://gateway.pipeworx.io/commoncrawl/mcp"
    }
  }
}

What this endpoint actually serves

tools/list at https://gateway.pipeworx.io/commoncrawl/mcp returns the tools in the table above plus the shared Pipeworx meta-tools — askpipeworx, discovertools, search_within, remember/recall and the rest of the gateway-wide set. So the tool count you see is larger than this table: a single-pack endpoint currently lists roughly 30 shared tools alongside the pack's own. The connection's initialize response states its exact scope, and is the authoritative answer for a given day.

This is deliberate, not multiplexing by accident. The meta-tools are what let a scoped connection answer a question this pack does not cover — via ask_pipeworx, which routes across the whole catalog — without you adding a second MCP server. There is currently no way to mount a pack endpoint without them; if the extra schemas cost you more context than the routing is worth, connect to the full gateway once rather than to several pack endpoints.

Shortened. The full README is on GitHub.