Sitemap Stats MCP server

@pipeworx/sitemap-stats

Reads a domain's sitemap(s) and reports their STRUCTURE, not their content — URL counts by path prefix, the fingerprint that separates a normal site from a programmatic content farm (a farm concentrates a large majority of its URLs under one or two path prefixes).

Part of Pipeworx — an MCP gateway connecting AI agents to 1576+ live data sources.

Tools

(robots.txt Sitemap: directives, falling back to /sitemap.xml then /sitemapindex.xml), follows one level of sitemap-index nesting, and returns totalurls, sitemapcount, a top-20 path-prefix histogram (topprefixes, each with count and share), topprefixshare (the single number that separates a normal site from a generated one), and lastmodmin/lastmodmax where present. Caps total URLs read (default 150,000, hard ceiling 500,000 via maxurls) and always sets capped: true with a capreason when it stops early, so a truncated read never looks like a small site. A domain with no sitemap returns found: false with a reason — not an error.

• sitemapstats({ domain, maxurls? }) — resolves a domain's sitemap(s)

Auth

Keyless. No vendor, no key.

Data sources

every call — a direct pass-through, not a copy.

• The target domain's own robots.txt and sitemap XML files, read live on

Traps found while building this

are common — the tool follows a one level down to its child files, but does not recurse further (nested indexes-of-indexes are non-standard and rare).

• Sitemap protocol caps a single file at 50,000 URLs, so real multi-file sites

header, so the runtime's automatic decompression never triggers — decoded by hand via Workers-native DecompressionStream('gzip') when the URL ends in .gz or the response's content-type mentions gzip.

• A raw .xml.gz served as a static file carries no Content-Encoding: gzip

a run that hits it stops mid-file rather than skipping whole files — so total_urls under a cap is always the true count of what was actually read, never silently smaller than that.

• The URL cap applies to total URLs read across every file, not per file, and

aren't always in the wild; relative ones are resolved against the domain's origin.

• robots.txt Sitemap: lines are supposed to be absolute per spec but

Quick Start

Add to your MCP client (Claude Desktop, Cursor, Windsurf, etc.):

{
  "mcpServers": {
    "sitemap-stats": {
      "url": "https://gateway.pipeworx.io/sitemap-stats/mcp"
    }
  }
}

What this endpoint actually serves

tools/list at https://gateway.pipeworx.io/sitemap-stats/mcp returns the tools in the table above plus the shared Pipeworx meta-tools — askpipeworx, discovertools, search_within, remember/recall and the rest of the gateway-wide set. So the tool count you see is larger than this table: a single-pack endpoint currently lists roughly 30 shared tools alongside the pack's own. The connection's initialize response states its exact scope, and is the authoritative answer for a given day.

This is deliberate, not multiplexing by accident. The meta-tools are what let a scoped connection answer a question this pack does not cover — via ask_pipeworx, which routes across the whole catalog — without you adding a second MCP server. There is currently no way to mount a pack endpoint without them; if the extra schemas cost you more context than the routing is worth, connect to the full gateway once rather than to several pack endpoints.

Or connect to the full Pipeworx gateway to get every pack's tools listed directly, instead of just this one's:

{
  "mcpServers": {
    "pipeworx": {
      "url": "https://gateway.pipeworx.io/mcp"
    }
  }
}

Both URLs reach the same gateway and the same 1576+ data sources. The only difference is which pack's tools are listed directly; ask_pipeworx reaches all of them from either one.

Shortened. The full README is on GitHub.