MCP.Market

Crawl Census MCP server

online · 100% uptime
by taylorsmithgg·io.github.taylorsmithgg/crawl-census·v1.0.0
No reviews yet

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

B77/100grade B
Write a reviewRun it
What users say
No reviews yet
Be the first
Safety scan
B77/100

full report

Adoption
New

0 stars

Reviews

Write one

Nobody has reviewed Crawl Census yet.

If you have run it, two minutes of your experience saves the next person an afternoon.

Crawl Census tools (7, 2 write)

write = sends, deletes, buys or posts
  • agent_profileFree

    What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.

  • census_factsFree

    Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.

  • census_statsFree

    Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.

  • crawl_preflightFree

    Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.

  • scan_sitewrite actionFree

    Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.

  • site_reportFree

    Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.

  • submit_domainswrite actionFree

    Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.

Public scan report

scanner v0.1.10 · 2026-11-01 · same rubric, same numbers if you re-run it

1 high
  • –Code scanremote-only server, no package to scann/a
  • Live reliabilityremote reachable in 511ms20/20
  • Tool poisoning7 tool descriptions checked15/15
  • Auth qualityopen endpoint exposes 2 write-action tools with no auth3/15
  • Maintenancelast push 70 days ago12/15
  • Maintainer identityregistry namespace matches repository owner; GitHub account older than a year8/10

Findings (1)

  • highWrite-action tools reachable without authenticationauth.open-write
Overall 77/100. Components that don't apply are left out of the denominator. Any critical finding is an F.RubricAppeal a findingJSON

How Crawl Census compares

Against every graded Scraping and browsing MCP server, from our own scans and uptime checks. Updated after each scan run.

Safety score
77/100
higher than 50% of 866 Scraping and browsing servers
Tools
7
the typical Scraping and browsing server has 6
Tools that change things
29%
typical: 0%. Fewer write tools means less an agent can break
Uptime
100%
of our recent checks; the typical remote Scraping and browsing server answers 100%
Response time
176 ms
faster than 57% of 384 Scraping and browsing servers

What its tools let an agent do: 5 read · 2 change

Install Crawl Census in Claude Code, Cursor or VS Code

claude mcp add --transport http crawl-census https://crawlcensus.com/mcp
Add to Cursor

What the publisher says

From the Crawl Census repository's README, as published. We do not edit it. Read it on GitHub

crawl-census-client

Ask before you fetch. A drop-in client that stops your crawler spending requests on doors that are shut, and stops it routing around content someone is trying to sell.

Reading robots.txt answers one question and hides two others. Measured across 23,482 domains by Crawl Census:

network edge.** A parser sees permission; the fetch returns 403. You pay for the round trip and get nothing.

  • **2,874 domains permit AI agents in robots.txt and then refuse those same agents at the

not a refusal. Treating it as a block walks away from content the operator wants to sell you. Retrying around it takes something they are charging for.

  • 208 domains answer an AI user agent with HTTP 402 Payment Required. That is a price,

No dependencies. No key required.

MCP server

The same measurement is exposed as a remote MCP server, so an agent can ask before it fetches rather than after it fails. Listed in the official MCP registry as io.github.taylorsmithgg/crawl-census.

{ "mcpServers": { "crawl-census": { "url": "https://crawlcensus.com/mcp" } } }

No authentication for read tools. Streamable HTTP.

Install

npm i github:taylorsmithgg/crawl-census-client
pip install git+https://github.com/taylorsmithgg/crawl-census-client

Use

import { politeFetch } from "crawl-census-client";

const r = await politeFetch("https://example.com/", { agent: "gptbot" });
if (r.skipped) console.log(r.verdict, r.reason);   // disallow | refuse | pay
else           process(await r.response.text());
from crawl_census import polite_fetch

r = polite_fetch("https://example.com/", agent="gptbot")
if r.skipped:
    print(r.verdict, r.reason)
else:
    process(r.body)

Skipping is returned, not raised. It is the normal outcome for a large share of the web, and a crawl loop should be able to count skips without a try/except around every URL.

Split a queue before crawling it

One call per 1,000 domains instead of one per host:

const { crawl, skip, pay, unknown } = await partition(urls, { agent: "gptbot" });
p = partition(urls, agent="gptbot")
p.crawl, p.skip, p.pay, p.unknown

Or just take the file

For a fetcher that only needs a deny list in memory, skip the per-domain calls entirely:

curl https://crawlcensus.com/agents/gptbot/blocklist.txt   # one domain per line, commented header
const sync = await syncBlocklist("gptbot");   // full list once
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // then deltas only, a few hundred bytes
sync = BlocklistSync("gptbot")
if host in sync: skip()
sync.refresh()          # {'added': 3, 'removed': 1, 'size': 3310, 'cursor': ...}

The delta feed is https://crawlcensus.com/agents//changes.json?since= and each response carries next_since, so a long-running crawler stays current on a few hundred bytes an hour instead of re-downloading the list.

That file covers robots.txt only. Edge refusal and HTTP 402 are per-request behaviours and still need preflight or politeFetch.

What a crawl costs the census

Shortened. The full README is on GitHub.

Nothing above is checked by us. What we check is on the safety report.

Crawl Census: common questions

Is Crawl Census MCP server safe?
Mostly: it is graded B (77/100). Read the Crawl Census safety report
How do I install Crawl Census?
It runs remotely at crawlcensus.com. Add it to Claude Code, Cursor, VS Code or Claude Desktop with the snippets above, or call it through the mcp.market gateway without installing anything.
Does Crawl Census need an API key?
Not as far as the registry entry and our scan can tell: no credentials are declared or required.
Is Crawl Census maintained?
The last commit was 73 days ago (2026-08-23). The latest release is v1.0.0.
Is Crawl Census up?
100% of our last 28 checks got an answer. We check remote servers about four times a day.
What can I use instead of Crawl Census?
Servers from other publishers that do the same job: fastCRW MCP server, CRW Web Scraper MCP server and Svipall MCP server.
How does Crawl Census compare with other Scraping and browsing MCP servers?
safety score 77/100 (higher than 50% of 866 Scraping and browsing servers); tools 7 (the typical Scraping and browsing server has 6); tools that change things 29% (typical: 0%. Fewer write tools means less an agent can break). Full comparison.

Alternatives to Crawl Census

Same job from other publishers: the closest match first, then the best rated.

  • fastCRW
    Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
    More from us →
    A
  • CRW Web Scraper
    Open-source web scraper for AI agents with scrape, crawl, and map tools
    A
  • Svipall
    Read any page past its anti-bot wall, crawl sites, search without an API key, solve captchas locally
    C
  • Crawlberg
    Scrape, crawl, and map websites to Markdown or JSON via local CLI.
    A
  • Kreuzcrawl
    Scrape, crawl, and map websites to Markdown or JSON via local CLI.
    A