MCP.Market

polite-crawler MCP server

by mark1russell7·io.github.mark1russell7/polite-crawler·v0.3.0
No reviews yet

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

A92/100grade A
Write a reviewInstall
What users say
No reviews yet
Be the first
Safety scan
A92/100

not scanned yet

Adoption
New

0 stars

Reviews

Write one

Nobody has reviewed polite-crawler yet.

If you have run it, two minutes of your experience saves the next person an afternoon.

polite-crawler tools

No tool declarations could be read from the package source. They show once the server is installed.

Not scanned yet. New listings are scanned within a day or two of arriving.

How polite-crawler compares

Against every graded MCP server, from our own scans and uptime checks. Updated after each scan run.

Safety score
92/100
higher than 95% of 48,255 servers

Release history: 5 versions in the registry since 2026-10-13, latest 2026-11-02, typically one every 6 days.

Install polite-crawler in Claude Code, Cursor or VS Code

Runs npx -y polite-crawler-mcp on your machine. Read the scan report first; the gateway never runs local packages.

claude mcp add polite-crawler -- npx -y polite-crawler-mcp
Add to Cursor

What the publisher says

From the polite-crawler repository's README, as published. We do not edit it. Read it on GitHub

crawler

This repository holds polite-crawler, a polite web crawler for research and for AI agents, and its site.

polite-crawler fetches a source one time, keeps it, and reads it many times. It obeys robots.txt, sends one request at a time to each site, and answers each later question from a local full-text index.

  • The package: packages/polite-crawler is on npm as polite-crawler.
  • The site: packages/site is the documentation, with interactive demos that use the real code: .
  • The study: Who lets AI agents in? The crawler read the robots.txt files of the top 1,000 domains. The page shows the verdict for 15 agents, with the rule that decided it.
npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler
npx polite-crawler find "robots exclusion"
claude mcp add polite-crawler -- npx -y polite-crawler mcp

What the crawler does for you and for the sites:

  • It stops a host after a challenge, a 401, a 402 or a 403, for 1 to 24 hours. It does not try a different identity.
  • By default, it fetches only public addresses. --allow-private permits loopback and private addresses.
  • The MCP server gives page text in an envelope, so an assistant can tell data from instructions. A page that refuses AI input stays held back until a person confirms.
  • The library is polite-crawler, and the MCP server is polite-crawler/mcp.

The package README tells the details.

Development

The repository is a pnpm workspace. It needs Node.js 22.19 or later and pnpm 10 or later.

pnpm install
pnpm browser   # the headless shell for the renderer test
pnpm check     # typecheck, lint, test, lint:ste, build and the smoke test of the package

The prose of the repository (the README files, the doc comments and the site) follows ASD-STE100 Simplified Technical English. The linter ste-lint examines it in CI.

  • CONTRIBUTING.md tells how to set up, test and write a change.
  • SECURITY.md tells how to report a vulnerability privately.
  • CHANGELOG.md records the changes in each version.

License

MIT

Nothing above is checked by us. What we check is on the safety report.

polite-crawler: common questions

Is polite-crawler MCP server safe?
Yes, by our scan: it is graded A (92/100). Read the polite-crawler safety report
How do I install polite-crawler?
It runs on your machine. Copy the Claude Code, Cursor, VS Code or Claude Desktop config from the install section.
Does polite-crawler need an API key?
No secret keys are declared. It reads 6 settings from the environment.
Is polite-crawler maintained?
The last commit was 22 days ago (2026-10-13). The latest release is v0.3.0.