MCP.Market

TheCrawler MCP server

by manchittlab·io.github.manchittlab/thecrawler·v0.5.8·2 stars
No reviews yet

Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.

A86/100grade A
Write a reviewInstall
What users say
No reviews yet
Be the first
Safety scan
A86/100

full report

Adoption
Growing

2 stars35 downloads/wk

Reviews

Write one

Nobody has reviewed TheCrawler yet.

If you have run it, two minutes of your experience saves the next person an afternoon.

TheCrawler tools (8, 1 write)

write = sends, deletes, buys or posts

Read from the package source without running it. The installed server may list more.

  • crawl

    Scrape one or more URLs. Returns rich structured data: title, description, text, markdown, links, images, meta, OG/Twitter cards, JSON-LD, microdata, headings, tables, forms, redirect chain, hreflang, pagination links, commerce data (price/rating/sku from JSON-LD), analytics tracker detection (16 trackers), and optional email-like/phone-like public text fields when explicitly enabled. Auto-handles

  • crawl_markdown

    Extract clean markdown for a single URL. Strips boilerplate (nav/header/footer/cookie banners). Returns markdown text only — no surrounding metadata. Ideal for feeding a single page to an LLM. For multi-URL or structured output, use `crawl`, `extract_structured`, or `extract_extraction_contract` instead.

  • crawl_sitemap

    Fetch URL list from a sitemap.xml (supports sitemap-index files), then optionally scrape them. Use scrape=false to just get the URL list (cheap), scrape=true to also extract content from each.

  • diagnose_extraction_contractwrite action

    Run a no-LLM readiness diagnostic for one extraction contract before spending LLM tokens. Returns per-URL verdicts, scores, blockers, branchable error types, missing readiness signals, and a workflow-level recommended next step. This does not claim a site will work; it tells the agent whether to extract, retry with browser/proxy, review manually, or stop.

  • extract_extraction_contract

    Crawl URLs and extract against a built-in validated extraction contract with an OpenAI-compatible LLM. Returns parsed data plus required-field validation and missingRequiredFields evidence. Use diagnose_extraction_contract first when the source is unknown.

  • extract_structured

    LLM-powered structured extraction. Crawls each URL → cleans to markdown → asks an OpenAI-compatible LLM (llama.cpp / vLLM / LM Studio / Ollama / OpenAI) to return ONLY a JSON object matching your jsonSchema or prompt. Returns parsed typed data, not raw markdown. LLM endpoint configurable per-call or via env vars (THECRAWLER_LLM_BASEURL, THECRAWLER_LLM_MODEL, THECRAWLER_LLM_API_KEY).

  • list_extraction_contracts

    List built-in extraction contracts agents can use for validated schema extraction. Use this before diagnose_extraction_contract or extract_extraction_contract.

  • search_and_crawl

    Search Google for a query, then scrape the top N results. Returns structured data per result page. Without a SerpAPI key, falls back to scraping Google's HTML — fragile, can break when Google updates their HTML. For production agent use, supply serpApiKey.

Public scan report

scanner v0.1.9 · 2026-09-26 · same rubric, same numbers if you re-run it

no findings
  • Code scan32 source files scanned25/25
  • –Live reliabilityno gateway calls yet and no remote to proben/a
  • –Tool poisoningtools not inspected (local package is not executed); not countedn/a
  • Auth qualitylocal package, no credentials required12/15
  • Maintenancelast push 86 days ago12/15
  • Maintainer identityregistry namespace matches repository owner7/10
Overall 86/100. Components that don't apply are left out of the denominator. Any critical finding is an F.RubricAppeal a findingJSON

How TheCrawler compares

Against every graded AI and agents MCP server, from our own scans and uptime checks. Updated after each scan run.

Safety score
86/100
higher than 75% of 4,460 AI and agents servers
Tools
8
the typical AI and agents server has 6
Tools that change things
13%
typical: 0%. Fewer write tools means less an agent can break
GitHub stars
2
more than 65% of 2,711 AI and agents servers
Downloads a week
35
more than 25% of 1,538 AI and agents servers with a package

What its tools let an agent do: 7 read · 1 change

Install TheCrawler in Claude Code, Cursor or VS Code

Runs npx -y thecrawler on your machine. Read the scan report first; the gateway never runs local packages.

claude mcp add thecrawler -- npx -y thecrawler
Add to Cursor

What the publisher says

From the TheCrawler repository's README, as published. We do not edit it. Read it on GitHub

TheCrawler — AI-ready web scraper with validated extraction contracts

Scrape web pages, run LLM-powered structured extraction, or diagnose whether URLs are ready for a built-in extraction contract before spending LLM tokens. Open source engine (AGPL-3.0). $0.005 per successfully scraped page on Apify.

Start with a safe test: run one public URL with dryRun: true on Apify, or clone the current GitHub source and run the local CLI/MCP build from engine/. A small proof pack is in examples/diagnostic-challenge, including a sample readiness report at examples/diagnostic-challenge/sample-report.md.

$500 extraction readiness sprint

Use this when you need to know whether one real public-web workflow is worth automating before you spend engineering time on extraction.

  • Scope: up to 25 public URLs and one target output shape.
  • First step: send a public fit check through the structured issue form or use the private fit-check path on the sprint page.
  • Payment: requested only after the workflow looks like a fit, by one-off $500 payment link or invoice.
  • Output: a readiness report with ready, mixed, blocked, or not-worth-automating-yet guidance.
  • Credit: if the workflow continues into setup or hosted usage, the $500 is credited toward that next step.

The public offer thread is GitHub issue #1. The proof pack includes a sample readiness report showing the report shape before a buyer sends URLs.

Public fit checks should use this shape:

Workflow type:
Public URLs (up to 25):
Target output shape / required fields:
Known blockers or constraints:
Timing:

Do not include login credentials, private URLs, personal data, or raw customer data in GitHub issues.

What makes this different

Shortened. The full README is on GitHub.

Nothing above is checked by us. What we check is on the safety report.

TheCrawler: common questions

Is TheCrawler MCP server safe?
Yes, by our scan: it is graded A (86/100). Read the TheCrawler safety report
How do I install TheCrawler?
It runs on your machine. Copy the Claude Code, Cursor, VS Code or Claude Desktop config from the install section.
Does TheCrawler need an API key?
Not as far as the registry entry and our scan can tell: no credentials are declared or required.
Is TheCrawler maintained?
The last commit was 125 days ago (2026-07-02). The latest release is v0.5.8.
What can I use instead of TheCrawler?
Servers from other publishers that do the same job: PDF MCP server, DataSinking MCP server and agent-device MCP server.
How does TheCrawler compare with other AI and agents MCP servers?
safety score 86/100 (higher than 75% of 4,460 AI and agents servers); tools 8 (the typical AI and agents server has 6); tools that change things 13% (typical: 0%. Fewer write tools means less an agent can break). Full comparison.

Alternatives to TheCrawler

Same job from other publishers: the closest match first, then the best rated.

  • PDF
    Agentic RAG over one PDF or a whole folder: hybrid search, selective page reads, tables, OCR.
    A
  • DataSinking
    Full-text financial reports (US, China, Japan, Korea, Taiwan, UK) as clean Markdown for RAG agents.
    A
  • agent-device
    Open source device layer for AI agents: control mobile, web, TV, and desktop apps over MCP
    A
  • Basic Memory
    Local-first knowledge management with bi-directional LLM sync via Markdown files.
    B
  • IWE (iwe.md)
    Markdown knowledge base as agent memory. Runs against the notes directory it is started in.
    A