Octocrawl MCP server

Octocrawl — Web-to-LLM Context Extraction

A transparent, verifiable web extraction system built for RAG and Agent workflows.

Try it without installing anything at octocrawl.dev, read the documentation, or connect your agent to hosted Octocrawl in one line, no key needed to start (how, for Claude Code, Cursor, OpenCode and Codex):

claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp

Why This Exists

Most crawlers report "success" when they return empty pages, challenge screens, or the wrong content. Octocrawl makes failure visible and fixable:

Before (typical crawler):

✓ Fetched example.com/article
  Status: 200 OK
  Content: 953 bytes

After (Octocrawl):

✗ Fetched example.com/article
  Status: blocked (cloudflare_challenge)
  Lane: http → escalated to browser_local
   Evidence: artifacts=[] (no screenshot or DOM snapshot was produced)
  Cost: 847 tokens, 2.3s, $0.0042
  Fix: needs user login or proxy (tier 1b/2)

What's Different

1. Failure is a first-class outcome — emptyverified, blocked, failed with reasons, not silent empties; a page answered with an error status keeps its httpStatus and Markdown as evidence, never as success, and so does a page on which Octocrawl finds no main content (failed/emptyunverified with the whole page's Markdown), a main region the page hides with hidden and an article region that only names the page included (headings of fewer than 100 characters and almost nothing beside them, no image the Markdown carries, in-page jump links such as "Skip to Filters" set aside)
2. Five false-success checks — challenge text, wrong-page content, missing facts, truncation, yield-below-floor
3. Execution ladder — HTTP → browser → user auth → proxy, with automatic routing, per-attempt trace, and task-level cost accounting
4. Ground-truth benchmark — a 56-case fixture suite (soft 404s, challenge pages, SPAs, timeouts, zip bombs, tables) with verified false-success rates
5. Honest evidence — artifacts: [] is an explicit empty artifact list, not a promise that every failed page has a screenshot or DOM snapshot; browser bytesWire: null means wire bytes were not measured
6. One Evidence Record — every scrape result, batch item and crawl page carries evidenceRecord (final URL, redirect chain, fetch time, status and reason, lane, robots.txt decision, raw and output hashes, field evidence), stated the same way in every lane and described by a versioned JSON Schema; see the reference

Quick Start

The no-install, single-page web preview runs at octocrawl.dev (five previews a day; see Public preview for how it is deployed). Besides Markdown it returns a page's links and metadata, and up to 20 fields read from the page without a model. The same site serves the documentation: MCP connection steps for four clients, four task guides, and limits. The pages are generated from Markdown in this repository during the public web build; npm run public:preview:local serves them at http://127.0.0.1:8798/docs/.

Install nothing first: with Node.js 22.13+ or 24+,

npx octocrawl scrape https://example.com --markdown
pip install octocrawl-client      # the Python client of a running API (npx octocrawl serve)
npm install @octocrawl/sdk        # the TypeScript client; @octocrawl/mcp is the MCP server

Shortened. The full README is on GitHub.