CrawlCheck MCP server

crawlcheck-core

The dependency-free modules of CrawlCheck, the verification layer for the agentic web, extracted from the production Worker. These are the readers behind its signed answer about what a site permits and serves to AI crawlers and agents. No build step, no dependencies, ESM, Node 20+.

The hosted service fetches a site as 15 crawler identities and grades 22 sections across three stages — reach (can the crawler get in), read (what it actually receives), quote (can an answer engine lift it). This repository is the open core: the readers and checks that need no network, no store and no key, published so the measurements can be reproduced.

What is here today: the robots.txt policy reader, the crawler identifier / log-line forgery verifier, the quotable-content reader, the image-signal reader, and the markdown-negotiation and agent-surface probes. What stays in the service: the scoring — the thresholds each reading is judged against, the weight every section carries in the grade, and the corpus calibration those numbers were set from — plus fetching as fifteen crawler identities, the record over time, monitoring, the dataset, and billing. The rule is simple: a reading of your page is a fact you are entitled to check, and the calibration is the product.

Install

npm install github:emmanuelorta/crawlcheck-core

Not on the npm registry yet, so install it from the repository. Every subpath below resolves from that install.

robots — which crawlers your robots.txt actually admits

import { crawlerPolicy } from "crawlcheck-core/robots";

const policy = crawlerPolicy(robotsTxt, /* unreadable */ false, "/wp-admin/");
policy.agents.find(a => a.ua === "GPTBot");
// { ua:"GPTBot", label:"OpenAI training corpus", role:"train",
//   allowed:true, allowed_root:true, named:true, rules_shadowed:true }
policy.shadowed;
// [{ uas:["gptbot"], missing:["/wp-admin/","/feed/"], missing_count:2 }]

For each of 92 named agents (answer engines, search indexes, training crawlers, SEO tools, social unfurlers, regional engines) it reports: which user-agent group applies (named), whether the given path and the root are allowed under that group, and whether the group shadows the * rules.

The shadowing defect

A crawler obeys only its most specific group. The moment you write

User-agent: *
Disallow: /wp-admin/

User-agent: GPTBot
Allow: /

GPTBot no longer sees Disallow: /wp-admin/ — the group stops applying to it entirely. Sites that add an allowlist for AI agents routinely void every rule they thought still held. shadowedGroups() lists each named group and the disallows it lost. This was found on our own sites first: how to block AI crawlers and the glossary entry for user-agent group.

API

What it does not do

Shortened. The full README is on GitHub.