doc-scraper MCP server

LLM Documentation Scraper (doc-scraper)

A configurable, concurrent, and resumable web crawler written in Go. Specifically designed to scrape technical documentation websites, extract core content, convert it cleanly to Markdown format suitable for ingestion by Large Language Models (LLMs), and save the results locally.

Overview

This project provides a powerful command-line tool to crawl documentation sites based on settings defined in a config.yaml file. It navigates the site structure, extracts content from specified HTML sections using CSS selectors, and converts it into clean Markdown files.

Why Use This Tool?

• Built for LLM Training & RAG Systems - Creates clean, consistent Markdown optimized for ingestion
• Preserves Documentation Structure - Maintains the original site hierarchy for context preservation
• Production-Ready Features - Offers resumable crawls, rate limiting, and graceful error handling
• High Performance - Uses Go's concurrency model for efficient parallel processing

Goal: Preparing Documentation for LLMs

The main objective of this tool is to automate the often tedious process of gathering and cleaning web-based documentation for use with Large Language Models. By converting structured web content into clean Markdown, it aims to provide a dataset that is:

• Text-Focused: Prioritizes the textual content extracted via CSS selectors
• Structured: Maintains the directory hierarchy of the original documentation site, preserving context
• Cleaned: Converts HTML to Markdown, removing web-specific markup and clutter
• Locally Accessible: Provides the content as local files for easier processing and pipeline integration

Key Features

Shortened. The full README is on GitHub.