webcrawler-deep-crawl skill

by browser-act·browser-act/skills·6.0k stars·MIT

Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, description, author, language, canonical URL, OG) and in-scope outbound links. Use when user mentions deep crawl website, recursive crawl, crawl a whole site, scrape entire website, scrape docs site, scrape documentation, scrape knowledge base, scrape blog, build RAG corpus, build vector database from website, knowledge base for chatbot, GPT knowledge files, llms.txt, sitemap crawl, BFS crawl, scrape with depth or page limit, include exclude URL globs, remove boilerplate, strip navigation header footer, website to markdown, website to text, multi-page extraction, bulk page scraping, clean markdown from URL, docs site to markdown corpus, site to clean corpus. Also applies to building RAG pipelines, indexing a customer site, syncing docs into a vector store, generating training corpora from any docs hub, or expanding a single start URL into a clean corpus of every reachable in-scope page.

A100/100content scan

Is the webcrawler-deep-crawl skill safe?

Clean: nothing in its files matched our rules. We read 5 files in the folder on 2026-09-28.

No findings.

Install the webcrawler-deep-crawl skill

A skill is a folder. Copy it into your agent's skills folder and the agent loads it when the task matches its description.

git clone --depth 1 https://github.com/browser-act/skills.git /tmp/skills
mkdir -p ~/.claude/skills
cp -r /tmp/skills/solutions/search-research/webcrawler-deep-crawl ~/.claude/skills/webcrawler-deep-crawl

available in every project

In the Claude apps, zip the folder and upload it from the Skills settings. The folder on GitHub

The instructions your agent would load

SKILL.md as published, without the frontmatter. Read it on GitHub

Website Deep Crawl

Input: one or more start URLs (+ optional scope, depth, page-count, globs, removal selectors). Output: per-page records {url, crawl, metadata, text, markdown, html, outboundLinks} for every page reached within scope.

Language

All process output to user (progress updates, process notifications) follows the user's language.

Objective

From a small set of start URLs, breadth-first crawl every reachable in-scope page, strip boilerplate (navigation, header, footer, cookie banners, etc.), and emit per-page LLM-ready content (text / markdown / HTML) plus structured metadata — suitable for feeding RAG pipelines, vector databases, or chatbot knowledge bases.

Prerequisites

One or more start URLs are provided by the caller.
Target pages are publicly reachable, OR the running browser is already logged in for any pages behind authentication.
A working directory is available for writing per-page JSON records and the crawl state file.

Pre-execution Checks

1. Tool Readiness

If browser-act has been confirmed available in the current session → skip this step.

Invoke browser-act via Skill tool to load usage. If installation or configuration issues arise, follow its guidance to resolve then retry.

2. Login Verification (when prerequisites include login requirement)

If login status for the target site has been confirmed in the current session → skip this step.

Otherwise: open the target site and observe the page login status:

Logout/sign-out entry, user avatar, or username exists → logged in, continue execution
Login/register entry exists with no logout entry → not logged in, inform the user that login is needed first, assist the user in completing the login flow

User refuses or cannot log in → terminate execution.

Capability Components

This Skill's operational boundary = what the user can manually do in their browser. It only reads data already displayed to the user on the page, never bypassing authentication or access controls. Its role is equivalent to copy-pasting on the user's behalf — the data is already on screen, automation merely saves time. JS code is encapsulated in Python files under the scripts/ directory, invoked via browser-act --session eval "$(python scripts/xxx.py {params})". $(...) is bash syntax; use the bash tool for execution. The eval token below refers to the browser-act CLI eval subcommand — always include the browser-act --session prefix and the "$(...)" substitution.

Below are all atomic capabilities discovered and verified during the exploration phase, listed by command template with parameters. Simply invoke them as needed — no need to read scripts/*.py source code or re-verify. Only inspect scripts when execution fails for troubleshooting. Combine freely as needed during execution.

API: discover URLs from /llms.txt

Probes {origin}/llms.txt (a convention used by LLM-friendly documentation sites) and returns every URL it lists. Fast path — try this first.

eval "$(python scripts/discover-llms-txt.py 'https://example.com')"

Parameters:

positional origin: the site origin (scheme + host), e.g. https://docs.example.com

Output example:

{
  "error": false,
  "source": "llms.txt",
  "count": 88,
  "urls": ["https://docs.example.com/intro", "https://docs.example.com/install"]
}

On failure (file missing or HTTP error): {"error": true, "message": "llms.txt not available (HTTP 404)", "urls": []} — move on to sitemap discovery.

API: discover URLs from /sitemap.xml

Probes {origin}/sitemap.xml and {origin}/sitemap_index.xml, follows nested sitemap indexes, and collects every URL.

eval "$(python scripts/discover-sitemap.py 'https://example.com' --max-urls 5000)"

Parameters:

positional origin: site origin
--max-urls: hard cap to stop ballooning sitemap indexes, default 5000

Output example:

{
  "error": false,
  "source": "sitemap.xml",
  "count": 86,
  "urls": ["https://example.com/page-a", "https://example.com/page-b"]
}

On failure: {"error": true, "message": "No sitemap found at standard paths", "urls": []} — fall back to DOM link discovery.

DOM: discover URLs from the current page

Reads from the currently loaded page DOM, normalizes them to absolute URLs, drops fragments, asset extensions, and out-of-scope links, and optionally applies include / exclude glob filters. Use when llms.txt and sitemap are both unavailable, or to extend the queue with links discovered while crawling.

eval "$(python scripts/discover-links.py 'https://example.com/docs/' --include-globs '[]' --exclude-globs '["/changelog/"]')"

Parameters:

positional start_url: scope URL. Only links under its directory (or equal to it) are kept.
--include-globs: JSON array of glob patterns; if non-empty, a link must match at least one to be kept. Default [] (no include filter).
--exclude-globs: JSON array of glob patterns; matching links are dropped. Default [].

Glob semantics: matches any characters (including /), matches any except /, ? matches one character. Example: https://example.com/{docs,api}/**.

Output example:

{
  "error": false,
  "source": "dom",
  "page": "https://example.com/docs/intro",
  "scope_base": "https://example.com/docs/",
  "count": 12,
  "links": ["https://example.com/docs/intro", "https://example.com/docs/install"]
}

DOM: extract clean content + metadata + outbound links from the current page

The core extractor. Run this on every crawled page after wait stable. Returns the page body in the requested format(s), structured metadata, and the in-scope outbound links found on this page (so callers can extend the BFS queue without a second DOM pass).

eval "$(python scripts/extract-page-content.py 'https://example.com/docs/' --output-format markdown --remove-selectors '.cookie-banner,#chat-widget' --include-globs '[]' --exclude-globs '[]')"

Parameters:

positional start_url: scope URL — used to filter the outboundLinks array to in-scope links only.
--output-format: one of markdown, text, html, all. Default markdown. all includes every body field.
--remove-selectors: comma-separated CSS selectors to delete from the chosen content root before extraction (in addition to the built-in boilerplate list). Use this to strip site-specific chrome (e.g. .cookie-banner, #chat-widget).
--keep-selector: a single CSS selector identifying the main content area. If set, only this element's content is extracted (overrides the built-in content-root heuristic). Use this when the site has a known main wrapper, e.g. article.docs-content.
--include-globs / --exclude-globs: same semantics as discover-links; applied to the returned outboundLinks array.

Content-root heuristic (used when --keep-selector is not provided), in priority order: , [role="main"], , #content, .content, .

Built-in boilerplate removal (always applied) includes: nav, header, footer, aside, script, style, noscript, iframe, [role="navigation"], [role="banner"], [role="contentinfo"], .cookie*, .advertisement, .modal, .popup, .share, .social, .breadcrumb, .toc, [aria-hidden="true"], etc.

Output example:

{
  "error": false,
  "url": "https://example.com/docs/intro",
  "crawl": {
    "loadedUrl": "https://example.com/docs/intro",
    "loadedTime": "2026-06-25T04:37:23.643Z",
    "referrerUrl": null
  },
  "metadata": {
    "canonicalUrl": "https://example.com/docs/intro",
    "title": "Introduction — Example Docs",
    "description": "Get started with Example.",
    "author": null,
    "keywords": [],
    "languageCode": "en",
    "publishedAt": null,
    "modifiedAt": null,
    "ogImage": "https://example.com/og.png",
    "ogType": "website"
  },
  "text": "Introduction\n\nGet started with Example…",
  "markdown": "# Introduction\n\nGet started with Example…",
  "outboundLinks": [
    "https://example.com/docs/install",
    "https://example.com/docs/quick-start"
  ]
}

[AI Intervention] On pages with infinite scroll or lazy-loaded sections, before invoking this script: scroll down repeatedly (until page height stops growing or a max-scroll cap is hit) so the dynamic content is in the DOM. The script reads what is currently rendered — it cannot trigger lazy loading on its own.

Composite: full deep crawl from start URL(s)

End-to-end flow. The Agent orchestrates discovery → BFS queue → per-page extraction → persistence. Records are written one per page so that crashes can resume from where they stopped.

Step 1 — seed the queue:

For each start URL, perform discovery in this priority order and merge results. Stop discovery once the queue has enough URLs to honor max_pages.

a. eval "$(python scripts/discover-llms-txt.py '{origin}')" — instant full list when available. b. eval "$(python scripts/discover-sitemap.py '{origin}' --max-urls {cap})" — broad coverage. c. If both fail or return zero in-scope URLs: navigate to the start URL, wait stable, then eval "$(python scripts/discover-links.py '{start_url}' --include-globs '{globs}' --exclude-globs '{globs}')".

Filter all discovered URLs to scope: every URL must start with the start URL's origin + dirname/, and must satisfy include / exclude globs.

More skills from browser-act/skills

A1688-product-detailExtracts comprehensive wholesale product data from 1688.com product detail pages: title, tiered pricing, SKU variants with dimensions/weight, product images, seller info, shop scores, buyer protection, cross-border flags, product attributes, coupon/promotion data, and review stats. Use when user mentions 1688, 1688.com, wholesale China, alibaba wholesale, B2B China sourcing, Chinese wholesale scraper, 1688 product scrape, 1688 offer, 1688 detail, extract 1688 data, pull 1688 listings, get wholesale price, 1688 supplier info, factory stats 1688, 1688 SKU variants, 1688 product attributes, 1688 shop score, DSR score 1688, 1688 buyer protection, 1688 cross-border, 1688 dropship. Also applies to: scraping bulk product data from 1688 by offer ID list, monitoring 1688 supplier metrics, extracting 1688 pricing tiers for resale analysis.
Aairbnb-listing-detailFetches complete Airbnb listing details for a given numeric listing ID via the internal GraphQL API, returning title, room type, description, amenities, photos, coordinates, city, house rules, highlights, ratings, review count, bedroom configuration, and property overview. Use when user mentions Airbnb listing details, Airbnb property info, Airbnb room details, get Airbnb listing data, Airbnb amenities list, Airbnb house rules, Airbnb property description, Airbnb detail page scraper, Airbnb rooms detail, Airbnb property page data, Airbnb listing info, fetch Airbnb room details, pull Airbnb listing.
Aairbnb-search-listingExtracts Airbnb accommodation search results from a destination query via SSR-embedded data, returning listing ID, URL, name, coordinates, rating, price, photos, and badge info for each result, plus pagination cursors for multi-page retrieval. Use when user mentions Airbnb search results, Airbnb listings, vacation rental search, short-term rental listings, scrape Airbnb, get Airbnb data, find rentals on Airbnb, Airbnb destination search, Airbnb property list, Airbnb stays search, Airbnb accommodation results, pull Airbnb listings, collect Airbnb search data, Airbnb scraper, Airbnb search page extraction, Airbnb search by destination.
Aamazon-alexa-qaAmazon Alexa for Shopping Q&A automation: submits questions to Amazon's Alexa/Rufus AI shopping assistant and collects response text; supports optional keyword search context (navigate to search results page before asking for category-specific answers). Use when user mentions Amazon Alexa, Rufus, Amazon shopping assistant, Amazon AI chat, ask Amazon, Amazon Q&A, automate Alexa questions, Rufus chatbot, Amazon assistant automation, collect Alexa responses, bulk question submission to Amazon, keyword search context, category research. Also applies to extracting Amazon product recommendations from conversational AI, automating repeated queries to Amazon's AI shopping feature, collecting Alexa shopping responses at scale, or market research within a specific product category.
Aamazon-asin-lookup-api-skillThis skill helps users extract structured product details from Amazon using a specific ASIN (Amazon Standard Identification Number). Use this skill when the user asks to get Amazon product details by ASIN, lookup Amazon product title and price using ASIN, extract Amazon product ratings and reviews count for a specific ASIN, check Amazon product availability and current price, get Amazon product description and features via ASIN, enrich product catalog with Amazon data using ASIN, monitor Amazon product price changes for specific ASINs, retrieve Amazon product brand and material information, fetch Amazon product images and specifications by ASIN, validate Amazon ASIN and get product metadata.
Aamazon-best-selling-products-finder-api-skillThis skill helps users extract structured best-selling product data from Amazon via the BrowserAct API. Agent should proactively apply this skill when users express needs like search for best selling products on Amazon, extract Amazon product data based on keywords, find top rated Amazon products, monitor Amazon competitor prices and sales, discover trending products on Amazon marketplace, extract Amazon product titles prices and ratings, gather Amazon product sales volume for market research, search Amazon best sellers in specific region, collect Amazon product reviews and promotion details, analyze Amazon product availability and badges, get Amazon product data for market analysis.
Aamazon-bestseller-listingAmazon Best Sellers listing scraper: extract product cards from any Amazon Best Sellers (zgbs) or /gp/bestsellers/ category page — returns rank (position on chart), asin, title, url, image, imageAlt, price, stars, reviewCount, ratingRaw per item, plus category metadata (categoryName, categoryFullName, categoryUrl) and pagination state (currentPage, hasNextPage, nextPageUrl). Works across all Amazon regional TLDs (amazon.com, amazon.co.uk, amazon.de, amazon.co.jp, amazon.fr, amazon.it, amazon.es, amazon.ca, amazon.com.au, amazon.in, etc.). Use when user mentions Amazon Best Sellers, Amazon bestsellers, Amazon top 100, Amazon zgbs, Amazon /zgbs/, Amazon /gp/bestsellers/, Amazon Best Sellers Rank, Amazon BSR, Amazon top ranked products, Amazon top-selling products, Amazon chart, Amazon category ranking, Amazon best sellers by category, Amazon best sellers electronics, Amazon best sellers kitchen, Amazon best sellers toys, scrape Amazon bestsellers, extract Amazon top 100, Amazon rank scraper, Amazon best seller list, Amazon leaderboard, Amazon trending products, discover trending Amazon products, Amazon niche discovery, Amazon top ranked ASINs. Also applies to competitive intelligence via ranking snapshots, spotting up-and-coming products, sourcing bestseller ASINs for further enrichment, tracking rank changes over time, and building bestseller-per-category datasets.
Aamazon-buy-box-monitor-api-skillThis skill helps users extract basic product details other sellers prices and seller ratings from Amazon via ASIN automatically using the BrowserAct API. Agent should proactively apply this skill when users express needs like query Amazon buy box information, monitor Amazon product prices, extract Amazon product details by ASIN, check other sellers prices on Amazon, get Amazon seller ratings and feedback count, monitor buy box ownership for a specific ASIN, track Amazon fulfillment methods for competitors, compare Amazon product prices across different sellers, retrieve Amazon buy box availability status, analyze Amazon seller profile details.
Aamazon-competitor-analyzerScrapes Amazon product data from ASINs using browseract.com automation API and performs surgical competitive analysis. Compares specifications, pricing, review quality, and visual strategies to identify competitor moats and vulnerabilities.
Aamazon-listing-competitor-analysis-skillThis skill helps users analyze Amazon competitor listings by ASIN and produce structured competitive intelligence plus strategic opportunity points for their own go-to-market. The Agent should proactively apply this skill when users want to analyze a competitor Amazon listing by ASIN, understand what a top-ranked product does right in content keywords or visuals, find market gaps and unmet buyer needs, turn competitor research into opportunity maps for their brand, identify keyword placement patterns on rival listings, extract SEO insights from Amazon product pages, reverse-engineer competitor bullet and title strategies, mine competitor reviews for buyer psychology, compare seller and A plus content patterns, run gap analysis before launching a new SKU, research why a listing wins conversion signals, synthesize whitespace you can own versus the diagnosed listing, or say just look at this ASIN with a competitive or optimization angle.
Aamazon-product-api-skillThis skill helps users extract structured product listings from Amazon, including titles, ASINs, prices, ratings, and specifications. Use this skill when users want to search for products on Amazon, find the best selling brand products, track price changes for items, get a list of categories with high ratings, compare different brand products on Amazon, extract Amazon product data for market research, look for products in a specific language or marketplace, analyze competitor pricing for keywords, find featured products for search terms, get technical specifications like material or color for product lists.
Aamazon-product-detailAmazon product detail page scraper: extract full product data from any open Amazon product detail URL (any /dp/{asin} or /gp/product/{asin} page across all Amazon regional TLDs) — returns asin, url, title, brand, price, listPrice, stars, reviewsCount, starsBreakdown (5/4/3/2/1 star percentages), answeredQuestions, inStock, inStockText, delivery, fastestDelivery, returnPolicy, breadCrumbs, features (bullet points), description, bookDescription, thumbnailImage, highResolutionImages, galleryThumbnails, productOverview (Brand/Model/etc.), attributes (tech spec table), attributesMapped (flat key-value), bestsellerRanks (rank + category + url), variantAttributes (currently selected color/size/style), variantAsins, seller (name + id + url), isAmazonChoice, amazonChoiceText, monthlyPurchaseVolume, hasAPlusContent, hasBrandStory, aiReviewsSummary, reviewsLink, productPageReviews (sample), videosCount, locationText, loadedCountryCode. Works on amazon.com, amazon.co.uk, amazon.de, amazon.co.jp, amazon.fr, amazon.it, amazon.es, amazon.ca, amazon.com.au, amazon.in, amazon.com.mx, amazon.com.br, amazon.nl, amazon.se, amazon.sg, amazon.ae, amazon.sa, amazon.pl, amazon.tr, amazon.eg. Use when user mentions Amazon product page, Amazon /dp/, Amazon dp URL, Amazon ASIN scraper, Amazon product detail, Amazon PDP, Amazon product data, Amazon product info, Amazon product fields, Amazon product attributes, Amazon full field extraction, Amazon per-ASIN enrichment, Amazon rating breakdown, Amazon stars breakdown, Amazon bestseller rank, Amazon BSR, Amazon variants, Amazon variant ASINs, Amazon color size options, Amazon feature bullets, Amazon A+ content, Amazon brand story, Amazon AI review summary, Amazon bought in past month, Amazon monthly sales volume, Amazon Amazon's Choice badge, Amazon seller info, scrape Amazon product, enrich Amazon ASIN, Amazon ASIN details, Amazon product review data. Also applies to bulk ASIN enrichment from a list of URLs, competitive product research, brand catalog audits, price and stock monitoring per ASIN, and building a normalized product dataset from a list of Amazon URLs.

All agent skills → · MCP servers