searchts MCP server

searchts

<!-- mcp-name: io.github.capad-xyz/searchts -->

The missing layer between AI and the web. A Python CLI and library that lets an AI agent read and search the internet, fronted by a fully open-source "unlocker" that gets through common bot-walls with no paid proxy and no API key.

▶ Watch the full 1-minute demo

Why searchts?

• Reads pages behind common bot walls
• Reads complete ChatGPT / Claude / Gemini / Grok / Poe / DeepSeek / Perplexity / Copilot shared conversations
• Works with Claude, Codex, and MCP agents
• Extracts clean Markdown, ready to feed a model
• Searches the web without API keys
• Downloads a page's assets (images, fonts, palette)
• Transcribes videos, subtitles-first

Why it's free

AI agents constantly need to read web pages, but the naive way they fetch is trivially blocked by modern anti-bot systems (Cloudflare, PerimeterX, DataDome). Paid unlocker services solve this, but the thing they really charge for is a large pool of clean residential IP addresses. searchts runs on your own machine, from your own connection, at personal volume, so it sidesteps that cost and gets through most of those walls for free.

The unlocker

searchts reads any URL through an escalating ladder and stops at the first tier that returns real content:

1. curlcffi**: a fetch that impersonates a real Chrome's TLS/JA3 and HTTP2 fingerprint. Beats user-agent and fingerprint filters. Fast, local, private.
2. Jina Reader: a JavaScript-rendering relay (r.jina.ai), for pages that only fill in content after running JS. Default on — the target URL is sent to Jina's servers on this rung. Opt out with SEARCHTSNOJINA=1 or config jina: false (local curl + stealth only).
3. stealth browser: an undetected headless Chromium (patchright), launched lazily only when the cheaper tiers fail, for live JS / Cloudflare managed challenges.

If no tier comes back with real content, an optional human-in-the-loop step opens a real browser so you can clear the page once and continue. That covers interactive CAPTCHAs and soft walls alike: a login page served as HTTP 200 is not a challenge, but it is still a page only a human gets past. Block detection is phrase-based (not vendor-name based), so legitimate pages that merely embed a bot-sensor script are not falsely rejected. Content is extracted to clean Markdown with trafilatura.

Walls (F12 playbook, not a bypass): fail loud on login/challenge/thin. Do not cut a release that claims Reddit/LinkedIn now read (N7). Order: stealth already retries page.content after a navigation race (P3.11) → next is a persistent Chromium profile so clearance can survive across reads (F1, not shipped) → then --human / device session for extras only (F7, never silent, never inside readurl). Never paid residential as default (N1). Never a keyed commercial unlocker as default (N3**).

AI-chat share links

Shortened. The full README is on GitHub.