Web-reading tools for a sovereign agent

Zoya runs three tools for the web — web_fetch, mcp_lightpanda, mcp_scrapling — plus a sovereign-first search chain. The other eighty-odd projects in this space were evaluated and rejected. This is the decision record: what runs, what was cut, and the dimension that decided each call. Four facts about Zoya did most of the filtering before quality was assessed: it is sovereign, single-user, MCP-native, and it is already the reasoning agent rather than a tool that needs one bolted on.

The constraints did most of the work

Four facts about what Zoya is remove most of the market before quality is assessed:

• Sovereign / self-hosted. No managed cloud services. That deletes the entire managed-browser and managed-extraction tier on contact — Browserbase, Bright Data, ScrapingBee, Diffbot, and friends.
• Single-user personal agent, not a scraping farm or a RAG-ingestion pipeline. "Run 100+ concurrent headless" and "crawl an entire docs site" aren't the workload; reading a URL inside a conversation is.
• MCP-native. Zoya is an MCP client; new tools plug in over stdio through a glibc sidecar. Anything that ships an MCP server drops in cleanly; anything that needs wrapping pays a tax.
• Zoya is already the agent. Its runtime wraps an LLM and runs the loop itself, so it already reasons. It does not need a second LLM to drive a browser, and it turns fetched content into typed JSON natively. This single fact rules out two whole categories (agent harnesses and LLM extractors) as redundant.

What we run

A three-tier escalation ladder for fetching, plus a search backend:

• web_fetch — an in-process HTTP GET (curl-shaped), no JavaScript. The default for static pages, JSON APIs, RSS, downloads. Cheapest and fastest; tried first.
• mcp_lightpanda_* — Lightpanda, a Zig headless browser, for JS-rendered pages and interaction (goto, markdown, fill, click, detectForms, evaluate). Used when web_fetch comes back with an empty shell or a "please enable JavaScript" page.
• mcp_scrapling_* — Scrapling, for bot-protected sites (TLS fingerprinting, Cloudflare/Turnstile) and adaptive extraction. The tier above a plain browser.
• Search: web_search runs a sovereign-first chain — SearXNG (self-hosted meta-search) → Brave Search API → DuckDuckGo — falling through on empty results. The agent can also force a specific engine per call (backend), so it can cross-check two engines on a hard lookup.

The agent-facing rule is one line: try cheap, escalate only when the cheaper tier fails.

Why Lightpanda

Zig, Apache-2.0, self-hostable, CDP-compatible, ~9× less RAM than headless Chrome. Headless Chrome costs ~200 MB per instance — wrong tool for a single-host agent that needs a DOM occasionally. Lightpanda is beta: coverage gaps, some sites crash it, a few block it. Acceptable, because the tier below it (web_fetch) and above it (scrapling) cover the misses.

Why Scrapling

BSD-3, built-in MCP server, three fetcher tiers in one package: HTTP (curl-impersonate-style TLS fingerprinting, no browser), Stealthy (Cloudflare bypass), Dynamic (full browser). Its structured extraction is deterministic — element fingerprinting, not an LLM — so it costs nothing per call and survives layout drift. It closes the two gaps above Lightpanda — anti-bot fetch and selector extraction — in one sovereign tool that installs into the sidecar like the document-to-markdown server.

The dimensions we scored on

Every candidate was weighed on the same axes. For Zoya, the first four behaved as near-vetoes; the rest were tie-breakers.

1. Sovereignty — self-hostable vs managed-only. (veto: managed-only is out.)
2. License — permissive (BSD/MIT/Apache) vs copyleft (AGPL) vs commercial. (AGPL tolerated for a true service, avoided for a dependency.)
3. LLM dependency — deterministic vs needs an LLM at runtime. (veto-ish: a second runtime LLM is redundant — Zoya already drives one.)
4. MCP-native — ready to plug in vs needs wrapping.
5. Anti-bot capability — none / TLS-level / full stealth / managed-bypass.
6. Adaptivity — breaks on layout change vs adapts.
7. Scale shape — single-page vs spider vs concurrent pool. (deprioritized: single-user.)
8. Language / container fit — does it slot into the glibc sidecar; does it ship a usable image.
9. Maintenance — release cadence, star velocity, sponsor health.

What we skipped, and why

• All managed / cloud (Browserbase, Bright Data, Firecrawl-cloud, Tavily/Exa-managed, ScrapingBee, Apify, ZenRows, Diffbot, AgentQL, Kadoa) — breaks sovereignty. This is the whole point of the project, so it's a hard no, not a trade-off.
• Agent harnesses (Browser Use, Stagehand, Skyvern, Magnitude, Anthropic computer-use) — Zoya is the agent that drives tools. Bolting on another LLM-driven browser is an architectural inversion (and a second model bill). Hermes Agent stays a reference for how a memory-first multi-surface agent is built, not a dependency.
• Browser drivers (Playwright/Patchright/Puppeteer/Selenium/chromedp/Rod) — only needed if you script a browser. Zoya drives Lightpanda and Scrapling through MCP instead, so a driver layer would be dead weight.
• Crawl4AI / Firecrawl — genuinely good, but single-URL→markdown is already covered three ways (web_fetch / lightpanda / scrapling). Their unique value is recursive multi-page crawl and RAG ingestion, which isn't a current use case (our knowledge base is populated out-of-band). Firecrawl is also AGPL and heavy. Deferred, with a written revisit trigger.
• Dedicated structured extractors (ScrapegraphAI, Firecrawl-Extract) — redundant. Fetch a page through the ladder, and Zoya's own model emits the typed JSON; Scrapling covers the deterministic/selector case. A separate extractor is a second LLM doing what the agent already does.
• Other engines — Obscura (Rust, stealthier CDP) is the one to watch as a future swap for Lightpanda if the beta gaps bite; Servo and Ladybird aren't production-ready headless.
• Unbrowse / API-discovery — clever (call a site's hidden API directly), but niche; revisit per-site if it ever matters.

Adjacent: document parsers

This stack is web-reading; the sibling problem is converting documents — PDF, DocX, PPTX — into clean Markdown for the agent to read. Zoya runs markitdown as an MCP sidecar for that today. Docling (IBM Research, MIT, MCP-capable) is the more capable peer: stronger on table structure, layout reconstruction, and embedded figures — the obvious swap if markitdown's output ever costs us fidelity on tables or scientific PDFs. Both are parsing tools, not retrieval ones. A clean, structurally-perfect Markdown'd document is still untrusted content until it passes the memory store's trust gate — see Agent memory is an attack surface.

What would change our mind

Skips aren't permanent — they're recorded with the condition that would reopen them:

• Exa — the moment semantic / research-discovery search (find by meaning, find-similar) becomes a real workload. It's the one search API that adds something the keyword stack can't; until then its SaaS key + cost aren't worth it. (Tavily was evaluated and skipped — it's ≈ Brave with cleaner snippets, i.e. redundant; Firecrawl is a crawler, not search — see below.)
• Crawl4AI — the moment Zoya needs to bulk-ingest whole sites/docs into memory (crawl → chunk → store). That's the one job the read-one-URL loop doesn't scale to.
• Obscura — if Lightpanda's beta coverage gaps start costing us real pages.

These live in a decision log where every entry carries an explicit "revisit when" trigger. A deferral you can't reopen is just a guess you got attached to.

---

Appendix: the landscape we evaluated

The full reference the decision was made against. Sorted by layer; each row carries the dimensions above.

Decision tree

Need to extract data from web pages?
│
├─ Just need clean text/markdown for an LLM to read?
│    → Jina Reader (cheapest), Firecrawl, Crawl4AI (self-hosted)
│
├─ Need typed JSON matching a schema?
│    ├─ Schema known, layout stable  → Scrapling (selectors + adaptive)
│    ├─ Schema known, layout volatile → ScrapegraphAI (LLM extraction)
│    └─ Schema unknown, exploratory  → Diffbot, AgentQL
│
├─ Need to click, fill forms, log in?
│    ├─ Deterministic script → Playwright + Patchright (anti-detect)
│    ├─ LLM-driven actions   → Browser Use, Stagehand
│    └─ Heavy anti-bot       → Browserbase, or Scrapling StealthyFetcher
│
├─ Need to crawl an entire site?
│    ├─ Self-hosted → Scrapling Spiders, Scrapy, Crawl4AI
│    └─ Managed     → Firecrawl, Apify
│
├─ Avoid the browser (use the site's hidden API)?
│    → Unbrowse, or DIY with curl-impersonate
│
└─ Just need search results, not page scraping?
     → Tavily, Exa, SearXNG (self-hosted), Brave Search API

1. Browser engines

The thing that parses HTML and runs JavaScript. Drop-in CDP-compatible engines.

Pick rule: Chromium unless you have a strong reason. Lightpanda/Obscura when you'll spawn >50 concurrent and memory matters. Firefox/WebKit for fingerprint diversity.

2. Browser drivers

Libraries that control engines via CDP or WebDriver.

Pick rule: Playwright + Patchright if anti-detection matters; plain Playwright otherwise; curl-impersonate if you can skip JS entirely.

3. URL → clean markdown / text

Pick rule: Crawl4AI for self-hosted at any scale; Jina Reader for prototyping; Firecrawl for managed recursive crawl; Trafilatura when you already have the HTML.

4. URL → typed JSON (structured extraction)

Pick rule: ScrapegraphAI for volatile layouts with LLM budget; Scrapling for stable layouts, free; Diffbot/AgentQL for commercial reliability. For an LLM agent: it already extracts — you mostly need a good fetch.

5. Full scraping frameworks

Pick rule: Scrapling for new projects; Scrapy for legacy/team familiarity; Katana for security recon, not data extraction.

6. Agent harnesses (LLM drives the browser)

Pick rule: Browser Use for OSS production; Stagehand if you're already on Browserbase; Hermes Agent as an architectural reference for Zoya rather than a competitor.

7. Cloud browser infrastructure (managed)

Headless Chrome as a service. Skipped wholesale — sovereignty.

8 – 10. Self-hosted agentic browsers, API-discovery, search

11. MCP servers (cross-category index)

Self-hostable (sovereign-friendly): Scrapling, Crawl4AI, Firecrawl (self-host backend), Jina Reader (self-host backend), ScrapegraphAI, Playwright (MS), Chrome DevTools, Puppeteer, SearXNG, Browser Use, Hermes Agent, Unbrowse (self-host mode).

Managed-only: Browserbase, Browserless (cloud), Bright Data, Apify, Tavily, Exa, Linkup, ScrapingBee, Firecrawl (managed), Jina (managed).

Picker logic for common tasks

Notable combinations

• Crawl4AI + ScrapegraphAI — cheap markdown for the 90%, LLM extraction on layout-volatile pages.
• Lightpanda + Playwright — drop Lightpanda in as the CDP endpoint for an existing Playwright script (beta — expect gaps).
• Patchright + Scrapling DynamicFetcher — stack stealth patches under Scrapling's stealth layer.
• SearXNG + Crawl4AI — search, then fetch the top N. Fully sovereign pipeline.
• Scrapling MCP + Zoya approval gates — the sovereign agent stack: MCP exposes the fetchers, the agent wraps risky calls in human-in-the-loop. This is the one we shipped.

Sources

• Lightpanda — github.com/lightpanda-io/browser
• Scrapling — github.com/D4Vinci/Scrapling (v0.4.8, May 2026)
• Crawl4AI — github.com/unclecode/crawl4ai
• Firecrawl — firecrawl.dev · ScrapegraphAI — scrapegraphai.com
• Browser Use — github.com/browser-use/browser-use · Playwright MCP — github.com/microsoft/playwright-mcp