Web-reading tools for a sovereign agent

Zoya runs three tools for the web (web_fetch, mcp_lightpanda, mcp_scrapling) plus a sovereign-first search chain. The other eighty-odd projects in this space were evaluated and rejected. This is the decision record: what runs, what was cut, and the dimension that decided each call. Four facts about Zoya did most of the filtering before quality was assessed: it is sovereign, single-user, MCP-native, and it is already the reasoning agent rather than a tool that needs one bolted on.

The constraints did most of the work

Four facts about what Zoya is remove most of the market before quality is assessed:

• Sovereign / self-hosted. No managed cloud services. That deletes the entire managed-browser and managed-extraction tier on contact: Browserbase, Bright Data, ScrapingBee, Diffbot, and friends.
• Single-user personal agent, not a scraping farm or a RAG-ingestion pipeline. "Run 100+ concurrent headless" and "crawl an entire docs site" aren't the workload; reading a URL inside a conversation is.
• MCP-native. Zoya is an MCP client; new tools plug in over stdio through a glibc sidecar. Anything that ships an MCP server drops in cleanly; anything that needs wrapping pays a tax.
• Zoya is already the agent. Its runtime wraps an LLM and runs the loop itself, so it already reasons. It does not need a second LLM to drive a browser, and it turns fetched content into typed JSON natively. This single fact rules out two whole categories (agent harnesses and LLM extractors) as redundant.

What we run

A three-tier escalation ladder for fetching, plus a search backend:

• web_fetch: an in-process HTTP GET (curl-shaped), no JavaScript. The default for static pages, JSON APIs, RSS, downloads. Cheapest and fastest; tried first.
• mcp_lightpanda_*: Lightpanda, a Zig headless browser, for JS-rendered pages and interaction (goto, markdown, fill, click, detectForms, evaluate). Used when web_fetch comes back with an empty shell or a "please enable JavaScript" page.
• mcp_scrapling_*: Scrapling, for bot-protected sites (TLS fingerprinting, Cloudflare/Turnstile) and adaptive extraction. The tier above a plain browser.
• Search: web_search runs a sovereign-first chain, SearXNG (self-hosted meta-search) → Brave Search API → DuckDuckGo, falling through on empty results. The agent can also force a specific engine per call (backend), so it can cross-check two engines on a hard lookup.

The agent-facing rule is one line: try cheap, escalate only when the cheaper tier fails.

Why Lightpanda

Zig, Apache-2.0, self-hostable, CDP-compatible, ~9× less RAM than headless Chrome. Headless Chrome costs ~200 MB per instance, the wrong tool for a single-host agent that needs a DOM occasionally. Lightpanda is beta: coverage gaps, some sites crash it, a few block it. Acceptable, because the tier below it (web_fetch) and above it (scrapling) cover the misses.

Why Scrapling

BSD-3, built-in MCP server, three fetcher tiers in one package: HTTP (curl-impersonate-style TLS fingerprinting, no browser), Stealthy (Cloudflare bypass), Dynamic (full browser). Its structured extraction is deterministic (element fingerprinting, not an LLM), so it costs nothing per call and survives layout drift. It closes the two gaps above Lightpanda (anti-bot fetch and selector extraction) in one sovereign tool that installs into the sidecar like the document-to-markdown server.

The dimensions we scored on

Every candidate was weighed on the same axes. For Zoya, the first four behaved as near-vetoes; the rest were tie-breakers.

1. Sovereignty: self-hostable vs managed-only. (veto: managed-only is out.)
2. License: permissive (BSD/MIT/Apache) vs copyleft (AGPL) vs commercial. (AGPL tolerated for a true service, avoided for a dependency.)
3. LLM dependency: deterministic vs needs an LLM at runtime. (veto-ish: a second runtime LLM is redundant, Zoya already drives one.)
4. MCP-native: ready to plug in vs needs wrapping.
5. Anti-bot capability: none / TLS-level / full stealth / managed-bypass.
6. Adaptivity: breaks on layout change vs adapts.
7. Scale shape: single-page vs spider vs concurrent pool. (deprioritized: single-user.)
8. Language / container fit: does it slot into the glibc sidecar; does it ship a usable image.
9. Maintenance: release cadence, star velocity, sponsor health.

What we skipped, and why

• All managed / cloud (Browserbase, Bright Data, Firecrawl-cloud, Tavily/Exa-managed, ScrapingBee, Apify, ZenRows, Diffbot, AgentQL, Kadoa): breaks sovereignty. This is the whole point of the project, so it's a hard no, not a trade-off.
• Agent harnesses (Browser Use, Stagehand, Skyvern, Magnitude, Anthropic computer-use): Zoya is the agent that drives tools. Bolting on another LLM-driven browser is an architectural inversion (and a second model bill). Hermes Agent stays a reference for how a memory-first multi-surface agent is built, not a dependency.
• Browser drivers (Playwright/Patchright/Puppeteer/Selenium/chromedp/Rod): only needed if you script a browser. Zoya drives Lightpanda and Scrapling through MCP instead, so a driver layer would be dead weight.
• Crawl4AI / Firecrawl: genuinely good, but single-URL→markdown is already covered three ways (web_fetch / lightpanda / scrapling). Their unique value is recursive multi-page crawl and RAG ingestion, which isn't a current use case (our knowledge base is populated out-of-band). Firecrawl is also AGPL and heavy. Deferred, with a written revisit trigger.
• Dedicated structured extractors (ScrapegraphAI, Firecrawl-Extract): redundant. Fetch a page through the ladder, and Zoya's own model emits the typed JSON; Scrapling covers the deterministic/selector case. A separate extractor is a second LLM doing what the agent already does.
• Other engines: Obscura (Rust, stealthier CDP) is the one to watch as a future swap for Lightpanda if the beta gaps bite; Servo and Ladybird aren't production-ready headless.
• Unbrowse / API-discovery: clever (call a site's hidden API directly), but niche; revisit per-site if it ever matters.

Adjacent: document parsers

This stack is web-reading; the sibling problem is converting documents (PDF, DocX, PPTX) into clean Markdown for the agent to read. Zoya runs markitdown as an MCP sidecar for that today. Docling (IBM Research, MIT, MCP-capable) is the more capable peer: stronger on table structure, layout reconstruction, and embedded figures, the obvious swap if markitdown's output ever costs us fidelity on tables or scientific PDFs. Both are parsing tools, not retrieval ones. A clean, structurally-perfect Markdown'd document is still untrusted content until it passes the memory store's trust gate (see Agent memory is an attack surface).

What would change our mind

Skips aren't permanent. They're recorded with the condition that would reopen them:

• Exa: the moment semantic / research-discovery search (find by meaning, find-similar) becomes a real workload. It's the one search API that adds something the keyword stack can't; until then its SaaS key + cost aren't worth it. (Tavily was evaluated and skipped: it's roughly Brave with cleaner snippets, i.e. redundant; Firecrawl is a crawler, not search, covered below.)
• Crawl4AI: the moment Zoya needs to bulk-ingest whole sites/docs into memory (crawl → chunk → store). That's the one job the read-one-URL loop doesn't scale to.
• Obscura: if Lightpanda's beta coverage gaps start costing us real pages.

These live in a decision log where every entry carries an explicit "revisit when" trigger. A deferral you can't reopen is just a guess you got attached to.

---

Appendix: how to choose

The full reference the decision was made against. Sorted by layer; each row carries the dimensions above.

Decision tree

Need to extract data from web pages?
│
├─ Just need clean text/markdown for an LLM to read?
│    → Jina Reader (cheapest), Firecrawl, Crawl4AI (self-hosted)
│
├─ Need typed JSON matching a schema?
│    ├─ Schema known, layout stable  → Scrapling (selectors + adaptive)
│    ├─ Schema known, layout volatile → ScrapegraphAI (LLM extraction)
│    └─ Schema unknown, exploratory  → Diffbot, AgentQL
│
├─ Need to click, fill forms, log in?
│    ├─ Deterministic script → Playwright + Patchright (anti-detect)
│    ├─ LLM-driven actions   → Browser Use, Stagehand
│    └─ Heavy anti-bot       → Browserbase, or Scrapling StealthyFetcher
│
├─ Need to crawl an entire site?
│    ├─ Self-hosted → Scrapling Spiders, Scrapy, Crawl4AI
│    └─ Managed     → Firecrawl, Apify
│
├─ Avoid the browser (use the site's hidden API)?
│    → Unbrowse, or DIY with curl-impersonate
│
└─ Just need search results, not page scraping?
     → Tavily, Exa, SearXNG (self-hosted), Brave Search API

1. Browser engines

The thing that parses HTML and runs JavaScript. Drop-in CDP-compatible engines.

Pick rule: Chromium unless you have a strong reason. Lightpanda/Obscura when you'll spawn >50 concurrent and memory matters. Firefox/WebKit for fingerprint diversity.

2. Browser drivers

Libraries that control engines via CDP or WebDriver.

Pick rule: Playwright + Patchright if anti-detection matters; plain Playwright otherwise; curl-impersonate if you can skip JS entirely.

Appendix: extraction and frameworks

3. URL → clean markdown / text

Pick rule: Crawl4AI for self-hosted at any scale; Jina Reader for prototyping; Firecrawl for managed recursive crawl; Trafilatura when you already have the HTML.

4. URL → typed JSON (structured extraction)

Pick rule: ScrapegraphAI for volatile layouts with LLM budget; Scrapling for stable layouts, free; Diffbot/AgentQL for commercial reliability. For an LLM agent: it already extracts, so you mostly need a good fetch.

5. Full scraping frameworks

Pick rule: Scrapling for new projects; Scrapy for legacy/team familiarity; Katana for security recon, not data extraction.

Appendix: harnesses, cloud, and search

6. Agent harnesses (LLM drives the browser)

Pick rule: Browser Use for OSS production; Stagehand if you're already on Browserbase; Hermes Agent as an architectural reference for Zoya rather than a competitor.

7. Cloud browser infrastructure (managed)

Headless Chrome as a service. Skipped wholesale on sovereignty.

8 to 10. Self-hosted agentic browsers, API-discovery, search

11. MCP servers (cross-category index)

Self-hostable (sovereign-friendly): Scrapling, Crawl4AI, Firecrawl (self-host backend), Jina Reader (self-host backend), ScrapegraphAI, Playwright (MS), Chrome DevTools, Puppeteer, SearXNG, Browser Use, Hermes Agent, Unbrowse (self-host mode).

Managed-only: Browserbase, Browserless (cloud), Bright Data, Apify, Tavily, Exa, Linkup, ScrapingBee, Firecrawl (managed), Jina (managed).

Appendix: pickers and combinations

Picker logic for common tasks

Notable combinations

• Crawl4AI + ScrapegraphAI: cheap markdown for the 90%, LLM extraction on layout-volatile pages.
• Lightpanda + Playwright: drop Lightpanda in as the CDP endpoint for an existing Playwright script (beta, expect gaps).
• Patchright + Scrapling DynamicFetcher: stack stealth patches under Scrapling's stealth layer.
• SearXNG + Crawl4AI: search, then fetch the top N. Fully sovereign pipeline.
• Scrapling MCP + Zoya approval gates: the sovereign agent stack: MCP exposes the fetchers, the agent wraps risky calls in human-in-the-loop. This is the one we shipped.