Metadata-Version: 2.4
Name: dlbrowser
Version: 4.0.0
Summary: Enterprise-Grade Multi-Backend Browser Engine for AI Agents and Web Automation
Author: Dekrypt Labs
License: MIT
Project-URL: Homepage, https://dekryptlabs.com/dlbrowser
Project-URL: Documentation, https://dlbrowser.dev
Project-URL: Source, https://github.com/dekryptlabs/dlbrowser
Project-URL: Changelog, https://github.com/dekryptlabs/dlbrowser/releases
Project-URL: Issues, https://github.com/dekryptlabs/dlbrowser/issues
Keywords: web-scraping,browser-automation,ai-agent,stealth,web-crawler,enterprise,cloudflare-bypass,tls-impersonation,mcp-server,model-context-protocol,headless-browser
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: lxml>=4.9
Requires-Dist: httpx>=0.27
Requires-Dist: markdownify>=0.11
Requires-Dist: pyyaml>=6.0
Provides-Extra: full
Requires-Dist: playwright>=1.40; extra == "full"
Requires-Dist: curl-cffi>=0.7; extra == "full"
Requires-Dist: nodriver>=0.38; python_version >= "3.10" and extra == "full"
Requires-Dist: cloudscraper>=1.2; extra == "full"
Requires-Dist: tls-client>=0.2; extra == "full"
Provides-Extra: mcp
Requires-Dist: mcp>=1.0; extra == "mcp"
Provides-Extra: billing
Requires-Dist: stripe<13,>=7; extra == "billing"
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == "langchain"
Requires-Dist: langchain-community>=0.3; extra == "langchain"
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.12; extra == "llamaindex"
Provides-Extra: web
Requires-Dist: fastapi>=0.110; extra == "web"
Requires-Dist: uvicorn>=0.27; extra == "web"
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.1; extra == "postgres"
Provides-Extra: all
Requires-Dist: dlbrowser[full,langchain,llamaindex,mcp,web]; extra == "all"
Provides-Extra: enterprise
Requires-Dist: dlbrowser[full,mcp]; extra == "enterprise"
Requires-Dist: curl-cffi>=0.7; extra == "enterprise"
Requires-Dist: tls-client>=0.2; extra == "enterprise"
Requires-Dist: cloudscraper>=1.2; extra == "enterprise"
Requires-Dist: firecrawl-py>=1.0; extra == "enterprise"
Dynamic: license-file

# DLBrowser — Reliable, metered web access for AI agents

Give your agent eyes. One MCP connection, auto-recovers through Cloudflare blocks, captchas, and JS-heavy pages that break every other tool. Every request metered against an API-key plan — transparent credits, no surprises.

```bash
# One line. Works everywhere.
npx dlbrowser connect --key dlb_xxx
pip install dlbrowser
dlbrowser fetch https://example.com
```

---

## Why DLBrowser

Every AI agent eventually hits the wall of the live web.

Cloudflare, DataDome, and PerimeterX block headless browsers. Captchas stop naive fetches. One site works with curl, another needs a real browser, another needs residential proxies. You have no idea which backend will work until you try — and no record of what each run costs.

DLBrowser solves this with a self-healing engine that tries the fastest backend first, falls back through increasingly stealthy alternatives, and logs every outcome into a queryable performance database that makes the router smarter over time.

**It is not a scraper. It is not a crawler. It is an acquisition infrastructure platform — the capability layer that gives AI agents reliable, observable, and billable access to any web resource.**

---

## Quick Start

### For AI Agents (one line)

```bash
# Add to Claude Code
claude mcp add dlbrowser --transport http \
  --url https://api.dlbrowser.dev \
  --header "Authorization: Bearer dlb_xxx"

# Or use the universal installer
npx dlbrowser connect --key dlb_xxx

# Now your agent has 11 tools:
# dlbrowser_fetch · dlbrowser_scrape · dlbrowser_smart_extract
# dlbrowser_screenshot · dlbrowser_markdown · dlbrowser_crawl
# dlbrowser_search · dlbrowser_serp · dlbrowser_status
# dlbrowser_account · dlbrowser_topup
```

### For Developers

```bash
# Install
pip install "dlbrowser[full]"
python3 -m playwright install chromium

# One-shot fetch
dlbrowser fetch https://news.ycombinator.com

# Check your usage
dlbrowser usage --user usr_abc --period monthly

# Start the MCP server
dlbrowser mcp --transport http --port 8929
```

### For Chrome Users

Install the extension from the Chrome Web Store (`Cmd+Shift+D` to scrape any page) or load it unpacked from `extension/`.

---

## How It Works

```
Agent / CLI / Extension / MCP Client
          │
    ┌─────┴─────┐
    │  MCP Gateway │  ← Paywalled (401/402/403)
    │  11 tools    │  ← free: status, account, topup
    └─────┬─────┘
          │
    ┌─────┴─────┐
    │   Router   │  ← Adaptive scoring: success×0.4 + latency×0.2 + cost×0.2 + freshness×0.2
    └─────┬─────┘
          │
    ┌─────┴──────────────────────────────┐
    │  curl_cffi → tls_client            │  ← HTTP-level, 50ms, no JS
    │  → playwright → nodriver → firecrawl│  ← Browser-level, 1.5-2.5s, full JS
    └─────┬──────────────────────────────┘
          │
    ┌─────┴─────┐
    │  Ledger   │  ← per-run cost, margin, trace_id
    └─────┬─────┘
          │
    ┌─────┴─────┐
    │  Invoice  │  ← credits × unit price
    └───────────┘
```

Every operation produces a `run_id`, `trace_id` (OpenTelemetry-compatible), `backend`, `credits_consumed`, and `estimated_cost`. The router learns per-domain — more usage means better routing.

---

## Backends

| Backend | Speed | Stealth | JS | Best For |
|---------|-------|---------|----|----------|
| **curl_cffi** | 50ms | Medium | No | Simple sites, APIs — 30x faster than browser |
| **tls_client** | 200ms | Medium | No | TLS fingerprint bypass |
| **undetected-playwright** | 1.5s | High | Yes | JS-heavy sites, SPAs |
| **nodriver (CDP)** | 2.5s | Highest | Yes | Aggressive anti-bot, direct CDP |
| **Firecrawl** | varies | Paid | Yes | Last resort (pay per-use) |

---

## Features

| | |
|---|---|
| **Self-healing fallback** | 5 backends auto-escalate on block. Never silently fails. |
| **Adaptive routing** | Per-domain scoring. Learns which backends work for which sites. |
| **SPA detection** | Detects Next.js/React shells and auto-escalates to a real browser. |
| **Atomic quota** | 0% billing overshoot. Reservation-based — no surprise bills. |
| **Per-run cost tracking** | Every request records infrastructure cost and gross margin. |
| **Captcha solving** | 3 providers (2captcha, anti-captcha, capsolver) for hCaptcha/reCAPTCHA/Turnstile. |
| **LLM extraction** | Natural language → structured JSON. "Extract product prices and ratings." |
| **Deep crawling** | BFS/DFS/Best-First with crash recovery and robots.txt compliance. |
| **Markdown conversion** | Full HTML→Markdown with tables, code blocks, mermaid diagrams. |
| **OpenTelemetry tracing** | Every run is traceable end-to-end. |
| **Open source (MIT)** | Self-host with full feature parity. Your data never leaves your infra. |
| **Framework adapters** | LangChain, LlamaIndex — drop-in tools for any agent framework. |

---

## Pricing

| Plan | Price | Credits/mo | Best for |
|------|-------|------------|----------|
| **Scout** | Free | 2,500 (one-time) | Evaluation, hobby projects |
| **Analyst** | $29/mo | 100,000 | Indie devs, small teams |
| **Operator** | $85/mo | 500,000 (+ captcha) | Production workloads |
| **Command** | $299/mo | 2,000,000 (+ proxy) | Scale teams |
| **Enterprise** | Custom | Unlimited | Dedicated infra, SLA, support |

Overage: $0.001/credit. Annual: 20% off. No credit multipliers — what you see is what you pay.

---

## Enterprise

- **Multi-tenant** — Organizations, teams, seats, roles
- **API keys** — Create, revoke, rotate, scope by permission, set expiry
- **Audit logs** — Immutable event ledger: who did what, when, at what cost
- **Usage dashboard** — Credits, cost, success rate, top domains
- **Webhooks** — HMAC-SHA256 signed events with dead-letter queue
- **SLA monitoring** — Scheduled page diffing with noise-tolerant change detection
- **Self-hosted** — Docker one-liner, full feature parity

---

## When NOT to Use DLBrowser

- You need **pre-built datasets** → use Bright Data/Apify
- You need a **visual scraping IDE** → use Browserflow/Octoparse
- You need **screenshot-as-a-service** → use Urlbox
- You need an **enterprise proxy network** → use Bright Data

DLBrowser is for teams running AI agents that need reliable, observable, cost-controlled web access at scale.

---

## Benchmarks

*[Coming soon — running the full test suite against Firecrawl, Crawl4AI, and naive fetch on 50 production sites]*

The codebase supports 5-backend fallback, adaptive routing, and per-run cost tracking that no competitor has. We'll publish benchmark data once the managed gateway is live.

---

## Documentation

| Resource | Link |
|----------|------|
| Getting Started | [dlbrowser.dev/getting-started](https://dlbrowser.dev) |
| API Reference | [docs.dlbrowser.dev](https://dlbrowser.dev) |
| Architecture | [docs.dlbrowser.dev/architecture](https://dlbrowser.dev) |
| Self-Hosted Deploy | [DEPLOYMENT.md](./DEPLOYMENT.md) |
| Comparison | [dlbrowser.dev/comparison](https://dlbrowser.dev) |
| Enterprise | [dlbrowser.dev/enterprise](https://dlbrowser.dev) |
| MCP Setup | [docs.dlbrowser.dev/mcp](https://dlbrowser.dev) |

---

## The Team

Built by [Dekrypt Labs](https://dekryptlabs.com). Contact: sales@dekryptlabs.com

## License

MIT — Dekrypt Labs
