Metadata-Version: 2.4
Name: browser-optimizer-mcp
Version: 0.1.1
Summary: Browser Optimizer MCP — reduces LLM token usage by 80-98% for browser automation
Author: Manthan Railkar, Ayush Mhatre
License: MIT
Project-URL: Homepage, https://github.com/yourusername/browser-optimizer-mcp
Project-URL: Repository, https://github.com/yourusername/browser-optimizer-mcp
Project-URL: Bug Tracker, https://github.com/yourusername/browser-optimizer-mcp/issues
Keywords: mcp,browser,playwright,llm,ai-agent,automation,token-optimization,fastmcp
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: playwright>=1.40.0
Requires-Dist: mcp>=1.0.0
Requires-Dist: beautifulsoup4>=4.12.0
Requires-Dist: httpx>=0.25.0
Requires-Dist: loguru>=0.7.0
Requires-Dist: lxml>=4.9.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: xxhash>=3.0.0
Requires-Dist: cachetools>=5.0.0
Dynamic: license-file

# Browser Optimizer MCP

[![PyPI version](https://img.shields.io/pypi/v/browser-optimizer-mcp.svg)](https://pypi.org/project/browser-optimizer-mcp/)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

## ⚡ Quick Install

```bash
pip install browser-optimizer-mcp
browser-optimizer install
```

The `install` command automatically:
- ✅ Checks your Python version (≥ 3.11 required)
- ✅ Installs Playwright browser binaries (`chromium`)
- ✅ Detects **Claude Desktop** and auto-writes its MCP config
- ✅ Detects **Antigravity IDE** and auto-writes its MCP config
- ✅ Prints manual setup instructions for **Cursor**
- ✅ Verifies the installation end-to-end

Once installed, start the server with:

```bash
browser-optimizer start
```

> **Tip:** Run `browser-optimizer doctor` at any time to diagnose your setup.

---

An optimization middleware layer built on top of **FastMCP** and **Playwright**. It sits between AI agents (LLMs) and browser automation frameworks to drastically reduce token usage, execution latency, and API inference costs while maintaining high accuracy for browser workflows.

---

## 🔄 How It Works & Process Flow

### 1. Step-by-Step Execution Pipeline
When an AI agent requests a page analysis, the Optimizer executes the following pipeline:
1. **Request Intake**: The AI agent calls `extract_context` with a target URL.
2. **Browser Navigation**: The browser manager opens or reuses a Playwright page and navigates to the URL.
3. **HTML & Accessibility Tree Capture**: The extractor captures the raw HTML markup and generating a semantic ARIA snapshot of the page body.
4. **xxhash Fingerprinting**: The semantic cache hashes the raw HTML to uniquely identify the page state.
   * **Cache Hit**: If the hash matches a stored signature, the server returns the cached context in less than `1ms`, skipping DOM parsing and cleaning completely.
   * **Cache Miss**: If it's a new page or the content changed, the extraction process continues.
5. **Context Compression**: The compressor strips out styling, scripts, SVGs, and header/footer boilerplate. It extracts only interactable elements (buttons, inputs, dropdowns, links).
6. **Task Classification**: The rule-based classifier analyzes the interactive elements to score and categorize the page type (e.g. login, product search, checkout).
7. **Delta Diff Calculation**: The difference engine compares the fresh UI elements with the last observed state of this URL, outputting only added or removed controls.
8. **Metrics Logging**: The metrics module records raw bytes, compressed bytes, and savings ratios.
9. **Payload Delivery**: A compact JSON package containing the optimized UI elements, ARIA snapshot, and page metadata is returned to the agent.

### 2. Process Flow Diagram
```mermaid
sequenceDiagram
    autonumber
    actor Agent as AI Agent (LLM)
    participant Server as MCP Server (main.py)
    participant Cache as Semantic Cache (cache.py)
    participant Browser as Browser Manager (manager.py)
    participant Ext as Page Extractor (extractor.py)
    participant Comp as Context Compressor (compressor.py)
    participant Diff as Difference Engine (diff.py)

    Agent->>Server: extract_context(url)
    Server->>Browser: navigate(url)
    Browser-->>Server: page loaded
    Server->>Cache: lookup(url, current_html)
    alt Cache Hit (HTML Unchanged)
        Cache-->>Server: Return cached context
    else Cache Miss (HTML Changed / New URL)
        Server->>Ext: extract(page)
        Ext-->>Server: raw html, title, url, ARIA tree
        Server->>Comp: compress(extracted_data)
        Comp-->>Server: cleaned UI elements list, text summary
        Server->>Cache: store(url, html, compressed_context)
    end
    Server->>Diff: compute_diff(url, current_ui)
    Diff-->>Server: added & removed UI elements
    Server-->>Agent: Return optimized JSON context
```

---

## Use Cases

Here are 10 key use cases where the Browser Optimizer MCP can be utilized:

1. **AI Agent Cost Reduction**: Reduces token usage on dense web pages (like e-commerce portals or social media feeds) by up to 98%, dramatically lowering LLM API costs.
2. **Structured Web Scraping**: Empowers LLM-based scrapers to locate and extract content from specific elements without parsing scripts, styles, or bloated HTML trees.
3. **Automated UI/E2E Testing**: Helps developers run fast assertion checks on UI changes by using delta diffing to detect structural regressions.
4. **Form Auto-Filling**: Provides clean, interactive element trees, allowing agents to fill forms, log in, or interact with custom widgets without parsing complex nested divs.
5. **Real-time Page Monitoring**: Monitors active pages for updates using delta diffing, alerting agents only when new interactive components are added or removed.
6. **Web Search & RAG Pipelines**: Acts as an efficient web crawler that strips boilerplates (headers, footers, etc.) and provides clean context for Retrieval-Augmented Generation.
7. **Accessibility Compliance Audits**: Exposes semantic ARIA accessibility trees, letting automated agents inspect pages for accessibility compliance.
8. **E-Commerce Monitoring**: Navigates product listings, classifies page types, and extracts price/stock updates with minimal payload sizes.
9. **Dynamic Single Page Application (SPA) Automation**: Handles modern JavaScript-heavy frameworks by running Playwright locally, caching states, and delivering optimized components.
10. **Multi-Agent Browser Sharing**: Serves as a standardized MCP bridge for multi-agent workflows, letting separate agents share, observe, and control the same browser session.

and many more...

---

## 🛠️ MCP Tools Reference

The server registers and exposes the following tools:

| Tool | Parameters | Return Type | Description |
| :--- | :--- | :--- | :--- |
| `extract_context` | `url` (string) | `CompressedContext` | Navigates to a URL, performs cleanup and compression, runs page classification, and returns optimized UI and ARIA trees. |
| `page_diff` | `url` (string) | `PageDiff` | Returns deltas (added/removed elements) compared to the last observed state of this URL. |
| `execute_action` | `action` (string), `selector` (optional), `value` (optional) | `ActionResult` | Executes standard interactions (`click`, `type`, `select`, `scroll`, `wait`, `navigate`) on the active page. |
| `summarize_page` | `url` (string) | `Dict` | Produces an instant text summary listing interactive element counts and text content snippets. |
| `classify_page` | `url` (string) | `ClassificationResult` | Evaluates UI elements to identify the page category (e.g. login, search, survey). |
| `wait_until_ready` | `url` (string), `timeout` (optional) | `ActionResult` | Navigates to a page and waits for browser stabilization. |
| `cache_lookup` | `url` (string) | `Dict` | Directly queries the semantic cache for stored context. |
| `get_metrics` | None | `Dict` | Retrieves telemetry (bytes saved, cache hit rate, total actions). |

---

## 📂 About the Code & Module Architecture

The codebase is structured modularly under the `app/` directory:

* **`app/browser/manager.py`**: Controls the lifecycle of the async Playwright browser. Reuses page contexts to avoid startup overhead and manages navigation timeouts.
* **`app/extractor/extractor.py`**: Siphons raw HTML and utilizes Playwright's latest `.aria_snapshot()` API to capture accessibility trees.
* **`app/compressor/compressor.py`**: Houses DOM filters. Strips unneeded tags (scripts, styles, headers, footers, SVGs) and outputs a list of structured interactive UI controls.
* **`app/classifier/classifier.py`**: Employs a heuristics-based scoring algorithm to classify pages into `LOGIN`, `SEARCH`, `SURVEY`, `CHECKOUT`, `PRODUCT`, or `DASHBOARD` states.
* **`app/diff/diff.py`**: Compares consecutive observations on the same URL and generates a delta report (added and removed elements) using composite element fingerprints.
* **`app/cache/cache.py`**: Manages an in-memory `cachetools.TTLCache` indexed by URL and validated via 64-bit `xxhash` signatures of page HTML.
* **`app/executor/executor.py`**: Executes browser interactions (`click`, `type`, `select`, `scroll`, `wait`, `navigate`) deterministically.
* **`app/schemas/schemas.py`**: Declares Pydantic data models enforcing contract compliance across modules and tools.
* **`app/metrics/metrics.py`**: Logs context size reductions, cache hits/misses, and calculates cumulative byte savings.

---

## ⚙️ Setup & Installation

### 1. Prerequisites
* Python 3.11 or newer.
* Playwright dependencies installed on your system.

### 2. Installation Steps
```bash
# Clone the repository
git clone https://github.com/yourusername/browser-optimizer-mcp.git
cd browser-optimizer-mcp

# Create and activate virtual environment
python -m venv venv
venv\Scripts\activate     # On Windows
source venv/bin/activate  # On macOS/Linux

# Install dependencies
pip install -r requirements.txt

# Install Playwright browser binaries
playwright install chromium
```

### 3. Configuration
Create a `.env` file in the project root:
```env
LOG_LEVEL=INFO
HEADLESS=True
CACHE_ENABLED=True
CACHE_TTL=300
CACHE_MAX_SIZE=100
BROWSER_TIMEOUT=30000
```

---

## 🖥️ Client Integration

### 1. Claude Desktop Setup
Add the configuration to your `claude_desktop_config.json` file (located at `%APPDATA%\Claude\claude_desktop_config.json` on Windows or `~/Library/Application Support/Claude/claude_desktop_config.json` on macOS):

#### For Windows (uses the `cmd` wrapper to handle folder spaces):
```json
{
  "mcpServers": {
    "browser-optimizer": {
      "command": "cmd",
      "args": [
        "/c",
        "C:\\path\\to\\browser-optimizer-mcp\\venv\\Scripts\\python.exe",
        "-m",
        "app.server.main"
      ],
      "env": {
        "PYTHONPATH": "C:\\path\\to\\browser-optimizer-mcp"
      }
    }
  }
}
```

#### For macOS / Linux:
```json
{
  "mcpServers": {
    "browser-optimizer": {
      "command": "/path/to/browser-optimizer-mcp/venv/bin/python",
      "args": ["-m", "app.server.main"],
      "env": {
        "PYTHONPATH": "/path/to/browser-optimizer-mcp"
      }
    }
  }
}
```


### 2. Antigravity IDE Setup
Add this to your `mcp_config.json` file (located at `%USERPROFILE%\.gemini\config\mcp_config.json` on Windows or `~/.gemini/config/mcp_config.json` on macOS/Linux):

```json
{
  "mcpServers": {
    "browser-optimizer": {
      "command": "C:\\path\\to\\browser-optimizer-mcp\\venv\\Scripts\\python.exe",
      "args": ["-m", "app.server.main"],
      "env": {
        "PYTHONPATH": "C:\\path\\to\\browser-optimizer-mcp"
      }
    }
  }
}
```

### 3. Cursor Setup
1. Go to **Settings** -> **Features** -> **MCP**.
2. Click **+ Add New MCP Server**.
3. Configure the parameters:
   * **Name**: `browser-optimizer`
   * **Type**: `command`
   * **Command**: `C:\path\to\browser-optimizer-mcp\venv\Scripts\python.exe -m app.server.main`
4. Set the environment variable `PYTHONPATH` = `C:\path\to\browser-optimizer-mcp`.
5. Click **Save**.

---

## 📊 Benchmark & Tool Comparison

Directly comparing the **Browser Optimizer MCP** against traditional browser automation agents highlights the efficiency gains:

### 1. Performance Comparison

| Metric / Feature | Standard Browser Tools (Direct DOM/Screenshots) | Browser Optimizer MCP |
| :--- | :--- | :--- |
| **Average Token Count (Google)** | ~50,000+ tokens | **~120 tokens** (97.7% reduction) |
| **Average Token Count (HN)** | ~9,000+ tokens | **~1,500 tokens** (87.8% reduction) |
| **Observation Payload Type** | Raw DOM or Base64 screenshots | **Clean JSON UI controls + ARIA snapshot** |
| **Incremental Observations** | Resends entire DOM or new screenshot | **Returns only element deltas (added/removed)** |
| **Re-observation Latency** | Full DOM download and parse (~1.5s) | **In-memory cache lookup** (~0.12ms) |
| **Page Classification** | Requires LLM API call & reasoning tokens | **Instant, local rule-based heuristics** (0 tokens) |
| **Action Execution** | LLM must reason step-by-step | **Deterministic rule-based execution** |
| **Inference Cost** | High ($0.15 - $1.00+ per step) | **Extremely Low** (80-95% cost reduction) |

### 2. Running the Benchmark Suite
To run the benchmark suite against live public pages and verify these savings on your local machine:
```powershell
$env:PYTHONPATH="."
venv/Scripts/python scripts/benchmark.py
```

---

## 🧪 Testing & Deployment

### Run Unit Tests
```bash
pytest tests/ -v
```

### Docker Deployment
```bash
docker compose -f docker/docker-compose.yml up --build
```

---

## 📄 License

Distributed under the MIT License. See [LICENSE](LICENSE) for more details.

Copyright (c) 2026 Manthan.
