Metadata-Version: 2.4
Name: webscope-mcp
Version: 0.6.0
Summary: MCP server (stdio, standard library only) that gives a coding agent short answers about a project
License-Expression: MIT
Project-URL: Repository, https://github.com/partprogramming09/webscope
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# Webscope

<!-- mcp-name: io.github.partprogramming09/webscope -->

A lightweight, token-efficient MCP server (stdio, JSON-RPC 2.0, Python standard library only). It provides coding agents with short, surgical answers: grouped search hits, isolated symbols instead of whole files, native in-browser testing on the user's open Chrome, and bounded project scripts.

Tool names, parameters, and messages are in English.

**Scope.** The core (`search`, `read`, `run`, `e2e_chrome`) works on any stack. The included adapters are Laravel-first: 16 of the 23 target Laravel, Laravel + Vue, or Vue. Astro, plain Python and other stacks get the core, the Astro map, and their own scripts declared in `webscope.json`.

**Status.** The Laravel adapters are a reference pack: they came out of one application, were validated only against it, and their automated tests use toy file trees. There are no external users yet.

---

## Measured Effect

**Why not the agent's own grep?** Grep and read answer «where is X». They cannot compute an answer that crosses layers or reads the database in one step. Measured below: 5 calls against 1 for a column sheet, 6 against 1 for «who reads this column». Where plain search is enough (map of a module), there is no measurable difference: 6 calls against 7.

Measured 2026-10-20 (second run of the same method): four read-only questions about a real Laravel + Vue monorepo, asked through `claude -p` (Sonnet) in a clean session, five runs each per condition (40 runs), order randomized per pair, one warm-up run per condition discarded, ground truth fixed by hand from sources that are not adapters (a grep of the code, direct SQL counts, a manual check of each file). Hypotheses and thresholds were written before the run: the column sheet and the consumers question cost at most half as much with webscope (met: 25 % and 19 % of the median cost); the module map differs by less than 20 % (met: 14 % lower). The method, the environment and the numbers per run are in [`bench/`](bench/); the table below is printed by `bench/report.py` from `bench/results/runs-v3.json`.

| Question | USD without: median (min–max) | USD with: median (min–max) | Calls without → with (median) | Complete answers without → with |
| :--- | :--- | :--- | :--- | :--- |
| Backend module and tables of a frontend module | 0.050 (0.045–0.058) | 0.043 (0.037–0.050) | 6 → 7 | 5/5 → 5/5 |
| Rows with a value per column of a table | 0.053 (0.035–0.185) | 0.013 (0.013–0.023) | 5 → 1 | 5/5 → 5/5 |
| Type clashes between rules and columns | 0.165 (0.111–0.257) | 0.093 (0.038–0.272) | 11 → 7 | not scored |
| Backend files that read or write a column | 0.088 (0.065–0.245) | 0.017 (0.013–0.017) | 6 → 1 | 4/5 → 5/5 |

Totals over the 40 runs (sums): 2.26 USD and 146 calls without webscope and 0.93 USD and 80 calls with it. The ranges do not overlap for the column sheet and the consumers question; they overlap for the module map and the type audit, where no difference can be claimed. The first run of this method (`bench/results/runs.json`, 2026-10-20 as well) gave the same picture: 2.02 USD against 1.00 USD in total.

Read with care:

- One model, one project, and questions chosen because an adapter covers each. Questions with no adapter are not measured; the module map is the realistic one for them.
- USD includes cached tokens. The first run showed cost rising with each repetition; this run does not (without webscope per repetition: 0.42, 0.38, 0.43, 0.59, 0.44 USD). The expensive repetitions are the ones with the most cache-creation tokens, which makes cache effects the probable cause of the outliers (five points per condition, not tested further); there is no trend.
- «Complete answers» means every item of the hand-fixed truth was found. The one incomplete answer without webscope missed a file.
- An earlier run of 24 pairs scored against adapter output is discarded: that scoring was circular. The type audit is not scored because it has no independent truth; its precision was measured separately (see Checks below).

### Checks

Measured on the same application, 2026-10-20, labels set by hand from the code (not by an adapter). Counts are small: read them as orders of magnitude.

- `audit_types`: before the change, 28 of 36 labelled findings were real (78 %). The false ones came from joining a column by name across tables, from services that transform the value before the column (`true` to `'on'`), from filter-only requests and from rules that already bound the value. It now narrows to the table of the model the request names and marks as `heuristic` any finding it cannot tie to one table (nested keys, no model match, `boolean`, `in`/`exists`/closure rules); only the others set exit code 1. On the sample used to tune it: 14 of 15 real; on 12 fresh findings: 10 of 12; together 24 of 27 (89 %). The last rule change (`in`/`exists`) was not measured on a new sample. Target was 90 %: not reached.
- `unique_indexes`: 9 of 22 findings labelled before the labeller stopped: 8 real, 1 false (`CATCHES?` guessed from the folder).
- `tenant_queries`: 23 of 92 lines labelled: all 23 are real default-connection queries on tenant tables; none is reachable from a job, command or export, so the risk it warns about did not show up in this application.
- Engines: `fixtures/laravel` (Docker; Laravel 12 with MySQL 8, PostgreSQL 16 and SQLite) checks that `tinker --read-only` is refused by the database, that the default run rolls back and that only `--commit` persists, by row counts. Run it with `sh fixtures/laravel/run-engine-checks.sh`.
- CI is configured for Linux and macOS with Python 3.9 to 3.13 plus the engine job; it has not run on GitHub yet. Before pushing, the suite was run on macOS with 3.9 to 3.13 and on Linux (Docker) with 3.9 and 3.13.

---

## Quick Start

### 1. Initialize your project
Run in your project root to detect your stack (Laravel, Vue, Astro or Python get stack-specific scripts; anything else gets the generic core) and generate `webscope.json`:

```sh
# Via uvx (no install needed)
uvx webscope-mcp --init

# Or locally with Python
python3 webscope.py --init
```

### 2. Register with your MCP client

```sh
# Claude Code (recommended, writes project .mcp.json)
claude mcp add --scope project webscope -- uvx --from git+https://github.com/partprogramming09/webscope@v0.6.0 webscope-mcp --manifest "$PWD/webscope.json"

# With pip
pip install git+https://github.com/partprogramming09/webscope.git@v0.6.0
claude mcp add --scope project webscope -- webscope-mcp --manifest "$PWD/webscope.json"
```

The examples pin the tag `v0.5.1` on purpose. Replace it with the latest tag from the [Releases](https://github.com/partprogramming09/webscope/releases) page; the same tag appears in every client example below, and a test fails if one of them differs from the package version.

Start `claude` and verify with `/mcp`: `webscope` will be connected with `search`, `read`, and `run`.

---

## Tools

| Tool | Description |
| :--- | :--- |
| `search(text, where?, limit?, format?, context?, regex?)` | Case-insensitive search grouped by file with relevance ranking. `format="concise"` returns top 5 files. Skips secrets (`.env`, `*.key`). |
| `read(file, symbol?, lines?, max_lines?, tail?, contains?, format?)` | Surgical read: extracts a single function, method, class (PHP, TS, JS, Vue, Python), Markdown section, line range, or log tail (`tail`). |
| `run(name?, args?, detail?)` | Runs a declared read-only script from the manifest. Output trimmed to head and tail limits. |
| `run_write(name, args)` | Runs scripts declared in `writes`. Requires `WEBSCOPE_WRITE=1` and interactive user confirmation. |

Every response starts with a one-line actionable guide and is capped at `token_limit` (~4 characters per token).

---

## In-Browser Testing & Chrome Connector (`e2e_chrome.py`)

Webscope includes an in-browser automation engine that replaces heavy screenshot-based tools with lightweight DOM extraction:

### Native Chrome Connector (`--evaluate`)
Runs JavaScript in a tab of a Chrome that you control, through the Chrome DevTools Protocol (CDP), so logins and 2FA survive between runs. It needs a Chrome started with remote debugging **and its own profile**: from Chrome 136 the debugging port is ignored on the default profile, so you sign in once in a dedicated profile that then keeps its cookies. It does not attach to your everyday Chrome window.

```sh
# macOS
open -na "Google Chrome" --args --remote-debugging-port=9222 --user-data-dir="$HOME/.chrome-webscope"
# Linux
google-chrome --remote-debugging-port=9222 --user-data-dir="$HOME/.chrome-webscope"
# Windows
chrome.exe --remote-debugging-port=9222 --user-data-dir=%USERPROFILE%\.chrome-webscope
```

A new profile opens Chrome's welcome page, not your app: open the app in a tab yourself and sign in. Set the app's base URL once (`project.base_url` in the manifest or `WEBSCOPE_BASE_URL=http://localhost:5173`); without it nothing runs. Another debugging port: `WEBSCOPE_CDP_PORT`. A raw script takes the tab as its last argument (`--evaluate "<js>" localhost:5173`), the sub-commands take `--url` (`--evaluate --map --url localhost:5173`); neither ever runs in «the active tab». If the port does not answer, the command prints the launch lines above; on macOS it then falls back to AppleScript in your normal Chrome, which needs View > Developer > Allow JavaScript from Apple Events.

Checked on 2026-10-20 on macOS with Chrome 155 and a new profile: the port answered, a raw `--evaluate` and `--evaluate --map --url` ran over CDP against a local app (the signed-out login page), and with no debugging port the AppleScript fallback ran in the normal Chrome, and after `--evaluate --forms <route>` the helper `window.__f.state()` returned the buttons of the login page. `__f` is not there until a sub-command installs it. Not checked: Linux, Windows, a signed-in session through the new profile.

`--tabs` lists the open tabs (title and URL, read-only) and `--open <url>` focuses the tab already at that URL or opens one; the URL must be under `project.base_url`. Without `--url`, the tab is found from `project.base_url`: exactly one tab must match, otherwise the command lists them and stops. When no debugging port answers on macOS, every one of these says on stderr that it is acting on your normal Chrome session through AppleScript. `sh docs/chrome-cold-start.sh` repeats the whole path on a throwaway profile (5 steps, checked on 2026-10-20 with Chrome and CDP; the AppleScript path was checked by hand only: it listed 11 real tabs after a separator bug was fixed).

What sets it apart from other browser tools is not the session but what runs in the page:

```sh
# Install the helpers (`__e2e` capture and `__f`) in the tab. Give it one route: without routes it walks every route
# of the app, which can take longer than the 120 s default
python3 webscope_adapters/e2e_chrome.py --evaluate --forms /demo/auth/login --url localhost:5173

# Then inspect the current screen; a raw script takes the tab as its last argument
python3 webscope_adapters/e2e_chrome.py --evaluate "window.__f.state()" "localhost:5173"

# Run a complete JS test flow from a file
python3 webscope_adapters/e2e_chrome.py --evaluate test_flow.js "localhost:5173"
```

### Key Capabilities
- **Screen & Form Helpers (`__f`)**: Finds inputs by visible label, fills values, coordinates clicks with active settle wait (`calma()`), and verifies persistence.
- **Network Capture (`__e2e`)**: Intercepts `/api/` calls, latency (`ms`), and console errors without screenshots.
- **Self-Healing Across Reloads**: Injected helpers tolerate full page reloads (`window.location.reload()`) with automatic re-initialization.
- **Zero-Token Replay (`--replay <name>`)**: Replays recorded user flows in one call with 0 AI tokens.
- **Playwright Export (`--export <capture> <url>`)**: Converts recorded steps into clean `.spec.js` Playwright tests.

### The Essential Testing Contract: Full Lifecycle (CRUD + Sub-modules)
A module is **never considered tested** by merely reading a view or clicking a button. Webscope enforces a strict full-lifecycle standard:
1. **Create (C)**: Fill mandatory and conditional fields, submit, verify HTTP 200/201, and capture the generated record ID.
2. **Edit (U)**: Reopen the record, mutate fields, and verify data persistence in both UI and database.
3. **Exercise Sub-modules & Lifecycle**: Navigate sub-routes (`:id`), exercise child panels/tabs (e.g. characteristics, tariffs, pre-settlements, notes, print), and trigger state transitions (e.g. approve, reopen, duplicate).
4. **Delete & Clean (D)**: Delete the test record, confirm UI removal, and verify database clean state (`ZZ_E2E`).

---

## Configuration (`webscope.json`)

All behavior is project-contained. See [`webscope.example.json`](webscope.example.json).

```json
{
  "root": ".",
  "trim": { "head": 10, "tail": 70 },
  "token_limit": 5000,
  "storage": ".agents/webscope",
  "test_prefix": "ZZ_E2E",
  "search": { "dirs": ["src"], "extensions": ["ts", "vue", "php", "py"] },
  "allowed_env": ["DATABASE_URL"],
  "scripts": {
    "report": {
      "cmd": ["python3", "-I", "scripts/report.py"],
      "description": "Summarises a database table",
      "usage": "<table>",
      "rejects": ["--out"],
      "writes": ["--fix"],
      "timeout": 120
    }
  }
}
```

| Key | Description |
| :--- | :--- |
| `root` | Project root relative to manifest directory. |
| `extra_roots` | Folders outside `root`, relative to the manifest directory, that `search` and `read` may also read (for example a logs folder). Secret files stay blocked. |
| `token_limit` | Hard limit on response size in approximate tokens. |
| `trim` | Default head (`head`) and tail (`tail`) lines kept from script outputs. |
| `storage` | Internal storage for `module-map.json`, `flows/`, `walkthroughs/`. Autodetects `.agents/webscope/`, `.claude/webscope/`, or `.webscope/`. |
| `test_prefix` | Prefix for test entities (default `ZZ_E2E`). |
| `search.dirs`, `extensions` | Folders and extensions searched by default. |
| `allowed_env` | Environment variables passed to child scripts. |
| `redact` | Optional list of regular expressions; every match in a script's output is replaced by `[redacted]`. |
| `scripts.<name>` | Fixed command list (`cmd`), `rejects` (blocked flags), `writes` (write flags). |

### The `project` section

Unknown keys in `project` (a typo such as `base_urll`) are reported on stderr when the server starts and by `webscope --doctor`, which also checks that the manifest parses, that each script file exists, that `docker` or `php` are installed when the scripts need them, and whether a Chrome answers on the debugging port.

The included adapters read the `project` section (container, boot code, module folders, frontends, audit settings…). Every key is optional and has a default. Each one is documented, with its default, who uses it and a minimal example, in [`docs/project.md`](docs/project.md).

Database adapters run inside a transaction that is rolled back. `sql` also puts the database session in read-only mode, so the database itself refuses writes. Only `cleanup_test_rows --delete --confirm` and `tinker --commit` (both behind `run_write`) commit.


---

## Included Adapters (`{adapters}`)

Ready-made scripts shipped in the package. Use `{adapters}` in `cmd`:

Grouped by the stack they need. The core tools (`search`, `read`, `run`) need none.

**Any stack**

| Script | Purpose |
| :--- | :--- |
| `e2e_chrome.py` | Browser walk, form automation (`__f`), replay, and open Chrome connector (`--evaluate`). Any SPA. |
| `triage.py` | Classifies failing pieces of a run summary as known, CHANGED or NEW (`project.known_failures`). |
| `session_cost.py` | Token and USD cost auditor per model for Claude Code sessions. |
| `agent_usage.py`, `webscope_usage.py`, `session_state.py` | Subagent usage, webscope usage and session bootstrap for Claude Code. |

**Laravel** (a database adapter needs `project.container` or `project.workdir`)

| Script | Purpose |
| :--- | :--- |
| `laravel.py` (`map_back`) | Resolves endpoint → controller → service → model → database table. |
| `tinker.py` | Runs PHP in `artisan tinker` inside a transaction rolled back unless `--commit`; `project.boot` prepares the connection. |
| `sql.py` | Read-only SQL (one SELECT/SHOW/DESCRIBE/EXPLAIN/WITH) through `tinker.py`, in a read-only database session; prints TSV with a header. |
| `fields.py` | Column sheet of a table: type, nullability, rows with a value, backend readers and writers. |
| `consumers.py` | Who reads and who writes a column: `file:line` in the backend (and `project.legacy`). |
| `unique_indexes.py` | Unique indexes on tables with a status column and whether the backend covers each (`project.status_columns`). |
| `tenant_queries.py` | `DB::` queries on tenant tables without an explicit connection (`project.tenant_tables`). |
| `snap.py` | Snapshot and diff of a query: what an on-screen action really wrote. |
| `db_check.php` | SELECT-only database integrity checks by prefix marker from `.env` (MySQL / Postgres). |
| `payload_check.py` | Compares captures against FormRequest rules, Resources, and columns. |
| `cleanup_test_rows.py` | Lists and safely cleans test rows matching the test prefix. |

**Laravel + Vue**

| Script | Purpose |
| :--- | :--- |
| `module_graph.py` (`graph`) | Screen → frontend module → endpoints → backend module → tables (`project.frontends`, `project.stores`). |
| `audit_types.py` | Crosses columns with FormRequest rules, Resource casts and frontend numeric conversions, with real data. |
| `simulate.py` | Lists what could break in a module from request, form and schema; `--out` writes the walk spec. |
| `changed.py` | Specs affected by uncommitted changes, resolved to screens (`project.repos`, `project.spec_files`). |

**Vue 3 + TS**: `vue_ts.py` (`map`) maps routes, subroutes, clickable actions, API calls and E2E coverage.
**Astro**: `astro.py` (`map_web`) maps file-based routes, components, form fields and client scripts.

---

## Subagents Integration

Webscope keeps subagents light. Two separate numbers:

- **The server itself** costs about 1.6k tokens per session: `tools/list` plus `instructions`, measured 2026-10-20 with a 23-script manifest (the adapters it declares plus the project's own scripts) and with the 25-script `webscope.example.json`. It does not grow with the number of included adapters: they are not in `tools/list`; only the scripts your manifest declares add a short line to `instructions`.
- **A subagent** that declares `tools:` in its frontmatter loads about 9.3k tokens at start, against 28.8k when it loads every tool and skill name. That number depends on how many tools the agent declares, not on webscope's adapters.

### Exporting the `walkthrough` Agent
Export the ready-made walk agent without overwriting existing agents:

```sh
# Autodetects .agents/ or .claude/agents/
webscope-mcp --agent

# Or targeting a specific folder
webscope-mcp --agent [directory]
```

- **Non-destructive**: Never overwrites existing agent files unless `--force` is passed.
- **Isolation**: Leaves all other user-defined agents untouched.

### Declaring Tools in Agents
In your `.agents/<name>.md` or `.claude/agents/<name>.md`:
```yaml
---
name: browser
tools: Read, Grep, Glob, mcp__webscope__search, mcp__webscope__read, mcp__webscope__run
model: sonnet
effort: high
---
```

---

## Configuration in Other Clients

### Claude Desktop (`claude_desktop_config.json`)
```json
{
  "mcpServers": {
    "webscope": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/partprogramming09/webscope@v0.6.0", "webscope-mcp", "--manifest", "/absolute/path/to/webscope.json"]
    }
  }
}
```

### Cursor / VS Code / Cline (`.cursor/mcp.json` or `.vscode/mcp.json`)
```json
{
  "mcpServers": {
    "webscope": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/partprogramming09/webscope@v0.6.0", "webscope-mcp", "--manifest", "${workspaceFolder}/webscope.json"]
    }
  }
}
```

### Zed (`settings.json`)
```json
{
  "context_servers": {
    "webscope": {
      "command": {
        "path": "uvx",
        "args": ["--from", "git+https://github.com/partprogramming09/webscope@v0.6.0", "webscope-mcp", "--manifest", "/absolute/path/to/webscope.json"]
      }
    }
  }
}
```

---

## Updating

```sh
# uvx: refresh package cache
uv cache clean webscope-mcp

# pip: force reinstall
pip install --upgrade --force-reinstall git+https://github.com/partprogramming09/webscope.git@v0.6.0
```

---


## Security Model

- **Root Sandbox**: `search` and `read` resolve strictly within `root` and reject outside paths.
- **Invisible Secrets**: Automatically hides `.env*`, `*.pem`, `*.key`, `*.p12`, `id_rsa*`, `id_ed25519*`, `auth.json`, and credentials.
- **No Shell Injection**: Scripts execute as fixed argument lists without intermediate shell.
- **Read-Only Default**: Destructive arguments (`writes`) require `run_write`, `WEBSCOPE_WRITE=1`, and explicit approval.
- **SQL guard (`sql.py`)**: a regex accepts one statement starting with SELECT, SHOW, DESCRIBE, DESC, EXPLAIN or WITH and rejects `;`. It is a convenience filter, not a parser: `WITH ... DELETE` passes it. The real guard is that the database session is read-only (`SET SESSION TRANSACTION READ ONLY` on MySQL/MariaDB, `default_transaction_read_only` on PostgreSQL, `query_only` on SQLite) inside a rolled-back transaction. Verified on MySQL: a `WITH ... DELETE` fails with «Cannot execute statement in a READ ONLY transaction». Not covered by rollback: file output such as `SELECT ... INTO OUTFILE`, locks and sleeps, PostgreSQL sequences and non-transactional engines. For a hard guarantee, connect with a database user that only has SELECT.
- **`allowed_env` is a deliberate escape.** Scripts get only `PATH`, `HOME`, `LANG`, `LC_ALL`, `TMPDIR` and the variables you list in `allowed_env` (for example `DATABASE_URL`). Those values are real, and the output of a script goes back to the agent. Webscope does **not** redact script output: «Invisible Secrets» applies to `search` and `read` only. A script that dumps its environment, or a stack trace that prints a DSN, reaches the agent. Declare only what a script needs. Only `cleanup_test_rows.py` masks, on its own, the values of columns that look like secrets (`pass`, `token`, `secret`, `clave`).
- **`tinker.py`** only runs PHP files from the system temp folder or the project's storage folder, never arbitrary project code.

### Recommended `.gitignore`
```gitignore
# Temporary Webscope artifacts
captures/
.webscope/captures/
```

Commit `webscope.json`, `.claude/webscope/walkthroughs/` (or `.agents/webscope/`), and `module-map.json`.

---

## Tests

```sh
python3 -I test_webscope.py
sh fixtures/laravel/run-engine-checks.sh   # Docker: MySQL, PostgreSQL and SQLite
```

## License

[MIT](LICENSE)
