Metadata-Version: 2.5
Name: view-as-ai
Version: 1.0.0
Summary: Preview website content as AI web browsers and LLM agents see it, calibrated against ChatGPT web browsing
Project-URL: Homepage, https://github.com/Serbyte-Development/view-as-ai
Project-URL: Repository, https://github.com/Serbyte-Development/view-as-ai
Project-URL: Issues, https://github.com/Serbyte-Development/view-as-ai/issues
Author: Serbyte Development
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Keywords: ai-web-browser,chatgpt,html-to-markdown,llm,llm-ready,web-scraping,website-to-markdown,website-to-text
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.11
Requires-Dist: html2text==2025.4.15
Requires-Dist: httpx<1,>=0.28
Requires-Dist: lxml<7,>=5.3
Description-Content-Type: text/markdown

# View as AI

[![CI](https://github.com/Serbyte-Development/view-as-ai/actions/workflows/ci.yml/badge.svg)](https://github.com/Serbyte-Development/view-as-ai/actions/workflows/ci.yml)
[![License: GPL-3.0-or-later](https://img.shields.io/badge/License-GPL--3.0--or--later-blue.svg)](LICENSE)

**View Website As AI.** Preview the model-readable information a browsing system is likely to receive from a webpage.

View as AI is a lightweight Python CLI and library empirically calibrated against native ChatGPT `web.run open()` output. It runs locally, uses direct HTTP origin HTML, and makes no OpenAI API calls.

The calibration target is information presence, omission, ordering, links, images, and controls. Markdown details and line layout can differ from native ChatGPT output.

## Install

Requires Python 3.11+.

Install the CLI from PyPI:

```sh
uv tool install view-as-ai
view-as-ai https://example.com
```

For library use:

```sh
uv add view-as-ai
```

For development:

```sh
uv sync --group dev
uv run view-as-ai https://example.com
```

## CLI

```sh
# Public webpage
view-as-ai https://example.com

# Plain model-readable text
view-as-ai https://example.com --format text

# Structured output with reference URLs
view-as-ai https://example.com --format json

# Local or built HTML
view-as-ai ./dist/index.html --base-url https://example.com/

# HTML from stdin
curl -fsSL https://example.com | view-as-ai - --base-url https://example.com/

# Save output
view-as-ai https://example.com -o preview.txt
```

The default `view` format adds line numbers for inspection:

```text
Example Domain (https://example.com/)
View as AI preview; Total lines: 5
L0: # Example Domain
L1:
L2: This domain is for use in documentation examples without needing permission. Avoid use in operations.
L3:
L4: 【0†Learn more†iana.org】
```

`--format json` returns the page URL, model-readable text, title, and the reference URL map:

```json
{
  "url": "https://example.com/",
  "text": "# Example Domain\n\n...",
  "title": "Example Domain",
  "urls": {
    "0": "https://iana.org/domains/example"
  }
}
```

The CLI follows HTTP redirects, accepts HTML/XHTML responses, and uses a 30-second default timeout. Run `view-as-ai --help` for the complete option list. `python -m view_as_ai` exposes the same CLI.

## Python API

```python
from view_as_ai import process_html, prune_html

html = "<h1>Pricing</h1><button>Start trial</button>"
page = process_html(prune_html(html), "https://example.com/pricing")

print(page.title)
print(page.text)
print(page.urls)
```

`process_html()` is pure and performs no I/O. It returns a `PageContents` dataclass with `url`, `text`, `title`, and `urls` fields. `prune_html()` applies the conservative provider-style pruning used by the CLI.

## What it approximates

The stable runtime path is deliberately small:

```text
HTTP origin HTML
    ↓
conservative provider-style pruning
    ↓
OpenAI-derived HTML-to-model-text formatter
    ↓
model-readable text + reference URLs
```

The parser preserves useful model-visible information such as headings, prose, lists, tables, code, links, image descriptions, buttons, inputs, selects, and document order. Pruning removes a narrow set of recurring navigation and UI patterns supported by calibration evidence.

Client-side JavaScript execution, shadow DOM traversal, and fetching iframe documents are outside the runtime. Browser-rendered DOM was removed after broad calibration because it generally reduced resemblance to the recorded native output.

> [!IMPORTANT]
> View as AI approximates page representation. Search ranking, citation likelihood, cached crawl versions, crawler policy decisions, and OpenAI's private indexing behavior are outside scope.

## Calibration and limitations

View as AI is a best-effort approximation based on recorded native ChatGPT browsing observations. The broad stabilization calibration reached roughly **99% average information-level resemblance after normalization**. That figure is an aggregate calibration result, and individual pages can diverge.

Expected sources of difference include:

- Sites serving different HTML to different user agents or crawlers.
- Content created only after client-side JavaScript runs.
- Shadow DOM and fetched iframe contents.
- Images whose meaning exists only in pixels instead of `alt` or `title` text.
- Unusual navigation, widgets, or page structures outside the calibration set.
- Whitespace, Markdown structure, heading levels, reference numbering, and line counts.

The active repository keeps compact native golden fixtures plus synthetic parser and pruner regressions. The larger reverse-engineering corpus and oracle tooling are archived separately for future recalibration if native behavior changes materially.

## Verification and maintenance

```sh
uv sync --group dev
uv run pytest -q
uv run ruff check src tests
uv run ruff format --check src tests
uv build
```

Routine maintenance is evidence-driven: reproduce meaningful information differences against native output, add the smallest generic fix, and preserve conservative pruning. The installed wheel contains the runtime package and required third-party attribution files.

## Source lineage and attribution

The formatter began from OpenAI's public `gpt-oss` `gpt_oss/tools/simple_browser/page_contents.py` at revision `750cfe908fdc9dd1f0e9bfcd92a4bb1adb0aa81c` and was adapted for View as AI's standalone parser and empirical calibration. OpenAI source attribution, the upstream Apache-2.0 license, and dependency notices are preserved in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) and `licenses/`.

View as AI is distributed under [GPL-3.0-or-later](LICENSE).

Developed & maintained by [Serbyte Development](https://www.serbyte.net/) · [GitHub](https://github.com/Serbyte-Development)
