Metadata-Version: 2.4
Name: parsehub-dsl
Version: 1.0.1
Summary: A domain-specific language for web scraping, inspired by ParseHub
Author: ParseHub DSL Team
License-Expression: MIT
Keywords: web-scraping,dsl,parsehub,compiler
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Compilers
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: lark>=1.1.0
Requires-Dist: playwright>=1.40.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Dynamic: license-file

# ParseHub DSL

ParseHub DSL is a bounded browser-automation DSL for structured web extraction.

Its officially supported runtime contract is:

- **Playwright**
- **Chrome / Chromium-family browsers**

The project is built around:

- a ParseHub-inspired interaction model
- a dedicated DSL grammar and compiler pipeline
- a Playwright-backed browser execution runtime
- a verified, bounded set of supported DSL examples and automation flows

The repo is no longer in broad modernization mode. Its latest bounded release
mainline was:

> **release-to-1.0 convergence**

That lane has now converged locally: the repo metadata and release-facing docs
are aligned on a bounded `1.0.1` stable-release posture for the already
supported modern DSL surface.

## Current posture

- **Phase 1**: complete
- **Phase 2**: closeout documented; the former config-string runtime boundary is now closed by proof
- **Phase 3**: capability promotion review is complete
- **Phase 4**: docs / workflow / release-surface convergence is complete
- **Post-Phase-4 bounded follow-ups**: semantic authority, runtime authority, browser-confidence hardening, compiler/runtime hardening, and release-readiness slices are complete
- **Latest bounded operator-surface expansion**: local profile/session management now includes the closed RFC-anchored Phase 3 successor lane (`config.session_profile` default-profile selection + shared cross-profile session query seam; metadata indexing remains deferred behind the scale gate)
- **Latest closed release lane**: `release-to-1.0-convergence`
- **Current repo metadata**: `parsehub-dsl==1.0.1` / `Production/Stable`
- **Latest externally published artifact**: `parsehub-dsl==1.0.0` (the next repo-side patch target is `1.0.1`)
- **Historical prior publication milestone**: `parsehub-dsl==0.2.0b1`
- **Publication rule**: repo verification now targets `1.0.1`, while external stable publication remains a separate operator action; the latest published artifact stays `1.0.0` until that upload completes

## Runtime support contract

ParseHub DSL is **not** positioned as a general-purpose “automate anything on
any website” platform.

The current official support contract is narrower:

- supported runtime: **Playwright**
- supported browser family: **Chrome / Chromium-family**
- supported automation surface: the **bounded DSL surface verified by this repository**
- supported browser-confidence proof lane: **deterministic local headless browser verification**

Important note:

- the repository may retain internal browser abstractions for testing and
  architecture
- those internal seams do **not** imply public support for Selenium, Firefox,
  WebKit, or general multi-backend parity
- the current verified browser-confidence lane installs and launches
  **Chromium via Playwright**
- stealth-related code paths and flags remain **optional** and **best-effort**;
  they do **not** widen the support contract or create anti-bot / CAPTCHA /
  platform-defense success guarantees

## Runtime operator controls

The supported operator surface is still bounded, but it includes practical
runtime controls:

- `--locale`, `--timezone-id`, and `--user-agent` for browser-context shaping
- `--browser-channel` and `--browser-executable` for **local Chromium-family
  launches**
- `--cdp-url` to attach to an existing **Chromium CDP endpoint** instead of
  launching a local browser
- `--disable-resources`, `--block-domain`, and `--extra-header` for bounded
  request-layer control
- `--disable-stealth` for turning off optional best-effort stealth tweaks
- `--profile`, `--session`, and `--profile-root` for local profile/session-backed runs
- `parsehub profile ...` / `parsehub session ...` for local profile/session lifecycle management

Important operator notes:

- `--browser-channel` and `--browser-executable` are **launch-only** controls;
  they do not apply when `--cdp-url` is used
- `--headed` is a **local debugging** toggle for a local Playwright launch; it
  does **not** widen the supported proof story beyond the deterministic local
  headless browser-confidence lane
- these controls improve operator ergonomics and reproducibility, but they do
  **not** create public-site reliability, anti-bot, or multi-browser parity
  guarantees

## Stable-release posture

Use these files as the authority chain:

- `docs/START_HERE.md`
- `docs/README.md`
- `docs/CURRENT_STATUS.md`
- `docs/ROADMAP.md`
- `docs/RELEASE_CONTRACT.md`
- `docs/SUPPORT_POLICY.md`
- `CHANGELOG.md`
- `docs/EXPRESSION_MODEL.md`
- `docs/CAPABILITY_MATRIX.md`
- `docs/OMX_WORKING_CONVENTION.md`
- `docs/PROFILE_SESSION_ENVIRONMENT_MANUAL.md` as the primary user-manual + design-appendix package
- `docs/PROFILE_SESSION_ENVIRONMENT_GUIDE.md` as a secondary companion for the full conceptual and CLI relationship
- `docs/PROFILE_SESSION_ENVIRONMENT_QUICKSTART.md` as a secondary companion for the shortest operator path
- `docs/PROFILE_SESSION_ENVIRONMENT_DOMAIN_MODEL.md` as a secondary companion for the designer-facing model
- `docs/encoding.md` (when diagnosing DSL/config/source-file encoding issues)

Additional operator references:

- `docs/SUPPORTED_EXAMPLES.md`
- `docs/PARAMETER_INPUTS.md`
- `docs/TEST_LANES.md`
- `examples/USAGE_GUIDE.md`

## Install

### Recommended: editable install

```bash
cd F:/parsehub/parsehub-dsl

python -m venv .venv
.venv\Scripts\activate

pip install -e ".[dev]"
python -m playwright install chromium
```

### Alternative: requirements + editable package

```bash
cd F:/parsehub/parsehub-dsl

python -m venv .venv
.venv\Scripts\activate

pip install -r requirements.txt
python -m playwright install chromium
pip install -e .
```

## Fast operator path

### Installed CLI help

```bash
parsehub --help
```

### Smallest supported execution path

```bash
parsehub example_simple.ph
```

### Persist the full structured result

```bash
parsehub example_simple.ph --output json --output-file temp/example_simple.json
```

### Create a local profile and capture a reusable login session

```bash
parsehub profile create demo_profile --name "Demo Profile" --login-url https://example.com/login
parsehub session login demo_profile
parsehub session verify sess_... --profile demo_profile
parsehub session export sess_... --profile demo_profile --output-dir temp/session-export
parsehub authenticated.ph --profile demo_profile
```

Use `docs/PROFILE_SESSION_ENVIRONMENT_MANUAL.md` as the primary authority
for this topic. `docs/PROFILES_AND_SESSIONS.md` remains the specialized
companion for storage/lifecycle/verification/export semantics, while the guide,
quickstart, and domain-model docs remain secondary companions for narrower
reading paths.

Manual login capture also supports bounded overrides such as `--locale`,
`--timezone-id`, `--user-agent`, `--browser-channel`, `--browser-executable`,
`--disable-resources`, repeated `--extra-header`, repeated `--block-domain`,
and `--disable-stealth`.

### Inspect parameter metadata without running the browser

```bash
parsehub examples/parameter_demo_current.ph --params-help
python -m parsehub.cli examples/parameter_demo_current.ph --params-help
```

Module entry is the development fallback after editable install (or with
`PYTHONPATH=src` set). The installed `parsehub` console script is the canonical
release-facing operator path.

## Output behavior

The CLI now has two distinct output surfaces:

1. **stdout preview**
   - human-readable execution summary
   - grouped preview of collected rows
   - suitable for quick inspection
2. **`--output-file` persisted result**
   - full structured JSON or CSV output
   - suitable for downstream processing or saved artifacts

Important rule:

- `--output` controls the format of `--output-file`
- stdout remains a human-readable preview rather than raw JSON/CSV

## DSL source-file encoding contract

- ParseHub DSL source files (`.ph`) are read as **UTF-8**
- files saved as **UTF-8 with BOM** are also accepted
- UTF-8 remains the only supported source-file encoding contract
- non-UTF-8 `.ph` files produce an explicit `save the .ph file as UTF-8`
  guidance message
- non-UTF-8 `--config` JSON files produce an explicit `save the .json file as
  UTF-8` guidance message
- non-UTF-8 internal grammar/resource files are reported as an internal
  install/project-file integrity problem rather than user DSL input drift

## Default verification commands

### Stable smoke baseline

```bash
pytest
```

### Broader regression bundle

```bash
python run_phase1_regressions.py
```

### Required release gates

```bash
python scripts/release/run_release_gates.py
```

### Fresh-venv install / CLI / compile / run happy path

```bash
python scripts/release/run_release_happy_path.py
```

### Publish-surface verification before manual release

```bash
python scripts/release/run_publish_surface_check.py
```

### Explicit browser-confidence lane

```bash
pytest tests/runtime/test_playwright_driver.py tests/runtime/test_e2e_execution.py -m browser_confidence -q
```

Interpretation:

- `pytest` is the stable default baseline
- `python run_phase1_regressions.py` is the broader verified safety net
- `python scripts/release/run_release_gates.py` is the repo-owned release gate runner
- the browser-confidence lane remains **non-default for plain `pytest`**
- the same browser-confidence lane is now also invoked explicitly by release-gate automation
- the current explicit proof lane is **Playwright-backed** and uses locally
  installed **Chromium**
- `python scripts/release/run_publish_surface_check.py` validates the stable-release package artifacts and metadata locally
- none of these commands imply public-site scraping reliability

## Supported example quickstart

Representative supported examples include:

- `simple.ph`
- `product_list.ph`
- `nested_data.ph`
- `pipe_functions_demo.ph`
- `else_branch_demo.ph`
- `simple_param_test.ph`
- `enhanced_click_demo.ph`
- `label_system_demo.ph`
- `wait_time_units_demo.ph`
- `enhanced_click_demo_current.ph`
- `label_system_demo_current.ph`
- `wait_time_units_demo_current.ph`
- `parameter_demo_current.ph`
- `template_data_flow_current.ph`
- `search_interaction_scroll_current.ph`
- `runtime_engine_authority_current.ph`

Useful compile commands:

```bash
python compile.py examples/parameter_demo_current.ph --verbose
python compile.py examples/template_data_flow_current.ph --verbose
```

See `docs/SUPPORTED_EXAMPLES.md` for the verified example-by-example guidance.

## Known boundaries

The current release target is intentionally bounded.

Still out of scope:

- public-site scraping reliability guarantees
- headed/manual browser execution as a default guarantee
- anti-bot / CAPTCHA / stealth success guarantees
- stealth-related code paths, flags, or tests as a contract-widening signal
- official multi-browser runtime parity across Chrome / Firefox / WebKit
- official Selenium runtime support
- YAML config support
- legacy inline globals such as `global counter = 0`
- legacy dollar-ref syntax such as `$counter`
- legacy parameter aliases such as `number` / `boolean`

## Docs

- [start here / current project reality](docs/START_HERE.md)
- [docs index](docs/README.md)
- [current status snapshot](docs/CURRENT_STATUS.md)
- [roadmap / active next-step authority](docs/ROADMAP.md)
- [release contract / stable-release target](docs/RELEASE_CONTRACT.md)
- [support policy / runtime support contract](docs/SUPPORT_POLICY.md)
- [changelog / release-facing history](CHANGELOG.md)
- [expression model / runtime contract](docs/EXPRESSION_MODEL.md)
- [capability matrix](docs/CAPABILITY_MATRIX.md)
- [OMX working convention / repo-root workflow](docs/OMX_WORKING_CONVENTION.md)
- [test lanes / verification surfaces](docs/TEST_LANES.md)
- [supported examples quickstart](docs/SUPPORTED_EXAMPLES.md)
- [parameter inputs and precedence](docs/PARAMETER_INPUTS.md)
- [profiles and sessions](docs/PROFILES_AND_SESSIONS.md)
- [examples usage guide](examples/USAGE_GUIDE.md)
- [root-level legacy tests cleanup note](docs/ROOT_LEGACY_TESTS.md)
- [legacy syntax migration guide](docs/LEGACY_SYNTAX_MIGRATION.md)

## Repository layout

```text
parsehub-dsl/
├── src/parsehub/     # grammar, parser, semantic analysis, IR, runtime, codegen
├── tests/            # smoke / integration / runtime verification lanes
├── examples/         # supported and historical DSL examples
├── docs/             # repo-visible authority docs
├── scripts/          # release, verification, and maintenance scripts
├── .github/          # CI / workflow configuration
├── compile.py        # compile / natural-language generation entrypoint
├── run_phase1_regressions.py
└── README.md
```

## Practical next-step rule

When in doubt:

1. trust executable evidence first
2. trust top-level current-facts docs second
3. preserve the bounded release target
4. start any widened work from a new bounded plan instead of silently reopening old lanes
