Metadata-Version: 2.4
Name: websource
Version: 0.1.1
Summary: Standalone website discovery, fetching, parsing, and adapter contracts.
Project-URL: Homepage, https://github.com/romamo/websource
Project-URL: Repository, https://github.com/romamo/websource
Project-URL: Issues, https://github.com/romamo/websource/issues
Project-URL: Documentation, https://github.com/romamo/websource#readme
Author-email: Roman Medvedev <pypi@romavm.dev>
License-Expression: MIT
License-File: LICENSE
Keywords: adapters,extraction,pydantic,structured-data,website
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Typing :: Typed
Requires-Python: >=3.14
Requires-Dist: defusedxml<1,>=0.7
Requires-Dist: httpx<1,>=0.28
Requires-Dist: pydantic<3,>=2.11
Requires-Dist: selectolax<1,>=0.3
Provides-Extra: browser
Requires-Dist: playwright<2,>=1.40; extra == 'browser'
Description-Content-Type: text/markdown

# websource

Standalone website discovery, fetching, parsing, rendering, content extraction,
structured extraction, refresh checking, and adapter contracts.

`websource` is built around strict Pydantic models and protocol interfaces. The
foundation layer owns value objects, enums, request/result models, adapter
descriptors, protocol interfaces, and error contracts. Runtime HTTP, browser,
provider, and website-specific adapters sit outside `websource.foundation`.

Structured extraction is modeled separately from readable content extraction, so
article markdown, recipes, product records, property listings, vehicle ads, jobs,
events, and custom schemas can share one transport contract without forcing every
page into an article-shaped model.

## Install

```bash
pip install websource
```

Optional browser rendering uses Playwright:

```bash
pip install "websource[browser]"
python -m playwright install chromium
```

For local development:

```bash
uv sync --all-extras --group dev
uv run --group dev pytest
```

## CLI

```bash
websource adapters list --json
websource inspect https://example.com --json
websource discover https://example.com --mode crawl --max-depth 1 --jsonl
websource fetch https://example.com --json
websource fixture replay tests/fixtures/sites/example.com/article.html --url https://example.com/articles/one --json
websource provider run tavily tavily-http https://api.tavily.example/extract extract --input-url https://example.com/article --param schema=article --json
```

The CLI emits JSON envelopes for validation and runtime errors so agents and
automation can handle failures deterministically.
`inspect` performs a shallow seed URL probe: robots facts and verified source
hints stay in fields such as `robots.sitemap_urls` and `sitemap_urls`, while
unverified conventional fallbacks are labeled as `candidate_sitemap_urls`.
When no sitemap hint has been verified, `sitemap_urls` is `null`, not `[]`.
Use `inspect --include-http-headers` to append request and response headers for
the HTTP exchanges made during inspection.

## Python API

```python
from websource.adapters import DefaultWebsiteClient
from websource.foundation import FetchRequest

client = DefaultWebsiteClient()
result = await client.fetch(FetchRequest(url="https://example.com"))
page = await client.parse_page(result)
content = await client.extract_content(page)
```

Custom adapters can subclass `SiteSpecificAdapter` and advertise only the
capabilities they override. Managed providers can use
`HttpManagedProviderAdapter` while provider-specific packages stabilize.
Built-in vertical schemas are available through `websource.schemas` for recipes,
products, property listings, vehicle listings, job postings, and events.

## Release Checks

```bash
uv run --group dev ruff check src tests
uv run --group dev mypy src tests
uv run --group dev pytest
uv run --group dev bandit -r src
uv build
scripts/verify-package-install.sh
```

## Documentation

- [Docs index](docs/README.md)
- [Foundation contracts](docs/architecture/foundation-contracts.md)
- [Website client architecture](docs/architecture/website-client-architecture.md)
- [Adapter design](docs/architecture/adapter-design.md)
- [CLI and tool surface](docs/reference/cli-and-tools.md)
- [Vertical schemas](docs/reference/vertical-schemas.md)
- [JSON schemas](docs/reference/json-schemas.md)
- [Release policy](docs/reference/release.md)
- [Implementation plan](docs/plans/implementation-plan.md)

## License

MIT
