Metadata-Version: 2.4
Name: pagespring
Version: 0.1.1
Summary: Acquire and normalize online software manuals into clean, convertible source files — the acquisition front-end to pagespeak.
Author: Mike Farr
License-Expression: MIT
Project-URL: Repository, https://github.com/phierceweb/pagespring
Project-URL: Changelog, https://github.com/phierceweb/pagespring/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/phierceweb/pagespring/issues
Keywords: manuals,documentation,ingest,acquisition,rag,markdown
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: pyyaml>=6.0
Requires-Dist: pf-core[cli]~=0.4.1
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: pre-commit>=3.7; extra == "dev"
Dynamic: license-file

# pagespring

Acquire and normalize online software manuals into clean, convertible source
files — the **acquisition front-end to
[pagespeak](https://github.com/phierceweb/pagespeak)**.

Point it at a manual's URL. A *pattern* recognizes the source type, *acquires*
the raw pages (stdlib `urllib`), and *normalizes* them into ONE clean
HTML/markdown file with absolute asset URLs under `incoming/<slug>/`. That clean
file is the deliverable; converting it into the finished RAG corpus is a separate
step (pagespeak) that consumes `incoming/` on its own — pagespring never runs it.

Lean by design: `pf-core[cli]` + `beautifulsoup4`, stdlib fetch, no ML stack.

## Intended use

pagespring is for **publicly available documentation** — vendor manuals, help
centers, open textbooks, API specs. It fetches only what the source serves to
any reader: there is no login/session handling, no paywall traversal, and no
bot-detection evasion. It is a **polite client**: it identifies itself with a
`pagespring/<version>` User-Agent (see `.env.example` to override), honors
`429 Retry-After`, backs off on server errors, paces crawl requests, and caps
crawl sizes.

It is a *user-invoked, one-manual-at-a-time* archiver — closer to "Save Page
As" than to an autonomous crawler — so it does not consult `robots.txt`
(which governs bots that discover URLs on their own; you supply the URL).
Before mirroring a site, check its terms of use. What you may do with the
acquired copy (personal RAG corpus, internal search, redistribution) is
governed by the source's license — the deliverable under `incoming/` stays on
your machine, and nothing is re-published by this tool.

## Install

```bash
pip install pagespring
```

## Quick start

```bash
pagespring ingest https://docs.tableplus.com   # acquire + normalize → incoming/tableplus/
pagespring localize <slug>                      # pull a deliverable's images later (resumable; --all)
pagespring patterns                             # list the source patterns
pagespring classify <url>                       # which pattern handles a URL (no fetch)
pagespring status                               # what's been acquired
```

Deliverables land in `./incoming/<slug>/` under the directory you run from.

## Dev

```bash
bin/setup   # clone → venv + editable install with dev extras
bin/test    # pytest
bin/lint    # ruff check + ruff format --check + mypy (strict)
```

See [docs/usage.md](docs/usage.md) for the full command set and
[docs/architecture.md](docs/architecture.md) for the acquire → normalize flow
and how to add a new source pattern.
