Metadata-Version: 2.3
Name: escrape-plan
Version: 0.0.1
Summary: Tool for extracting data from HTML documents
Author: a.p.p
Author-email: a.p.p <a.phlaming.phoenix@gmail.com>
Requires-Dist: beautifulsoup4
Requires-Dist: lxml
Requires-Dist: pandoc
Requires-Dist: pyyaml
Requires-Dist: bpython>=0.26,<1 ; extra == 'dev'
Requires-Dist: ruff>=0.17.3,<1 ; extra == 'dev'
Requires-Dist: twine>=7.0.0,<8 ; extra == 'dev'
Requires-Python: >=3.14
Project-URL: homepage, https://gitlab.com/a.phlaming.phoenix/escrape-plan
Project-URL: repository, https://gitlab.com/a.phlaming.phoenix/escrape-plan
Provides-Extra: dev
Description-Content-Type: text/markdown

# eScrape Plan

A tool for extracting data from HTML pages in complex and extensible ways according to plans crafted to describe that data.

eScrape Plan starts at the top of an HTML document and compares every tag in a "flattened" sequence against a series of **markers**, which are HTML tags you expect to find. The content between those markers are stored internally as **sections** in a **document**. Each section can then be acted on by **extractors**, which pull data out of the HTML and store them. Further, after a document has been fully parsed and all of this data has been extracted, the data can be run through **outputters** to produce artifacts from the data.

Today, eScrape Plan can do things like:

- Parse a long document into smaller sections
- Categorize those sections internally by name/ID
- Run find/replace operations on that content (to do things like normalize newline characters in the source)
- Reorganize this content arbitrarily and produce new HTML files with any arrangement of that content
- Export textual/HTML content into SQLite databases for later querying
- Convert HTML `<table>` content into SQLite databases

> [!NOTE]
> **A Note from a.p.p on LLM Contributions**
>
> I would prefer not to use LLMs to generate code for this project for a variety of reasons that are not appropriate to list in detail here. That said, unfortunately, LLM usage is becoming unavoidable for me in my daily job developing software due to external pressures, and so it does behoove me to experiment with this technology.
>
> At this point, I remain unconvinced of the value of LLM generated code, but I have found some mixed value in using LLMs for code review. I will often have an LLM form merge requests and post code review comments for me. In these cases, there should always be an attribution to the bot. I only use open weight models running on my own hardware.


## Usage

I'll have [better documentation](https://gitlab.com/a.phlaming.phoenix/escrape-plan/-/work_items/11) on before too much longer. Until then...

```
pip install escrape-plan
```
