Metadata-Version: 2.5
Name: scrapemend
Version: 0.0.1
Summary: Self-healing web scraping for Python. Describe what you want in plain language; when a site changes and your selectors break, an LLM repairs them automatically.
Project-URL: Homepage, https://github.com/FengYue427/scrapemend
Project-URL: Repository, https://github.com/FengYue427/scrapemend
Project-URL: Issues, https://github.com/FengYue427/scrapemend/issues
Author: FengYue427
License: MIT
License-File: LICENSE
Keywords: ai,crawler,data-extraction,llm,scraping,self-healing,web-scraping
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: selectolax>=0.3.21
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == 'openai'
Description-Content-Type: text/markdown

# scrapemend

**Self-healing web scraping for Python.** Describe the data you want once.
scrapemend extracts it with cached CSS selectors — and when a site changes
its markup and those selectors break, a language model inspects the new page,
repairs the selectors, and caches the fix. The next run is free again.

> Traditional scrapers break every time a site is redesigned. scrapemend
> treats a broken selector as a cache miss, not a crash.

---

## Why

Every scraper hits the same wall: you write selectors, the site ships a
redesign, your selectors silently return nothing, and you go rewrite them by
hand. Cloud "AI scraping" services solve the robustness problem but are
expensive, send your target data through someone else's servers, and run an
LLM call on *every* request.

scrapemend takes the middle path:

- **Cache first.** Known selectors run with zero LLM cost — it's just parsing.
- **Heal on miss.** The LLM is only called when selectors stop matching.
- **Self-hosted.** It's a library, not a service. Your keys, your data.

## Install

```bash
pip install scrapemend
```

## Quickstart

```python
from scrapemend import Scraper, Schema

schema = (
    Schema("product")
    .add("title", "the product name")
    .add("price", "the product price with currency")
)

scraper = Scraper()  # offline stub by default; plug in a real LLM to heal
result = scraper.extract("https://shop.example/p/1", schema)

print(result["title"], result["price"])
print("healed:", result.healed, "tokens:", result.tokens)
```

You describe *what* you want in plain language. You never hand-write a CSS
selector — scrapemend discovers them, and re-discovers them when they break.

## How it works

```
extract(url, schema)
      │
      ├─ cached selectors exist? ──► apply them
      │                               ├─ all fields matched ──► return (free)
      │                               └─ some field missed ──┐
      │                                                      ▼
      └─ no cache ─────────────────────────────────► LLM heals selectors
                                                      apply, cache, return
```

Because healing only runs on a miss, steady-state scraping costs nothing in
tokens. You pay the model exactly when a site actually changes.

## Status

Early alpha (`0.0.1`). The core pipeline — cache, parse, heal, re-cache — works
and is covered by tests that run fully offline. A real OpenAI adapter is the
next step; the LLM layer is provider-agnostic by design.

## Roadmap

- [x] Cache-first / heal-on-miss pipeline
- [x] Offline, key-free test path
- [ ] OpenAI adapter
- [ ] Local-model adapter
- [x] HTML condensing for cheaper prompts
- [ ] Pluggable cost/budget guards

## License

MIT
