Metadata-Version: 2.4
Name: crawlr
Version: 0.1.0
Summary: Crawlr: an AI-powered, self-healing web scraper with an e-commerce price-intelligence vertical
Project-URL: Homepage, https://github.com/ardfaiyaz/crawlr
Project-URL: Repository, https://github.com/ardfaiyaz/crawlr
Project-URL: Issues, https://github.com/ardfaiyaz/crawlr/issues
Author: Ard Tree
License: MIT License
        
        Copyright (c) 2026 Ard Tree
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: cli,ecommerce,llm,price-intelligence,price-monitoring,scraper,self-healing,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Requires-Dist: fastapi>=0.111
Requires-Dist: httpx>=0.27
Requires-Dist: jinja2>=3.1
Requires-Dist: pydantic>=2.7
Requires-Dist: python-dotenv>=1.0
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: selectolax>=0.3.21
Requires-Dist: tenacity>=8.3
Requires-Dist: typer>=0.12
Requires-Dist: uvicorn>=0.30
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: js
Requires-Dist: playwright>=1.44; extra == 'js'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.1; extra == 'postgres'
Description-Content-Type: text/markdown

<p align="center">
  <img src="crawlr-icon-logo.png" alt="crawlr" width="120" />
</p>

<h1 align="center">crawlr</h1>

<p align="center">
  <strong>Crawlr</strong> is an AI-powered, <strong>self-healing</strong> web scraper with an
  e-commerce / price-intelligence vertical.
</p>

## Why Crawlr is different

Most "AI scrapers" pipe every page through an LLM — slow, expensive, and
non-deterministic. Crawlr uses the LLM (or an offline heuristic) **only to
generate and repair CSS selectors**, then extracts pages deterministically with
those cached selectors. When a site changes its markup and the selectors break,
Crawlr detects it and **regenerates the selectors automatically** (self-healing).

- **Cheap & fast:** deterministic extraction on every run; the model runs once per site/schema and only again on breakage.
- **Resilient:** self-healing survives layout changes that break traditional scrapers.
- **General-purpose core, vertical products:** the same engine powers any schema; e-commerce ships out of the box.
- **Runs offline:** works with zero API keys via a heuristic selector generator; add an OpenAI/Anthropic key for higher accuracy.
- **Continuous monitoring:** schedule scrapes, store time-series snapshots, and detect price/stock changes.

## Features

- **Self-healing extraction** with LLM or offline heuristic selector generation.
- **Alerting** on changes via webhook, Slack, and email, with threshold rules (e.g. only price drops above N%).
- **Validation & confidence scoring** per run, surfaced in the CLI and dashboard.
- **LLM cost guardrails:** per-run call budget, content-hash cache to avoid re-billing identical pages, and spend accounting.
- **Anti-bot resilience:** proxy rotation, robots.txt compliance, randomized delay jitter, and optional User-Agent rotation.
- **Concurrent monitoring** of many sites via a bounded async runner.
- **Built-in scheduler daemon** (`crawlr monitor --daemon`) — no external cron required.
- **User-defined schemas** in YAML/JSON — add new verticals (jobs, real estate, leads) without code.
- **Dashboard** with add-site form, run-now buttons, health indicators, and price-history sparklines.
- **Pluggable storage:** SQLite by default, Postgres via `CRAWLR_DATABASE_URL`; Docker + docker-compose included.

## Watchlist — the easy way

Track a competitor's price and stock in one command:

```bash
crawlr watch "https://store.com/product/123"                 # track price + stock
crawlr watch "https://store.com/product/123" --target 25     # alert at/below $25
crawlr watch "https://store.com/product/123" --restock       # alert when back in stock
crawlr watchlist                                             # see current price, movement, stock
crawlr monitor --daemon                                      # keep checking in the background
```

Or use the **dashboard** (`crawlr serve`) — a black‑and‑white, iOS‑styled watchlist: paste a
product URL, pick a **trigger** from the dropdown (the filter for when you want to be alerted),
optionally set a target price, and click **Watch**.

### Trigger filter

Choose per watch (CLI `--trigger` or the dashboard dropdown):

| Trigger | Alerts when |
|---------|-------------|
| `any_change` | any watched field changes |
| `price_drop` | the price goes down |
| `price_below` | price is at/below your target |
| `price_above` | price is at/above your target |
| `back_in_stock` | the item becomes available |
| `out_of_stock` | the item sells out |

### Rules template — "what happens in different circumstances"

For richer logic across many situations, create an editable rules file:

```bash
crawlr init          # writes crawlr.rules.yaml
```

```yaml
default_action: ignore
rules:
  - when: price_drops_below
    amount: 25
    action: alert
  - when: back_in_stock
    action: alert
  - when: price_increases
    action: ignore
```

When `crawlr.rules.yaml` exists it takes precedence over per‑watch triggers, giving you a single
place to describe exactly what should happen in each circumstance.

## Architecture

```
fetch (static -> auto JS, proxies, robots, jitter) -> simplify DOM -> selector cache?
   |-- hit  -> deterministic extract --(broken?)--> self-heal
   |-- miss -> LLM/heuristic generate selectors -> cache -> extract
                              |
              validate + confidence score -> store run (SQLite/Postgres)
                              |
                     diff vs previous -> log changes -> alert sinks
```

Modules:

| Module | Responsibility |
|--------|----------------|
| `fetcher.py` | Static HTTP + retries, proxy rotation, robots.txt, jitter, auto-escalation to Playwright |
| `simplifier.py` | Reduce HTML to a compact outline for the LLM (80–95% fewer tokens) |
| `llm.py` | Pluggable OpenAI/Anthropic selector generation + offline heuristic fallback |
| `usage.py` | LLM call budget + token/spend accounting |
| `extractor.py` | Self-healing deterministic extraction core |
| `validate.py` | Schema validation + confidence scoring |
| `selector_cache.py` | Selector cache keyed by host+schema and by page-content hash |
| `db.py` | SQLite/Postgres connection + dialect abstraction |
| `storage.py` | Sites, runs, records (time series), change log |
| `monitor.py` | Change detection + sync/async runners |
| `scheduler.py` | Polling daemon that runs due sites |
| `alerts.py` | Webhook / Slack / email sinks + threshold rules |
| `schemas.py` | Unified registry: built-in verticals + user YAML/JSON schemas |
| `verticals/ecommerce.py` | Ready-made `product` and `product_list` schemas |
| `cli.py` / `api.py` | Typer CLI + FastAPI dashboard |

## Install

Install the CLI in one line — no clone needed:

```bash
pipx install "git+https://github.com/ardfaiyaz/crawlr.git"
# once published to PyPI:
pipx install crawlr        # or: pip install crawlr
```

For development (from a clone):

```bash
python -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'
# optional extras
pip install -e '.[js]' && playwright install chromium   # JS rendering
pip install -e '.[postgres]'                              # Postgres backend
```

## Usage

```bash
# One-off scrape (prints confidence + validity + LLM spend)
crawlr scrape https://example-store.com/search?q=laptop --schema product_list

# Monitor sites
crawlr add https://example-store.com/product/123 --schema product --interval 30
crawlr monitor                       # run all due sites once (concurrently)
crawlr monitor --daemon --poll 60    # run continuously (built-in scheduler)

# Schemas
crawlr schemas                       # list built-in + user schemas
crawlr validate-schema ./my-schema.yaml

# Inspect + dashboard
crawlr sites
crawlr changes
crawlr serve                         # http://127.0.0.1:8000
```

### Defining a custom schema (no code)

Drop a YAML file into `CRAWLR_SCHEMA_DIR` (default `<data dir>/schemas`):

```yaml
name: jobs
item_selector: ".job-card"
fields:
  - name: title
    description: the job title
    type: text
    required: true
  - name: salary
    description: annual salary
    type: number
```

Then `crawlr scrape <url> --schema jobs`.

## Configuration reference

| Variable | Default | Description |
|----------|---------|-------------|
| `CRAWLR_DATA_DIR` | `./.crawlr` | SQLite DB, selector cache, schemas |
| `CRAWLR_DATABASE_URL` | — | `postgresql://...` to use Postgres instead of SQLite |
| `CRAWLR_SCHEMA_DIR` | `<data>/schemas` | Directory scanned for user YAML/JSON schemas |
| `CRAWLR_LLM_PROVIDER` | `none` | `openai`, `anthropic`, or `none` (heuristic) |
| `CRAWLR_LLM_API_KEY` | — | API key for the chosen provider |
| `CRAWLR_LLM_MODEL` | provider default | Model override |
| `CRAWLR_LLM_MAX_CALLS` | `2` | Max LLM calls per scrape (cost guardrail) |
| `CRAWLR_PROXIES` | — | Comma-separated proxy URLs to rotate through |
| `CRAWLR_RESPECT_ROBOTS` | `true` | Honor robots.txt |
| `CRAWLR_JITTER` | `0.75` | Max random extra delay (s) per request |
| `CRAWLR_ROTATE_UA` | `false` | Rotate realistic User-Agent strings |
| `CRAWLR_ALERT_WEBHOOK` | — | Generic webhook URL for change alerts |
| `CRAWLR_ALERT_SLACK` | — | Slack incoming-webhook URL |
| `CRAWLR_ALERT_EMAIL_TO` | — | Comma-separated recipient emails |
| `CRAWLR_SMTP_HOST` / `_PORT` / `_USER` / `_PASSWORD` / `_FROM` | — | SMTP settings for email alerts |
| `CRAWLR_ALERT_MIN_DROP` | `0.0` | Only alert on price drops ≥ this fraction (0.1 = 10%) |

## Docker

```bash
# Dashboard + Postgres + background scheduler
docker compose up --build
# Dashboard at http://localhost:8000
```

## Development

```bash
pip install -e '.[dev]'
pytest        # offline test suite (extraction, self-heal, validation, alerts, schemas, async, dashboard)
ruff check .  # lint
```

The test suite runs fully offline (no network, no LLM key): `fetch` is
monkeypatched with local fixtures and selector generation uses the heuristic
path. The same portable SQL is exercised on SQLite in CI and on Postgres in
production.

## Releasing to PyPI

Publishing is automated via GitHub Actions using **PyPI Trusted Publishing**
(OIDC — no API tokens to manage). One-time setup:

1. On PyPI, create the project (or add a *pending publisher*) under
   **Publishing → GitHub**, with: owner `ardfaiyaz`, repo `crawlr`,
   workflow `publish.yml`, environment `pypi`.
2. Then cut a release by pushing a version tag:

   ```bash
   git tag v0.1.0
   git push origin v0.1.0
   ```

The workflow builds the sdist + wheel, runs `twine check`, and publishes to PyPI.

> Note: confirm the `crawlr` name is available on PyPI first. If it's taken,
> change `name` in `pyproject.toml` (the CLI command can stay `crawlr`).

## Website &amp; docs

Crawlr is a **command-line product**. The `web/` directory is a single static landing page
(hero, features, how to use it, and an FAQ) — it deploys free on Vercel:

1. Import the repo at [vercel.com/new](https://vercel.com/new).
2. Set **Root Directory** to `web`.
3. Framework preset: **Other** (no build step — it's static HTML/CSS).
4. Deploy. You get a public URL like `https://crawlr.vercel.app`.

Locally you can preview it with any static server, e.g. `python -m http.server -d web 3000`.

## Roadmap

- Additional verticals shipped as YAML presets
- Richer dashboard charts and filtering
- Distributed workers / task queue for very large fleets
