Metadata-Version: 2.4
Name: crawlr
Version: 0.1.1
Summary: Crawlr: an AI-powered, self-healing web scraper with an e-commerce price-intelligence vertical
Project-URL: Homepage, https://github.com/ardfaiyaz/crawlr
Project-URL: Repository, https://github.com/ardfaiyaz/crawlr
Project-URL: Issues, https://github.com/ardfaiyaz/crawlr/issues
Author: Ard Tree
License: MIT License
        
        Copyright (c) 2026 Ard Tree
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: cli,ecommerce,llm,price-intelligence,price-monitoring,scraper,self-healing,web-scraping
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Requires-Dist: fastapi>=0.111
Requires-Dist: httpx>=0.27
Requires-Dist: jinja2>=3.1
Requires-Dist: pydantic>=2.7
Requires-Dist: python-dotenv>=1.0
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: selectolax>=0.3.21
Requires-Dist: tenacity>=8.3
Requires-Dist: typer>=0.12
Requires-Dist: uvicorn>=0.30
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: js
Requires-Dist: playwright>=1.44; extra == 'js'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.1; extra == 'postgres'
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://raw.githubusercontent.com/ardfaiyaz/crawlr/main/crawlr-icon-logo.png" alt="crawlr" width="120" />
</p>

<h1 align="center">crawlr</h1>

<p align="center">
  An AI-powered, <strong>self-healing</strong> web scraper for e-commerce price &amp; stock monitoring.
</p>

<p align="center">
  <a href="https://pypi.org/project/crawlr/"><img src="https://img.shields.io/pypi/v/crawlr?color=2a6833" alt="PyPI version" /></a>
  <a href="https://pypi.org/project/crawlr/"><img src="https://img.shields.io/pypi/pyversions/crawlr?color=2a6833" alt="Python versions" /></a>
  <a href="https://github.com/ardfaiyaz/crawlr/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-2a6833" alt="License: MIT" /></a>
  <a href="https://01crawlr.vercel.app/"><img src="https://img.shields.io/badge/website-live-2a6833" alt="Website" /></a>
</p>

<p align="center">
  <a href="https://01crawlr.vercel.app/"><strong>Website</strong></a> &middot;
  <a href="https://pypi.org/project/crawlr/"><strong>PyPI</strong></a> &middot;
  <code>pip install crawlr</code>
</p>

---

Crawlr uses an LLM (or a free offline heuristic) **only to generate and repair CSS selectors**,
then extracts pages deterministically with those cached selectors. When a site changes its layout
and the selectors break, Crawlr detects it and **regenerates them automatically** — so your
monitoring never silently dies.

- **Runs free & offline** — no API key required; add an OpenAI/Anthropic key for higher accuracy.
- **Self-healing** — survives redesigns that break traditional scrapers.
- **Price & stock alerts** — via webhook, Slack, or email, on the trigger you choose.
- **General-purpose** — e-commerce ships in; add new verticals as simple YAML.

## Install

```bash
pipx install crawlr        # or: pip install crawlr
```

## Quickstart

```bash
crawlr watch "https://store.com/product/123" --target 25   # track price + stock
crawlr watchlist                                           # current price, movement, stock
crawlr monitor --daemon                                    # keep checking in the background
crawlr serve                                               # optional local dashboard
```

## Alert triggers

Pick one per watch with `--trigger` (or in the dashboard):

| Trigger | Alerts when |
|---------|-------------|
| `any_change` | any watched field changes |
| `price_drop` | the price goes down |
| `price_below` | price is at/below your target |
| `price_above` | price is at/above your target |
| `back_in_stock` | the item becomes available |
| `out_of_stock` | the item sells out |

For richer logic, run `crawlr init` to create an editable `crawlr.rules.yaml` that maps
circumstances to actions (and overrides per-watch triggers):

```yaml
default_action: ignore
rules:
  - when: price_drops_below
    amount: 25
    action: alert
  - when: back_in_stock
    action: alert
```

## Custom schemas (no code)

Add new verticals by dropping a YAML file into your schema directory:

```yaml
name: jobs
item_selector: ".job-card"
fields:
  - name: title
    description: the job title
    type: text
    required: true
  - name: salary
    description: annual salary
    type: number
```

Then `crawlr scrape <url> --schema jobs`. List schemas with `crawlr schemas`.

## Configuration

Set via environment variables or a `.env` file:

| Variable | Default | Description |
|----------|---------|-------------|
| `CRAWLR_LLM_PROVIDER` | `none` | `openai`, `anthropic`, or `none` (offline heuristic) |
| `CRAWLR_LLM_API_KEY` | — | API key for the chosen provider |
| `CRAWLR_DATABASE_URL` | — | `postgresql://...` to use Postgres instead of SQLite |
| `CRAWLR_ALERT_WEBHOOK` | — | Webhook URL for change alerts |
| `CRAWLR_ALERT_SLACK` | — | Slack incoming-webhook URL |
| `CRAWLR_PROXIES` | — | Comma-separated proxy URLs to rotate |
| `CRAWLR_RESPECT_ROBOTS` | `true` | Honor robots.txt |

## Development

```bash
pip install -e '.[dev]'
pytest          # offline test suite
ruff check .    # lint
```

## License

MIT
