Metadata-Version: 2.4
Name: harvestweb
Version: 0.1.1
Summary: Multi-provider website review and Firecrawl CLI for repo-local research bundles
Project-URL: Homepage, https://github.com/OuCodes/harvestweb
Project-URL: Repository, https://github.com/OuCodes/harvestweb
Project-URL: Issues, https://github.com/OuCodes/harvestweb/issues
Project-URL: Changelog, https://github.com/OuCodes/harvestweb/blob/main/CHANGELOG.md
Author: OuCodes
License: MIT
License-File: LICENSE
Keywords: crawl4ai,firecrawl,research,review,scraping,website
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Utilities
Requires-Python: >=3.11
Requires-Dist: beautifulsoup4>=4.13
Requires-Dist: markitdown>=0.1.2
Requires-Dist: requests>=2.32
Provides-Extra: crawl4ai
Requires-Dist: crawl4ai>=0.8.6; extra == 'crawl4ai'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: pylint>=3.3; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.4; extra == 'dev'
Description-Content-Type: text/markdown

# harvestweb

`harvestweb` is a small CLI and Python package for website review work.

It gives you one installable tool that can:

- review a site with provider fallbacks
- call Firecrawl directly for `scrape`, `search`, `map`, and `crawl`
- save markdown + JSON bundles under a repo-local `data/harvestweb/` directory

This package is meant to be reusable across client ops repos without dragging along brand-specific code or repo-specific state.

## Install

```bash
pip install harvestweb
```

Optional Crawl4AI fallback support:

```bash
pip install "harvestweb[crawl4ai]"
```

For local development:

```bash
python3.12 -m venv .venv
./.venv/bin/python -m pip install --upgrade pip
./.venv/bin/python -m pip install -e ".[dev]"
```

## Quick start

Hosted Firecrawl requires `FIRECRAWL_API_KEY`:

```bash
export FIRECRAWL_API_KEY=fc-your-key
harvestweb review https://example.com --write
harvestweb scrape https://example.com --write
harvestweb map https://example.com --write
harvestweb crawl https://example.com --write
```

## Commands

### Review a site with fallbacks

Default provider order:

1. `firecrawl`
2. `crawl4ai`
3. `requests`

```bash
harvestweb review https://example.com --write
harvestweb review https://example.com --focus "homepage messaging" --write
harvestweb review https://example.com --provider-order firecrawl,requests --write
```

### Firecrawl commands

```bash
harvestweb scrape https://example.com --write
harvestweb search "best electrolyte drink" --limit 5 --write
harvestweb map https://example.com --write
harvestweb crawl https://example.com --limit 50 --write
```

## Output

By default, bundles are written under:

```text
data/harvestweb/
```

Subfolders are grouped by kind:

- `reviews/`
- `scrapes/`
- `searches/`
- `maps/`
- `crawls/`

Override the root with:

- `--output-root ...` on the CLI
- or `HARVESTWEB_OUTPUT_ROOT=/custom/path`

## Why this package exists

This tool is designed for people who work across multiple client repositories and want one clean, pip-installable website review utility instead of copying scripts between ops repos.

## Development

Run tests:

```bash
./.venv/bin/python -m pytest
```

Build distributions:

```bash
./.venv/bin/python -m build
```

## Release docs

- changelog: [`CHANGELOG.md`](./CHANGELOG.md)
- release checklist: [`RELEASE_CHECKLIST.md`](./RELEASE_CHECKLIST.md)
